Talend DI –
Testing and Continuous Integration
Talend, Global Leader in Open Source Integration Solutions
Kai Wähner
Principal Consultant
kwaehner@talend.com
@KaiWaehner
www.kai-waehner.de
Agenda
■ Testing
■ Integration into Maven / CI
■ Add external / referenced files
Agenda
■ Testing
■ Integration into Maven / CI
■ Add external / referenced files
Blackbox tests
■ Create specific test jobs
 They run other jobs (via tRunJob or Joblet) and use tAssert / tAssertCatcher.
 You could write to a tFileOutput, or even write a status to a table. i.e. a simple
JobName|Status 2 columns table that is cleared down before every CI run. Of
course, you can do more complicated stuff. Or just throw a Java Exception to
break the build.
■ Context variables for mocking
 Create contexts, e.g. „test“ and „prod“ environment
■ Test Components




tAssert + tAssertCatcher
tDie / tWarn / „Java Exceptions“ + tLogCatcher
OnError: Write error message to File / DB or throw Java Exception
React in CI to Java Exception / stopped Java process / Exception / error messages
in File or DB
„Whitebox tests“
■ Test only parts of a job
 „whitebox test replacement“
■ The only (good) solution is to use joblets if you want to test
only parts of a job
 Transformation is made into the joblet
 Normal job use the transformation joblet, and might expand it to watch the full
job
 A Test job can be designed using the joblet + normalized input and output +
tAssertCatcher.
■ tRunJob is no good alternative here
 See next slide for differences between tRunJob and Joblet
Excursus: tRunjob vs. Joblet
■
■



On the technical point of view, a joblet is just an extract of the job, but the components
which are in the joblet are going to be generated in the code as if they were designed in the
job itself. It's just as if you expand/collapse a part of the subjob.
The tRunJob excute a child job, which is a separate Java class. So the main job is going to
instanciate the child job and then execute it using the runJob method
Joblet generate code inside the job itself, so you can do things like that :
a joblet can be use to load context from a file or a table
a joblet can be used to manage custom log, with tlogcatcher as first component
a customer have created lot of joblet with input and output flow then use them as "lego" in job
■ tRunJob is a different unit of execution with it's own contexts and
■


variables. Joblet is a just GUI factorisation of components. to create a
reusable transformation.
if you have to process file or/DB and do the same action
e.g. Job1 File ->Joblet->Target AND Job2 Db->Joblet->Target
it is only possible with joblet, except if you copy and paste but you have twice the maintenance
if you update your joblet it updates both job
Example
Alternative: JUnit (or other test frameworks)
■ Low level coding
■ Use benefits / additional features of these frameworks
 and integrate into Maven / CI easily
■ How to use with Talend DI jobs?
 A DI Job is a Java process
 Start a DI Job within a JUnit test (i.e. start an external Java process)
• Executing Operating System Commands From Java: http://blog.art-ofcoding.eu/executing-operating-system-commands-from-java/
 Analyse results (e.g. File o Database content using JUnit‘s assert and use other
JUnit features)
Agenda
■ Testing
■ Integration into Maven / CI
■ Add external / referenced files
Use Talend‘s Maven Export Snippet
■ Build jobs from source code with Maven
■ Integrate directly into CI easily
■ Extend pom.xml snipped with additional logic
 The snipped is generated by Talend Studio and added to the zip file when you
export a job
 See Customizing build script templates (Talend > Import/Export):
https://help.talend.com/pages/viewpage.action?pageId=13806090
Use Talend‘s CommandLine
➜
The Talend CommandLine is the Talend Studio without the GUI
•
•
•
•
Export jobs
Create and run jobs
Execute test jobs
TODO where is a list of all commands
• How to use Command Line from Maven?
• Tutorial: http://blog.nanthrax.net/2013/10/talend-esb-continousintegration-part2-maven-and-commandline/
• Example: Pass contexts to Talend job via CommandLine:
http://www.vikramtakkar.com/2013/04/how-to-pass-contextparameters-to.html#sthash.mTW7BGTS.dpuf
• When to use Command Line instead of generated Maven
Snipped?
• If you have to use features which are not possible via Maven snippet.
• TODO ?!Anything beyond compiling and packaging the job from SVN ?!
© Talend 2013
Follow-up actions ...
➜
... after running a DI Test Job
➜
React via CI / Maven to
• Error messages in File / DB
• Java Exceptions
• etc.
© Talend 2013
Agenda
■ Testing
■ Integration into Maven / CI
■ Add external / referenced files
External / referenced files
■ Problem
 The job contains references to external files (e.g. Log4j.xml). The file is not added
to the JAR when you export the job from Studio.
■ Workarounds
 Include your log4 configuration in a JAR and add it to the job thanks to the
tLibraryLoad component
 Add log4j.xml to Talend metadata (with a context variable). Add this xml
metadata to your export. You can change context later, so it is not hardcoded
TODO does this work?
 You can adjust the generated Maven snipped (pom.xml) via settings  Talend 
Export  Maven Script. Add an external file within this script. See
http://maven.apache.org/plugins/maven-r … ctory.html
 Use Command Line to add log4j.xml before creating the JAR TODO is this
possible? Should be…
Talend and log4j
■ Community log4j component
 “unfortunately, this component is improperly designed and not so useful” – cite
by Talend’s lead component developer
■ Talend 5.4.1 will support log4j implicitly
 log4j in TAC
• Configuration:
https://help.talend.com/display/TalendAdministrationCenterUserGuide54EN/2.3.13+
+Setting+up+the+Log4j+parameters
• Logs TAC messages (technical and business messages)
 Studio: Project Settings  Log4j  Add / configure APPENDERS
 Activate log4j in advanced settings in run view
 Be aware: not available for all components in 5.4.1
• TODO Which components support log4j ?!
 Use elasticsearch / logstash / Kibana to analyse and monitor logs in realtime with
graphical dashboards
Talend and log4j
Talend and log4j
Q&A
 NOW
 Just ask 
 LATER
 kwaehner@talend.com
 @KaiWaehner
 Xing / LinkedIn