Architectural Principles
Functions And Libraries
CTL2 functions are grouped and distributed in function libraries. CloverETL engine can be extended
with new CTL2 function library plugins extending the ctlfunction extension point.
The libraries are also used for visual grouping in CloverETL Designer’s expression editor dialog.
Library name is automatically used as the name of the group of functions being displayed.
FIXME: Expression Editor Screenshot
Functions in interpreted vs. compiled mode
The CTL2 can be executed in interpreted or compiled mode.
In compiled mode the CTL2 compiler translates CTL2 code directly into Java code. The code is then
dynamically compiled and loaded. In the translation process the calls to CTL2 functions are
translated directly to a Java method calls from respective library.
In the interpreted mode the CTL2 interpreter constructs AST from the CTL2 code. By traversing the
AST it evaluates CTL2 expressions. Any calls to CTL2 library functions are executed via objectwrapper of the function. The objec-wrapper is a Java-object representation of the CTL2 function that
can be stored in an AST node. The object wrappers contains no logic; it’s puprpose is to retrieve
arguments (actual parameters) from interpreter current stack then delegate the call to
corresponding Java method implementing the actual function.
The default execution mode of CTL2 is interpreted, this means the object-wrappers must be
implemented for each newly added function.
DateAdd function: Java method with function implementation
public static final Date dateAdd(TLFunctionCallContext context, Date lhs,
Long shift, DateFieldEnum unit) {
Calendar c = ((TLCalendarCache)context.getCache()).getCalendar();
c.setTime(lhs);
c.add(unit.asCalendarField(), shift.intValue());
return c.getTime();
}
DateAdd function: Object-wrapper delegating to the Java method
class DateAddFunction implements TLFunctionPrototype {
@Override
public void execute(Stack stack, TLFunctionCallContext context)
{
final DateFieldEnum unit = (DateFieldEnum)stack.pop();
final Long shift = stack.popLong();
final Date lhs = stack.popDate();
stack.push(dateAdd(context, lhs,shift,unit));
}
}
Function Annotations
CTL2 discovers new functions and their signatures automatically via Java reflections andd
annotations. Methods that should be treated as CTL2 function implementation must be correctly
annotated. Unannotated functions are ignored by the scanning process. The following annotations
are recognized by CTL2:
org.jetel.ctl.extensions.TLFunctionAnnotation(String functionDescription) : Annotation is used to tag
methods which implement the logic of CTL2 functions.


Optional String argument carries the description which is automatically displayed in Designer
tooltip text for the function
Additionally the first parameter of the method must be of TLFunctionCallContext type
USEFUL: With debug level TRACE the Engine plugin system reports information about discovered,
ignored and erroneously declared methods.
Example of DateLib.dateAdd() function declaring implicit context and 3 arguments:
@TLFunctionAnnotation("Adds to a component of a date (e.g. month)")
public static final Date dateAdd(TLFunctionCallContext context, Date
lhs, Long shift, DateFieldEnum unit) {
Calendar c = ((TLCalendarCache)context.getCache()).getCalendar();
c.setTime(lhs);
c.add(unit.asCalendarField(), shift.intValue());
return c.getTime();
}
org.jetel.ctl.extensions.TLFunctionInitAnnotation: No-arg annotation used to tag methods serving as
initializers for functions. The initializer methods are called during component’s init() time, before the
first call to the function implementation method. Can be used to initialize libraries, backend API and
to create function cache (see Context and Cache). Besides the annotation the initializer must respect
the following conventions:


Method must declare only a single parameter of type
org.jetel.extensions.ctl.TLFunctionCallContext.
Method name must use „Init“ suffix after function name it belongs to. E.g. method
dateAddInit() (note uppercase „I“) is an initializer for the dateAdd() function.
Example of DateAddInit caching instance of Calendar:
@TLFunctionInitAnnotation
public static final void dateAddInit(TLFunctionCallContext context) {
context.setCache(new TLCalendarCache());
}
org.jetel.ctl.extensions.TLFunctionParametersAnnotation(String[] paramNames): Annotation for
specifying names of function parameters as these cannot be determined automatically by Java
reflection.
Library and Function Lifecycle
The function lifecycle has the following steps:
1. Library initialization:
a. Scan for available library extensions
b. Scan each library for declared functions
2. Function compilation:
a. Interpreted mode: instantiate function’s object-wrapper; or
b. Compiled mode: generate call to implementing method
3. Function initialization:
a. Interpreted mode:call init() on object-wrapper; or
b. Compiled mode: call initializer method
4. Execution:
a. Interpreted mode: call execute() on object-wrapper; or
b. Compiled mode: invoke implementing method
5. End of execution
Engine Initialization
New libraries are discovered by Engine’s plugin system during Engine initialization. It is done by
scanning for extensions on ctlfunction extension point. New libraries should extend the
org.jetel.ctl.extensions.TLFunctionLibrary. The default implementation in the TLFunctionLibrary scans
all methods in the descendant for presence of TLFunctionAnnotation.
All annotated methods are additionally introspected for their parameters via Jav reflections. This
information is used by CTL2 type checker.
Function compilation
During code compilation the CTL2 compiler resolves the name of the function to its implementation.
It establishes the binding by searching all libraries and their functions by function name.


If no-match is found the compiler generates a compilation error – „Function not declared“.
In case multiple matches for the function are found, the correct version to call is determined
by CTL2 type checker (see Parameters and Overloading)
In interpreted mode after finding the function the CTL2 compiler in instantiates the object-wrapper
for the function. This is done by calling by calling getExecutable(String functionName) on function’s
library. Compiler creates a new instance is of object-wrapper for each call to the corresponding
function. This means two calls to an identical function do NOT share their object-wrappers.
This behavior is identical to compiled mode where the compiler generates call to initializer for every
call to the CTL2 library function found in the source code.
Code initialization
In compiled mode the CTL2 compiler generates a call to Java method representing the initializer.
In interpreted mode the CTL2 executor invokes the init() method on object-wrapper. The wrapper
should delegate this call to corresponding initializer method for the function. This way the initializer
logic is localized in the same method.
When executed, the initializer is presented with a TLFunctionCallContext instance that can be used to
store cached instances of Java objects . The cache should be used to store objects which are
expensive to create during repetitive calls to the function during execution (e.g.
java.util.regex.Pattern, java.util.Calendar, ...). The cache should be populated and set set via
TLFunctionCallContext.setCache(TLCache) method.
There is a variety of cache implementations already present (TLCalendarCache,TLRegexpCache,
TLObjectCache<T>). Developers who need to implement own cache shoud extend TLCache type.
Code execution
In compiled mode the CTL2 compiler generates a call to Java method representing the
implementation of the function.
In interpreted mode the CTL2 interpreter invokes the execute() method on the object-wrapper
passing a reference to current stack and TLFunctionCallContext. The responsibility of the objectwrapper is to pop the necessary function arguments, delegate the call to corresponding Java method
with function logic and push function’s retrurn value to the stack (for non-void functions). Consider
the following when popping function arguments:


The number of arguments is either known during development or can be determined via
TLFunctionCallContext.getParams().length for vararg functions
The arguments on the stack are popped in reverse order; last argument of the function is
popped first from the stack
End of execution
The last record processed by the CTL2 code results in the last invokaction of the function. There is no
additional post-execute or cleanup logic.
If you require such functionality, consider implementing your logic as custom Component which
provides such lifecycle mechanisms.
Parameters and Overloading
The CTL2 supports the following concepts for functions:



Function overloading
Vararg functions (aka. 3-dot functions)
Passing containers by reference
Overloading
Overloading allows declaring multiple version of a function with the same name but varying
parameters. CTL2 follows the same rules for overloading as in Java language: the versions of the
function must differ at least in one parameter (i.e. difference only in return type is insufficient).
No specific coding is required from the developer when overloading functions. The CTL2 type checker
will generate call to the correct version of the function depending on actual arguments in the
function call.
Library functions can be additionally overloaded by local declaration in the CTL2 code. In case a local
function declares identical parameters as library function the local function takes precedence. No
error or warning is issued by the compiler
The overloaded versions of the function may be located in different libraries; i.e. it is possible to
overload standard library function with own functions. This practice is however not recommended as
it may collide with new additions within standard CloverETL product.
Vararg functions
CTL2 support functions with variable number of arguments. Such functions are automatically
recognized and handled by the scanning process. In order to implement a custom vararg function see
the existing implementation of the StringLib.concat() or ContainerLib.insert().
The developers implementing avararg function must take care to pop correct number of arguments
from the stack in object-wrapper. This number of actual arguments can be retrieved from
TLFunctionCallContext.getParams().length
Passing containers by reference
Container types (map, list, record) are passed to functions by reference. It is therefore possible to
modify their state (e.g. insert element to a list, change field value in a record).
IMPORTANT: This is also true in case user passes input and output record of the component ($in.0,
$out.0)!
External Dependencies
CTL2 library functions may rely on external libraries. It is necessary to declare such dependencies in
library plugin descriptor (plugin.xml)
Example: dependencies of StringLib.json2xml
<runtime>
<library path="bin/"/>
<library path="lib/json.jar"/>
</runtime>
Deprecating Functions
Deprecated CTL2 library functions can be annoted by java.lang.Deprecated annotation. This
information is used by CloverETL Designer and CTL2 compiler. The compiler will issue a warning
about usage of deprecated function. The Designer will render the function in dialogs using
strikethrough font.
Implementation
Implementation of a custom CTL2 function requires the following steps:
1.
2.
3.
4.
Implement function library
Implement and annotate function method
Implement object wrapper
(Optional) Implement and annotate function initializer
5. Implement name-to-object lookup
6. Create Engine plugin descriptor
Where to start
Good starting point are the existing functions from org.jetel.ctl.extensions.StringLib and
org.jetel.ctl.extensions.ContainerLib:





StringLib.upperCase(): simple function accepting a single paramter and returning a value
StringLib.concat(): vararg function accepting any number of String arguments
StringLib.find(): function with initializer used to precompile and cache regex patterns
ContainerLib.clear(): overloaded function accepting either lists or maps
ContainerLib.insert(): vararg function working with lists
The plugin descriptor of the StringLib library also shows how to declare external dependencies.
Implement function library



Create new Eclipse project for your library.
Add cloveretl.engine and cloveretl.ctlfunction projects as dependencies
The library class should extend the TLFunctionLibrary class
Implement and annotate function method



Implement your function body as public static method in the library class
It is required the first parameter of the method must be of TLFunctionCallContext type. Even
in case the actual parameter is not used
Use TLFunctionAnnotation(string) to annotate the method as CTL2 function. Provide business
description of the function in annotation parameter
Implement object wrapper



Implement function’s object wrapper as inner class of the library with default access
Do NOT declare the inner class as private or static; this prevents CTL2 interpreter to create
instances of the object wrapper
The object wrapper must extend the TLFunctionPrototype class
o Implement the execute() method to retrieve parameters from stack, call
corresponding static Java method representing function body and optionally push
retrurn value to interpreter stack
o Implement the init() method only in case your function uses initializer. In that case
call the static Java method representing the initializer and pass it the reference to
TLFunctionCallContext from argument
(Optional) Implement function initializer




Initializer method is not required. Use it in case your function needs special initialization
before its execution (e.g. initialize external library, precompile and cache its parameters). See
StringLib.find() for reference.
Implement the initializer body as public static final method in the library
It is required that
o The intializer method uses„Init“ suffix after function name it initializes; e.g. findInit()
is an initializer method for the find() function
o The initializer method declares only single parameter of type TLFunctionCallContext
and no return type (void).
Use TLFunctionCallContext.setCache() to store cached parameters/references to the context.
See descendants of the TLCache class for existing cache implementations.
Implement name-to-object lookup
Implement the object wrapper lookup in the getExecutable(String) method in your library class:



Return an instance of corresponding object wrapper for the function
Always return a new instance of the wrapper object
Throw IllegalArgumentException(„Unknown function“ + functionName) to handle lookup to
non-existent function
Create Engine plugin descriptor
Your library should be packaged as CloverETL engine:



Copy or create a new plugin.xml plugind descriptor in the root of Eclipse project for the
library
External dependencies
o If your library requires external classes it is a good idea to package them as JAR files
placed in a „lib“ folder of your project
o We recommend placing any 3-rd party to the „lib“ folder as well
o Use the <runtime> element to declare all external dependencies of your library
o If two custom library function requires two different versions of the same 3rd party
library, you have to split them to two CTL2 libraries. All functions withing the same
library share the same classpath
Extension point
o Your must extend the ctlfunction extension point
o Provide library name and reference a fully qulified class name to your library as
parameters
Checking your implementation
Is my library and function correctly registered in Engine and Designer?
CloverETL Engine reports all discovered plugins during initialization, including the CTL2 function
libraries and all introspected functions. In order for the message to appear the DEBUG log level must
be enabled.
To check if the function is correctly registered in Designer, run eclipsec.exe. This starts Eclipse with
console window so that the Engine log is visible. Correctly registered function will appear in the
Expression Editor which can be started simply by launching the editor for Filter expression attribute
of ExtFilter component.
Common errors:



Missing annotation on functions
Wrong parameters in Java implementation; the first parameter is not of
TLFunctionCallContext type
Missing or invalid plugin descriptor for the library
OK message:
DEBUG [main] - Plugin eu.javlin.demolib loaded.
id - eu.javlin.demolib
version - 0.0.0.devel
provider-name - mtomcanyi
ctlfunction { libraryName = demo; className =
eu.javlin.DemoLibrary; }
Error message for missing plugin descriptor:
ERROR [main] - IO error occure in plugin manifest reading file:/C:/Projects/CloverETL/workspaces/workspace-librarydemo/cloveretl.demolibrary/plugin.xml.
(C:\Projects\CloverETL\workspaces\workspace-librarydemo\cloveretl.demolibrary\plugin.xml (The system cannot find the file
specified))
DE