Diirt status
data integration in real time
Gabriele Carcassi
diirt: data integration in real-time
• Connection/data processing layer for general
purpose Java clients
– http://diirt.org
• Includes pvmanager/graphene/…
• Main two clients:
– CS-Studio (Java based Eclipse RCP)
– WebPODS (Web-socket + JSON “gateway”)
Data integration in real time (Diirt) architecture
CS-Studio (general purpose clients)
Web-pods (aka web gateway)
graphene
list
formula
Processing
csv
vTypes
Data Definition
JSON
pvmanager
Visualization
Aggregation
…
Channel
Finder
Cmd line
(exec)
DB (jdbc)
services support
…
Local
File
Web Pods
PV Access
datasources support
Request/response
…
PV Access
Probe
Data serialization/conversion
Ch Access
Publish/subscribe
BOY
Pods (Protocol Oriented Distribution Service)
Funding situation
• Funding at BNL was drastically cut
– Closed what I could
– Cleanly dropped the rest
• I’ll report on each area
– Current state
– What we wanted to do but won’t get done
Overall project
• General improvements
– All projects in a single repository
• Easier to move code where it belongs
• Easier to make releases
• (Maven lesson learnt: change version in feature branches)
– All elements configurable through a single
configuration directory structure DIIRT_HOME
– Documentation review (Wiki for user documentation,
javadocs for developer documentation, documented
source code examples in the repository)
VTypes
• Review types with Java 8 in mind
– Use javax.time
– Support for unsigned integers
– Reorganize type hierarchy to overcome some
implementation and usage issues
– Reorganize utility methods within the type
hierarchy (instead of utility classes)
• Work was started, currently unfunded
• Frozen in org.diirt.vtype.next package
VTypes
• Extract VType conversions from JCA/PVA
support
– Allow to use same exact type conversion for those
who do not need/want the whole stack
• Currently unfunded
Datasources (pvmanager)
• Final review to the datasource spec
– Generalize type conversion for writes (so that all
datasources behave the same)
– Use Java 8 standard classes where possible
– Generalize caches from sources of external data (i.e. better
support for archive data, better link between services, …)
– Finalized documentation and examples for writing
datasources
– Performance improvements
– Streamline error handling (i.e. do more work in the
framework, clients are simpler, error handling more
homogeneous)
• Currently unfunded
Archiver integration
• Client side temporal cache that can be filled
by different archiver sources and real-time
updates
• Under development by ITER.
Services
• Final review to the services spec
–
–
–
–
–
Uses Java 8 standard classes for callback
Defines order for arguments and results
Allows both synch and asynch execution
Allows both synch and asynch implementation
Automatically matches/wraps execution with
implementation (e.g. asynch execution on synch
implementation)
– Finalized documentation and examples
• No obvious functionality missing: this part can be
considered done
Sync execution example
public static void main(String[] args) {
System.out.println("Locating service method...");
ServiceMethod method =
ServiceRegistry.getDefault().findServiceMethod("math/add");
System.out.println("Service method found: " + method);
System.out.println("Preparing arguments...");
Map<String, Object> arguments = new HashMap<>();
arguments.put("arg1", ValueFactory.newVDouble(1.0));
arguments.put("arg2", ValueFactory.newVDouble(2.0));
System.out.println("Arguments: " + arguments);
System.out.println("Executing service...");
Map<String, Object> result = method.executeSync(arguments);
System.out.println("Result: " + result);
Locating service method...
Service method found: add(VNumber arg1, VNumber arg2): VNumber
result
Preparing arguments...
Arguments: {arg2=VDouble[2.0, 2015/03/25 09:35:09.243],
arg1=VDouble[1.0, 2015/03/25 09:35:09.229]}
Executing service...
Result: {result=VDouble[3.0, 2015/03/25 09:35:09.321]}
Output
}
Async execution example
public static void main(String[] args) {
System.out.println("Locating service method...");
ServiceMethod method =
ServiceRegistry.getDefault().findServiceMethod("math/add");
System.out.println("Service method found: " + method);
System.out.println("Preparing arguments...");
Map<String, Object> arguments = new HashMap<>();
arguments.put("arg1", ValueFactory.newVDouble(1.0));
arguments.put("arg2", ValueFactory.newVDouble(2.0));
System.out.println("Arguments: " + arguments);
System.out.println("Executing service...");
method.executeAsync(arguments, result -> {
System.out.println("Result: " + result);
}, ex -> {
ex.printStackTrace();
});
Locating service method...
Service method found: add(VNumber arg1, VNumber arg2): VNumber
result
Preparing arguments...
Arguments: {arg2=VDouble[2.0, 2015/03/25 09:35:09.243],
arg1=VDouble[1.0, 2015/03/25 09:35:09.229]}
Executing service...
Result: {result=VDouble[3.0, 2015/03/25 09:35:09.321]}
Output
}
Services
• Supported services:
– v4 PVAccess services
– ChannelFinder
– Database queries
– Command-line execution
• Plenty of copy and paste examples
Formula
• Final review to the function spec and parser
– Reorganized implementation, more consistent and
reusable
– Parser generates Abstract Syntax Tree (AST)
• AST can be manipulated and then converted to the
actual function
– Finalized documentation and examples
• Current functionality is done
Formula
Formula
• Other enhancements
– Writeable formulas
– Passive scanning for formula (performance
improvement)
– User defined functions (e.g. site specific function
to handle custom v4 PVStructure, naming
conventions, …)
• Currently unfunded
Graphene
• Contributions from students
– Improvements to intensity graph color map
• Import color map from Matlab
• Support for both relative and absolute value color maps (e.g.
RED is at 10% of the range, or at value 25)
– Time axis and plot
• Supports automatic time scale (e.g. sec -> min -> hours ->
day -> week -> month -> year)
• Support for daylight savings time, different time zones
– Preliminary JavaFX components based on the graphs
• Student work finished
• Other work needed to integrate new features
HSVRadian.XML:
<?xml version="1.0" encoding="UTF-8"?>
<!-- Varies the hue component of the HSV color model.
The map is useful for displaying periodic functions,
and expects the value to be in radians. -->
<colormap position="absolute" colorNaN="BLACK">
<color position="0.0" value="RED"/>
<color position="1.04719755119659774" value="YELLOW"/>
<color position="2.09439510239319549" value="GREEN"/>
<color position="3.14159265358979323" value="CYAN"/>
<color position="4.18879020478639098" value="BLUE"/>
<color position="5.23598775598298873" value="MAGENTA"/>
<color position="6.28318530717958647" value="RED"/>
</colormap>
JET.XML:
<?xml version="1.0" encoding="UTF-8"?>
<colormap position="relative" colorNaN="BLACK">
<color position="0.0" value="rgb(0,0,138)"/>
<color position="0.2" value="BLUE"/>
<color position="0.4" value="CYAN"/>
<color position="0.6" value="YELLOW"/>
<color position="0.8" value="RED"/>
<color position="1.0" value="rgb(138,0,0)"/>
</colormap>
JavaFX line graph integration:
JavaFX
• There is a general desire to develop widgets
and tools in JavaFX
– Can be used within SWT/Eclipse/CS-Studio
– More full featured than SWT
– Can be developed/tested/used outside Eclipse
– Allows testing contribution to diirt outside of CSStudio
• Work was started, currently unfunded
JavaFX
• What is done
– Almost all basic tools re-implemented and
expanded in JavaFX
– Determined best approach for JavaFX integration
• general purpose binding between channel updates and
JavaFX properties
JavaFX
• Probe
Browseable
event history
JavaFX
• Service probe
Autogenerates
field based on
service data
Currently handles only
scalar strings and numbers
JavaFX
• Formula Function Browser
JavaFX
• Service Browser
JavaFX
• What needs to be done
– Set of general purpose JavaFX widgets that
understand VTypes and channels/formula/services
• Can be used in standalone Java UI and in Eclipse/CS-Studio
• Can be used to create BOY widgets
• Ideally, use the same definitions for JavaFX/BOY and Web
– Finalize Graphene JavaFX widgets
– Finalize toolbox
• Work was started, currently unfunded
CS-Studio
• CS-Studio integration (what’s done)
– Renaming packages
– Jars are OSGi bundles (no extra build step)
– Bridge from Java standard dependency injection
(ServiceLoader) to OSGi implemented
– Proof of concept for updating CS-Studio codebase
to new version
• These items are done
CS-Studio
• CS-Studio integration (what’s not done)
– Finish the integration
– Testing
– Replace old SWT application with JavaFX substitutes
• Items that were on my horizon
– Integrate example BOY screens
– BOY JavaFX widgets
– Simplify error reporting for connection problems (e.g.
collect pvmanager/caj/pva logs to report to
developers)
• Work currently unfunded
WebPODS
• Web gateway based on WebSocket + JSON and
pure HTML/CSS widgets
• Main benefits:
– Uses web standards (any web client in any language)
– Widgets can be configured through standard CSS
– Data access outside of control network (possibly
WAN)
– Web pages do not need to live on the same server
where the gateway is
WebPODS
• Specs available on the WIKI:
– https://github.com/diirt/diirt/wiki/WebPodsProtocol-Specification-v1
– https://github.com/diirt/diirt/wiki/JSONserialization-for-vTypes-v1
Chrome extension: Simple Web Socket Client
WebPODS when installed
HTML Probe (Danielle Connolly, UMich)
Text monitor examples
LED examples
WebPODS
Specify server location
Adding text-monitor
Adding led
Specify color for “LOW” enum value
Using CS-Studio off site through Web Pods
Web Pods server configuration
[xxx@diirt ~]# more .diirt/pods/web/mappings.xml
<?xml version='1.0' encoding='UTF-8'?>
<mappings version="1">
<mapping channel="calibration/constants"
substitution="file:///path/to/file/table.csv"
permission="READ_ONLY"/>
<mapping channel="public-.*" permission="READ_WRITE"/>
<mapping channel="cf-tag-(.\w)" substitution="=cfQuery($1)"
permission="READ_ONLY"/>
<mapping channel="sim/(.*)" substitution="sim://$1"
permission="READ_ONLY"/>
</mappings>
Access security planned, not yet implemented
• Use wss (like https) for authentication
• Use username/role/unix group/host for authorization
Be careful not to expose too much
Web Pods
•
What’s done:
–
–
–
–
–
–
•
What needs to be done:
–
–
–
–
–
•
Version 1 specification done and implemented
Preliminary Javascript API
Finalized HTML/CSS compliant widgets (text-monitor, led, table, bubble-graph)
Being used in production at UMich for 6 months under light load. No downtime.
Client-side Java API
Client-side and server-side formulas
More widgets
Client-side security (server side is implemented)
Load tests
Review Javascript API
Client-side reconnected and ping/pong
What could be done:
– Investigate binary protocols (for large data or pre-calculated graphs)
– Make single widget spec for HTML and JavaFX widgets