Karabo: The European XFEL
software framework
Burkhard Heisen
2nd Collaboration Meeting of the European XFEL
April 24, 2013
Karabo: The European XFEL software framework
What I always start with…
2
A typical use case:
Control
drive hardware and
complex experiments
monitor variables &
trigger alarms
DAQ
data readout
online processing
quality monitoring
(vetoing)
allow some
control & show
hardware status
Accelerator Undulator Beam Transport
show online data
whilst running
DM
DAQ
DM
Control
SC
Tight integration of applications
Easy, efficient and comfortable usage of those
Burkhard Heisen (WP76)
storage of experiment
& control data
data access,
authentication
authorization etc.
setup computation &
show scientific results
SC
processing pipelines
distributed and GPU
computing
specific algorithms
(e.g. reconstruction)
Karabo: The European XFEL software framework
Outline – Follow the requirements
Phase
Independent
Proposal phase
Identify needs, Accept/Reject/Postpone
several month before
Preparation phase
Prepare configurations, custom developments
several month – few weeks before
Setup phase
Verify setup, find signal, alignments
few days – few hours before
Data taking phase
Clean up phase
Data analysis phase
Publication phase
Burkhard Heisen (WP76)
Data storage, sample switching/delivery
Remove custom h/w, bring s/w back to safe
Data analysis workflows, data management
Data cleanup
3
Karabo: The European XFEL software framework
Phase independent
4
GUI
Requirement
 Distinction between different users with associated
access restrictions
 Solution
 Karabo is user centered, i.e. any access to
control, (meta-)data, or computing infrastructure
requires login.
username
 Karabo interacts with a central database
userId
password
managing all user data.
accessRole
currentTime
session
 Access is controlled using a role-based system.
location
 Roles are assigned based on login context. The
context involves username, current time and
current location.
Central DB
 The same user for example may have more rights
during his beamtime and at a specific computer
1.
Authorizes
than at other places and during different times.
2.
Computes context based

access role
Burkhard Heisen (WP76)
Karabo: The European XFEL software framework
Preparation phase

5
Requirement
 Setup of experiment configuration, scan macros, etc.
 Solution
 XFEL system operators can prepare configurations and scan macros
according to information of the proposal.
 Configurations and macros are managed within Karabo’s central database.
 The control system is fully scriptable. The default language is Python.
 Control commands can be issued sequentially, or concurrently (e.g. move
several axes at the same time)
 Scripting can be done procedurally (python program) or interactively (IPython)
 Being Python any complicated constructs can be coded (i.e. n-D scans with
conditionals, etc.)
 Scripting can be done within the GUI
 Main API: get, set, monitor, execute, instantiate, kill
Burkhard Heisen (WP76)
Karabo: The European XFEL software framework
Preparation phase (special needs)


Requirement
 Implementation of new control devices and/or integration of new
algorithms to analysis pipeline
Solution
 Karabo is split into a core framework and extension plugins (devices).
 XFEL developers (or even the beamline users) can create new devices
for Karabo that can be injected into the runtime and will enable new
control or analysis capabilities.
 Several code templates exist for different categories of devices.
 Devices can be written in either C++ or pure Python (transparent to the
Karabo runtime).
 Devices impose a standard for self-description (properties, commands,
state-machine). GUI widgets and behavior such as auto-completion and
interactive scripting is automatically generated using this information.
 Devices can be deployed at any computer and are automatically made
available and ready to use for any control-clients.
Burkhard Heisen (WP76)
6
Karabo: The European XFEL software framework
Setup phase

Requirement
 Verification of h/w and s/w functionality, alignments, signal search, etc.
 Solution
 Sanity functionality within Karabo ensures correct technical setup (hosts
available, correct plugins loaded etc.)
 XFEL beamline scientist can use the GUI for inspecting parameters,
trendlines, histograms and images in conjunction with prepared macros
to verify correct behavior
 The GUI system uses the Device self-description to provide basic access
to all features.
 Custom panel creation needs no coding but is a process that is similar to
the way you create a PowerPoint slide.
 Panels can be loaded and saved per user. Sharing with other users is
also possible.
Burkhard Heisen (WP76)
7
Karabo: The European XFEL software framework
Multi-purpose GUI system
8
Configuration
Custom composition area
Navigation
Logging / Scripting console
Notifications
Kerstin Weger (WP76)
Documentation
Karabo: The European XFEL software framework
Property/Command composition
Display widget (Trend-Line)
Editable widget
drag & drop
Display widget
Kerstin Weger (WP76)
9
Karabo: The European XFEL software framework
Property/Command composition
Display widget
(Image View)
Display widget
(Histogram)
Kerstin Weger (WP76)
drag & drop
10
Karabo: The European XFEL software framework
Custom panels – PowerPoint like
Open/Save panel view
11
Cut, copy, paste, remove item
Group items
Change between
“Design/Live” mode
Bring to front/back
Insert text, line, rectangle, …
Kerstin Weger (WP76)
Rotate, scale item
Karabo: The European XFEL software framework
Data taking phase

Requirements
 Data collection, modification of variables, switching samples, online data
analysis
 Solutions
 Karabo is tightly integrated into the data management system. Within the
GUI, experiment runs can be started and stopped.
 Whilst running, Karabo monitors and saves all (slow-)control data which can
later be associated to the experiment raw data.
 Raw data is streamed to the PC Layer, formatted and is saved in the online
storage as HDF5 files.
 Karabo scientific workflow system can directly be fed with the raw data
allowing highly parallel, pipelined processing whilst data taking.
 The GUI allows monitoring experiment statistics and data-analysis runs.
 Results of online data analysis can be used for tuning the experiment and
vetoing bad data (those won’t even travel to the offline storage)
Burkhard Heisen (WP76)
12
Karabo: The European XFEL software framework
Device (workflow) composition
Workflow node (device)
drag & drop
Draw connection
Kerstin Weger (WP76)
13
Karabo: The European XFEL software framework
Data analysis phase

Requirements
 Pre-analysis on-site up to preliminary results, full analysis off-site
 Solutions
 The same workflow system as already available during data taking can be
used in offline mode. Workflows are now fed by data as available from the
offline storage (or - if still available - from the online storage).
 Data analysis for the user starts with already calibrated data
 Data calibration is also implemented as a workflow (but maintained by XFEL
and accessing calibration parameters from Karabo’s central resources)
 The analysis system will provide common image processing algorithms
making also use of GPU technology
 Community algorithms can easily be integrated to Karabo and shared
amongst scientists. XFEL.EU plans to build up a central resource for state of
the art analysis tools.
Burkhard Heisen (WP76)
14
Karabo: The European XFEL software framework
Processing workflows
15
CPU
IO whilst computing
Pixel parallel processing
(one GPU thread per pixel)
Notification about new data possible to obtain
GPU
Burkhard Heisen (WP76)
Karabo: The European XFEL software framework
Yes, it’s there!
Burkhard Heisen (WP76)
16
Karabo: The European XFEL software framework
Conclusions and timelines

Challenges are important to learn about detailed requirements
 pnCCD (combines control, DAQ, calibration and analysis)
 Beckhoff test stands
 Slice test (gives ideas about data throughput)
 2-tile LPD, later ¼ Mpx LPD (imaging detector)
 CrystFEL integration (data analysis workflows)

First release of Karabo on 30th July
 Will cover what was mentioned in this talk
 Need more time to place the available technical solutions in
larger contexts and workflows
Burkhard Heisen (WP76)
17
Karabo: The European XFEL software framework
18
Thank you for your kind attention.
Burkhard Heisen (WP76)