Workflow Management
Concepts and Requirements
For Scientific Applications
The Two Tier View of a Workflow
Management system (generic)
Workflow layers
– Control-flow layer
– Application and Software Tools layer
– I/O System layer
– Storage and Network Resource layer
Anatomy of a scientific workflow
management system
Flow Tier
Task A:
Generate
Time-Steps
Work Tier
Simulation
Program
Parallel
NetCDF
Task B:
Move TS
Data
Mover
PVFS
Task C:
Analyze TS
Post
Processing
LN
+
Parallel
R
SRM
Task D:
Visualize TS
Terascale
Browser
HDF5
Libraries
Control Flow
Layer
Applications &
Software Tools
Layer
I/O System
Layer
Storage & Network
Resouces
Layer
Architecture of a workflow
management system
Design time
(Process modeling)
Workflow engine
Tracking
Tools
Process
Design Tool
Invoke Tasks
Process
Database
Post run time
Run time
Application code
Software Tools
Workflow execution
history database
Data Mining
&
Analysis Tool
Main Services Offered by
Workflow Management Systems
• Workflow design tools (GUI interfaces)
• Automatic sequencing of component
invocation
• Synchronizing data flow between
components
• Tracking and reporting mechanisms
Discussion items
• Control flow tier
–
–
–
–
–
Granularity of tasks, sub-workflows
Task Invocation - Web Services, Corba, Wrappers, Callbacks
Human tasks: Notifications and alerts, steering
Dataflow streaming granularity
Performance expectations (provoke alerts)
• Work Tier
–
–
–
–
–
–
–
Workflow engine for scientific applications
Integrated dataflow management
Failure detection and recovery
Data-driven flow control
Performance-driven flow control
Workflow optimization
Run-time resource coordination
Discussion items
Research
and
Development
- Granularity of tasks, sub-workflows
X
- Task Invocation
X
- Human tasks: Notifications, alerts, steering
- Dataflow streaming granularity
X
X
- Performance expectation
X
- Workflow engine for scientific applications
X
- Integrated dataflow management
- Failure detection and recovery
X
X
- Data-driven flow control
- Performance-driven flow control
X
X
- Workflow optimization
- Run-time resource coordination
Deployment
and
maintenance
Hardening
and
Packaging
X
X
Other topic discussed
• Simulations setup
– reserve resources, schedule run
• Performance monitoring needed
– e.g. Disks getting full
– e.g. Some task is stalled or too slow
• Interoperation of components
– Match outputs to inputs (e.g. CCA)
• Allow feedback loops
• How to specify what to do in case of
failures/exceptions
Example Workflow Systems
Considered
• Scientific
–
–
–
–
–
Ptolemy/Kepler – UC Berkeley
DAGMAN – U Wisc, used in Griphyn
SciRun – U Utah, collaborate with CCA
(Taverna)
(Triana)
• Commercial
– FileNet
– Oracle Workflow
– IBM’s MQ flow