Support for I/O in Workflow Engines

This work will be embedded in the ACES research group.

If you are interested in this topic or similar topics, contact Dr. Julian Kunkel.

Workflow engines orchestrate tasks defined as dependency graphs efficiently. In High-Performance Computing, such engines are, for example, used to define the workflows of generating a weather prediction and execute them on supercomputers consisting of 1000s of servers. However, the current workflow engines neglect the requirements for input/output (I/O) in their description and, thus, lack the opportunity of optimization. In this work, the CYLC workflow engine is extended to cover I/O requirements in the description and graphical representation and develop a first scheduling algorithm that aims to minimize I/O between steps.

Firstly, a model and graphical representation are developed describing the relevant facets of I/O that should be covered. Next, the existing description language is extended with the model. A scheduling algorithm is developed (e.g., in Python) and evaluated that considers the The developed scheduler will act as a prototype showcasing the benefits when including I/O for the existing CYLC community.

While all skills needed to complete this project can be obtained during the time of the project, some skills are beneficial:

• Programming languages: Python (intermediate level)
• Linux (intermediate)

An MSc candidate is expected to bring soft skills (in decreasing order of importance):

• Communication
• Problem-Solving
• Time management