Overview

SimStack II

SimStack II is a modern, Python-based workflow system with a web-based graphical user interface (GUI). The GUI is generated automatically from your Python workflow code and supports interactive inspection of inputs, outputs, tables, and plots.

Key features

  • GUI-driven execution: Workflows are submitted via a graphical user interface, which is auto-generated from the Python code.

  • Persistent results: Inputs and results are stored automatically in a non-SQL database so they can be reused and re-analyzed.

  • Flexible workflows: Workflows can be generated dynamically at runtime, nested, and executed across different remote resources.

  • Artifacts for analysis: Tables, plots, and other artifacts can be created during or after workflow execution to support data analysis.

  • Simple authoring model: Workflows are written as decorated plain Python functions.

Roadmap

Future versions are planned to (roughly in this order):

  • integrate research data management

  • provide tools to wrap other workflow systems

  • implement artifact generation via the GUI

  • enable data aggregation via the GUI

  • enable workflow development via the GUI

For a quick start, see Simstack II Setup and start using SimStack II via the graphical user interface.

Motivation

Computational scientists produce data that ultimately becomes figures and tables in publications. To aid this process many workflow environments already exist: Wikipedia lists a wide range of systems such as Airflow, Luigi, Nextflow, and Snakemake. Many widely used tools are file-based: a workflow is defined in a static configuration file and executed by an engine. This approach can be robust and has been very successful, but it often becomes limiting when workflows must be assembled dynamically or adapted during execution.

In parallel, the AI/ML ecosystem has accelerated the development of powerful Python-native workflow systems such as covalent, prefect, and pyiron. These systems embrace Python directly, enabling more dynamic and expressive computation pipelines.

Our experience with SimStack I suggests there are two only partially overlapping communities:

  • expert workflow developers (in python or otherwise) who struggle to develop user-interfaces make these workflows accessible to end-users

  • a larger group of workflow users, but may not have the software engineering background to develop the workflows on their own.

Bridging developers and users

SimStack II aims to bridge this gap by making workflow execution and result exploration accessible through a GUI, while keeping workflow implementation in Python. This enables rapid development of workflows using the full power of the python ecosystem. We believe this approach is much more powerful than attempting to construct complex workflows in a graphical user interface.

Developers who can construct workflows via a GUI typically have the ability to write them directly in Python. A GUI for workflow development thus adds overhead without providing much benefit. A GUI for workflow execution and analysis, however, can substantially lower the barrier for many users.

In developing a workflow system for scientific applications, we also need to consider another issue: Workflows in IT settings (e.g., server or database maintenance) tend to be stable: the goal is an environment where everything works all the time.

Scientific workflows often operate in a different environment: methods, hypotheses, and analysis requirements evolve continuously. A workflow may succeed technically while producing results that invalidate the motivating hypothesis—triggering new analysis and iteration.

This reality makes persistence, traceability, and interactive re-analysis essential.

Design goals

SimStack II is designed with the following priorities:

  • Workflows are initialized, monitored, and managed through a GUI on a variety of compute resources.

  • Results are persisted and accessible for re-analysis, ideally through the GUI.

  • Workflow components are implemented in Python, with minimal coding overhead.

  • Workflows expose results in a UI that is (to a large extent) automatically generated on the basis of the python code additional frontend development.

Simstack Architecture

Simstack II was implemented using the following architecture:

GUI Concept Design

Overview of the graphical user interface concept

Starting at the bottom right, then end user interacts with a web-based graphical user interface (GUI) implemented using Next-JS. The GUI interacts with a backend implemented using FastAPI, which in turn interacts with a MongoDB database to persist data. The backend also interacts with remote resources, so called runners, where the python code is actually executed. The graphical user interface is decribed in more detail in using_gui_section.

In a typical installation Simstack II requrires only one instance of the Next-JS frontend, one instance of the FastAPI backend and one instance of the MongoDB database server. Each user has his/her own database in the MongoDB server. The runners are installed for each user on the required remote resources and automatically update themselves from one or several GIT repositories, which may be different for different users.