Miscellaneous

BUGS:

Optional fields in embedded models are not handled correctly

https://github.com/art049/odmantic/issues/370 https://github.com/art049/odmantic/issues/484

References

The challenge in Simstack II is to exploit the features of Next-JS frontend without forcing the developer to write Next-JS or server components. In Simstack II each user has his/her own MongoDB database, there is an additional database used exclusively for authentication.

Note

Only one instance of a mongodb database server, of the fastapi server and of the next-js frontend is required per installation.

The workflow developer customizes the appearance of workflows for submission, data analysis and data visualization using three main mechanisms:

• React JSON Schema Form

• AG-Grid is used to generate interactive tables (via Artifacts)

• AG-Charts is used to generate interactive graphs (via Artifacts)

Customizing Simstack Models

Note

General information about Simstack models

Simstack models are built on top of odmantic, which uses pydantic for data validation. By decorating a class with @simstack_model, you enable additional UI-related features. You can customize how a model appears in the GUI by implementing json_schema and ui_schema methods. Refer to Persisting and Visualizing Data for more details.

Running the above code as a workflow we would like to achieve the following:

• code can be executed on remote resources (possibly in parallel)

• users can inspect the results in a GUI

• results can be reused when the workflow failed or in other workflows

• the workflow can be run from a GUI

• (workflows can be created from existing components in a GUI)

To address the first points, all data must be serializable. This can be achieved by pickling the inputs/outputs but then the data is hidden in the pickled string and not searchable. Here we have chosen odmantic classes to persist the data in a database. See: persisting-results-section. This has the charm that routes to access this data can be automatically generated.

Overall we need four “servers” to run the workflow:

• The “workflow server” that manages and executes workflows

• The “runners” that execute parts of workflows on remote resources

• The “database server” provides an API to the database for the GUI

• The webserver that serves the GUI

@node(parameters=Parameters(resource="my_resource", queue="slurm-queue"))
def my_node(arg1: Model1, arg2: Model2, **kwargs) -> SimstackResult:
    # do something with inputs
    return outputs

Note

• Each node has a unique task_id which is the ObjectId of entry in the node_registry table in the database

• Before control is passed to a node, the calling function will create a directory as workdir/node_name/node_id which is unique to the node.

• all files generated by the node should be written into this directory.

• nodes can by synchronous or async functions.

Allowed Arguments

• All positional arguments of the wrapped function (my_node) must be registered odmantic Models. These arguments are automatically persisted in the database.

• The my_node function must have **kwargs even if they are not used in the function.

• **kwargs is used to pass information to process the node, but should not be used to pass arguments that affect the result of the function. Presently, Model-type arguments in the **kwargs are not persisted in the database which precludes the execution of such nodes on resources which are different from the function is called.

Important

Only positional arguments should affect the outcome of the function execution

Return Types

While it is easy to check whether the variables passed to nodes can be persisted in the database, this can be checked for results only in post-processing.