Miscellaneous
==============


BUGS:
-----

Optional fields in embedded models are not handled correctly

https://github.com/art049/odmantic/issues/370
https://github.com/art049/odmantic/issues/484

References
----------



The challenge in Simstack II is to exploit the features of Next-JS frontend
without forcing the developer to write Next-JS or server components. In
Simstack II each user has his/her own MongoDB database, there is an
additional database used exclusively for authentication.

.. note:: Only one instance of a mongodb database server, of the fastapi
   server and of the next-js frontend is required per installation.


The workflow developer customizes the appearance of workflows for submission,
data analysis and data visualization using three main mechanisms:

* `React JSON Schema Form <https://rjsf-team.github.io/react-jsonschema-form/>`_
* `AG-Grid <https://www.ag-grid.com/>`_ is used to generate interactive tables (via Artifacts)
* `AG-Charts <https://charts.ag-grid.com/>`_ is used to generate interactive graphs (via Artifacts)

Customizing Simstack Models
---------------------------

.. note:: General information about Simstack models

Simstack models are built on top of `odmantic`, which uses `pydantic` for data validation.
By decorating a class with `@simstack_model`, you enable additional UI-related features.
You can customize how a model appears in the GUI by implementing `json_schema` and `ui_schema` methods.
Refer to :ref:`persisting_data` for more details.



Running the above code as a workflow we would like to achieve the following:

* code can be executed on remote resources (possibly in parallel)
* users can inspect the results in a GUI
* results can be reused when the workflow failed or in other workflows
* the workflow can be run from a GUI
* (workflows can be created from existing components in a GUI)

To address the first points, all data must be serializable. This can be achieved by pickling the inputs/outputs but
then the data is hidden in the pickled string and not searchable. Here we have chosen odmantic classes to persist the
data in a database. See: :ref:`persisting-results-section`. This has the charm that routes to access this data can be
automatically generated.

Overall we need four "servers" to run the workflow:

* The "workflow server" that manages and executes workflows
* The "runners" that execute parts of workflows on remote resources
* The "database server" provides an API to the database for the GUI
* The webserver that serves the GUI


.. code-block:: python

    @node(parameters=Parameters(resource="my_resource", queue="slurm-queue"))
    def my_node(arg1: Model1, arg2: Model2, **kwargs) -> SimstackResult:
        # do something with inputs
        return outputs



.. note::
  - Each node has a unique task_id which is the ObjectId of entry in the
    node_registry table in the database
  - Before control is passed to a node, the calling function will create
    a directory as workdir/node_name/node_id which is unique to the node.
  - all files generated by the node should be written into this directory.
  - nodes can by synchronous or async functions.


**Allowed Arguments**

- All **positional** arguments of the wrapped function (my_node) must be
  **registered** odmantic Models. These arguments are automatically persisted
  in the database.
- The my_node function must have \*\*kwargs even if they are not used in the
  function.
- \*\*kwargs is used to pass information to process the node, but
  should not be used to pass arguments that affect the result of the
  function. Presently, Model-type arguments in the \*\*kwargs are not persisted
  in the database which precludes the execution of such nodes on resources
  which are different from the function is called.

.. important::
  Only positional arguments should affect the outcome of the function execution

**Return Types**

While it is easy to check whether the variables
passed to nodes can be persisted in the database, this can be checked for
results only in post-processing.

