Datasets

SimStack II uses a structured way to handle datasets through the DataSet, DataSetSection, and DataSetSelection models defined in simstack.models.dataset.

DataSet

The DataSet model is the top-level container for data.

It consists of:

  • metadata: A DataSetMetadata object that defines the type and validation rules for the dataset.

  • sections: A dictionary of DataSetSection objects, keyed by section name.

It behaves like a dictionary of sections. Think of a dataset as an excel file, where each DataSetSection is a sheet.

Metadata

Metadata in SimStack II is handled by the DataSetMetadata and DataSetMetadataTemplate classes.

  • DataSetMetadata: This is an EmbeddedModel used within a DataSet. It behaves like a dictionary and stores key-value pairs of information about the dataset (e.g., experimental conditions, parameters). It also ensures structural consistency of the data.

  • DataSetMetadataTemplate: This model defines the expected structure and schema for a specific type of dataset metadata. When a new DataSet is saved, its metadata is validated against the corresponding template.

The metadata provides: * Validation: Ensures that all datasets of the same type have consistent metadata keys and value types. * JSON Schema: Automatically generates JSON schemas for the stored metadata. * Dict-like API: Supports standard dictionary operations like __getitem__, keys(), items(), and update().

DataSetSection

A DataSetSection represents a collection of rows within a dataset. Each row is a dictionary of models, but only the model ids are stored when saving.

Key features: * Type consistency: All rows in a section must have the same model types for the same keys. * Lazy loading: Models are cached and only loaded from the database when needed. * Table representation: It can automatically generate column definitions and table entries for UI components (like ag-grid).

To add data to a section, you can use the add_row method:

section.add_row({"model_a": instance_a, "model_b": instance_b})

DataSetSelection

The DataSetSelection model is used to reference specific items within a DataSet. Instead of duplicating data, it stores the dataset_id and a list of indices for each section.

This is particularly useful for workflows where a user selects a subset of a dataset to be processed by a subsequent node.

Example usage:

selection = DataSetSelection(dataset_id=my_dataset.id)
selection.dataset_selection_fields.append(
    DataSetSelectionField(section_name="default", indices=[0, 2, 5])
)

API Reference

class simstack.models.dataset.DataSet(*, field_name: str = 'dataset', metadata: DataSetMetadata, sections: dict[str, ~simstack.models.dataset.DataSetSection]=<factory>, id: ObjectId = <factory>)[source]

Bases: Model

clear()[source]
collect_structure() → Dict[str, Dict[str, str]][source]
async custom_model_dump(**kwargs) → Dict[str, Any]

Custom model dump method to handle the conversion of model instances to dictionaries. This method recursively traverses dictionaries and lists to convert any nested model instances to their dictionary representation.

Parameters:

  • self – The model instance

  • kwargs – Additional keyword arguments

Returns:

A dictionary representation of the model instance

property dataset_type: str
field_name: str = <odmantic.field.FieldProxy object>
classmethod from_dict(data: dict, **kwargs) → Any

Create an instance of the model from a dictionary. Handles nested models and enum values.

classmethod from_model(model: Model, **kwargs) → Model
get(key: str, default: DataSetSection = None) → DataSetSection[source]
id: ObjectId = <odmantic.field.FieldProxy object>
items() → ItemsView[str, DataSetSection][source]
classmethod json_schema()

Generates a JSON schema for the given class and its fields, but eliminates all fields which are models, embedded models, or references to models.

Parameters:

cls

Returns:

keys() → KeysView[str][source]
metadata: DataSetMetadata = <odmantic.field.FieldProxy object>
model_config = {'arbitrary_types_allowed': False, 'collection': None, 'extra': 'forbid', 'indexes': None, 'json_schema_extra': None, 'parse_doc_with_default_factories': False, 'str_strip_whitespace': False, 'title': None, 'validate_assignment': True, 'validate_default': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

pop(key: str, default=None) → DataSetSection[source]
async save(db)[source]
sections: dict[str, DataSetSection] = <odmantic.field.FieldProxy object>
setdefault(key: str, default: DataSetSection = None) → DataSetSection[source]
classmethod ui_make_title(ui_schema: Dict[str, Any], field: str, title: str) → dict

Adds a title to the JSON schema.

Parameters:

  • cls – The class to which the JSON schema belongs

  • ui_schema – The original ui_schema schema

  • title – Title to be added

Returns:

Modified JSON schema with title

classmethod ui_schema() → dict[source]
update(other: Dict[str, DataSetSection] | DataSet = None, **kwargs) → None[source]
values() → ValuesView[DataSetSection][source]
class simstack.models.dataset.DataSetSection(*, model_types: dict[str, str]=<factory>, data: dict[str, dict[str, ~odmantic.bson.ObjectId]]=<factory>, column_defs: list[dict[()]] = <factory>, table_entries: list[list[dict[()]]] = <factory>)[source]

Bases: EmbeddedModel

Represents a section of a dataset containing dictionaries of models.

A DataSetSection is a list of dictionaries where for each key, the values are of the same model type.

Variables:

  • model_types – Dictionary mapping keys to model class names.

  • data – Dictionary mapping names to dictionaries mapping keys to ObjectIds.

add_row(item: Dict[str, Model | None], name: str | None = None) → None[source]

Add a dictionary of models to this section.

Parameters:

  • item – Dictionary of model instances to add.

  • name – Optional name for the item. If None, a UUID will be generated.

Raises:

  • ValueError – If the model types don’t match the section’s expected types.

  • TypeError – If a non-None item value is not a Model instance.

clear() → None[source]
column_defs: list[dict[()]] = <odmantic.field.FieldProxy object>
async custom_model_dump(**kwargs) → Dict[str, Any]

Custom model dump method to handle the conversion of model instances to dictionaries. This method recursively traverses dictionaries and lists to convert any nested model instances to their dictionary representation.

Parameters:

  • self – The model instance

  • kwargs – Additional keyword arguments

Returns:

A dictionary representation of the model instance

data: dict[str, dict[str, ObjectId]] = <odmantic.field.FieldProxy object>
async db_find_postprocess(db: Database)[source]
classmethod from_dict(data: dict, **kwargs) → Any

Create an instance of the model from a dictionary. Handles nested models and enum values.

classmethod from_model(model: Model, **kwargs) → Model
get(name: str, default: Any = None) → Any[source]
get_item(name: str) → Dict[str, Model][source]
items() → ItemsView[str, Dict[str, Model]][source]
classmethod json_schema()

Generates a JSON schema for the given class and its fields, but eliminates all fields which are models, embedded models, or references to models.

Parameters:

cls

Returns:

keys() → KeysView[str][source]
async load_to_cache(db: Database) → None[source]

Load all items from the database into the cache assuming that data is already loaded.

async make_column_defs()[source]

Generate ag-grid column definitions for all model types in this section.

async make_table_entries()[source]
model_config = {'arbitrary_types_allowed': False, 'collection': None, 'extra': 'forbid', 'indexes': None, 'json_schema_extra': None, 'parse_doc_with_default_factories': False, 'str_strip_whitespace': False, 'title': None, 'validate_assignment': True, 'validate_default': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_types: dict[str, str] = <odmantic.field.FieldProxy object>
pop(name: str, default: Any = Ellipsis) → Any[source]
popitem() → Tuple[str, Dict[str, Model]][source]
async save(db)[source]

Save all models in the cache to the database and update data.

setdefault(name: str, default: Dict[str, Model | None] = None) → Dict[str, Model][source]
table_entries: list[list[dict[()]]] = <odmantic.field.FieldProxy object>
classmethod ui_make_title(ui_schema: Dict[str, Any], field: str, title: str) → dict

Adds a title to the JSON schema.

Parameters:

  • cls – The class to which the JSON schema belongs

  • ui_schema – The original ui_schema schema

  • title – Title to be added

Returns:

Modified JSON schema with title

classmethod ui_schema()

Generates a UI schema that uses GenericForm for fields with ui_schema function. Also preserves any existing UI schema configurations from the class.

Parameters:

cls – The model class to generate UI schema for

Returns:

The generated UI schema

Return type:

dict

update(other: Dict[str, Dict[str, Model | None]] | DataSetSection) → None[source]
values() → ValuesView[Dict[str, Model]][source]
class simstack.models.dataset.DataSetSelection(*, field_name: str = 'dataset_selection', dataset_id: ObjectId, dataset_selection_fields: list[DataSetSelectionField] = <factory>, id: ObjectId = <factory>)[source]

Bases: Model

async custom_model_dump(**kwargs) → Dict[str, Any]

Custom model dump method to handle the conversion of model instances to dictionaries. This method recursively traverses dictionaries and lists to convert any nested model instances to their dictionary representation.

Parameters:

  • self – The model instance

  • kwargs – Additional keyword arguments

Returns:

A dictionary representation of the model instance

dataset_id: ObjectId = <odmantic.field.FieldProxy object>
dataset_selection_fields: list[DataSetSelectionField] = <odmantic.field.FieldProxy object>
field_name: str = <odmantic.field.FieldProxy object>
classmethod from_dict(data: dict, **kwargs) → Any

Create an instance of the model from a dictionary. Handles nested models and enum values.

classmethod from_model(model: Model, **kwargs) → Model
async get_dataset()[source]
get_selected_elements(section_name: str = None) → List[Tuple[Model, ...]][source]

Retrieve all selected model groups from the dataset.

Parameters:

section_name – Optional section name to filter results. If None, returns all sections.

Returns:

List of tuples of model instances for all selected elements

id: ObjectId = <odmantic.field.FieldProxy object>
classmethod json_schema()

Generates a JSON schema for the given class and its fields, but eliminates all fields which are models, embedded models, or references to models.

Parameters:

cls

Returns:

model_config = {'arbitrary_types_allowed': False, 'collection': None, 'extra': None, 'indexes': None, 'json_schema_extra': None, 'parse_doc_with_default_factories': False, 'str_strip_whitespace': False, 'title': None, 'validate_assignment': True, 'validate_default': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

classmethod ui_make_title(ui_schema: Dict[str, Any], field: str, title: str) → dict

Adds a title to the JSON schema.

Parameters:

  • cls – The class to which the JSON schema belongs

  • ui_schema – The original ui_schema schema

  • title – Title to be added

Returns:

Modified JSON schema with title

classmethod ui_schema() → dict[source]
class simstack.models.dataset_metadata.DataSetMetadata(*, field_name: str, data: dict[str, str | int | float | bool | ~odmantic.bson._datetime]=<factory>, is_validated: bool = False, structure: dict[str, dict[str, str]]=<factory>)[source]

Bases: EmbeddedModel

clear()[source]

Clear data dict.

copy_data() → Dict[str, str | int | float | bool | datetime][source]

Return a copy of the data dict.

async custom_model_dump(**kwargs) → Dict[str, Any]

Custom model dump method to handle the conversion of model instances to dictionaries. This method recursively traverses dictionaries and lists to convert any nested model instances to their dictionary representation.

Parameters:

  • self – The model instance

  • kwargs – Additional keyword arguments

Returns:

A dictionary representation of the model instance

data: dict[str, str | int | float | bool | _datetime] = <odmantic.field.FieldProxy object>
field_name: str = <odmantic.field.FieldProxy object>
freeze(new_structure: Dict[str, Dict[str, str]]) → bool[source]
classmethod from_dict(data: dict, **kwargs) → Any

Create an instance of the model from a dictionary. Handles nested models and enum values.

classmethod from_model(model: Model, **kwargs) → Model
get(key: str, default=None)[source]

Get item from data dict with default.

get_json_schema()[source]
get_key_type(key: str) → type[source]

Get the type of a specific key.

get_schema_for_key(key: str) → dict[source]

Get JSON schema for a specific key.

property initialized: bool

Check if the model has been fully constructed.

is_type_compatible(key: str, value) → bool[source]

Check if a value is type-compatible with existing key.

is_validated: bool = <odmantic.field.FieldProxy object>
items()[source]

Get items from data dict.

classmethod json_schema()

Generates a JSON schema for the given class and its fields, but eliminates all fields which are models, embedded models, or references to models.

Parameters:

cls

Returns:

keys()[source]

Get keys from data dict.

model_config = {'arbitrary_types_allowed': False, 'collection': None, 'extra': None, 'indexes': None, 'json_schema_extra': None, 'parse_doc_with_default_factories': False, 'str_strip_whitespace': False, 'title': None, 'validate_assignment': True, 'validate_default': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

pop(key: str, *args)[source]

Pop item from data dict.

popitem()[source]

Pop item from data dict.

setdefault(key: str, default=None)[source]

Set default value for key if not exists.

structure: dict[str, dict[str, str]] = <odmantic.field.FieldProxy object>
classmethod ui_make_title(ui_schema: Dict[str, Any], field: str, title: str) → dict

Adds a title to the JSON schema.

Parameters:

  • cls – The class to which the JSON schema belongs

  • ui_schema – The original ui_schema schema

  • title – Title to be added

Returns:

Modified JSON schema with title

classmethod ui_schema()

Generates a UI schema that uses GenericForm for fields with ui_schema function. Also preserves any existing UI schema configurations from the class.

Parameters:

cls – The model class to generate UI schema for

Returns:

The generated UI schema

Return type:

dict

update(*args, **kwargs)[source]

Update data dict with validation.

async validate_dict(new_structure: Dict[str, Dict[str, str]]) → bool[source]
values()[source]

Get values from data dict.

class simstack.models.dataset_metadata.DataSetMetadataTemplate(*, dataset_type: str, model_json: dict[str, ~typing.Any], structure: dict[str, dict[str, str]] = <factory>, id: ~odmantic.bson.ObjectId = <factory>)[source]

Bases: Model

dataset_type: str = <odmantic.field.FieldProxy object>
id: ObjectId = <odmantic.field.FieldProxy object>
model_config = {'arbitrary_types_allowed': False, 'collection': None, 'extra': None, 'indexes': None, 'json_schema_extra': None, 'parse_doc_with_default_factories': False, 'str_strip_whitespace': False, 'title': None, 'validate_assignment': True, 'validate_default': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_json: dict[str, Any] = <odmantic.field.FieldProxy object>
structure: dict[str, dict[str, str]] = <odmantic.field.FieldProxy object>