Program Execution¶

When developers write new nodes that execute external programs they cannot know the specifics of the resources on which these nodes are to be run. Simstack separates the program execution logic from the specifics of a particular resource. The ResourceConfig class handles resource-specific configurations from a config.toml file. It provides methods to retrieve program parameters, setup scripts, and run commands in a controlled environment (optionally using temporary directories).

The config toml is structured by resources, i.e. if you have resources hpc1 and small-computer and my-laptop sections starting with [hpc1], [small-computer], [my-laptop] contain the data for each of these nodes. The convenience functions return the data for the actual resource. In principle you are totally free which information you store in config.toml for your own nodes, but the convenience functions implement often used features, such as

using scratch space for program execution and where that is
startup scripts
the actual command to execute to run the programs

Note

An instance of ResourceConfig is provided by the context

The documentation below explains the current defaults. Standard nodes are then simply executed by:

from pathlib import Path
from simstack.core.context import context

input_files: List[str | FileStack] = [...]
output_files: List[str | FileStack] = [...]
result = context.resource_config.run("program-name", input_files, output_files)

ResourceConfig Class¶

The ResourceConfig class is located in simstack.util.resource_config.

class simstack.util.resource_config.ResourceConfig(config_path: Path, resource: str)[source]¶

Bases: object

ResourceConfig is responsible for managing configuration settings, setup, execution, and post-processing parameters for specified resources.

This class is designed to read configuration data from a TOML file, provide an interface to access resource-specific parameters, and execute resource-related operations such as setup and running commands. It encapsulates functionality for handling temporary directories, file manipulation, and subprocess execution.

_config¶

The loaded configuration dictionary.

Type:: Dict[str, Any]

_resource¶

The name of the current resource.

Type:: str

get_postprocessing_params() → Dict[str, Any][source]¶: Returns the post-processing dict for the specified resource. Expected structure in TOML: [resource_name.post-processing] or [resource_name.postprocessing]

get_program(program_name: str) → Dict[str, Any][source]¶: Returns the dict from resource.program.name for program with name and the current resource. Expected structure in TOML: [resource_name.program.program_name]

get_setup_params() → Dict[str, Any][source]¶: Returns the setup dict for the specified resource. Expected structure in TOML: [resource_name.setup]

property os: str¶: Returns the OS of the current resource, defaults to ‘linux’.

Executes the run command with optional temporary directory usage and file handling. Retrieves parameters from the configuration for the specified program.

Parameters:

program_name – Name of the program to run.
input_files – List of input files (str or FileStack). Overrides TOML input_files if provided.
output_files – List of output files (str or FileStack). Overrides TOML output_files if provided.
node_runner – Optional NodeRunner instance for execution.

setup(node_runner: Any | None = None)[source]¶: Executes the setup scripts for the current resource.

property tmp_base_dir: Path¶

tmp_dir(task_id: ObjectId | str) → Path[source]¶

Initialization¶

The ResourceConfig is initialized with a path to the configuration file (or directory containing config.toml) and the name of the resource.

from pathlib import Path
from simstack.util.resource_config import ResourceConfig

config = ResourceConfig(Path("/path/to/project"), "local")
config.program = "orca"

Methods¶

os: Property that returns the operating system of the resource (defaults to “linux”).
program: Property to get or set the current program name.
setup(node_runner=None): Executes the setup scripts defined for the resource.
run(program_name=None, input_files=None, output_files=None, node_runner=None): Executes the program command, handling file transfers and temporary directories based on configuration.
get_program(): Retrieves configuration for the current program.
get_setup_params(): Retrieves setup configuration for the current resource.
get_postprocessing_params(): Retrieves post-processing configuration.

Integration with NodeRunner¶

If a node_runner instance is passed to setup() or run(), it will use node_runner.subprocess() to execute commands, which provides better logging and integration with SimStack’s task management.

Config.toml Structure¶

The config.toml file is organized by resource name. Each resource can have several sections:

General Resource Settings¶

[resource_name]
os = "linux"  # optional, defaults to "linux"

Setup Section¶

Contains scripts to be executed before any program run and configuration for temporary directories.

[resource_name.setup]
scripts = [
    "module load orca",
    "export MY_VAR=value"
]
tmp_base_dir = "/path/to/scratch" # Can also be a command like "set TMP_BASE_DIR=..."

Post-processing Section¶

Contains settings for cleanup after execution.

[resource_name.post-processing] # or [resource_name.postprocessing]
scratch_cleanup = true

Program Section¶

Defines configuration for specific applications on the resource.

[resource_name.program.orca]
use_tmp = true
run_command = "orca orca.inp > orca.out"

Example config.toml¶

[local]
os = "windows"

[local.setup]
scripts = ["set USER=%USERNAME%"]
tmp_base_dir = "C:\\Temp"

[local.program.orca]
use_tmp = true
run_command = "orca orca.inp > orca.out"