Metadata-Version: 2.4
Name: sollertia-forgery
Version: 1.0.0rc5
Summary: Provides tools for processing and managing the data acquired using the Sollertia data acquisition platform.
Project-URL: Homepage, https://github.com/Sun-Lab-NBB/sollertia-forgery
Project-URL: Documentation, https://sollertia-forgery-api-docs.netlify.app/
Author: Ivan Kondratyev, Natalie Yeung, Kushaan Gupta
Maintainer-email: Ivan Kondratyev <ik278@cornell.edu>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: data,dataset,processing,sollertia
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering
Classifier: Typing :: Typed
Requires-Python: <3.15,>=3.14
Requires-Dist: ataraxis-base-utilities<7,>=6.0.2
Requires-Dist: ataraxis-communication-interface<7,>=6
Requires-Dist: ataraxis-data-structures<7,>=6.2.0rc1
Requires-Dist: ataraxis-time<7,>=6
Requires-Dist: ataraxis-video-system<5,>=4
Requires-Dist: cindra<3,>=2.0.0rc3
Requires-Dist: filelock<4,>=3
Requires-Dist: genicam==1.6.0
Requires-Dist: intel-cmplr-lib-rt<2027,>=2026; sys_platform != 'darwin'
Requires-Dist: mcp[cli]<2,>=1
Requires-Dist: natsort<9,>=8
Requires-Dist: numba<1,>=0.64
Requires-Dist: numpy<3,>=2
Requires-Dist: opencv-python<5,>=4.13
Requires-Dist: pandas<4,>=3
Requires-Dist: paramiko<6,>=5
Requires-Dist: polars<2,>=1
Requires-Dist: psutil<8,>=6
Requires-Dist: sollertia-shared-assets<10,>=9.1.0rc4
Requires-Dist: tables<4,>=3
Requires-Dist: tabulate<1,>=0
Requires-Dist: tbb4py<2024,>=2023; sys_platform != 'darwin'
Description-Content-Type: text/markdown

# sollertia-forgery
STUB

## Usage

This section has been transferred from sollertia-shared-assets and requires verification before 1.0.0 release!

### Configuring Server Access

To access the remote compute server, first author the server configuration. The configuration is stored inside the
'server_configuration.yaml' file under the Sollertia platform working directory, and is created with the
`slf server configure` command. It records the username, the password, the host, the absolute path to the server's
data root, and the name of the shared conda environment every remote job activates.

### Running a Remote Batch

The remote path runs the same prepared jobs the local batch engine runs, so one job graph, one core table, and one
memory model serve both. Locality is a property of execution rather than of planning.

A run has three steps, each exposed as a Model Context Protocol tool and backed by the `slf` CLI:

1. **Prepare.** `prepare_remote_batch_tool` refreshes the project's two tables on the server and pulls them. Planning
   (`slf plan`) reads each unit's acquisition data, registers the jobs that unit can actually run on its processing
   tracker, and records each job's cores, memory, and upstream jobs. State generation (`slf manifest create`, or
   `slf dataset-state` for a dataset) turns those trackers into a table. A job absent from that table is a job the
   unit cannot run, which is what lets the submitting host resolve a batch without opening anything on the server.
   The two tables join on the job identifier and become one descriptor per job, registered under a batch identifier.
2. **Submit.** `execute_remote_jobs_tool` submits each job as its own SLURM allocation, sized from its own estimate,
   in dependency order. Each job names the allocations of the upstream jobs the batch holds through an `afterok`
   dependency, so the scheduler sequences the graph and nothing has to stay running locally for the batch to finish.
3. **Read.** `get_remote_processing_status_tool` reports what the scheduler observed while a run is in flight.
   `sync_remote_state_tool` regenerates the project's manifest, job table, plan, and dataset state on the server and
   mirrors them into the working directory, where every read tool reads them exactly as it reads a local project.

Because the scheduler owns the run once it accepts the jobs, every accepted allocation is recorded in a submission
ledger at `<working directory>/remote_state/submission_ledger.yaml`, written under a file lock like every other shared
artifact this library keeps. That is what keeps concurrent batches all queryable, keeps a batch findable after this
process exits, and keeps the allocations already accepted recorded when the scheduler rejects a later job of the same
batch. Finished batches are pruned once the ledger holds more of them than it retains, and an unfinished batch is
never dropped.

A job whose upstream stage the run can neither queue nor find already succeeded is reported as blocked rather than
submitted, which matches what a local batch does with the same job.

### Running Headless Jobs

A headless job is a job that does not require any user interaction during runtime. Currently, all headless jobs in the 
sollertia platform rely on pip-installable packages that expose a callable Command-Line Interface to carry out
some type of data processing. In this regard, **running a headless job is equivalent to calling a CLI command on your local 
machine**, except that the command is executed on a remote compute server. Therefore, the primary purpose of the API 
exposed by this library is to transfer the target command request to the remote server, execute it, and monitor the 
runtime status until it is complete.

For example, the [cindra package](https://github.com/Sun-Lab-NBB/cindra) maintained in the sollertia platform exposes a CLI to 
process 2-Photon data from experiment sessions. During data processing by the 
[sollertia-forgery](https://github.com/Sun-Lab-NBB/sollertia-forgery) library, a remote job is sent to the server that uses the CLI 
exposed by the cindra package to process target session(s).

### Creating Jobs
All remote jobs are sent to the server in the form of an executable *shell* (.sh) script. The script is composed on the 
local machine that uses this library and transferred to a temporary server directory using Secure Shell File 
Transfer Protocol (SFTP). The server is then instructed to evaluate (run) the script using SLURM job manager, via a 
Secure Shell (SSH) session.

Broadly, each job consists of three major steps, which correspond to three major sections of the job shell script:
1. **Setting up the job environment**. Each job script starts with a SLURM job parameter block, which tells SLURM 
   what resources (CPUs, GPUs, RAM, etc.) the job requires. When resources become available, SLURM generates a virtual
   environment and runs the rest of the job script in that environment. This forms the basis for using the shared
   compute resources fairly, as SLURM balances resource allocation and the order of job execution for all users.
2. **Activating the target conda environment**. Currently, all jobs are assumed to use Python libraries to execute the 
   intended data processing. Similar to processing data locally, each job expects the remote server to provide a 
   Conda environment preconfigured with necessary assets (packages) to run the job. Therefore, each job contains a 
   section that activates the user-defined conda environment before running the rest of the job.
3. **Executing processing**. The final section is typically unique to each job and calls specific CLI commands or runs 
   specific Python modules. Since each job is submitted as a shell script, it can do anything a server shell can
   do. Therefore, despite python-centric approach to data processing in the sollertia platform, a remote job composed via this library 
   can execute ***any*** arbitrary command available to the user on the remove server.

Use the *Job* class exposed by this library to compose remote jobs. **Steps 1 and 2** of each job are configured when
initializing the Job instance, while **step 3** is added via the `add_command()` method of the Job class:
```python
from pathlib import Path
from sollertia_forgery.server import Job

# Instantiates a job. The resource arguments become the SBATCH directive block, and 'dependencies' names the
# allocations that must complete successfully before this job runs.
job = Job(
    job_name="0000-Session-motion_energy-1",
    output_log=Path("/server/root/processing_batches/batch01/0000-Session-motion_energy-1.out"),
    error_log=Path("/server/root/processing_batches/batch01/0000-Session-motion_energy-1.err"),
    working_directory=Path("/server/root/processing_batches/batch01"),
    conda_environment="slf_server",
    cpu_threads=16,
    ram=6,
    time=480,
    dependencies=("1000",),
)

# Adds the command the job runs. Commands added this way run under shell error checking, so the job exits with the
# status of the first command that fails.
job.add_command("slf process -sp /server/root/Project/Animal/Session -w 16 -np -id a1b2c3d4 video")
```

The rendered script removes itself through an exit trap rather than through a trailing command, so its exit status
stays the status of the work it ran. That is what a dependent allocation is sequenced against.

### Submitting and Monitoring Jobs

To submit a job, use a **Server** instance. It reads the server configuration authored above and supports the context
manager protocol, so the connection closes however the block ends:
```python
from sollertia_forgery.server import Server, JobStatus, TERMINAL_JOB_STATUSES, get_server_configuration

with Server(configuration=get_server_configuration()) as server:
    job = server.submit_job(job=job)

    # Queries every allocation of a batch in one accounting call, keyed by the identifier the scheduler assigned.
    statuses = server.get_job_statuses(slurm_job_ids=[job.job_id])
    if statuses[job.job_id] in TERMINAL_JOB_STATUSES:
        print(f"Job finished as {statuses[job.job_id]}.")
```

`get_job_statuses()` returns a `JobStatus` per allocation. Alongside the states accounting reports, it resolves
`BLOCKED` for a queued job whose dependency can no longer be satisfied, which accounting still calls pending.

**Note!** Composing jobs by hand is the low-level path. Prefer the remote batch tools described above, which size
every allocation from the data it will process and build the dependency graph from each pipeline's own job ordering.

**Critical!** Since running remote jobs is largely equivalent to executing them locally, all users are highly encouraged
to test their job scripts locally before deploying them server-side. If a script works on a local machine, it is likely
that the script would behave similarly and work on the server.
