Metadata-Version: 2.4
Name: wetlands
Version: 2.3.2
Summary: Create isolated Pixi environments and run Python functions without dependency conflicts.
Project-URL: Homepage, https://arthursw.github.io/wetlands/latest/
Project-URL: Documentation, https://arthursw.github.io/wetlands/latest/
Project-URL: Repository, https://github.com/arthursw/wetlands
Project-URL: Issues, https://github.com/arthursw/wetlands/issues
Project-URL: Changelog, https://github.com/arthursw/wetlands/blob/main/CHANGELOG.md
Author-email: Arthur Masson <arthur.masson@inria.fr>
License-Expression: MIT
License-File: LICENSE
Classifier: Intended Audience :: Developers
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Python: >=3.9
Requires-Dist: packaging>=24.2
Requires-Dist: psutil>=6.1.0
Requires-Dist: tomli>=2.0; python_version < '3.11'
Requires-Dist: typing-extensions>=4.0; python_version < '3.11'
Provides-Extra: docs
Requires-Dist: mike>=2.1.3; extra == 'docs'
Requires-Dist: mkdocs-gen-files>=0.5.0; extra == 'docs'
Requires-Dist: mkdocs-include-markdown-plugin>=7.1.5; extra == 'docs'
Requires-Dist: mkdocs-literate-nav>=0.6.2; extra == 'docs'
Requires-Dist: mkdocs-material>=9.6.11; extra == 'docs'
Requires-Dist: mkdocstrings[python]>=0.29.1; extra == 'docs'
Requires-Dist: pygments<2.20; extra == 'docs'
Provides-Extra: shared-memory
Requires-Dist: numpy>=1.26; (python_version < '3.14') and extra == 'shared-memory'
Requires-Dist: numpy>=2.3.3; (python_version >= '3.14') and extra == 'shared-memory'
Description-Content-Type: text/markdown

![](Wetland.png)

# Wetlands

[![Wetlands tests](https://github.com/arthursw/wetlands/actions/workflows/ci.yml/badge.svg?event=push&branch=main)](https://github.com/arthursw/wetlands/actions/)
[![Wetlands PyPI](https://img.shields.io/pypi/v/wetlands.svg?color=%2334D058)](https://pypi.org/project/wetlands/)
[![Wetlands Python versions](https://img.shields.io/pypi/pyversions/wetlands.svg?color=%2334D058)](https://pypi.org/project/wetlands/)

Wetlands is a Python library for creating isolated environments with [Pixi](https://pixi.sh/) and running Python functions inside them.

This lets an application use libraries with incompatible dependencies at the same time.
For example, [Cellpose](https://www.cellpose.org/) and [StarDist](https://github.com/stardist/stardist) can each run in their own environment while exchanging ordinary Python values and NumPy arrays with the main application.

Wetlands creates these environments when needed, installs their dependencies, keeps worker processes ready for repeated calls, and cleans up their resources automatically.
It can be used in desktop applications, servers, and plugin systems.

> Wetlands is intended for code you trust.
> Isolated environments prevent dependency conflicts, but they do not restrict what code can access on your computer.

[Appose](https://github.com/apposed/appose) is an alternative for applications that need interprocess cooperation across Python, Java, or Groovy, including explicit zero-copy tensor sharing between languages.
Wetlands is focused on running Python functions and adds automatic NumPy transport, managed worker pools, and post-hoc debugger attachment.
See [Wetlands and Appose](docs/appose.md) for a short comparison, or visit the [Appose documentation](https://docs.apposed.org/).

Wetlands 2 provides:

- side-effect-light manager construction;
- observable and cancellable preparation and provisioning operations;
- reproducible Pixi projects and `pixi.lock` files;
- managed-environment discovery and safe asynchronous removal;
- warm worker pools;
- validated environment variables for individual worker indices;
- qualified installed-package targets and path targets for local development;
- automatic transport of ordinary Python values and NumPy arrays;
- blocking, callback-based, and `asyncio`-friendly execution.

The first preparation may download a verified Pixi executable, and the first provisioning of an environment downloads its declared packages.
These operations require network access and can take several minutes.
Wetlands stores Pixi, managed environments, locks, and runtime state below the manager root you choose.

## Installation

```sh
pip install wetlands
```

Install the optional host-side NumPy dependency when arrays cross the execution boundary:

```sh
pip install "wetlands[shared-memory]"
```

## Quick start

The manager constructor only validates and stores configuration.
Downloading or inspecting Pixi begins when `prepare()` or `provision()` is called.

```python
import numpy as np

from wetlands import EnvironmentManager, EnvironmentSpec

manager = EnvironmentManager(root="wetlands")

preparation = manager.prepare()
preparation.listen(lambda event: print(event.stage, event.message))
pixi = preparation.wait_for()

spec = EnvironmentSpec(
    python="3.12.*",
    conda=("numpy>=2",),
)
environment = manager.provision("numpy-example", spec).wait_for()

with environment.start(workers=1) as workers:
    image = np.arange(9, dtype=np.float32).reshape(3, 3)
    task = workers.submit_import(
        "numpy:negative",
        args=(image,),
    )
    result = task.wait_for()

np.testing.assert_array_equal(result, -image)
manager.close()
```

This example is self-contained: Pixi installs NumPy in the worker environment, and the qualified target imports NumPy inside that environment.
Your own worker package exposes ordinary Python functions, is declared in `EnvironmentSpec`, and is called by its installed `module:qualified.callable` name in the same way.

Wetlands owns the shared-memory details.
Inputs use copy-in semantics and returned arrays are independently owned by the caller.
The repository also contains a [complete local worker-package example](https://github.com/arthursw/wetlands/blob/main/examples/getting_started.py).

## Async applications

Preparation, provisioning, and execution objects are awaitable.
Their `events()` methods expose async event streams while the caller retains ownership of its event loop.

```python
import asyncio

from wetlands import EnvironmentManager, EnvironmentSpec


async def main() -> None:
    manager = EnvironmentManager(root="wetlands")
    try:
        preparation = manager.prepare()
        async for event in preparation.events():
            print(event.kind.value, event.message)
        await preparation

        environment = await manager.provision(
            "analysis",
            EnvironmentSpec(python="3.12.*", conda=("numpy",)),
        )

        workers = await asyncio.to_thread(environment.start, workers=2)
        try:
            task = workers.submit_import(
                "numpy:negative",
                args=([1.0, 2.0, 3.0],),
            )
            result = await task
            print(result)
        finally:
            await asyncio.to_thread(workers.close)
    finally:
        await asyncio.to_thread(manager.close)


asyncio.run(main())
```

Starting, attaching, detaching, and closing worker pools, and closing a manager, are blocking lifecycle calls.
Async applications should run those calls with `asyncio.to_thread()` as shown above.

Cancel an operation or execution task with `cancel()`.
A canceled provisioning operation becomes terminal only after its active process tree has stopped and its incomplete environment has been cleaned up.
For a running worker task, Wetlands first requests cooperative cancellation.
If the worker does not finish during the configured grace period, Wetlands terminates its process tree and starts a replacement worker.

## Pixi projects and lockfiles

`EnvironmentSpec` is the complete managed recipe:

```python
from pathlib import Path

from wetlands import EnvironmentSpec, LocalPackage, PostInstallCommand

spec = EnvironmentSpec(
    python="3.12.*",
    conda=("numpy>=2", "scikit-image", "pip"),
    pypi=("example-pypi-package==1.2.0",),
    channels=("conda-forge",),
    local=(LocalPackage(Path("../worker-package"), editable=True),),
    post_install=(PostInstallCommand(("python", "-m", "worker_package.prepare_assets")),),
    pixi_lock=Path("pixi.lock"),
)
```

When `pixi_lock` is supplied, Wetlands provisions from those exact locked dependencies.
If the recipe contains local packages, the supplied lockfile must already resolve those same local sources and editable settings.
It must also include Wetlands' exact managed worker-runtime dependencies, including its `debugpy` pin.
Without one, Pixi resolves the generated project and Wetlands preserves the resulting lockfile in the managed environment.

Wetlands owns the managed `debugpy` version, so applications must not declare it in `EnvironmentSpec`.
The managed runtime pin participates in recipe identity and causes environments to rebuild when it changes.

An environment is ready only after every installation and validation step succeeds and Wetlands atomically publishes its ready metadata.
Failed, canceled, or crash-interrupted provisioning is rebuilt on the next attempt rather than resumed.
The returned `ManagedEnvironment` exposes its canonical project and lockfile paths, Pixi executable and version, recipe hash, lockfile hash, and generation ID.

## Worker targets

Installed packages use a qualified target:

```python
task = workers.submit_import(
    "package.module:ClassName.method",
    args=(value,),
)
```

The module is imported inside the isolated worker.

Local development can use an explicit source path:

```python
task = workers.submit_path(
    "worker_code.py",
    "segment",
    kwargs={"image": image},
    cache=False,
)
```

Path targets are keyed by canonical path and content, so equal filename stems do not collide.

## Debug running workers

Wetlands can start a debugger after an application and its workers are already running.
No debug flag or debugger call is required in application or worker code.

```sh
wetlands workers --root ./wetlands --environment numpy-example
wetlands debug --root ./wetlands --environment numpy-example --worker WORKER_ID --editor vscode --source .
```

The debug adapter remains available for reconnection until its worker exits.
See [Debugging running workers](docs/debugging.md) and [Persistent workers and reconnection](docs/persistent_workers.md).

## Supported values

Execution arguments and results may contain:

- `None`, booleans, integers, floats, strings, and bytes;
- nested lists, tuples, and dictionaries with simple keys;
- NumPy arrays without object dtype.

Unsupported objects fail explicitly at the boundary.
Non-contiguous arrays are transported as contiguous arrays.
Intermediate task outputs are limited to simple values in Wetlands 2.

## Migration from Wetlands 1

Wetlands 2 is a major release with a deliberately smaller public API.
Applications should migrate explicitly instead of relying on compatibility shims.

See the [Wetlands 2 migration guide](docs/migration_v2.md).

## Development

Install the development environment with:

```sh
uv sync --frozen --group dev --extra shared-memory
```

Run the fast test suite with:

```sh
uv run --extra shared-memory pytest -m "not integration and not compat and not manual"
```

Run the representative real-Pixi integration suite with:

```sh
UV_PROJECT_ENVIRONMENT=.venv-py314 uv run --python 3.14 --extra shared-memory pytest tests/test_v2_pixi_integration.py
```

Run linting with:

```sh
uv run ruff check
uv run ruff format --check
uv run mypy src/wetlands
```

Build the package with:

```sh
uv build
```

## Documentation

The complete documentation is available at [arthursw.github.io/wetlands](https://arthursw.github.io/wetlands/latest/).
Contributor-facing architecture and codec boundaries are described in the [developer guide](docs/developer/architecture.md).

## License

Wetlands is licensed under the [MIT License](LICENSE).

See the [security policy](SECURITY.md) before executing third-party worker code or post-install commands.
