Metadata-Version: 2.4
Name: py-sharedmemory
Version: 4.3.1
Summary: Fast multiprocessing.Queue alternative that moves large Python objects and NumPy arrays through shared memory
Author: Tobias Würth
License-Expression: MIT
Project-URL: Homepage, https://github.com/tobiaswuerth/python_shared_memory_queue
Project-URL: Documentation, https://github.com/tobiaswuerth/python_shared_memory_queue#readme
Project-URL: Repository, https://github.com/tobiaswuerth/python_shared_memory_queue
Project-URL: Changelog, https://github.com/tobiaswuerth/python_shared_memory_queue/blob/main/CHANGELOG.md
Project-URL: Releases, https://github.com/tobiaswuerth/python_shared_memory_queue/releases
Project-URL: Issues, https://github.com/tobiaswuerth/python_shared_memory_queue/issues
Keywords: shared-memory,multiprocessing,queue,multiprocessing-queue,ipc,inter-process-communication,numpy,pickle,data-pipeline,machine-learning,reinforcement-learning
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3.15
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: System :: Distributed Computing
Classifier: Typing :: Typed
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: test
Requires-Dist: pytest>=7.4; extra == "test"
Requires-Dist: pytest-cov>=4.1; extra == "test"
Requires-Dist: pytest-timeout>=2.3; extra == "test"
Requires-Dist: numpy>=1.26; extra == "test"
Provides-Extra: benchmark
Requires-Dist: numpy>=1.26; extra == "benchmark"
Requires-Dist: matplotlib>=3.8; extra == "benchmark"
Provides-Extra: dev
Requires-Dist: py-sharedmemory[benchmark,test]; extra == "dev"
Requires-Dist: basedpyright>=1.29; extra == "dev"
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: mypy>=1.10; extra == "dev"
Requires-Dist: pre-commit>=3.7; extra == "dev"
Requires-Dist: ruff>=0.15; extra == "dev"
Requires-Dist: twine>=5; extra == "dev"
Dynamic: license-file

# py-sharedmemory

[![CI](https://github.com/tobiaswuerth/python_shared_memory_queue/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/tobiaswuerth/python_shared_memory_queue/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/py-sharedmemory)](https://pypi.org/project/py-sharedmemory/)
[![Python](https://img.shields.io/badge/dynamic/toml?url=https%3A%2F%2Fraw.githubusercontent.com%2Ftobiaswuerth%2Fpython_shared_memory_queue%2Fmain%2Fpyproject.toml&query=%24.project%5B%27requires-python%27%5D&label=python&logo=python&logoColor=white)](https://pypi.org/project/py-sharedmemory/)
[![Downloads](https://img.shields.io/pypi/dm/py-sharedmemory)](https://pypistats.org/packages/py-sharedmemory)
[![Coverage](https://img.shields.io/badge/coverage-100%25-brightgreen)](https://github.com/tobiaswuerth/python_shared_memory_queue/actions/workflows/ci.yml)
[![License](https://img.shields.io/github/license/tobiaswuerth/python_shared_memory_queue)](https://github.com/tobiaswuerth/python_shared_memory_queue/blob/main/LICENSE)
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
[![Checked with mypy](https://www.mypy-lang.org/static/mypy_badge.svg)](https://mypy-lang.org/)
[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit)](https://github.com/pre-commit/pre-commit)

**A fast `multiprocessing.Queue` alternative for large Python objects and NumPy arrays.**
Payloads travel between processes through shared memory instead of pipes; the queues
underneath carry only a few bytes of metadata per message. Multi-megabyte batches,
images, and simulation states arrive up to 3.6× faster on Windows and up to 7.4× faster on
Linux, on macOS even small messages beat a bounded `multiprocessing.Queue`, and transfers
too large for `multiprocessing.Queue` still work.

- **Familiar API:** `put` / `get` with `block`, `timeout`, `put_nowait` / `get_nowait`,
  `queue.Full`, and `queue.Empty`.
- **Any picklable object:** NumPy arrays, other protocol-5 buffers, and large `bytes` and
  `bytearray` objects are copied straight into shared memory, without an intermediate
  pickle copy.
- **Bounded memory:** a capacity limit applies backpressure, and received blocks are
  reused for later messages, so shared memory stays within `capacity × message size`.
- **One producer, many consumers:** consumer processes and threads share one receiver.
- **Portable:** Linux, macOS, and Windows on x86-64 and ARM64; `spawn`, `fork`, and
  `forkserver`; Python 3.11–3.15, including free-threaded 3.15t.
- **Lightweight:** no dependencies, fully typed (PEP 561), and 100% test coverage.

Built for **machine-learning data pipelines, reinforcement-learning actors and replay
buffers, scientific computing, and image or video processing**, where large payloads
make pipe I/O the bottleneck.

## Installation

```sh
pip install py-sharedmemory
```

> **Upgrading from 3.x?** The import package is now `py_sharedmemory` (previously
> `memory`). Replace `from memory import ...` with `from py_sharedmemory import ...`;
> the API is otherwise the same.

## Quick start

```python
import multiprocessing as mp

from py_sharedmemory import create_shared_memory_pair


def produce(sender):
    with sender:
        sender.put({"step": 42, "frames": bytearray(16_000_000)})
        sender.wait_for_all_ack()  # Keep the data alive until it is received


if __name__ == "__main__":
    ctx = mp.get_context("spawn")
    sender, receiver = create_shared_memory_pair(capacity=2, ctx=ctx)
    worker = ctx.Process(target=produce, args=(sender,))
    worker.start()
    with receiver:
        batch = receiver.get(timeout=30)
    worker.join()
    sender.close()
```

More complete, runnable programs:

- [`examples/basic.py`](https://github.com/tobiaswuerth/python_shared_memory_queue/blob/main/examples/basic.py):
  a producer process with timeouts and error handling.
- [`examples/numpy_batches.py`](https://github.com/tobiaswuerth/python_shared_memory_queue/blob/main/examples/numpy_batches.py):
  streaming NumPy batches with backpressure and an end-of-stream marker.
- [`examples/worker_pool.py`](https://github.com/tobiaswuerth/python_shared_memory_queue/blob/main/examples/worker_pool.py):
  fanning large tasks out to several consumer processes.

## When to use it

Use py-sharedmemory when individual messages are **about 1 MB or larger** on Linux and
Windows; on Windows ARM64 it pulls clearly ahead from about 10 MB. Typical payloads are
NumPy arrays, image or video frames, RL rollouts, large `bytes`, or big nested objects. On
**macOS** it was faster at every size measured, from 1 kB up; see
[Performance](#performance).

On Linux and Windows, prefer `multiprocessing.Queue` for small messages: up to about
100 kB it is 1.5–4× faster, because handling a shared-memory block for every message
costs more than the pipe.

On free-threaded Python (`python3.14t` and later), threads run Python code in parallel.
If everything your workers use is free-threading safe, threads and `queue.Queue` pass
objects by reference without any copy. Use processes, and this package, when you need
process isolation or rely on extensions that re-enable the GIL.

To let several processes read and write **the same** array in place, use
[`multiprocessing.shared_memory`](https://docs.python.org/3/library/multiprocessing.shared_memory.html)
directly. py-sharedmemory delivers an independent copy of each message.

## Performance

![Speedup of py-sharedmemory over multiprocessing.Queue for each platform and payload size from 1 kB to 100 MB](https://raw.githubusercontent.com/tobiaswuerth/python_shared_memory_queue/main/benchmarks/results/platforms.png)

Speedup over `multiprocessing.Queue`; above 1× means py-sharedmemory is faster.

| Platform | 1 kB | 10 kB | 100 kB | 1 MB | 10 MB | 100 MB |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| Linux x64 | 0.33× | 0.32× | 0.45× | **3.32×** | **5.30×** | **2.80×** |
| Linux arm64 | 0.32× | 0.36× | 0.52× | **2.57×** | **7.42×** | **2.68×** |
| Windows x64 | 0.25× | 0.28× | 0.51× | **2.50×** | **3.07×** | **3.34×** |
| Windows arm64 | 0.30× | 0.31× | 0.63× | **1.11×** | **3.06×** | **3.62×** |
| macOS arm64 | **11.87×** | **22.58×** | **17.02×** | **10.87×** | **5.34×** | **5.42×** |

- **Linux, x86-64 and ARM64:** 2.6–7.4× faster from 1 MB. Up to 100 kB,
  `multiprocessing.Queue` is 2–3× faster.
- **Windows x86-64:** 2.5–3.3× faster from 1 MB, and 2–4× slower up to 100 kB.
- **Windows ARM64:** about even at 1 MB, and 3–3.6× faster from 10 MB.
- **macOS (Apple Silicon):** 5–23× faster at every size. macOS has no timed semaphore
  wait, so a full bounded `multiprocessing.Queue` whose `put` has a timeout, as here,
  [polls with sleeps of up to 20 ms](https://github.com/python/cpython/blob/main/Modules/_multiprocessing/semaphore.c).
  py-sharedmemory enforces its capacity with a thread condition instead.

Measured by the [benchmark workflow](https://github.com/tobiaswuerth/python_shared_memory_queue/blob/main/.github/workflows/benchmark.yml)
on GitHub-hosted runners on 2026-10-18: py-sharedmemory 4.3.0, Python 3.14.8, NumPy 2.5.4,
contiguous float32 arrays, `spawn`, capacity 2 for both queues, and the median of 5 runs
that alternate between the two. The runners are shared virtual machines, so compare the
speedups, not absolute throughput. Run-to-run spread is mostly within 20% on Linux and
Windows and larger on macOS, whose speedups dwarf it. See the
[raw results](https://github.com/tobiaswuerth/python_shared_memory_queue/tree/main/benchmarks/results/platforms).
The workflow reruns on every release.

**Larger messages:** the speedup holds beyond the 100 MB that shared runners can hold. On
a Windows desktop, an earlier version stayed 1.8× faster at 100 MB and 1.5× faster at 1 GB,
and at 10 GB per message `multiprocessing.Queue` failed while py-sharedmemory delivered
each message in 7.5 s ([raw results](https://github.com/tobiaswuerth/python_shared_memory_queue/blob/main/benchmarks/results/windows-numpy.json)).

## How it works

1. `put()` pickles the object with protocol 5. Large buffers, such as NumPy array data and
   `bytes` or `bytearray` objects of 64 KiB or more, never enter the pickle; everything is
   copied once, from the original objects into one shared-memory block: a block from an
   earlier, received message when one fits, otherwise a new one.
2. Only the block's name and identity, the buffer sizes, and a sequence number travel
   through a `multiprocessing.Queue`.
3. `get()` maps the block, or reuses its mapping from an earlier message, copies the data
   into local memory, unpickles it, and sends an acknowledgement back.
4. A background thread in the producer makes the block available for reuse when the
   acknowledgement arrives. `capacity` limits how many messages can be outstanding.

Received objects never point into shared memory, so they stay valid after the block is
reused or freed and can be modified freely.

## Usage guide

### Passing endpoints to processes

Create the pair in the parent process and pass the endpoints to workers as `Process`
arguments, the same way as a `multiprocessing.Queue`. Use the same context for the pair
and the processes. A sender can be passed to another process only **before** its first
`put`; after that it belongs to the process that used it.

As with a `multiprocessing.Queue`, keep an endpoint referenced in the creating process until
the processes using the pair have started. With `spawn` and `forkserver` on Linux and macOS,
a starting child reopens the pair's semaphores by name, and the names disappear once the
creating process lets go of them. After that, the creating process may drop endpoints it
does not use, and may create as many further pairs as it needs.

Closing an endpoint affects only that endpoint. As with the unused end of a `Pipe`, the
parent may close its copy of the endpoint it handed to a child right after starting the
child, and keep using the other endpoint. A process releases its handles to the pair when
its last open endpoint closes.

```python
ctx = mp.get_context("spawn")  # "fork" and "forkserver" also work
sender, receiver = create_shared_memory_pair(capacity=4, ctx=ctx)
producer = ctx.Process(target=produce, args=(sender,))
consumer = ctx.Process(target=consume, args=(receiver,))
```

### Capacity and backpressure

`capacity` counts messages that were sent but not yet received. When it is reached,
`put()` blocks, so a fast producer cannot exhaust memory. Shared-memory use is at most
about `capacity × message size`. `capacity=0` or `None` means unbounded.

Received blocks are reused rather than freed, because writing into memory that is
already mapped is much faster than faulting in fresh pages. A block is reused for a
message that fills at least half of it, or for any message that fits a block of up to
1 MiB. The sender keeps at most 8 such blocks and 256 MiB, never more blocks than the
capacity allows, and frees them on `close()`. Blocks over 256 MiB are freed once received.

Receivers likewise keep the blocks they read mapped instead of mapping them again for
every message: each receiving thread keeps at most `capacity` (up to 8) blocks and
256 MiB, and `close()` releases them. On Windows, where a block lives while any process
maps it, a receiver therefore keeps a block its sender has freed alive until the receiver
reads other blocks or closes.

### Timeouts and non-blocking calls

```python
from queue import Empty, Full

try:
    sender.put(batch, timeout=5)  # Wait up to 5 s for capacity
except Full:
    ...

try:
    item = receiver.get_nowait()  # Same as get(block=False)
except Empty:
    ...
```

`timeout=None` and `timeout=math.inf` both wait without limit. Negative timeouts behave
like `0`, and a NaN timeout raises `ValueError`.

### Multiple consumers and threads

Each message is delivered to exactly one consumer. Several consumer processes can share
one receiver, as in [`examples/worker_pool.py`](https://github.com/tobiaswuerth/python_shared_memory_queue/blob/main/examples/worker_pool.py),
and both endpoints are thread-safe. Only one process may produce per pair; for several
producer processes, create one pair per producer.

### Shutting down cleanly

The producer owns every message until it is acknowledged, and `close()` frees unread
messages. Before the producer closes or exits:

```python
with sender:
    for item in items:
        sender.put(item)
    if not sender.wait_for_all_ack(timeout=30):
        raise TimeoutError("Consumer did not receive everything")
```

Closing never blocks. If a `with sender:` block or the producing process ends normally
while messages are still unacknowledged, they are discarded and the sender emits a
`RuntimeWarning`, so forgetting to drain does not lose data silently. An explicit
`close()` or an exception leaving the `with` block discards without a warning.

When both endpoints live in one process, drain before closing the sender, because closing
it frees unread messages. A sentinel such as `None` is a simple way to tell consumers that
the stream has ended. A thread waiting in `get()` without a timeout wakes with
`BrokenPipeError` when its receiver is closed, and a daemon thread still waiting when the
program ends simply ends with it.

### Crashes and deadlocks

- **Pass timeouts when a peer can die.** `get`, `put`, and `wait_for_all_ack` without a
  timeout wait forever for a process that no longer exists, as with a
  `multiprocessing.Queue`.
- **A crashed consumer loses at most the message it was handling.** Messages it never
  received go to the next consumer, including one started to replace it. If the crash
  also lost that message's acknowledgement, its slot of capacity stays taken until the
  sender closes; closing still never blocks and frees every block.
- **Killing an idle consumer is safe.** Receivers wait for data without holding the
  queue's read lock, so the other consumers keep receiving.
- **Stop consumers cooperatively when you can,** for example with one sentinel each. A
  consumer killed during the brief moment it reads a message, or on Linux and macOS while
  its acknowledgement is being written, can still leave the pair's queue locked, just as
  it would a `multiprocessing.Queue`.
- **A crashed producer's unread messages** stay readable on Linux and macOS, where a block
  outlives its creator until it is unlinked. On Windows they are freed with the producer,
  and `get` raises `FileNotFoundError` for each of them.

### NumPy arrays

Arrays keep their dtype, shape, and writability. Read-only arrays stay read-only, and
writable ones arrive as writable local copies. C- and Fortran-contiguous arrays keep
their layout and are copied straight into shared memory without an intermediate pickle
copy; other views, such as strided slices, arrive as contiguous copies. Each received
array has its own aligned memory, so keeping one array does not keep the rest of the
message alive.

### Errors

| Exception | Raised when |
| --- | --- |
| `queue.Full` | `put` found no capacity within `timeout` or with `block=False`. |
| `queue.Empty` | `get` found no message within `timeout` or with `block=False`. |
| `BrokenPipeError` | The endpoint is closed, the producer's acknowledgement channel failed, or the main thread was waiting in `get` while the process exits. |
| `FileNotFoundError` | `get` received a message that the producer freed by closing before it was read. |
| `OSError` (`ENOSPC`) | `put` found no room for the message in shared memory, for example a full `/dev/shm` in a container. The sender stays usable. |
| `ValueError` | A timeout is NaN. |
| `RuntimeError` | A second process tries to produce, or a used sender is passed to another process. |
| Pickling errors | The object cannot be pickled or unpickled. Both endpoints stay usable, and failed messages are still released. |
| `RuntimeWarning` (warning) | A `with sender:` block or the producing process ended normally and discarded unacknowledged messages. |

## API reference

### `create_shared_memory_pair(capacity=0, *, ctx=None)`

Returns a connected `(SharedMemorySender, SharedMemoryReceiver)`. `capacity` limits
outstanding messages (`0` or `None` is unbounded). `ctx` is the multiprocessing context
for the underlying queues; it defaults to `multiprocessing.get_context()`.

### `SharedMemorySender`

| Member | Description |
| --- | --- |
| `put(obj, block=True, timeout=None)` | Copy `obj` into a shared-memory block and send it. |
| `put_nowait(obj)` | `put(obj, block=False)`. |
| `wait_for_all_ack(timeout=None) -> bool` | Wait until every message is received and its block released; `False` on timeout. |
| `close()` | Free unread messages and reusable blocks silently and release resources. Leaves a receiver in the same process usable. Runs automatically at normal process exit. |
| `closed` | Whether the sender is closed. |
| `with sender:` | Closes the sender on exit; warns if a clean exit discards unacknowledged messages. |

### `SharedMemoryReceiver`

| Member | Description |
| --- | --- |
| `get(block=True, timeout=None)` | Return the next object as an independent local copy. |
| `get_nowait()` | `get(block=False)`. |
| `close()` | Release kept mappings, flush acknowledgements, and wake waiting `get` calls with `BrokenPipeError`. Leaves a sender in the same process usable. |
| `closed` | Whether the receiver is closed. |
| `with receiver:` | Closes the receiver on exit. |

## Platform notes and troubleshooting

**Docker and Kubernetes:** on Linux, shared memory lives in `/dev/shm`, which containers
limit to 64 MB by default. Raise it to at least `capacity × largest message`, for example
`docker run --shm-size=2g` or `shm_size: 2gb` in Docker Compose. In Kubernetes, mount an
`emptyDir` volume with `medium: Memory` at `/dev/shm`. Otherwise `put` raises
`OSError: [Errno 28] No space left on device`; the sender reserves a new block's pages up
front, so a full `/dev/shm` raises this error instead of killing the producer with `SIGBUS`.

**Linux:** outside containers, `/dev/shm` usually defaults to half of physical RAM.

**macOS:** fully supported and tested on Apple Silicon. The default start method is
`spawn`, so guard the entry point with `if __name__ == "__main__":`. An unbounded pair
holds at most 32,767 unread messages, the limit of macOS semaphores; beyond that `put`
waits or raises `queue.Full` like a full bounded pair.

**Windows:** shared memory is backed by the page file, and each block disappears once
the last handle closes, so nothing can leak after a crash.

**"resource_tracker: There appear to be N leaked shared_memory objects"** (or semaphore
objects): a process was killed or crashed before releasing the blocks or semaphores it
created, and Python's resource tracker freed them when the program ended. Nothing stays
behind; stop workers cooperatively, for example with a stop event, to avoid the warning.

**`FileNotFoundError: Shared-memory block was freed before it was received`:** the
producer closed or exited before this message was read. Call `wait_for_all_ack()` before
closing the sender; the sender also warns when this happens.

**`OSError: [Errno 24] Too many open files`:** on Linux and macOS each pair costs a
process 4 file descriptors for its queues, on macOS a few more for its semaphores, plus 1
per block the sender keeps for reuse and 1 per block a receiving thread keeps mapped, each
at most `capacity` (up to 8). Unread messages beyond that hold none, and a receiver that
runs out releases its kept mappings and retries. For many pairs on macOS, whose default
limit is 256, raise it with `ulimit -n`.

**`RuntimeError: A shared-memory pair supports one producing process`:** a second
process called `put`. Create a separate pair for each producer process.

**`Can't pickle local object` or similar:** messages must be picklable. Lambdas, local
functions, open files, and locks cannot be sent.

## Contributing

Bug reports, ideas, and pull requests are welcome. See
[CONTRIBUTING.md](https://github.com/tobiaswuerth/python_shared_memory_queue/blob/main/CONTRIBUTING.md)
for the development setup, tests, and release process, and the
[changelog](https://github.com/tobiaswuerth/python_shared_memory_queue/blob/main/CHANGELOG.md)
for version history. Report security issues privately as described in
[SECURITY.md](https://github.com/tobiaswuerth/python_shared_memory_queue/blob/main/SECURITY.md).

## License

[MIT](https://github.com/tobiaswuerth/python_shared_memory_queue/blob/main/LICENSE) © Tobias Würth
