NAME Data::Buffer::Shared - Type-specialized shared-memory buffers for multiprocess access SYNOPSIS use Data::Buffer::Shared::I64; # Create or open a shared buffer (file-backed mmap) my $buf = Data::Buffer::Shared::I64->new('/tmp/mybuf.shm', 1024); # Keyword API (fastest) buf_i64_set $buf, 0, 42; my $val = buf_i64_get $buf, 0; # Method API $buf->set(0, 42); my $v = $buf->get(0); # Lock-free atomic operations (integer types) buf_i64_incr $buf, 0; buf_i64_add $buf, 0, 10; buf_i64_cas $buf, 0, 52, 100; # Multiprocess if (fork() == 0) { my $child = Data::Buffer::Shared::I64->new('/tmp/mybuf.shm', 1024); buf_i64_incr $child, 0; # atomic, visible to parent exit; } wait; DESCRIPTION Data::Buffer::Shared provides type-specialized fixed-capacity buffers stored in file-backed shared memory (mmap(MAP_SHARED)), enabling efficient multiprocess data sharing on Linux. Linux-only. Requires 64-bit Perl. Features * File-backed mmap for cross-process sharing * Lock-free atomic get/set for numeric types (single-element) * Lock-free atomic counters: incr/decr/add/cas (integer types) * Seqlock-guarded bulk operations (slice, fill) * Write-preferring futex read-write lock with dead-process recovery * eventfd-based cross-process notification (optional) * Keyword API via XS::Parse::Keyword * Presized: fixed capacity, no growing Variants Data::Buffer::Shared::I8 - int8 Data::Buffer::Shared::U8 - uint8 Data::Buffer::Shared::I16 - int16 Data::Buffer::Shared::U16 - uint16 Data::Buffer::Shared::I32 - int32 Data::Buffer::Shared::U32 - uint32 Data::Buffer::Shared::I64 - int64 Data::Buffer::Shared::U64 - uint64 Data::Buffer::Shared::F32 - float Data::Buffer::Shared::F64 - double Data::Buffer::Shared::Str - fixed-length string The "Str" variant stores each value in a fixed $max_len-byte slot, NUL-padded, and trims trailing NULs on the way out. Values longer than $max_len are truncated to it. Embedded and leading NULs are preserved, so "a\0b" round-trips exactly -- but trailing NULs do not: "abc\0" reads back as "abc", and a value that is all NULs reads back as the empty string. For binary payloads that may end in NUL, store an explicit length alongside the value or use a numeric variant. Constructors my $buf = Data::Buffer::Shared::I64->new($path, $capacity); # file-backed my $buf = Data::Buffer::Shared::I64->new_anon($capacity); # anonymous my $buf = Data::Buffer::Shared::I64->new_memfd($name, $capacity); # memfd my $buf = Data::Buffer::Shared::I64->new_from_fd($fd); # reopen memfd The "Str" variant takes an additional $max_len argument giving the per-element fixed byte width: my $buf = Data::Buffer::Shared::Str->new($path, $capacity, $max_len); my $buf = Data::Buffer::Shared::Str->new_anon($capacity, $max_len); my $buf = Data::Buffer::Shared::Str->new_memfd($name, $capacity, $max_len); my $buf = Data::Buffer::Shared::Str->new_from_fd($fd, $max_len); "new_from_fd" duplicates the caller's fd internally; the caller keeps ownership of the passed fd. The "Str" variant requires the same $max_len the original was created with: it is recorded in the header as the element size and checked on attach, so passing a different $max_len dies with an "elem_size mismatch" error. The descriptor you pass is duplicated ("F_DUPFD_CLOEXEC"), so it stays yours to close and closing it does not disturb the handle. Lifecycle my $p = $buf->path; # backing file path, or undef for anon/memfd my $fd = $buf->fd; # memfd fd, or undef for anon/file-backed my $fd = $buf->memfd; # alias of fd() $buf->clear; # zero all elements (write-locked) $buf->sync; # msync(MS_SYNC) mmap to backing store $buf->unlink; # remove backing file (dies for anonymous buffers) my $h = $buf->stats; # diagnostic hashref "unlink" also works as a class method: "Data::Buffer::Shared::I64->unlink($path)". It croaks if the removal fails -- except when the file is already gone, which is what you asked for, so a cleanup path may safely run twice. "memfd" is an alias of "fd" (present on every variant): both return the backing file descriptor for a memfd-backed buffer (created with "new_memfd"), or "undef" for anonymous and file-backed buffers. API Replace "xx" with variant prefix: "i8", "u8", "i16", "u16", "i32", "u32", "i64", "u64", "f32", "f64", "str". buf_xx_set $buf, $idx, $value; # set element (lock-free atomic for numeric) my $v = buf_xx_get $buf, $idx; # get element (lock-free atomic for numeric) my @v = buf_xx_slice $buf, $from, $count; # bulk read (seqlock) buf_xx_fill $buf, $value; # fill all elements (write-locked) buf_xx_clear $buf; # zero all elements (write-locked) "set_slice" is a method only (its variadic argument list has no keyword form); it writes a run of elements starting at $from and returns true on success: $buf->set_slice($from, @values); # bulk write (write-locked) Integer variants also have: my $n = buf_xx_incr $buf, $idx; # atomic increment, returns new value my $n = buf_xx_decr $buf, $idx; # atomic decrement my $n = buf_xx_add $buf, $idx, $delta; # atomic add my $ok = buf_xx_cas $buf, $idx, $old, $new; # compare-and-swap my $p = buf_xx_cmpxchg $buf, $idx, $old, $new; # CAS, returns prior value my $n = buf_xx_atomic_and $buf, $idx, $mask; # atomic AND (integer variants) my $n = buf_xx_atomic_or $buf, $idx, $mask; # atomic OR my $n = buf_xx_atomic_xor $buf, $idx, $mask; # atomic XOR Raw / bulk: my $raw = buf_xx_get_raw $buf, $byte_off, $nbytes; # raw bytes, seqlock-guarded buf_xx_set_raw $buf, $byte_off, $raw; # raw bytes, write-locked $buf->add_slice($from, @deltas); # batch atomic add (integer variants; flat list) my $ptr = buf_xx_ptr $buf; # raw pointer to data, for FFI use my $ptr = buf_xx_ptr_at $buf, $idx; # pointer to element at index "get_raw" and "set_raw" address the data area in bytes, not element indices -- unlike every other accessor here. On an "I64" buffer "$buf->get_raw(4, 4)" returns bytes 4..7, which is the upper half of element 0 and the lower half of element 1, not elements 4..7. Multiply by the element size to address elements. Both are bounds-checked against the data area and croak rather than run past it. Zero-copy: my $sv = $buf->as_scalar; # mmap-aliased read-only scalar ref The returned scalar aliases the mapped bytes directly (no copy) and holds a reference to the buffer so the mapping stays alive while it is in use. Cross-process notification (all variants): my $efd = $buf->create_eventfd; # create + attach an eventfd, returns the fd $buf->attach_eventfd($fd); # attach an already-open eventfd my $efd = $buf->eventfd; # current eventfd, or undef if none $buf->notify; # signal (eventfd write) my $n = $buf->wait_notify; # drain the counter, non-blocking (undef if 0) These are a thin wrapper over an eventfd(2) descriptor stored in the handle, letting one process signal another that the buffer changed. The eventfd is created non-blocking, so "wait_notify" does not block: it reads and clears the counter, returning the accumulated notify count, or "undef" when the counter is zero (nothing pending) or no eventfd is attached. Nothing else in the API depends on them; watch the descriptor for readability in an event loop rather than expecting a blocking wakeup. Diagnostics: my $c = buf_xx_capacity $buf; my $s = buf_xx_mmap_size $buf; my $e = buf_xx_elem_size $buf; my $h = $buf->stats; # hashref: capacity/elem_size/mmap_size/variant_id/recoveries Persistence: $buf->sync; # msync(MS_SYNC) mmap to backing store Explicit locking (for batch operations): buf_xx_lock_wr $buf; # write lock + seqlock begin buf_xx_unlock_wr $buf; # seqlock end + write unlock buf_xx_lock_rd $buf; # read lock buf_xx_unlock_rd $buf; # read unlock The explicit locks are non-recursive and non-upgradable: calling "lock_wr" while holding "lock_rd" on the same handle, or calling "lock_wr" twice without an intervening "unlock_wr", self-deadlocks. Dropping the last reference to a handle while holding one of its locks leaks that handle's reader slot (and any held lock contribution) until the process exits. CONCURRENCY AND CRASH SAFETY Single-element numeric get/set and the atomic counter operations ("incr"/"decr"/"add"/"cas"/"cmpxchg"/"atomic_and"/"atomic_or"/"atomic_xor" ) are lock-free and safe to call concurrently from any number of processes. Bulk reads ("slice", "get_raw") are guarded by a seqlock and retry if a writer intervenes. Bulk writes ("set_slice", "fill", "clear", "set_raw") and the explicit "lock_wr"/"unlock_wr" region take a write lock. The write/read lock is a write-preferring futex read-write lock with dead-process recovery: if a process crashes while holding the lock, another process detects the dead holder and reclaims its contribution so the mapping does not deadlock. See "Reader-slot exhaustion" for the one narrow case this recovery cannot cover. An interrupted create is also recovered. A creator killed after the backing file is sized but before its header is committed leaves a full-size, all-zero file; "new" re-initializes such a file automatically, but only when it is owned by your effective uid and is still entirely zero, so a file holding data is never re-initialized. If the creator got as far as writing part of the header, the file cannot be told apart from a corrupt one and "new" refuses it with "uninitialized file is not empty". If that path has only ever been used for this buffer, the file never held data and removing it is safe -- but the same refusal is given for any file of the right size whose first bytes are zero, so check before deleting. An initialized buffer is never re-initialized, resized, or truncated by a later "new". On attach the stored geometry wins: the capacity you pass is ignored and the existing capacity is used, so check "$buf->capacity" if it matters. A variant, element-size or version mismatch is reported as an error. Disk space. The backing file is created sparse: "new" sizes it, but blocks are allocated only as you write, so a large buffer costs almost nothing on disk until it is used. The cost of that is a late failure, and how it reaches you depends on the filesystem. Where blocks are allocated at fault time -- tmpfs, so "/dev/shm" and many "/tmp" mounts -- a write to a page that cannot be backed raises "SIGBUS" and kills the process, because an "mmap" store has no way to report "ENOSPC". Where allocation is delayed to writeback (ext4, xfs), the store lands in page cache and the failure appears later: the write is lost, and "sync" is what reports it, croaking with the underlying error. Keep the filesystem sized for the buffer you asked for, and call "sync" when you need to know your writes reached disk. SEE ALSO Data::HashMap::Shared - concurrent hash table Data::Queue::Shared - FIFO queue Data::PubSub::Shared - publish-subscribe ring Data::ReqRep::Shared - request-reply Data::Sync::Shared - synchronization primitives Data::Pool::Shared - fixed-size object pool Data::Stack::Shared - LIFO stack Data::Deque::Shared - double-ended queue Data::Log::Shared - append-only log (WAL) Data::Heap::Shared - priority queue Data::Graph::Shared - directed weighted graph Data::RingBuffer::Shared - fixed-size overwriting ring buffer Data::BitSet::Shared - shared bitset (lock-free per-bit ops) SECURITY Backing files are created with mode 0600 (owner-only) by default, so only the creating user can open and attach them. To share a backing file across users, pass an explicit octal file mode such as 0660 as the last argument to "new"; the mode is applied only when the file is created or initialized, never to a buffer that is already in use. A pre-existing file is initialized -- and so has the mode applied to it via "fchmod" -- only when it is owned by your effective uid and is either empty or entirely zero (what an interrupted create leaves behind, see "CONCURRENCY AND CRASH SAFETY"); in every other case the file keeps its own permissions and its contents. The file is opened with "O_NOFOLLOW", so a symlink planted at the path is refused, and created with "O_EXCL"; the on-disk header is validated when the file is attached. Any process you grant write access to a shared mapping is trusted not to corrupt its contents while other processes are using it. Reader-slot exhaustion Reader-slot exhaustion (slotless readers): dead-process recovery attributes a crashed lock holder's contribution through its reader-slot. The slot table holds 1024 entries (one per concurrent reader process). If more than that many reader processes share one mapping at once, a reader that cannot claim a slot proceeds "slotless" -- it still takes the read lock but leaves no per-process record. If such a slotless reader is then killed while holding the read lock, its share of the lock cannot be attributed to a dead process, so writer recovery cannot reclaim it and writers may block until the mapping is recreated. Reaching this needs more than 1024 concurrent reader processes on one mapping plus a crash in the brief read-lock window; the dead-process slot reclaim keeps the table from filling with stale entries, so in practice it is very unlikely. Those preconditions cover the live-process route only. The count lives in the mapping and "new" validates the geometry, not this transient value, so a backing file damaged at rest -- bit rot, a partial copy, or a process that scribbled on the mapping -- can present a non-zero slotless count and block every writer the same way, with none of the above. If writers hang on a file no live reader is using, recreate it. AUTHOR vividsnow LICENSE This is free software; you can redistribute it and/or modify it under the same terms as Perl itself.