Metadata-Version: 2.4
Name: aethisdb
Version: 1.1.0
Summary: A lightweight, pure-Python document database with a compact binary, optionally password-encrypted file format
Author: Aethis Studio
License: MIT
Keywords: database,nosql,document,pure-python,embedded
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Database
Classifier: License :: OSI Approved :: MIT License
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Provides-Extra: test
Requires-Dist: pytest>=7.0; extra == "test"

# AethisDB

A lightweight, pure-Python document database with a custom binary file format (`.aethisdb`). Zero external dependencies — uses only the Python standard library.

## Features

- **Custom binary file format** with a 96-byte header and append-friendly data section — no JSON on disk
- **Unreadable files**: records, collection names and the index are ciphered, so a text or hex editor shows noise
- **Optional password protection** (`password=`) with PBKDF2-HMAC-SHA256 key derivation
- **Hash-based indexing** for fast exact-match queries on indexed fields
- **Full CRUD operations**: insert, find, update, delete
- **Rich query operators**: `$eq`, `$ne`, `$gt`, `$lt`, `$gte`, `$lte`, `$in`, `$nin`, `$exists`, `$and`, `$or`
- **Auto-commit or manual commit** modes
- **File compaction** to reclaim space from deleted records
- **Automatic compaction** based on tombstone ratio — ideal for small/embedded storage
- **zlib-compressed index block** for minimal on-disk footprint
- **Compact binary encoding** — varint integers and a per-block string table beat compact JSON
- **Space statistics** reporting via `db.stats()`
- **Context manager** support (`with Database(...) as db:`)
- **Persistent across sessions** — collections and indexes survive reopens

## Installation / Setup

AethisDB has no third-party dependencies. Simply place the `aethisdb/` package directory on your Python path, or install it in development mode:

```bash
pip install aethisdb
```

Then import:

```python
from aethisdb import Database, Record
```

## Quick Start

### Initializing a Database

```python
from aethisdb import Database

# Creates a new .aethisdb file (or opens an existing one)
db = Database("mydata.aethisdb")

# Create a collection
users = db.create_collection("users")

# List collections
print(db.list_collections())  # ['users']
```

### Inserting Records

```python
# Insert a single record — returns the auto-assigned record ID
uid = users.insert({"name": "Alice", "email": "alice@example.com", "age": 30})

# Insert multiple records at once
ids = users.insert_many([
    {"name": "Bob",   "email": "bob@example.com",   "age": 25},
    {"name": "Carol", "email": "carol@example.com", "age": 35},
])

# Get a record by ID
alice = users.get(0)
print(alice["name"])  # Alice
```

### Querying Records

```python
# Find by exact match
results = users.find({"name": "Alice"})
print(results[0]["email"])  # alice@example.com

# Find one (returns None if no match)
bob = users.find_one({"email": "bob@example.com"})

# Range query
adults = users.find({"age": {"$gte": 30}})

# Multiple conditions (implicit AND)
results = users.find({"age": {"$gte": 25}, "age": {"$lte": 35}})

# $in operator
results = users.find({"name": {"$in": ["Alice", "Bob"]}})

# $or operator
results = users.find({"$or": [{"age": 25}, {"age": 35}]})

# Get all records
all_users = users.all()

# Count records
print(users.count())  # 3
```

### Updating Records

```python
# Update matching records
count = users.update({"name": "Alice"}, {"age": 31})
print(count)  # 1

# Update by record ID
users.update(record_id=1, update_data={"age": 26})
```

> **Note:** Updates in AethisDB use a copy-on-write strategy — the old record is marked as deleted and a new record is appended with the updated data. Record IDs are preserved.

### Deleting Records

```python
# Delete by query
count = users.delete({"name": "Bob"})

# Delete by record ID
count = users.delete(record_id=2)

# Delete all records (use with care!)
users.delete()

# Count after deletion
print(users.count())  # 2
```

### Indexing

Hash indexes provide O(1) exact-match lookups:

```python
# Create an index on a field
users.create_index("email")

# Now queries on the indexed field use the index automatically
bob = users.find_one({"email": "bob@example.com"})  # fast

# List indexes
print(users.indexes)  # ['email']

# Drop an index
users.drop_index("email")

# Rebuild an index after bulk operations
users.create_index("email")
users.rebuild_index("email")
```

### Compaction

Deleted records leave tombstones in the file. Use `compact()` to reclaim space:

```python
# After many deletions
file_size_before = os.path.getsize("mydata.aethisdb")

db.compact()

file_size_after = os.path.getsize("mydata.aethisdb")
# File is smaller — tombstones removed
```

### Transaction / Auto-commit Control

```python
# Manual commit mode — changes are not persisted until commit()
db = Database("mydata.aethisdb", auto_commit=False)

users = db.get_collection("users")
users.insert({"name": "Alice"})
users.insert({"name": "Bob"})

# Verify data is not yet persisted
db.close()  # Without commit, changes are lost

# Correct usage:
db = Database("mydata.aethisdb", auto_commit=False)
users = db.get_collection("users")
users.insert({"name": "Alice"})
users.insert({"name": "Bob"})
db.commit()  # persist to disk
db.close()
```

### Context Manager

```python
with Database("mydata.aethisdb") as db:
    users = db.get_collection("users")
    users.insert({"name": "Alice"})
    alice = users.find_one({"name": "Alice"})
# db is automatically closed
```

### Space Efficiency (Small Storage)

AethisDB includes several features to minimize disk usage, making it ideal for embedded or small-storage applications:

#### Compact Binary Encoding (no JSON)

Record documents and the index block are written with AethisDB's own binary codec:

- integers as zig-zag varints instead of ASCII digits
- one string table per block, so repeated keys are stored once
- varint element counts instead of braces, brackets, quotes and commas
- the index block packs record IDs and byte offsets as varints, with no field names

Measured against the previous JSON layout (2,000 records with one `email` index): 370,092 B → 337,292 B, about 9% smaller, and unreadable.

#### Unreadable by Default

Every record, collection name and the index block is sealed with a SHA-256 keystream before it is written, so opening the file in Notepad, VS Code or a hex editor shows binary noise. See [Passwords and Encryption](#passwords-and-encryption).

#### Compressed Index Block

The index block at the end of the file is automatically zlib-compressed when compression reduces its size. Compression is transparent — no configuration required.

#### Auto-Compaction

When records are deleted, they are soft-deleted (marked as tombstones). Over time, tombstones waste space. AethisDB can **automatically compact** the file on close when the tombstone ratio exceeds a configurable threshold:

```python
from aethisdb import Database

# auto_compact=True: compact on close if deleted ratio > compact_ratio
# compact_ratio=0.3 (default): compact when 30%+ of records are deleted
db = Database("small.aethisdb", auto_compact=True, compact_ratio=0.25)

users = db.create_collection("users")
for i in range(10):
    users.insert({"name": f"user_{i}"})
for i in range(5):
    users.delete({"name": f"user_{i}"})
# 50% deleted > 25% threshold → file is compacted automatically on close
db.close()
```

#### Manual Commit Mode

For workloads with many writes, use manual commit mode to reduce index write frequency:

```python
# Only write the index to disk when commit() or close() is called
db = Database("small.aethisdb", auto_commit=False)

users = db.create_collection("users")
for i in range(1000):
    users.insert({"name": f"user_{i}"})

db.commit()  # one index write instead of 1000
db.close()
```

#### Space Statistics

```python
db = Database("small.aethisdb")
stats = db.stats()
print(stats)
# {
#   'file_size': 4122,
#   'header_size': 96,
#   'data_size': 3218,
#   'index_size': 808,
#   'record_count': 50,
#   'deleted_count': 0,
#   'deleted_bytes_estimate': 0,
#   'collections': 1,
#   'auto_compact': False,
#   'auto_commit': True,
#   'version': 3,
#   'cipher': 'obfuscated'
# }
db.close()
```

#### Manual Compaction

Force a compaction at any time:

```python
db = Database("small.aethisdb")
users = db.get_collection("users")
users.delete({"age": {"$lt": 18}})
db.compact()  # immediately rewrite file, removing tombstones
db.close()
```

## Full Example

```python
from aethisdb import Database

with Database("company.aethisdb") as db:
    employees = db.create_collection("employees")
    employees.create_index("email")

    # Seed data
    employees.insert_many([
        {"name": "Alice",   "email": "alice@corp.com", "department": "Engineering", "salary": 90000},
        {"name": "Bob",     "email": "bob@corp.com",   "department": "Sales",       "salary": 65000},
        {"name": "Charlie", "email": "carol@corp.com", "department": "Engineering", "salary": 85000},
    ])

    # Query with index
    alice = employees.find_one({"email": "alice@corp.com"})

    # Range query
    high_earners = employees.find({"salary": {"$gte": 80000}})

    # Update
    employees.update({"email": "bob@corp.com"}, {"salary": 70000})

    # Delete
    employees.delete({"department": "Sales"})

    # Verify
    remaining = sorted(e["name"] for e in employees.all())
    print(remaining)  # ['Alice', 'Charlie']
```

---

## File Format: `.aethisdb`

### Overview

The `.aethisdb` file format is a custom binary layout designed for append-friendly record storage with a persistent binary index at the end of the file. Nothing is stored as JSON, and every block except the fixed header fields is ciphered — a text or hex editor shows noise. The format consists of three sections:

```
┌──────────────────┬───────────────────┬─────────────────────┐
│  Header (96B)    │  Data Section     │  Index Block        │
│  (fixed size)    │ (append-only)     │   (end of file)     │
├──────────────────┼───────────────────┼─────────────────────┤
│ Magic number     │ Collection        │ version, collection │
│ Version          │   markers         │   count, then per   │
│ Cipher mode      │ Data records      │   collection: name, │
│ KDF iterations   │                   │   counters, record  │
│ Index offset     │                   │   id → offset pairs,│
│ Index length     │                   │   and hash indexes  │
│ Data start       │                   │   (all varints)     │
│ Collection count │                   │                     │
│ Salt, key check  │                   │                     │
└──────────────────┴───────────────────┴─────────────────────┘
```

### 1. Header (exactly 96 bytes)

The header is a fixed-size binary structure at the beginning of the file:

| Offset | Size    | Type     | Field            | Description                              |
|--------|---------|----------|------------------|------------------------------------------|
| 0      | 8       | `8s`     | Magic number     | Always `b"AETHISDB"`                    |
| 8      | 1       | `uint8`  | Version          | Format version (currently `3`)          |
| 9      | 1       | `uint8`  | Cipher           | `0` none, `1` obfuscated, `2` encrypted |
| 10     | 2       | `uint16` | Reserved         | Zero                                     |
| 12     | 4       | `uint32` | Flags            | Reserved flags (currently `0`)           |
| 16     | 4       | `uint32` | Iterations       | PBKDF2 rounds for encrypted files        |
| 20     | 8       | `uint64` | Index offset     | Byte offset to the index block           |
| 28     | 8       | `uint64` | Index length     | Size of the index block in bytes         |
| 36     | 8       | `uint64` | Data start       | Always `96` (where data begins)          |
| 44     | 8       | `uint64` | Collection count | Number of collections                    |
| 52     | 16      | `16s`    | Reserved         | Zero-padded                              |
| 68     | 16      | `16s`    | Salt             | Per-file key salt                        |
| 84     | 8       | `8s`     | Key check        | HMAC key verifier                        |
| 92     | 4       | `4s`     | Padding          | Zero                                     |

All integers are little-endian (struct format character `<`).

### 2. Data Section

The data section starts at byte offset 96 (right after the header). It contains a sequence of block entries. Two block types are supported:

#### Collection Marker Block

Written when a collection is created. The name is ciphered, so collection names are not visible in the file.

| Offset | Size   | Type     | Field         | Description                        |
|--------|--------|----------|---------------|------------------------------------|
| 0      | 1      | `uint8`  | Type          | `1` (COLLECTION_MARKER_TYPE)      |
| 1      | 2      | `uint16` | Stored length | Length of the sealed payload       |
| 3      | var    | `bytes`  | Payload       | Sealed(name + `uint64` next ID)    |

#### Record Entry Block

Written for each record insert. Records are appended sequentially. The 14-byte entry header stays plaintext so that a soft-delete can flip the status byte in place; the document itself is ciphered.

| Offset | Size   | Type     | Field         | Description                        |
|--------|--------|----------|---------------|------------------------------------|
| 0      | 1      | `uint8`  | Type          | `2` (RECORD_ENTRY_TYPE)           |
| 1      | 1      | `uint8`  | Status        | `0` = deleted, `1` = active       |
| 2      | 8      | `uint64` | Record ID     | Unique record identifier           |
| 10     | 4      | `uint32` | Stored length | Length of the sealed payload       |
| 14     | var    | `bytes`  | Payload       | Sealed(binary-encoded document)    |

### 3. Binary encoding

Documents use AethisDB's own codec (`aethisdb/codec.py`), never JSON:

| Tag    | Type   | Encoding                                       |
|--------|--------|------------------------------------------------|
| `0x00` | null   | —                                              |
| `0x01` | false  | —                                              |
| `0x02` | true   | —                                              |
| `0x03` | int    | zig-zag LEB128 varint                          |
| `0x04` | bigint | magnitude varint + sign bit                    |
| `0x05` | float  | IEEE-754 double, 8 bytes                       |
| `0x06` | string | varint index into the block string table       |
| `0x07` | bytes  | varint length + raw bytes                      |
| `0x08` | list   | varint count + values                          |
| `0x09` | map    | varint count + (string ref, value) pairs       |

Each block begins with a string table (varint count, then varint-prefixed UTF-8 strings), so keys and repeated values are stored once and referenced by index.

### 4. Index Block (end of file)

The index block is a hand-packed binary structure appended after the last data byte. The header's `index_offset` and `index_length` fields locate it. On every commit, the entire index is rewritten.

```
varint  format version
varint  collection count
  per collection:
    string-ref  name
    varint      next_id, record_count, deleted_count, marker_offset
    varint      position count
      varint    record ID
      varint    byte offset
    varint      index count
      per index: field name, then value key → list of record IDs
```

**Key fields (in memory):**

| Field              | Type     | Description                                            |
|--------------------|----------|--------------------------------------------------------|
| `marker_offset`    | integer  | Byte offset of the collection marker in the data section|
| `next_id`          | integer  | Next auto-increment record ID                          |
| `record_count`     | integer  | Number of active (non-deleted) records                  |
| `deleted_count`    | integer  | Number of soft-deleted records                         |
| `record_positions` | object   | Maps `record_id` (as string) to byte offset in file    |
| `indexes`          | object   | Hash indexes for indexed fields                        |

Before sealing, the index gets a one-byte flag: `0x00` raw binary, `0x01` zlib-compressed.

### How Writes Work

AethisDB uses an **append-friendly** strategy:

1. **Insert**: A record is encoded, sealed and appended to the end of the data section. Its position is recorded in `record_positions`.
2. **Update**: The old record's status byte is set to `0` (deleted) in-place. A new record with the updated data is appended at the end. The `record_positions` map is updated to point to the new record. The old record remains as a tombstone.
3. **Delete**: The record's status byte is set to `0` (deleted) in-place. It is removed from `record_positions` and from all hash indexes.
4. **Commit**: The entire index block is packed, optionally compressed and sealed, then written after the last data byte. The 96-byte header is updated with the new `index_offset` and `index_length`.

### How Reads Work

1. The header is read (first 96 bytes) to locate the index block and derive the key.
2. The index block is unsealed and unpacked to reconstruct collection metadata and hash indexes in memory.
3. For single-record lookups (`get(id)`), the byte offset is retrieved from `record_positions`, the file is seeked to that position, and the record is unsealed and decoded.
4. For indexed exact-match queries, the hash index directly maps field values to record IDs, avoiding any file scanning.
5. For non-indexed or range queries, all active records are scanned and filtered using the query engine.

### How Compaction Works

When `compact()` is called:

1. A temporary file is created.
2. All active records (status = `1`) are read from the current file and rewritten sequentially to the temp file, re-sealing each block at its new position.
3. Deleted records (tombstones) are skipped — they are not copied.
4. A fresh, compact index is written at the end of the temp file.
5. The temp file replaces the original file atomically (`os.replace`).
6. The result is a file with no wasted space from tombstones or orphaned index blocks.

### Legacy files

Databases written by version 1.0.x (64-byte header, JSON payloads) are detected and migrated automatically the first time they are opened: records and indexes are re-encoded to the binary layout and rewritten using the current cipher mode.

## Passwords and Encryption

`.aethisdb` files are unreadable without this library. Two modes, selected automatically:

```python
from aethisdb import Database

# Obfuscated (default) — no password needed, raw file is still unreadable
db = Database("data.aethisdb")

# Encrypted — password required every time the file is opened
db = Database("secret.aethisdb", password="hunter2")
print(db.is_encrypted)   # True
print(db.cipher)         # "encrypted"
```

Under the hood, a per-file random salt plus PBKDF2-HMAC-SHA256 derives a key, and every block is sealed with a SHA-256 counter-mode keystream bound to its file position. An 8-byte HMAC check in the header rejects a wrong password before any record is read.

Add or change a password at any time — the file is rewritten with a fresh key:

```python
db = Database("data.aethisdb")
db.change_password("hunter2")
db.close()
```

`cipher="none"` disables the cipher; `cipher="obfuscated"` and `cipher="encrypted"` force a mode for new files.

To look inside a file without opening a session:

```python
from aethisdb import file_info, peek

file_info("secret.aethisdb")               # header only, no password needed
peek("secret.aethisdb", password="pw")     # dict of collections and records
```


## Query Operators

| Operator | Type      | Description                                          | Example                         |
|----------|-----------|------------------------------------------------------|---------------------------------|
| (default)| Exact match| Field equals value                                   | `{"name": "Alice"}`             |
| `$eq`    | Equality  | Field equals value                                   | `{"age": {"$eq": 30}}`          |
| `$ne`    | Inequality| Field does not equal value                           | `{"age": {"$ne": 30}}`          |
| `$gt`    | Comparison| Field greater than value                             | `{"age": {"$gt": 25}}`          |
| `$lt`    | Comparison| Field less than value                                | `{"age": {"$lt": 35}}`          |
| `$gte`   | Comparison| Field greater than or equal to value                 | `{"age": {"$gte": 30}}`         |
| `$lte`   | Comparison| Field less than or equal to value                    | `{"age": {"$lte": 30}}`         |
| `$in`    | Set       | Field value is in list                               | `{"role": {"$in": ["a", "b"]}}` |
| `$nin`   | Set       | Field value is not in list                           | `{"role": {"$nin": ["a"]}}`     |
| `$exists`| Existence | Field exists (true) or doesn't exist (false)         | `{"email": {"$exists": true}}`  |
| `$and`   | Logical   | All sub-conditions must match                        | `{"$and": [cond1, cond2]}`      |
| `$or`    | Logical   | At least one sub-condition must match                | `{"$or": [cond1, cond2]}`       |

## Running Tests

```bash
cd /path/to/aethisdb
python -m unittest discover -s tests -v
```

## Architecture

```
aethisdb/
├── __init__.py      Package exports (Database, Collection, Record, peek, ...)
├── database.py      Top-level Database class — file lifecycle, collections, passwords
├── collection.py    Collection class — CRUD operations & query optimization
├── storage.py       StorageEngine — binary file format I/O (header, records, index)
├── codec.py         Compact binary encoder/decoder (replaces JSON on disk)
├── crypto.py        Key derivation + block cipher (keeps files unreadable)
├── legacy.py        One-time reader for v2 (JSON) files, used by auto-migration
├── inspect_file.py  file_info() / peek() read-only helpers
├── index.py         Hash-based Index & IndexManager for fast lookups
├── record.py        Record class — dict-like document with an ID
├── query.py         Query engine — condition parsing, evaluation, update application
├── utils.py         Constants and struct helpers for the binary format
└── exceptions.py    Custom exception hierarchy
```
