Metadata-Version: 2.1
Name: structured-log
Version: 1.17
Summary: Structured logging
Author: Alex Yung
Description-Content-Type: text/markdown
Provides-Extra: secure

# structured-log

Structured logging for Python applications with two interfaces:

- the compatible `set_logger(filename)` interface used by existing applications;
- an opt-in static interface with contextual fields, multiple sinks, redaction,
  reliable HTTP delivery, and flow control.

The distribution name is `structured_log`; the Python package is imported as
`structured_logging`.

## Installation

```bash
pip install structured_log
```

Encrypted disk spooling requires the optional security dependency:

```bash
pip install "structured_log[secure]"
```

## Existing interface

The original interface remains available without behavioural changes:

```python
import logging

from structured_logging.logging import set_logger

set_logger("application.jsonl")
logger = logging.getLogger(__name__)
logger.error("operation failed")
```

This interface adds a JSON `FileHandler` to the root logger. The bundled native
transport is a Linux AMD64 ELF library exported as
`structured_logging.pyd`. On an unsupported platform, native loading is skipped
and file logging continues.

## Extended static interface

The new pipeline is completely opt-in and does not reconfigure the root logger:

```python
from structured_logging.extended import Logging, LoggingConfig, LoggingContext
from structured_logging.sinks import ConsoleSink, HttpSink, RotatingFileSink

Logging.configure(
    LoggingConfig(
        sinks=(
            ConsoleSink(min_level="debug"),
            RotatingFileSink(
                "logs/application.jsonl",
                min_level="info",
                max_bytes=10 * 1024 * 1024,
                backup_count=5,
            ),
            HttpSink(
                "https://logs.example.test/events",
                min_level="warning",
                allowlist={
                    "event_id", "timestamp", "level", "message",
                    "service", "environment", "trace_id",
                },
            ),
        )
    )
)

with LoggingContext.scope(
    service="billing", environment="development", request_id="req-42"
):
    event_id = Logging.info("invoice created", invoice_id="inv-7")

print(Logging.stats())
Logging.shutdown()
```

Explicit event fields override contextual fields. The system-owned
`event_id`, `timestamp`, `level`, `logger`, and `message` fields cannot be
replaced through context or event metadata.

### Context propagation

`LoggingContext` stores values in `contextvars`, isolating concurrent threads
and asynchronous tasks. It supports a scoped lifecycle and explicit transfer to
worker seams:

```python
snapshot = LoggingContext.capture()
result = LoggingContext.run(snapshot, process_job)
result = await LoggingContext.run_async(snapshot, process_async_job)
```

Context values must be JSON-compatible. UUID, date/time, and Enum values are
normalized automatically.

### Standard-library logging adapter

Applications can explicitly connect a standard logger to the extended pipeline:

```python
import logging

from structured_logging.extended import Logging

logger = logging.getLogger("my_app")
logger.addHandler(Logging.handler())
```

The adapter includes the correct module, line number, and function name without
changing the root logger.

## Delivery guarantees

Console and rotating-file sinks are synchronous. HTTP sinks use:

- a bounded background queue;
- NDJSON batches;
- exponential retry with jitter and `Retry-After` support;
- a size- and age-limited disk spool;
- immediate batch flush for `ERROR` and `CRITICAL`;
- a bounded graceful shutdown;
- at-least-once delivery for spooled events.

At-least-once delivery can produce duplicates. Receivers should deduplicate by
the stable `event_id`.

Production configurations with buffered sinks require encrypted spooling:

```python
from structured_logging.delivery import DeliveryConfig, SpoolConfig
from structured_logging.extended import LoggingConfig
from structured_logging.sinks import HttpSink

config = LoggingConfig(
    sinks=(HttpSink("https://logs.example.test/events"),),
    delivery=DeliveryConfig(
        spool=SpoolConfig(
            directory="/var/spool/my-app/logs",
            encryption_key=key_from_secret_manager,
            require_encryption=True,
        )
    ),
    production=True,
)
```

The encryption key can be 32 raw bytes or a Fernet key. Do not store it beside
the spool.

## Data protection

Redaction runs before filters, queues, renderers, and disk storage. The default
rules mask common passwords, authorization headers, cookies, tokens, API keys,
credentials, and private keys. Messages and exception text are checked for
Bearer tokens, JWTs, credential-bearing URLs, and private-key blocks.

Custom rules support masking, removal, partial masking, and stable HMAC values:

```python
from structured_logging.security import DROP, HASH, RedactionConfig, RedactionRule

redaction = RedactionConfig(
    rules=(
        RedactionRule("profile.ssn", action=DROP),
        RedactionRule("email", action=HASH),
        RedactionRule("card_number", keep_last=4),
    ),
    hmac_key=key_from_secret_manager,
)
```

Runtime redaction failures are fail-closed: the affected event is dropped and a
payload-free diagnostic is emitted. Individual sinks can apply a strict field
allowlist after redaction.

## Flow control

Sampling, rate limiting, and deduplication are disabled by default:

```python
from structured_logging.flow import (
    DeduplicationConfig,
    FlowControlConfig,
    RateLimitRule,
    SamplingRule,
)

flow_control = FlowControlConfig(
    sampling=(SamplingRule(rate=0.1, levels=("debug", "info")),),
    rate_limits=(
        RateLimitRule(
            rate_per_second=20,
            burst=100,
            group_by=("logger", "level", "event_name"),
        ),
    ),
    deduplication=DeduplicationConfig(window_seconds=5),
)
```

Sampling is deterministic by `trace_id`, and flow-control state is independent
for every sink. `ERROR` and `CRITICAL` always bypass sampling, rate limiting,
and deduplication.

## Development

Run the test suite:

```bash
python -m unittest discover -s tests -v
```

More detailed configuration examples are available in
[EXTENDED_LOGGING.md](EXTENDED_LOGGING.md).
