Metadata-Version: 2.4
Name: vcti-path-format
Version: 1.2.0
Summary: File format identification framework with heuristic evaluators and feature validators for Python
Author: Visual Collaboration Technologies Inc.
Requires-Python: <3.15,>=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: vcti-plugin-catalog>=1.0.1
Requires-Dist: vcti-lookup>=1.0.1
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Requires-Dist: pytest-cov; extra == "test"
Provides-Extra: lint
Requires-Dist: ruff; extra == "lint"
Dynamic: license-file

# Path Format

File format identification framework with heuristic evaluators and feature validators for Python.

## Installation

```bash
pip install vcti-path-format>=1.2.0
```

### In `pyproject.toml` dependencies

```toml
dependencies = [
    "vcti-path-format>=1.2.0",
]
```

---

## Quick Start

```python
from pathlib import Path
from vcti.pathformat import (
    FormatDescriptor,
    FormatIdentifier,
    FormatRegistry,
    MatchConfidence,
)
from vcti.pathformat.evaluator import HeuristicEvaluator

# Define a format descriptor with validators
hdf5_descriptor = FormatDescriptor(
    id="hdf5-file",
    name="HDF5 File",
    evaluator=(
        HeuristicEvaluator()
        .check_magic_bytes(b"\x89HDF\r\n\x1a\n")  # GATE
        .check_extension([".h5", ".hdf5", ".he5"])  # EVIDENCE
    ),
    attributes={"path_type": "file", "structure": "hdf5"},
)

# Register in a format registry
registry = FormatRegistry()
registry.register(hdf5_descriptor)

# Identify a file
identifier = FormatIdentifier(registry)
results = identifier.identify_file_format(Path("data.h5"))

for result in results:
    print(f"{result.descriptor.name}: {result.confidence.name}")

# Get best match above a confidence threshold
best = identifier.get_best_match(
    Path("data.h5"),
    min_confidence=MatchConfidence.LIKELY,
)
```

---

## Core Concepts

### FormatDescriptor

Extends `Descriptor[Evaluator]` from vcti-plugin-catalog. Wraps an evaluator
with format metadata and attributes.

### FormatRegistry

Extends `Registry[FormatDescriptor]`. Central catalog of known formats with
attribute-based filtering via `registry.lookup`.

### FormatIdentifier

Evaluates a path against all (or filtered) registered formats and returns
results sorted by confidence.

### HeuristicEvaluator

Builder-pattern evaluator that aggregates validation evidence:

```python
evaluator = (
    HeuristicEvaluator()
    .check_magic_bytes(b"\x89PNG\r\n\x1a\n")  # GATE
    .check_extension([".png"])                   # EVIDENCE
    .add_validator(custom_validator)             # Custom
)
```

**Heuristic rules:**
- Failed GATE -> `CERTAINLY_NOT`
- All passed + GATE present -> `DEFINITE`
- All passed + no GATE -> `LIKELY`
- Some EVIDENCE failed -> `UNLIKELY`
- No validators -> `CANT_EVALUATE`

### Feature Validators

| Validator | Role | Tier | Checks |
|-----------|------|------|--------|
| `MagicBytesValidator` | GATE | IDENTIFICATION | File signature bytes |
| `ExtensionValidator` | EVIDENCE | IDENTIFICATION | File extension |

Custom validators implement the `FeatureValidator` protocol.

---

## Validation Tiers

Control evaluation depth with `max_tier`:

| Tier | Cost | Examples |
|------|------|---------|
| `IDENTIFICATION` | Cheap | Magic bytes, file extension |
| `STRUCTURE` | Medium | Schema validation, header parsing |
| `SEMANTIC` | Expensive | Content analysis, business logic |

```python
from vcti.pathformat import ValidationTier

# Only run cheap checks
results = identifier.identify_file_format(path, max_tier=ValidationTier.IDENTIFICATION)
```

---

## Custom Validators

Implement the `FeatureValidator` protocol to add domain-specific checks:

```python
from pathlib import Path
from vcti.pathformat.feature_validator import (
    FeatureValidator,
    ValidationResult,
    ValidationTier,
    ValidatorRole,
)

class HeaderValidator:
    """Checks for a text header line in the first line of a file."""

    id = "header-check"
    description = "Header line validator"
    role = ValidatorRole.EVIDENCE
    tier = ValidationTier.STRUCTURE

    def __init__(self, expected_header: str):
        self.expected_header = expected_header

    def validate(self, path: Path) -> ValidationResult:
        try:
            first_line = path.read_text(encoding="utf-8").split("\n", 1)[0]
            is_passed = first_line.strip() == self.expected_header
        except (OSError, UnicodeDecodeError):
            is_passed = False
        return ValidationResult(
            validator_id=self.id,
            role=self.role,
            is_passed=is_passed,
            details=f"Header {'matches' if is_passed else 'mismatch'}",
        )

# Use with the builder pattern
evaluator = (
    HeuristicEvaluator()
    .check_extension([".csv"])
    .add_validator(HeaderValidator("id,name,value"))
)
```

---

## Evaluator Caching

`HeuristicEvaluator` includes an LRU cache keyed by `(path, max_tier)`:

```python
# Default: 128 entries
evaluator = HeuristicEvaluator(cache_size=128)

# Disable caching
evaluator = HeuristicEvaluator(cache_size=0)

# Bypass cache for a single call
report = descriptor.evaluate(path, use_cache=False)

# Inspect and manage
info = evaluator.cache_info()  # (hits, misses, maxsize, currsize) or None
evaluator.clear_cache()
```

Cache entries become stale if file contents change. Call `clear_cache()` after
known file modifications, or pass `use_cache=False` for one-off re-evaluation.

---

## Pre-filtering with Rules

```python
from vcti.lookup import Rule

# Only evaluate formats with structure="hdf5"
results = identifier.identify_file_format(
    path,
    rules=[Rule("structure", "==", "hdf5")],
)
```

---

## Error Handling

The framework raises typed exceptions:

| Exception | When |
|-----------|------|
| `FileNotFoundError` | Path does not exist |
| `PathAccessError` | Path is not a file or directory, or cannot be read |
| `EvaluatorError` | Base class for evaluator errors |
| `ValidationError` | A validator raised an unexpected exception |
| `InvalidValidatorError` | Invalid validator passed to builder |

```python
from vcti.pathformat import PathAccessError

try:
    results = identifier.identify_file_format(path)
except FileNotFoundError:
    print("File not found")
except PathAccessError as e:
    print(f"Cannot access path: {e}")
```

---

## Ecosystem

This package is the identification engine in a three-repo system:

| Package | Role |
|---------|------|
| **vcti-path-format** | Framework: evaluators, validators, registry, identifier |
| [vcti-path-format-attributes](https://pypi.org/project/vcti-path-format-attributes/) | Vocabulary: standardized attribute enums |
| [vcti-path-format-descriptors](https://pypi.org/project/vcti-path-format-descriptors/) | Built-in format definitions (HDF5, CAX, etc.) |

---

## Dependencies

- [vcti-plugin-catalog](https://pypi.org/project/vcti-plugin-catalog/) (>=1.0.1) — descriptor/registry framework
- [vcti-lookup](https://pypi.org/project/vcti-lookup/) (>=1.0.1) — attribute-based filtering
