Metadata-Version: 2.3
Name: timber-common
Version: 1.1.5
Summary: Configuration-driven persistence library with ml tools (finance related) config driven db model registration, llm model choice, automatic encryption, caching, vector search, and GDPR compliance for Python applications
License: Apache-2.0
Keywords: orm,sqlalchemy,persistence,vector-search,semantic-search,encryption,gdpr,yaml-config,postgres,redis,configuration-driven,data-modeling,machine-learning,embeddings
Author: Pumulo Sikaneta
Author-email: pumulo@gmail.com
Maintainer: Pumulo Sikaneta
Maintainer-email: pumulo@gmail.com
Requires-Python: >=3.13,<4.0
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Financial and Insurance Industry
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Database
Classifier: Topic :: Database :: Front-Ends
Classifier: Topic :: Office/Business :: Financial
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Provides-Extra: all
Provides-Extra: pdf
Provides-Extra: pinecone
Provides-Extra: qdrant
Provides-Extra: weaviate
Requires-Dist: Authlib (>=1.6.5,<2.0.0)
Requires-Dist: Jinja2 (>=3.1.6,<4.0.0)
Requires-Dist: PyYAML (>=6.0.3,<7.0.0)
Requires-Dist: Werkzeug (>=3.1.3,<4.0.0)
Requires-Dist: boto3 (>=1.41.2,<2.0.0)
Requires-Dist: cryptography (>=46.0.2,<47.0.0)
Requires-Dist: fastembed (>=0.7.3,<0.8.0)
Requires-Dist: google-genai[grpc] (>=1.52.0,<2.0.0)
Requires-Dist: httpx (>=0.28.0,<0.29.0)
Requires-Dist: langchain (>=1.0.8,<2.0.0)
Requires-Dist: langchain-core (>=1.1.0,<2.0.0)
Requires-Dist: langchain-google-genai (>=3.1.0,<4.0.0)
Requires-Dist: langgraph (>=1.0.3,<2.0.0)
Requires-Dist: pandas (>=2.3.3,<3.0.0)
Requires-Dist: pgvector[sqlalchemy] (>=0.4.1,<0.5.0)
Requires-Dist: plaid-python (>=39.2.0,<40.0.0)
Requires-Dist: psycopg2-binary (>=2.9.11,<3.0.0)
Requires-Dist: pydantic (>=2.11.9,<3.0.0)
Requires-Dist: python-dotenv (>=1.1.1,<2.0.0)
Requires-Dist: redis (>=6.4.0,<7.0.0)
Requires-Dist: requests (>=2.32.5,<3.0.0)
Requires-Dist: scipy (>=1.16.3,<2.0.0)
Requires-Dist: sendgrid (>=6.12.5,<7.0.0)
Requires-Dist: setuptools (>=80.9.0,<81.0.0)
Requires-Dist: sqlalchemy (>=2.0.36,<3.0.0)
Requires-Dist: stripe (>=15.2.0,<16.0.0)
Requires-Dist: tiktoken (==0.12.0)
Requires-Dist: twilio (>=9.9.0,<10.0.0)
Requires-Dist: vonage (>=4.7.2,<5.0.0)
Requires-Dist: xhtml2pdf (>=0.2.16,<0.3.0) ; extra == "pdf" or extra == "all"
Requires-Dist: yfinance (>=0.2.66,<0.3.0)
Project-URL: Bug Tracker, https://github.com/pumulo/timber-common/issues
Project-URL: Changelog, https://github.com/pumulo/timber-common/blob/main/CHANGELOG.md
Project-URL: Documentation, https://github.com/pumulo/timber-common/tree/main/documentation
Project-URL: Discussions, https://github.com/pumulo/timber-common/discussions
Project-URL: Homepage, https://github.com/pumulo/timber-common
Project-URL: Repository, https://github.com/pumulo/timber-common
Description-Content-Type: text/markdown

# Timber

**Configuration-driven persistence library with automatic encryption, caching, vector search, and GDPR compliance**

[![PyPI version](https://badge.fury.io/py/timber-common.svg)](https://badge.fury.io/py/timber-common)
[![Python 3.13+](https://img.shields.io/badge/python-3.13+-blue.svg)](https://www.python.org/downloads/)
[![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)

---

## What is Timber?

Timber is a **configuration-driven persistence library** that eliminates boilerplate code by defining SQLAlchemy models in YAML instead of Python. It automatically provides encryption, caching, vector search, and GDPR compliance based on simple configuration flags.

**Transform this Python boilerplate:**

```python
class StockResearchSession(Base):
    __tablename__ = 'stock_research_sessions'
    id = Column(String(36), primary_key=True, default=uuid4)
    user_id = Column(String(36), ForeignKey('users.id'), nullable=False)
    symbol = Column(String(10), nullable=False)
    analysis = Column(JSON)
    created_at = Column(DateTime, default=datetime.utcnow)
    # ... 50+ more lines of boilerplate
```

**Into this YAML configuration:**

```yaml
models:
  - name: StockResearchSession
    table_name: stock_research_sessions
    
    # Enable features with one line
    encryption:
      enabled: true
      fields: [analysis]
    
    caching:
      enabled: true
      ttl_seconds: 3600
    
    vector_search:
      enabled: true
      content_field: analysis
    
    columns:
      - name: id
        type: String(36)
        primary_key: true
      - name: user_id
        type: String(36)
        foreign_key: users.id
      - name: symbol
        type: String(10)
      - name: analysis
        type: JSON
```

---

## Key Features

### 🎯 Configuration-Driven Models
- **Zero Python boilerplate** - Define models in YAML
- **Dynamic generation** - Models created at runtime
- **Full SQLAlchemy** - All SQLAlchemy features supported
- **Type-safe** - Validated configuration with clear errors

### 🔐 Automatic Encryption
- **Field-level encryption** - Specify fields to encrypt
- **Transparent** - Automatic encrypt/decrypt
- **Secure** - Uses Fernet (symmetric encryption)
- **No code changes** - Enable with one config line

### ⚡ Multi-Level Caching
- **Redis support** - Distributed caching
- **Local cache** - In-memory fallback
- **Automatic invalidation** - Cache cleared on updates
- **Configurable TTL** - Per-model cache duration

### 🔍 Vector Search
- **Semantic search** - Find by meaning, not keywords
- **Automatic embeddings** - Generated on insert
- **Multiple backends** - Qdrant, Weaviate, Pinecone
- **Hybrid search** - Combine vector + keyword

### ✅ GDPR Compliance
- **Data export** - User data export in JSON
- **Right to deletion** - Complete data removal
- **Audit trails** - Track data operations
- **Configurable** - Specify exportable fields

### 🏗️ Modular Services
- **Session Service** - User session management
- **Research Service** - Store analysis and research
- **Notification Service** - User notifications
- **Tracker Service** - Event tracking and analytics
- **Stock Data Service** - Financial data fetching

### 🌐 Multi-App Support
- **Shared infrastructure** - One library, many apps
- **Data isolation** - Clear boundaries between apps
- **Consistent patterns** - Same API across applications

### 📈 Multi-Provider Stock Data
- **Provider fallback chain** - yfinance, Alpha Vantage, Finnhub, Polygon
- **Per-call ordering** - Pass `provider_order=[...]` to control the chain
- **Skip-missing-key / fall-through** - Skips keyless providers, falls through on errors and empty results
- **Deterministic, normalized output** - Reproducible P/E, standardized info/news across sources

### 🧩 Domain Plugins
- **Discoverable extensions** - Domains register via the `timber.domains` entry-point group
- **No core edits** - Plugins write into shared registries; core never imports a domain
- **Dependency-injected** - Each plugin receives a `DomainContext` (registries + db/llm/config)
- **Registered at startup** - Models/services wired in before table creation (init Step 8.5)

A domain is an installed package that advertises one entry point per layer. Timber is the
substrate layer (`timber.domains` → models + services); grove (`grove.domains` → HTTP routers)
and acorn (`acorn.domains` → agent tools) follow the same shape against their own groups, and a
domain can also ship sky widgets. One package can plug into all of them with no edits to the core
libraries. Two domains live on this system today:

- **`oak-domain-investments`** - the first complete business domain: goal-linked watchlist,
  accumulation plans, value projection, growth scheduler, and holdings linkage (in production).
- **`oak-domain-legal-intake`** - a second, unrelated access-to-justice domain, proving the
  pattern is general (Phase 0, instrumentation only).

See the `oak-domain-plugins` skill for the full plugin contract.

---

## Quick Start

### Installation

```bash
pip install timber-common
```

### Basic Example

```python
from timber.common import initialize_timber, get_model
from timber.common.services.persistence import session_service

# 1. Initialize Timber with your model configs
initialize_timber(
    model_config_dirs=['./data/models'],
    database_url='postgresql://localhost:5432/mydb'
)

# 2. Use services immediately
session_id = session_service.create_session(
    user_id='user-123',
    session_type='research',
    metadata={'symbol': 'AAPL'}
)

# 3. Or access models directly
Session = get_model('Session')
session = session_service.get_session(session_id)
print(f"Created session for {session.metadata['symbol']}")
```

### Complete Workflow Example

```python
from timber.common import initialize_timber
from timber.common.services.persistence import (
    session_service,
    research_service,
    notification_service
)

# Initialize
initialize_timber(model_config_dirs=['./data/models'])

# Create research session
session_id = session_service.create_session(
    user_id='user-123',
    session_type='research',
    metadata={'symbol': 'AAPL'}
)

# Save research (automatically encrypted if configured)
research_id = research_service.save_research(
    session_id=session_id,
    content={
        'company': 'Apple Inc.',
        'analysis': 'Strong fundamentals...',
        'recommendation': 'Buy'
    },
    research_type='fundamental'
)

# Notify user (automatically stored)
notification_service.create_notification(
    user_id='user-123',
    notification_type='research_complete',
    title='Analysis Complete',
    message='Your AAPL analysis is ready'
)

print(f"✅ Research workflow complete!")
```

### Vector Search Example

```python
from timber.common.services.vector import vector_service

# Semantic search (finds by meaning, not just keywords)
results = vector_service.search(
    query="companies with strong AI capabilities",
    collection_name="research_documents",
    limit=10
)

for result in results:
    print(f"{result['payload']['title']}: {result['score']:.3f}")
```

---

## Multi-Provider Stock Data

Timber ships a unified stock-data service that fetches market data across multiple
providers — **yfinance, Alpha Vantage, Finnhub, and Polygon** — with automatic
fallback. yfinance is key-free and always available; the other three are used only
when their API key is configured.

```python
from common.services.data_fetcher import stock_data_service

# Historical OHLCV (returns a (DataFrame, error) tuple)
df, error = stock_data_service.fetch_historical_data("AAPL", period="1y")

# Company info / news / financials
info, error = stock_data_service.fetch_company_info("AAPL")
news, error = stock_data_service.fetch_news("AAPL", limit=10)
income, balance, cashflow, error = stock_data_service.fetch_financials("AAPL", period="yearly")
```

### Explicit provider ordering

Every fetch method accepts an optional `provider_order` list. When supplied, the
listed providers are tried in order; otherwise a key-derived primary/fallback
ordering is used. The chain **skips any provider whose API key is missing** and
**falls through to the next provider on an error or an empty result**, returning the
last error only if all providers fail.

```python
# Try Polygon first, then Alpha Vantage, then yfinance.
df, error = stock_data_service.fetch_historical_data(
    "AAPL",
    period="1y",
    provider_order=["polygon", "alphavantage", "yfinance"],
)
```

The available fetch methods are `fetch_historical_data`, `fetch_company_info`,
`fetch_news`, and `fetch_financials`, all of which accept `provider_order`.

### Deterministic P/E and cross-source normalization

Provider responses are normalized to a stable shape. Because a provider's live
`trailingPE` is tied to the intraday tick (and so varies run to run), Timber derives a
reproducible **`peRatio = previousClose / trailingEps`** (rounded, div-by-zero
guarded) whenever both inputs are available. `previousClose` and `trailingEps` are
always surfaced, and the provider's raw value is preserved under `trailingPE_live`.
News from all providers is deduplicated and ordered deterministically.

### Alpha Vantage throttling and delayed entitlement

Alpha Vantage returns **HTTP 200 with a single-key payload** when throttled, so
Timber treats `"Error Message"`, `"Note"`, and the newer daily-limit
**`"Information"`** responses as *errors* (not empty success). This lets the provider
chain fall through to the next source instead of silently degrading to blank fields.

Delayed-plan support is built in: set the Alpha Vantage `entitlement` config to
`"delayed"` and Timber automatically appends `entitlement=delayed` to the market-data
functions that require it (e.g. `GLOBAL_QUOTE` and the `TIME_SERIES_*` family),
while leaving fundamentals and news endpoints untouched.

---

## Domain Plugins

Timber can be extended to new business domains **without modifying the core library**.
A domain lives as its own installable package that *declares itself* and registers its
models, services, and operations into Timber's shared registries at startup. The
dependency direction is strictly one-way: domain plugins import Timber's core
interfaces; **core never imports a domain.**

### The plugin contract

A domain exposes an object satisfying the `DomainPlugin` protocol:

```python
from common.plugins import DomainPlugin, DomainContext

class LegalIntakeDomain:
    name = "legal_intake"        # stable domain key
    version = "0.1.0"            # domain/contract version

    def register(self, ctx: DomainContext) -> None:
        # Wire models/services/operations into the shared registries.
        ctx.service_registry.register("legal_intake", "intake", IntakeService(ctx.db))
        # ctx also exposes: model_registry, operation_registry, db, llm, config
        ...
```

`DomainContext` is a dataclass injected at startup carrying the registries the domain
*writes* to (`model_registry`, `operation_registry`, `service_registry`) and the shared
singletons it *reads* (`db`, `llm`, `config`). `register(ctx)` is called exactly once
per process, **after core models are loaded and before table creation**, so any models
a domain registers get their tables created.

### Discovery

Plugins are discovered two ways, in priority order:

1. **Entry points** (the production mechanism) — installed packages advertise
   themselves under the **`timber.domains`** entry-point group:

   ```toml
   # in a domain package's pyproject.toml
   [project.entry-points."timber.domains"]
   legal_intake = "oak_domain_legal_intake:LegalIntakeDomain"
   ```

2. **`TIMBER_DOMAIN_PLUGINS` env var** (local-dev / test fallback) — a comma-separated
   list of `module:attr` references:

   ```bash
   TIMBER_DOMAIN_PLUGINS="oak_domain_legal_intake:LegalIntakeDomain"
   ```

Plugins are de-duplicated by `name` (entry points win over env refs). With neither
mechanism present, discovery is a no-op and initialization is unchanged.

### The service registry

Domains register services under a domain namespace via the singleton
`ServiceRegistry` instead of monkeypatching the `common.services` module. Consumers
reach them through a live accessor:

```python
from common.services.registry import service_registry

intake = service_registry.domain("legal_intake").intake   # attribute access
intake = service_registry.get("legal_intake", "intake")    # or by (domain, name)
```

### Wiring at startup (`initialize_timber` Step 8.5)

`initialize_timber()` runs a **Step 8.5** that discovers domains and registers them
before table creation:

```python
from common.plugins import discover_domains, DomainContext
from common.services.registry import service_registry

ctx = DomainContext(
    model_registry=model_registry,
    operation_registry=operation_registry,
    service_registry=service_registry,
    db=db_service,
    llm=llm_service,
    config=config,
)
for plugin in discover_domains():
    plugin.register(ctx)   # registers models/services/operations
# ...table creation (Step 9) then picks up any newly registered domain models.
```

If no domains are installed, Step 8.5 is a no-op and Timber behaves as core-only.

---

## Documentation

### 📚 How-To Guides
- [Getting Started](documentation/how_to/01_getting_started.md) - Setup and first model
- [Creating Models](documentation/how_to/02_creating_models.md) - YAML model definitions
- [Using Services](documentation/how_to/03_using_services.md) - Persistence services

### 🏛️ Design Guides
- [System Architecture](documentation/design_guides/01_system_architecture.md) - Overall design
- [Config-Driven Models](documentation/design_guides/02_config_driven_models.md) - Model factory pattern
- [Persistence Layer](documentation/design_guides/03_persistence_layer.md) - Database architecture
- [Vector Integration](documentation/design_guides/04_vector_integration.md) - Semantic search
- [Multi-App Support](documentation/design_guides/05_multi_app_support.md) - Multiple applications

### 📖 Full Documentation Index
See [DOCUMENTATION_INDEX.md](documentation/DOCUMENTATION_INDEX.md) for complete documentation structure.

---

## Requirements

- **Python:** 3.13+
- **Database:** PostgreSQL 12+
- **Optional:** Redis (for distributed caching)
- **Optional:** Qdrant/Weaviate/Pinecone (for vector search)

---

## Installation Options

### Basic Installation

```bash
pip install timber-common
```

### With Vector Search (Qdrant)

```bash
pip install timber-common[qdrant]
```

### With All Optional Features

```bash
pip install timber-common[all]
```

### Development Installation

```bash
git clone https://github.com/pumulo/timber-common.git
cd timber-common
poetry install
```

---

## Configuration

### Environment Variables

Create a `.env` file:

```bash
# Database
DATABASE_URL=postgresql://user:password@localhost:5432/dbname

# Redis (optional)
REDIS_URL=redis://localhost:6379/0

# Vector Database (optional)
QDRANT_URL=http://localhost:6333

# Encryption
ENCRYPTION_KEY=your-fernet-key-here

# Feature Flags
ENABLE_ENCRYPTION=true
ENABLE_VECTOR_SEARCH=true
ENABLE_GDPR=true
CACHE_ENABLED=true
```

### Model Configuration

Create YAML files in `data/models/`:

```yaml
# data/models/user_models.yaml
version: "1.0.0"

models:
  - name: User
    table_name: users
    
    columns:
      - name: id
        type: String(36)
        primary_key: true
        default: uuid4
      
      - name: email
        type: String(255)
        unique: true
        nullable: false
      
      - name: created_at
        type: DateTime
        default: utcnow
```

---

## Use Cases

### Financial Applications
- Trading platforms
- Research tools
- Portfolio management
- Market analysis

### Content Platforms
- Document management
- Knowledge bases
- Content recommendation
- Semantic search

### Data Analytics
- User behavior tracking
- Event analytics
- Session management
- Activity monitoring

### Multi-Tenant Applications
- SaaS platforms
- Enterprise applications
- Multiple product lines
- Isolated data domains

---

## Architecture

```
┌─────────────────────────────────────────┐
│          Your Application               │
└─────────────────────────────────────────┘
                  │
                  ↓
┌─────────────────────────────────────────┐
│         Timber Library                  │
│  ┌──────────────┐  ┌─────────────────┐ │
│  │ Model Factory│  │  Services Layer │ │
│  └──────────────┘  └─────────────────┘ │
│  ┌──────────────┐  ┌─────────────────┐ │
│  │   Encryption │  │  Vector Search  │ │
│  └──────────────┘  └─────────────────┘ │
└─────────────────────────────────────────┘
                  │
                  ↓
┌─────────────────────────────────────────┐
│      Infrastructure                     │
│  PostgreSQL │ Redis │ Qdrant           │
└─────────────────────────────────────────┘
```

---

## Examples

### E-Commerce Platform

```yaml
models:
  - name: Product
    table_name: products
    
    vector_search:
      enabled: true
      content_field: description
    
    columns:
      - name: id
        type: String(36)
        primary_key: true
      - name: name
        type: String(255)
      - name: description
        type: Text
      - name: price
        type: Numeric(10, 2)
```

### Healthcare Application

```yaml
models:
  - name: PatientRecord
    table_name: patient_records
    
    encryption:
      enabled: true
      fields: [ssn, medical_history]
    
    gdpr:
      enabled: true
      user_id_field: patient_id
      export_fields: [name, date_of_birth, medical_history]
    
    columns:
      - name: id
        type: String(36)
        primary_key: true
      - name: patient_id
        type: String(36)
        foreign_key: patients.id
      - name: ssn
        type: String(11)
      - name: medical_history
        type: JSON
```

---

## Testing

```bash
# Run tests
poetry run pytest

# With coverage
poetry run pytest --cov=common --cov=modules

# Run specific test
poetry run pytest tests/test_models.py::test_create_model
```

---

## Contributing

We welcome contributions! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.

### Development Setup

```bash
# Clone repository
git clone https://github.com/pumulo/timber-common.git
cd timber-common

# Install dependencies
poetry install

# Run tests
poetry run pytest

# Format code
poetry run black .
poetry run isort .

# Type check
poetry run mypy common modules
```

---

## Performance

Timber is designed for production use with:

- **Connection pooling** - Efficient database connections
- **Query optimization** - Built-in best practices
- **Caching** - Multi-level cache strategy
- **Batch operations** - Efficient bulk processing

### Benchmarks

```
Operation               Time (ms)    Notes
─────────────────────────────────────────────
Simple INSERT           1-5          Single record
Batch INSERT (100)      10-20        Bulk insert
SELECT by ID            1-2          Indexed lookup
Vector search           5-15         Semantic search
Cached query            < 1          Redis/local cache
```

---

## Roadmap

### Version 0.2.0 (Q1 2025)
- [ ] MySQL and SQLite support
- [ ] GraphQL API generation
- [ ] CLI tools for model management
- [ ] Enhanced monitoring dashboard

### Version 0.3.0 (Q2 2025)
- [ ] Real-time data streaming
- [ ] Advanced analytics
- [ ] Built-in vector store (no external DB required)
- [ ] Docker and Kubernetes templates

### Future
- [ ] Multi-database transactions
- [ ] Distributed tracing
- [ ] Auto-scaling recommendations
- [ ] Visual model designer

---

## Support

### Get Help
- **Documentation:** [Full docs](documentation/)
- **Issues:** [GitHub Issues](https://github.com/pumulo/timber-common/issues)
- **Email:** pumulo@gmail.com

### Commercial Support
For enterprise support, training, or consulting:
- Email: pumulo@gmail.com

---

## License

Timber is released under the [Apache License 2.0](LICENSE).

```
Copyright 2025 Pumulo Sikaneta

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
```

See the [LICENSE](LICENSE) file for the full license text.

**License: Apache-2.0**

---

## Author

**Pumulo Sikaneta**

- Email: pumulo@gmail.com
- GitHub: [@pumulo](https://github.com/pumulo)
- Website: [Your website]

---

## Acknowledgments

Built with:
- [SQLAlchemy](https://www.sqlalchemy.org/) - The Python SQL toolkit
- [PostgreSQL](https://www.postgresql.org/) - The world's most advanced open source database
- [FastEmbed](https://github.com/qdrant/fastembed) - Fast embedding generation
- [Poetry](https://python-poetry.org/) - Python dependency management

---

## Citation

If you use Timber in academic research, please cite:

```bibtex
@software{timber2025,
  author = {Sikaneta, Pumulo},
  title = {Timber: Configuration-Driven Persistence Library},
  year = {2025},
  url = {https://github.com/pumulo/timber-common},
  version = {0.1.0}
}
```

---

## Star History

If you find Timber useful, please star the repository! ⭐

---

**Made with ❤️ by Pumulo Sikaneta**

**Copyright © 2025 Pumulo Sikaneta. Licensed under Apache-2.0.**
