Metadata-Version: 2.4
Name: pumpwood-deploy-complex-datalake
Version: 0.0.3
Summary: Package to assist deploy Pumpwood Complex Datalake on K8s
License: BSD-3-Clause License
License-File: LICENSE
Author: André Andrade Baceti
Author-email: a.baceti@murabei.com
Requires-Python: >=3.6
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Dist: pumpwood-deploy
Project-URL: Homepage, https://github.com/Murabei-OpenSource-Codes/pumpwood-deploy-complex-datalake
Description-Content-Type: text/markdown

# pumpwood-deploy-complex-datalake

Satellite deploy package for the **Pumpwood Complex Datalake**
microservice on Kubernetes. It generates manifests for the API app,
the complex-database dataloader worker, and secrets, then hands them
to
[`pumpwood-deploy`](https://github.com/Murabei-OpenSource-Codes/pumpwood-deploy)
for apply.

Developed by [Murabei Data Science](https://murabei.com). BSD-3-Clause.

<p align="center" width="60%">
  <img src="static_doc/sitelogo-horizontal.png" /> <br>

  <a href="https://en.wikipedia.org/wiki/Cecropia">
    Pumpwood is a native Brazilian tree
  </a> with a symbiotic relation to ants (Murabei)
</p>

---

## Objective and motivation

This package deploys Complex Datalake as a satellite of
`pumpwood-deploy`: one Python class renders Kubernetes YAML; the core
orchestrator applies it.

### Why this exists

Complex Datalake used to live inside the monolithic `pumpwood-deploy`
tree. Splitting it keeps image pins, worker count, and database
credentials next to this service without shipping unused annotation
workers.

### How it is used

A platform deploy script constructs
`PumpWoodComplexDatalakeMicroservice`, registers it with
`DeployPumpWood.add_microservice`, then calls
`create_deploy_files` / `deploy_microservices`. Operators pass
**image tags** (`app_version`, `datalake_dataloader_version`); the
**PyPI package** version is in `VERSION` (see [changelog.md](changelog.md)).

### Scope

In scope: Secret, app Deployment + Service, and the
complex-database dataloader worker.

Out of scope: Postgres/PgBouncer, RabbitMQ, storage ConfigMap, auth,
Kong, and annotation workers. Those come from `pumpwood-deploy`,
`pumpwood-deploy-auth`, or a follow-up satellite.

---

## What it deploys

| Manifest | Kubernetes resources |
|----------|----------------------|
| `pumpwood_complex__datalake__secrets` | Secret `pumpwood-complex-datalake` |
| `pumpwood_complex__datalake__deploy` | Deployment + Service `pumpwood-complex-datalake-app` |
| `pumpwood_complex__datalake_dataloader_worker` | Deployment `pumpwood-complex-datalake-worker-complex-database` |

The app serves HTTP APIs. The worker consumes RabbitMQ and uploads
rows in parallel chunks to Postgres.

```mermaid
flowchart LR
    subgraph pkg [pumpwood-deploy-complex-datalake]
        A[PumpWoodComplexDatalakeMicroservice]
    end
    subgraph core [pumpwood-deploy]
        B[DeployPumpWood]
    end
    subgraph cluster [Cluster]
        S[pumpwood-complex-datalake Secret]
        APP[pumpwood-complex-datalake-app]
        W[complex-database worker]
        RMQ[rabbitmq-main]
    end
    A --> B
    B --> S
    B --> APP
    B --> W
    RMQ --> APP
    RMQ --> W
```

---

## Prerequisites

This package does **not** stand alone. Before pods can start, the
cluster must already provide:

| Resource | Provided by |
|----------|-------------|
| `storage` ConfigMap | `StandardMicroservices` in `pumpwood-deploy` |
| `general-secrets` | `StandardMicroservices` |
| `rabbitmq-main-secrets` | `StandardMicroservices` |
| Storage keys (GCP / Azure / AWS) | `DeployPumpWood` storage config |
| Postgres for complex datalake | `PostgresDatabase` + `PGBouncerDatabase` |
| Auth (typical) | [`pumpwood-deploy-auth`](https://github.com/Murabei-OpenSource-Codes/pumpwood-deploy-auth) |

Storage bucket name and type are read from the cluster `storage`
ConfigMap. They are **not** constructor arguments of
`PumpWoodComplexDatalakeMicroservice`.

For local or CI databases, deploy Postgres/PgBouncer from
`pumpwood-deploy`. Embedded test-database parameters were removed
from this satellite.

---

## Installation

```bash
pip install pumpwood-deploy-complex-datalake
```

Requires `pumpwood-deploy` and Python 3.6+.

---

## Quick start

```python
import os
import simplejson as json
from dotenv import load_dotenv
from pumpwood_deploy.deploy import DeployPumpWood
from pumpwood_deploy.microservices.postgres.deploy import (
    PostgresDatabase, PGBouncerDatabase)
from pumpwood_deploy_complex_datalake import (
    PumpWoodComplexDatalakeMicroservice)

with open("secrets/production.json", "r") as file:
    secrets = json.loads(file.read())
load_dotenv()

deploy = DeployPumpWood(
    model_user_password=secrets["microservices--model"],
    rabbitmq_secret=secrets["rabbitmq_secret"],
    hash_salt=secrets["hash_salt"],
    storage_type="aws_s3",
    storage_deploy_args={
        "storage_bucket_name": "my-pumpwood-bucket",
        "access_key_id": secrets["aws_access_key_id"],
        "secret_access_key": secrets["aws_secret_access_key"],
    },
    k8_provider="aws",
    k8_deploy_args={
        "region": "us-east-1",
        "cluster_name": "my-cluster",
    },
    k8_namespace="pumpwood",
)

deploy.add_microservice(
    PostgresDatabase(
        db_username="pumpwood",
        db_password=secrets["postgres_password"],
        name="postgres-main",
        disk_name="postgres-disk",
        disk_size="150Gi",
    ))

deploy.add_microservice(
    PGBouncerDatabase(
        name="pgbouncer-pumpwood-complex-datalake",
        postgres_database="pumpwood_complex_datalake",
        postgres_secret="postgres-main",
        postgres_host="postgres-main",
    ))

deploy.add_microservice(
    PumpWoodComplexDatalakeMicroservice(
        app_version=os.getenv("PUMPWOOD_COMPLEX_DATALAKE_APP"),
        datalake_dataloader_version=os.getenv(
            "PUMPWOOD_COMPLEX_DATALAKE_WORKER"),
        repository="my-registry.example.com",
        db_host="pgbouncer-pumpwood-complex-datalake",
        db_database="pumpwood_complex_datalake",
        db_password=secrets["postgres_password"],
        microservice_password=secrets[
            "microservice--complex-datalake"],
        app_replicas=1,
        app_debug="FALSE",
        datalake_dataloader_replicas=1,
        datalake_dataloader_n_parallel=4,
    ))

deploy.create_deploy_files()
deploy.deploy_microservices()
```

### Environment variables

These are **container image tags**, not the PyPI package version:

```bash
PUMPWOOD_COMPLEX_DATALAKE_APP=2.0.69
PUMPWOOD_COMPLEX_DATALAKE_WORKER=2.0.25
```

If the rendered manifest matches the cluster, `kubectl apply` is a
no-op — safe for rolling image updates.

---

## Configuration reference

Constructor:
`PumpWoodComplexDatalakeMicroservice` in
`src/pumpwood_deploy_complex_datalake/deploy.py`.

### Required

| Parameter | Description |
|-----------|-------------|
| `app_version` | Image tag for `pumpwood-complex-datalake-app` |
| `datalake_dataloader_version` | Image tag for `pumpwood-complex-datalake-worker-complex-database` |

### Database and registry

| Parameter | Default | Description |
|-----------|---------|-------------|
| `db_host` | `pgbouncer-pumpwood-complex-datalake` | Postgres host (PgBouncer in prod) |
| `db_port` | `5432` | Postgres port |
| `db_database` | `pumpwood` | Database name |
| `db_username` | `pumpwood` | Database user |
| `db_password` | `pumpwood` | Database password |
| `microservice_password` | `microservice--complex-datalake` | Service user password |
| `repository` | `gcr.io/repositorio-geral-170012` | Docker registry for app and worker |

### Application

| Parameter | Default | Description |
|-----------|---------|-------------|
| `app_replicas` | `1` | Number of app pods |
| `app_debug` | `FALSE` | Debug flag (`TRUE` / `FALSE`) |
| `app_workers` | `10` | Granian workers (`GRANIAN_WORKERS`) |
| `app_timeout` | `300` | Request timeout in seconds |
| `app_limits_memory` | `60Gi` | Memory limit |
| `app_limits_cpu` | `12000m` | CPU limit |
| `app_requests_memory` | `20Mi` | Memory request |
| `app_requests_cpu` | `1m` | CPU request |

### Complex-database dataloader worker

| Parameter | Default | Description |
|-----------|---------|-------------|
| `datalake_dataloader_replicas` | `1` | Worker pod count |
| `datalake_dataloader_debug` | `FALSE` | Worker debug flag |
| `datalake_dataloader_n_parallel` | `4` | Parallel upload requests |
| `datalake_dataloader_chunk_size` | `1000` | Rows per parallel batch |
| `datalake_dataloader_query_limit` | `1000000` | Max rows per upload cycle |
| `markitdown_ocr_enabled` | `FALSE` | LLM Vision OCR for scanned PDFs (`MARKITDOWN_OCR_ENABLED`) |
| `markitdown_llm_model` | `gpt-4o` | OCR model (`MARKITDOWN_LLM_MODEL`) |
| `openai_api_key` | (empty) | Stored in Secret; mounted as `OPENAI_API_KEY` |
| `datalake_dataloader_limits_memory` | `60Gi` | Worker memory limit |
| `datalake_dataloader_limits_cpu` | `12000m` | Worker CPU limit |
| `datalake_dataloader_requests_memory` | `20Mi` | Worker memory request |
| `datalake_dataloader_requests_cpu` | `1m` | Worker CPU request |

OCR is off by default. Set `markitdown_ocr_enabled="TRUE"` and pass
`openai_api_key` to extract text from scanned PDFs (worker image
`2.0.23+`).

---

## Health check

The app Deployment readiness probe is:

```text
GET /health-check/pumpwood-complex-datalake-app/  (port 5000)
```

Use the same path for ingress and load balancer checks.

---

## Migration from monolithic `pumpwood-deploy`

Older scripts imported from the core package:

```python
# Before
from pumpwood_deploy.microservices.pumpwood_complex_datalake.deploy import (
    PumpWoodComplexDatalakeMicroservice)

# After
from pumpwood_deploy_complex_datalake import (
    PumpWoodComplexDatalakeMicroservice)
```

Constructor changes in this satellite:

- Worker image tag is `datalake_dataloader_version` (not
  `worker_datalake_dataloader_version`)
- `bucket_name` — use the cluster `storage` ConfigMap
- `test_db_*` — use `PostgresDatabase` / `PGBouncerDatabase`
- `worker_simple_dataloader_version` and
  `worker_complex_dataloader_version` — annotation workers are not
  in this package

The monolith rendered five manifests (three workers). This satellite
renders three. Add annotation workers in another package if needed.

---

## Related packages

| Package | Role |
|---------|------|
| [`pumpwood-deploy`](https://github.com/Murabei-OpenSource-Codes/pumpwood-deploy) | Orchestrator, Kong, RabbitMQ, Postgres |
| [`pumpwood-deploy-datalake`](https://github.com/Murabei-OpenSource-Codes/pumpwood-deploy-datalake) | Standard datalake microservice |
| [`pumpwood-deploy-auth`](https://github.com/Murabei-OpenSource-Codes/pumpwood-deploy-auth) | Authorization microservice |

Platform docs:
[Murabei Open Source — pumpwood-deploy](https://murabei-opensource-codes.github.io/pumpwood-deploy/).

Release history: [changelog.md](changelog.md).

---

## Development

```bash
pip install -e ../pumpwood-deploy
pip install -e .

PYTHONPATH="src:../pumpwood-deploy/src" \
  python3 -m unittest discover \
  -s src/pumpwood_deploy_complex_datalake/tests -p "test_*.py" -v

ruff check src/
```

---

## License

BSD-3-Clause — see [LICENSE](LICENSE).

