Metadata-Version: 2.4
Name: pumpwood-deploy-crawler-bigdata
Version: 0.0.1
Summary: Package to assist deploy Pumpwood Crawler BigData on K8s
License: BSD-3-Clause License
License-File: LICENSE
Author: André Andrade Baceti
Author-email: a.baceti@murabei.com
Requires-Python: >=3.6
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Dist: pumpwood-deploy
Project-URL: Homepage, https://github.com/Murabei-OpenSource-Codes/pumpwood-deploy-crawler-bigdata
Description-Content-Type: text/markdown

# pumpwood-deploy-crawler-bigdata

Satellite deploy package for the **Pumpwood Crawler BigData**
microservice on Kubernetes. It generates manifests for the API
application, person and company workers, and crawler secrets — then
hands them to
[`pumpwood-deploy`](https://github.com/Murabei-OpenSource-Codes/pumpwood-deploy)
for apply.

Developed by [Murabei Data Science](https://murabei.com). BSD-3-Clause.

<p align="center" width="60%">
  <img src="static_doc/sitelogo-horizontal.png" /> <br>

  <a href="https://en.wikipedia.org/wiki/Cecropia">
    Pumpwood is a native Brazilian tree
  </a> with a symbiotic relation to ants (Murabei)
</p>

---

## Objective and motivation

This package deploys the Crawler BigData app and its two RabbitMQ
workers onto a Pumpwood Kubernetes cluster.

### Why this exists
Queue CRUD lives in the API image; BigData fetch and ETL live in the
worker image. The cluster needs one app Service and two worker
Deployments that share the worker image and differ only by queue.

### How it is used
A cluster deploy script imports
`PumpWoodCrawlerBigdataMicroservice` and calls
`DeployPumpWood.add_microservice`. PgBouncer is added in that same
script via `PGBouncerDatabase` from `pumpwood-deploy`.

### Scope
**In scope:** Secret, app Deployment + Service, person worker,
company worker.

**Out of scope:** Postgres, PgBouncer, Kong, Auth, Datalake, Graph
Datalake, and the database itself.

---

## What it deploys

| Manifest | Kubernetes resources |
|----------|----------------------|
| `pumpwood_crawler_bigdata__secrets` | Secret `pumpwood-crawler-bigdata` |
| `pumpwood_crawler_bigdata__deploy` | Deployment + Service `pumpwood-crawler-bigdata-app` |
| `pumpwood_crawler_bigdata_person__worker` | Deployment `pumpwood-crawler-bigdata-worker-person` |
| `pumpwood_crawler_bigdata_company__worker` | Deployment `pumpwood-crawler-bigdata-worker-company` |

```mermaid
flowchart LR
    subgraph pkg [pumpwood-deploy-crawler-bigdata]
        A[PumpWoodCrawlerBigdataMicroservice]
    end
    subgraph core [pumpwood-deploy]
        B[DeployPumpWood]
    end
    subgraph cluster [Cluster]
        S[crawler Secret]
        APP[crawler-bigdata-app]
        WP[person worker]
        WC[company worker]
        RMQ[rabbitmq-main]
    end
    A --> B
    B --> S
    B --> APP
    B --> WP
    B --> WC
    APP --> RMQ
    RMQ --> WP
    RMQ --> WC
```

---

## Prerequisites

This package does **not** stand alone. Before crawler pods can start,
the cluster must already provide:

| Resource | Provided by |
|----------|-------------|
| `storage` ConfigMap | `StandardMicroservices` in `pumpwood-deploy` |
| `general-secrets` | `StandardMicroservices` |
| `rabbitmq-main-secrets` | `StandardMicroservices` |
| Storage keys (GCP / Azure / AWS) | `DeployPumpWood` storage config |
| Postgres for crawler | `PGBouncerDatabase` in the cluster script |
| Auth | [`pumpwood-deploy-auth`](https://github.com/Murabei-OpenSource-Codes/pumpwood-deploy-auth) |
| Datalake | [`pumpwood-deploy-datalake`](https://github.com/Murabei-OpenSource-Codes/pumpwood-deploy-datalake) |
| Graph Datalake | graph-datalake satellite |

Storage bucket name and type are read from the cluster `storage`
ConfigMap — they are **not** passed to
`PumpWoodCrawlerBigdataMicroservice`.

The database named in `db_database` must already exist on the
external Postgres. `PGBouncerDatabase` only pools connections.

---

## Installation

```bash
pip install pumpwood-deploy-crawler-bigdata
```

Requires `pumpwood-deploy`.

---

## Quick start

```python
import os
import simplejson as json
from dotenv import load_dotenv
from pumpwood_deploy.deploy import DeployPumpWood
from pumpwood_deploy.microservices.postgres.deploy import (
    PGBouncerDatabase)
from pumpwood_deploy_crawler_bigdata import (
    PumpWoodCrawlerBigdataMicroservice)

with open("secrets/production.json", "r") as file:
    secrets = json.loads(file.read())
load_dotenv()

deploy.add_microservice(
    PGBouncerDatabase(
        name="pgbouncer-pumpwood-crawler-bigdata",
        postgres_database="pumpwood_crawler_bigdata",
        postgres_secret="postgres-general",
        postgres_host=POSTGRES_HOST,
    ))

deploy.add_microservice(
    PumpWoodCrawlerBigdataMicroservice(
        app_version=os.getenv("PUMPWOOD_CRAWLER_BIGDATA_APP"),
        worker_version=os.getenv(
            "PUMPWOOD_CRAWLER_BIGDATA_WORKER"),
        repository="my-registry.example.com",
        db_host="pgbouncer-pumpwood-crawler-bigdata",
        db_database="pumpwood_crawler_bigdata",
        db_password=secrets["postgres_password"],
        microservice_password=secrets[
            "microservice--crawler-bigdata"],
        bigdata_access_token=secrets["bigdata_access_token"],
        app_replicas=1,
        app_debug="FALSE",
        worker_person_replicas=1,
        worker_company_replicas=1,
    ))

deploy.create_deploy_files()
deploy.deploy_microservices()
```

### Environment variables

```bash
PUMPWOOD_CRAWLER_BIGDATA_APP=0.0.5
PUMPWOOD_CRAWLER_BIGDATA_WORKER=0.0.17
```

If the rendered manifest matches what is already on the cluster,
`kubectl apply` produces no changes — safe for rolling image updates.

---

## Configuration reference

### Required

| Parameter | Description |
|-----------|-------------|
| `app_version` | Image tag for `pumpwood-crawler-bigdata-app` |
| `worker_version` | Image tag for `pumpwood-crawler-bigdata-worker` |

Workers need a non-empty `bigdata_access_token` to call BigData Corp.

### Database

| Parameter | Default | Description |
|-----------|---------|-------------|
| `db_host` | `postgres-pumpwood-crawler-bigdata` | Postgres host (PgBouncer in prod) |
| `db_port` | `5432` | Postgres port |
| `db_database` | `pumpwood` | Database name |
| `db_username` | `pumpwood` | Database user |
| `db_password` | `pumpwood` | Database password |
| `microservice_password` | `microservice--crawler-bigdata` | Service user password |
| `repository` | GCR default | Docker registry for app and worker |

### Application

| Parameter | Default | Description |
|-----------|---------|-------------|
| `app_replicas` | `1` | Number of app pods |
| `app_debug` | `FALSE` | Debug flag |
| `app_workers` | `10` | Granian workers (`GRANIAN_WORKERS`) |
| `app_timeout` | `300` | Request timeout (seconds) |
| `app_limits_memory` | `60Gi` | Memory limit |
| `app_limits_cpu` | `12000m` | CPU limit |
| `app_requests_memory` | `20Mi` | Memory request |
| `app_requests_cpu` | `1m` | CPU request |

### Workers

Person queue: `CrawlerBigDataPersonQueue__on_queue`.  
Company queue: `CrawlerBigDataCompanyQueue__on_queue`.

| Parameter | Default | Description |
|-----------|---------|-------------|
| `worker_person_replicas` | `1` | Person worker pod count |
| `worker_company_replicas` | `1` | Company worker pod count |
| `worker_debug` | `FALSE` | Worker debug flag |
| `worker_person_datasets` | image default | JSON list of person datasets |
| `worker_company_datasets` | image default | JSON list of company datasets |
| `worker_limits_memory` | `60Gi` | Worker memory limit |
| `worker_limits_cpu` | `12000m` | Worker CPU limit |
| `worker_requests_memory` | `20Mi` | Worker memory request |
| `worker_requests_cpu` | `1m` | Worker CPU request |

---

## Health check

The app Deployment exposes a readiness probe at:

```
GET /health-check/pumpwood-crawler-bigdata-app/  (port 5000)
```

Use this path for ingress and load balancer health checks.

---

## Related packages

| Package | Role |
|---------|------|
| [`pumpwood-deploy`](https://github.com/Murabei-OpenSource-Codes/pumpwood-deploy) | Orchestrator, Kong, RabbitMQ, Postgres |
| [`pumpwood-deploy-auth`](https://github.com/Murabei-OpenSource-Codes/pumpwood-deploy-auth) | Authorization microservice |
| [`pumpwood-deploy-datalake`](https://github.com/Murabei-OpenSource-Codes/pumpwood-deploy-datalake) | Datalake app and dataloader |

---

## Development

```bash
pip install -e ../pumpwood-deploy
pip install -e .

ruff check src/
```

---

## License

BSD-3-Clause — see [LICENSE](LICENSE).

