Metadata-Version: 2.4
Name: iris-vision-training
Version: 1.0.0
Summary: A comprehensive training service library for AI models in the Nedo Vision platform
Author-email: Willy Achmat Fauzi <willy.achmat@gmail.com>
Maintainer-email: Willy Achmat Fauzi <willy.achmat@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://gitlab.com/sindika/research/nedo-vision/nedo-vision-training-service
Project-URL: Documentation, https://gitlab.com/sindika/research/nedo-vision/nedo-vision-training-service/-/blob/main/README.md
Project-URL: Repository, https://gitlab.com/sindika/research/nedo-vision/nedo-vision-training-service
Project-URL: Bug Reports, https://gitlab.com/sindika/research/nedo-vision/nedo-vision-training-service/-/issues
Keywords: computer-vision,machine-learning,ai,training,deep-learning,object-detection,neural-networks,pytorch
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: grpcio<2.0.0,>=1.59.0
Requires-Dist: grpcio-tools<2.0.0,>=1.59.0
Requires-Dist: pika<2.0.0,>=1.3.0
Requires-Dist: rfdetr<1.8.0,>=1.2.0
Requires-Dist: boto3<2.0.0,>=1.28.0
Requires-Dist: pynvml<12.0.0,>=11.4.0
Requires-Dist: psutil<6.0.0,>=5.8.0
Requires-Dist: torch<3.0.0,>=2.0.0
Requires-Dist: torchvision<1.0.0,>=0.15.0
Requires-Dist: numpy<2.0.0,>=1.21.0
Requires-Dist: pillow<11.0.0,>=9.0.0
Requires-Dist: opencv-python<5.0.0,>=4.8.0
Requires-Dist: requests<3.0.0,>=2.31.0
Requires-Dist: tqdm<5.0.0,>=4.65.0
Provides-Extra: gpu
Requires-Dist: torch==2.3.1; extra == "gpu"
Requires-Dist: torchvision==0.18.1; extra == "gpu"
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: black>=22.0.0; extra == "dev"
Requires-Dist: isort>=5.10.0; extra == "dev"
Requires-Dist: mypy>=0.950; extra == "dev"
Requires-Dist: flake8>=4.0.0; extra == "dev"
Requires-Dist: pre-commit>=2.17.0; extra == "dev"

<div align="center">

# ai-vision-training-service

**Agen training model AI Vision yang berjalan di mesin GPU — mengubah dataset berlabel menjadi model deteksi objek (RF-DETR) siap pakai.**

Bagian dari ekosistem **IRIS — AI Vision Platform** PT Petrokimia Gresik
<br/>*(codename internal / paket: `nedo-vision-training`)*

![Stack](https://img.shields.io/badge/AI-RF--DETR%20(Roboflow)-6E44FF)
![Runtime](https://img.shields.io/badge/Python-%E2%89%A5%203.10-3776AB)
![DL](https://img.shields.io/badge/PyTorch-2.5.1%2Bcu121-EE4C2C)
![gRPC](https://img.shields.io/badge/gRPC-:50051-244C5A)
![Queue](https://img.shields.io/badge/RabbitMQ-exchange%20nedo.train-FF6600)
![License](https://img.shields.io/badge/library-MIT-green)

</div>

---

## 1. Executive Summary

> **Untuk pembaca awam:** Ini adalah "pabrik model AI". IRIS memakai kamera CCTV untuk mendeteksi pelanggaran keselamatan kerja (tidak pakai helm, masuk area terlarang). Supaya kamera bisa "mengenali" objek baru, IRIS butuh model AI yang dilatih. Repo ini adalah program yang duduk di komputer ber-GPU (kartu grafis kuat), menerima perintah "latih model dari kumpulan foto ini", lalu bekerja berjam-jam melatihnya dan mengirim hasilnya kembali.

**Apa ini.** `ai-vision-training-service` adalah *training agent* — sebuah proses Python yang dijalankan di mesin ber-GPU. Ia mendaftar ke **manager** IRIS, menunggu perintah training (*TrainingJob*), lalu melatih model deteksi objek berbasis **RF-DETR** (Real-time Detection Transformer dari Roboflow) di atas **PyTorch + CUDA**.

**Masalah yang dipecahkan.** GPU itu mahal dan langka. Alih-alih setiap developer melatih model manual di laptop, IRIS memusatkan pekerjaan berat ini ke satu/beberapa mesin GPU. Pengguna cukup menekan tombol di web IRIS; job otomatis dirutekan ke agen yang tepat, berjalan sendiri, dan progres-nya (grafik akurasi per-epoch) tampil live di dashboard.

**Peran dalam ekosistem IRIS.**

| Arah | Service | Hubungan |
|---|---|---|
| Upstream (memerintah) | **ai-vision-manager** (.NET 9) | Mengirim TrainingJob via RabbitMQ, menerima status/metrik/model via gRPC :50051 |
| Upstream (data) | **ai-vision-autodistill** | Menghasilkan dataset berlabel (auto-labeling) yang dilatih di sini |
| Sideways (storage) | **S3 / MinIO / GCS** | Sumber gambar dataset + tujuan upload `model.zip` |
| Downstream (konsumen model) | **ai-vision-worker-core** | Memakai model `.pth` hasil training untuk inference RTSP di edge |
| Downstream (visual) | **ai-vision-frontend** (React) | Menampilkan grafik metrik live (mAP50, precision, recall, F1) |

### Status saat ini

| Aspek | Nilai |
|---|---|
| Versi paket | `1.0.0` (`nedo_vision_training.__version__`) |
| Bahasa / runtime | Python ≥ 3.10 |
| LOC (perkiraan) | ~3.900 baris Python (di luar stub proto), ~4.500 termasuk proto |
| Arah komunikasi | **Outbound-only** — agen adalah *client* murni; tidak membuka port inbound |
| Port dipakai | gRPC `:50051` (ke manager), AMQP `:5672` (RabbitMQ), HTTPS/S3 endpoint |
| Dependensi kunci | RF-DETR, PyTorch 2.5.1+cu121, grpcio, pika, boto3, pynvml |
| Distribusi | PyPI (`pip install nedo-vision-training`) + git (upstream GitLab Sindika) |
| Maturity (CMMI) | **Level 2 — Managed** (lihat §5) |

### Stack ringkas

```
AI / Model     │ RF-DETR (RFDETRBase, Roboflow) · pycocotools · COCO format
Deep Learning  │ PyTorch 2.5.1+cu121 · torchvision 0.20.1+cu121 · CUDA 12.1
Runtime        │ Python ≥3.10 · multiprocessing (spawn) · threading
IPC ke manager │ gRPC (grpcio ≥1.59) — 3 service, port :50051, insecure channel
Job intake     │ RabbitMQ (pika ≥1.3) — exchange `nedo.train`, direct, key=agent_id
Storage        │ boto3 (S3 / MinIO / GCS) — download gambar, upload model.zip
Monitoring     │ pynvml (NVML) · psutil — GPU/CPU/RAM/latency, Jetson-aware
Packaging      │ setuptools / pyproject.toml · CLI entrypoint `nedo-trainer`
CI             │ GitHub Actions — ruff (non-blocking) + compileall
```

---

## 2. Proses Bisnis (BPMN)

> **Untuk awam & analis:** Diagram di bawah membaca dari kiri ke kanan seperti alur kerja. Seorang engineer menekan "latih" di web IRIS, sistem memilih mesin GPU yang tepat, mesin itu mengunduh foto + label, mengubahnya ke format standar, melatih model sambil melaporkan nilai akurasi tiap putaran, lalu menyimpan model jadi dan menandai pekerjaan **SELESAI**.

```mermaid
flowchart LR
    subgraph U["👤 User / ML Engineer"]
        A1([Mulai]) --> A2[Pilih dataset + parameter<br/>epoch, batch, split ratio]
        A2 --> A3[Klik 'Train' di web IRIS]
    end

    subgraph M["🧠 ai-vision-manager (.NET 9)"]
        B1[Buat TrainingJob<br/>status = queued] --> B2{Pilih agen GPU<br/>by agent_id}
        B2 --> B3[Publish job ke<br/>exchange nedo.train]
        B7[(Simpan metrik +<br/>publish ke frontend)]
        B9[Update model_file_path<br/>status = COMPLETED]
    end

    subgraph T["🖥️ Training Agent (repo ini, mesin GPU)"]
        C1[Terima job dari queue] --> C2[Download dataset via gRPC<br/>+ gambar dari S3]
        C2 --> C3[Konversi ke COCO]
        C3 --> C4[Train RF-DETR<br/>subprocess spawn]
        C4 --> C5{Tiap epoch:<br/>hitung mAP/P/R/F1}
        C5 -->|kirim gRPC| B7
        C5 --> C6[Simpan checkpoint terbaik]
        C6 --> C7[Zip + upload model.zip ke S3]
        C7 --> B9
    end

    subgraph F["📊 ai-vision-frontend (React)"]
        D1[Grafik metrik live<br/>per epoch]
    end

    A3 --> B1
    B3 -.AMQP.-> C1
    B7 -.realtime.-> D1
    B9 --> Z([Selesai])

    style C4 fill:#6E44FF,color:#fff
    style B9 fill:#2e7d32,color:#fff
    style Z fill:#2e7d32,color:#fff
```

**Tabel langkah proses**

| No | Aktivitas | Aktor | Sistem / Tool | Output |
|---|---|---|---|---|
| 1 | Pilih dataset & parameter, klik Train | User | Frontend React | Request buat TrainingJob |
| 2 | Buat job & pilih agen | Manager | .NET 9 / PostgreSQL | TrainingJob (queued) |
| 3 | Route job ke agen | Manager | RabbitMQ `nedo.train` (routing key = `agent_id`) | Pesan JSON di queue agen |
| 4 | Terima & parse job | Agent | pika consumer | `TrainParams` (job_id, algorithm, dataset_id, epoch, batch, split) |
| 5 | Ambil daftar item dataset | Agent | gRPC `DatasetService.GetDataset` | List `DatasetItem` (file_path + anotasi) |
| 6 | Unduh gambar | Agent | boto3 (S3/MinIO/GCS) | File gambar lokal |
| 7 | Konversi ke COCO | Agent | `COCODatasetHandler` | `_annotations.coco.json` per split |
| 8 | Latih model per-epoch | Agent | RF-DETR + PyTorch/CUDA (subprocess) | Checkpoint + metrik per epoch |
| 9 | Lapor metrik | Agent | gRPC `CreateMetricsLog` | Manager publish `nedo.training_job.metrics_log.update` → grafik live |
| 10 | Simpan & upload model | Agent | `shutil` zip + boto3 upload | `model/{job_id}.zip` di storage |
| 11 | Update path & status | Agent | gRPC `UpdateModelFilePath` + `CreateStatusLog(completed)` | Job **COMPLETED** |

---

## 3. Arsitektur

> **Untuk awam:** Bayangkan agen ini seperti karyawan pabrik yang menerima surat perintah (RabbitMQ), meminta bahan baku (dataset via gRPC + S3), mengerjakan produksi (training di GPU), lalu melapor ke atasan dan menyetor hasil ke gudang. Semua komunikasi keluar dari agen — tidak ada orang luar yang bisa "menelepon masuk" ke mesin ini.

### 3.1 High-level component

```mermaid
flowchart TB
    subgraph GPU["🖥️ Mesin GPU (host training agent)"]
        TS["TrainingService<br/>(orchestrator)"]
        WM["WorkerManager"]
        TR["Trainer<br/>(listener nedo.train)"]
        DS["DataSenderWorker<br/>(usage + latency)"]
        subgraph PROC["🔬 Subprocess per job (spawn)"]
            RF["RFDETRTrainer<br/>RF-DETR + PyTorch/CUDA"]
            CO["COCODatasetHandler"]
        end
    end

    MGR["🧠 ai-vision-manager<br/>.NET 9"]
    MQ["🐇 RabbitMQ<br/>exchange nedo.train"]
    S3["🗄️ S3 / MinIO / GCS"]
    FE["📊 Frontend React"]

    TS -->|"gRPC :50051<br/>GetConnectionInfo"| MGR
    TS --> WM --> TR & DS
    MGR -->|publish TrainingJob| MQ
    MQ -->|"AMQP :5672<br/>key=agent_id"| TR
    TR -->|multiprocessing.Process| PROC
    CO -->|"gRPC :50051<br/>GetDataset"| MGR
    CO -->|"download gambar"| S3
    RF -->|"gRPC :50051<br/>CreateMetricsLog / StatusLog"| MGR
    RF -->|"upload model.zip"| S3
    DS -->|"gRPC :50051<br/>SendSystemUsage"| MGR
    MGR -->|"metrics_log.update"| FE

    style RF fill:#6E44FF,color:#fff
    style MGR fill:#244C5A,color:#fff
```

### 3.2 Sequence — dari job masuk sampai COMPLETED

```mermaid
sequenceDiagram
    autonumber
    participant MGR as 🧠 Manager
    participant MQ as 🐇 RabbitMQ<br/>nedo.train
    participant TR as Trainer<br/>(consumer)
    participant PR as 🔬 Subprocess<br/>RFDETRTrainer
    participant DSVC as DatasetService<br/>(gRPC)
    participant S3 as 🗄️ Storage
    participant TJ as TrainingJobService<br/>(gRPC)

    Note over TR,MQ: Queue: nedo.train.queue.{agent_id}, direct, key=agent_id
    MGR->>MQ: publish TrainingJob (JSON)
    MQ-->>TR: deliver {id, algorithm, dataset_id, epoch, batch_size, split_ratio}
    TR->>PR: spawn multiprocessing.Process(job)
    PR->>TJ: CreateStatusLog(job_id, "initializing")
    PR->>DSVC: GetDataset(dataset_id)
    DSVC-->>PR: DatasetItem[] (file_path + bbox anotasi %)
    loop tiap item
        PR->>S3: download_file(file_path)
    end
    PR->>PR: konversi → COCO (_annotations.coco.json)
    PR->>TJ: CreateStatusLog(job_id, "running")
    loop tiap epoch
        PR->>PR: RF-DETR train() 1 epoch
        PR->>TJ: CreateMetricsLog(epoch, map_50, map_50_95, precision, recall, f1)
        TJ-->>MGR: simpan + publish nedo.training_job.metrics_log.update
    end
    PR->>PR: simpan checkpoint_best_total.pth → zip
    PR->>S3: upload model/{job_id}.zip
    PR->>TJ: UpdateModelFilePath(job_id, s3_path)
    PR->>TJ: CreateStatusLog(job_id, "completed")
    TR->>MQ: basic_ack(delivery_tag)
```

### 3.3 Struktur data — DatasetItem → COCO

> **Untuk spesialis ML:** Anotasi dari manager memakai bbox ternormalisasi `(x1,y1,x2,y2)` dalam skala 0–1. `COCODatasetHandler._convert_annotations` mengalikannya dengan dimensi gambar (dari `PIL.Image.size`) dan mengubahnya ke format COCO `[x, y, w, h]` piksel. Split train/val/test memakai `random.shuffle` dengan `seed=42` (reproducible). Kategori index 0 di-reserve `"objects"` (konvensi RF-DETR/Roboflow), kelas nyata mulai dari 1.

```mermaid
erDiagram
    DATASET ||--o{ DATASET_ITEM : berisi
    DATASET_ITEM ||--o{ ANNOTATION : punya
    DATASET_ITEM {
        string file_path "path gambar di S3"
    }
    ANNOTATION {
        string label "nama kelas"
        double b_box_x1 "0..1 (ternormalisasi)"
        double b_box_y1 "0..1"
        double b_box_x2 "0..1"
        double b_box_y2 "0..1"
    }
    COCO_JSON {
        array images "id, file_name, width, height"
        array annotations "bbox[x,y,w,h] px, category_id, area, iscrowd"
        array categories "id, name, supercategory"
    }
    ANNOTATION ||--|| COCO_JSON : "dikonversi ke"
```

### 3.4 Security & network boundary

> **Untuk awam:** Mesin GPU ini hanya "berbicara keluar" — ia menelepon manager, RabbitMQ, dan storage, tapi tidak menerima panggilan masuk. Ini mengurangi permukaan serangan.

```mermaid
flowchart LR
    subgraph HOST["🖥️ Host GPU (tanpa port inbound)"]
        AG["Training Agent"]
    end
    AG -->|"gRPC :50051 (insecure_channel)"| MGR["Manager"]
    AG -->|"AMQP :5672 (PlainCredentials)"| MQ["RabbitMQ"]
    AG -->|"HTTPS / S3 API (aws sig v4)"| ST["Storage"]

    NOTE["🔑 Auth: token agen (CLI --token)<br/>→ GetConnectionInfo → kredensial RabbitMQ + S3<br/>diterima runtime, tidak disimpan ke disk"]
    AG -.-> NOTE

    style NOTE fill:#fff3cd,color:#000
```

**Catatan keamanan (spesialis):**
- Channel gRPC memakai `grpc.insecure_channel` (tanpa TLS) — aman hanya bila jalur ke manager berada di jaringan tepercaya / VPN / host-lokal. *Upgrade path:* `grpc.secure_channel` + credential bila melewati jaringan publik.
- Kredensial RabbitMQ & S3 **tidak** disimpan di repo/`.env`; agen mengambilnya saat runtime lewat `GetConnectionInfo(token)`. Token adalah satu-satunya rahasia yang di-supply operator.
- `test.py` di root berisi token & host contoh — hanya untuk uji lokal; jangan commit token nyata.

### 3.5 Ports & Protokol

| Port | Protokol | Arah | Tujuan | Tool / Library |
|---|---|---|---|---|
| `50051` | gRPC (HTTP/2) | Outbound | `TrainingAgentService`, `DatasetService`, `TrainingJobService` ke manager | `grpcio`, `grpc.insecure_channel` |
| `5672` | AMQP 0-9-1 | Outbound | Konsumsi TrainingJob dari exchange `nedo.train` | `pika.BlockingConnection` |
| endpoint S3 | HTTPS / HTTP | Outbound | Download gambar dataset, upload `model.zip` | `boto3` (endpoint & region dari connection info) |
| — | — | Inbound | **Tidak ada** — agen tidak melayani port | — |

### 3.6 Tech Stack & Rationale

| Pilihan | Versi (sumber) | Alasan |
|---|---|---|
| **RF-DETR** (`RFDETRBase`) | git `1e63dbad…` / `>=1.2.0` | Detector transformer real-time Roboflow; akurasi tinggi tanpa anchor tuning; API `train()` dengan callback per-epoch |
| **PyTorch** | `2.5.1+cu121` (requirements.txt) | Backbone DL; build `+cu121` dipin agar tidak fallback ke CPU (lihat komentar requirements) |
| **torchvision** | `0.20.1+cu121` | Transform & util vision, kompatibel torch 2.5.1 |
| **CUDA** | 12.1 (`--extra-index-url .../cu121`) | Target host Linux + GPU NVIDIA |
| **grpcio** | `>=1.59,<2.0` | RPC biner cepat ke manager; kontrak proto `Sindika.AspNet.App005.Services` |
| **pika** | `>=1.3,<2.0` | Client RabbitMQ (AMQP) untuk intake job |
| **boto3** | `>=1.28,<2.0` | Klien S3 generik — bekerja untuk AWS S3, MinIO, GCS (S3-compat) via `endpoint_url` |
| **pynvml** | `>=11.4,<12.0` | Baca utilisasi/memori/suhu GPU via NVML; ada jalur khusus Jetson (`tegrastats`) |
| **psutil** | `>=5.8` | Monitoring CPU/RAM lintas OS |
| **numpy / pillow / opencv** | `<2.0` / `<11` / `>=4.8` | Prapemrosesan gambar & dimensi |

> **Catatan versi:** `requirements.txt` memakai pin ketat produksi (`torch==2.5.1+cu121`). `pyproject.toml` memakai rentang longgar (`torch>=2.0`) + extra `[gpu]` lawas (`torch==2.3.1`) untuk instalasi pustaka lewat PyPI. Untuk deploy di mesin GPU, **gunakan `requirements.txt`**.

---

## 4. Pola Algorithm Extensible (untuk spesialis ML)

> **Untuk awam:** Sistem ini dirancang agar mudah menambah "jenis otak AI" baru. Sekarang ada satu (RF-DETR); menambah yang lain (mis. YOLO) cukup menaruh satu file baru dengan pola nama tertentu — sistem otomatis menemukannya, tanpa mengubah kode inti.

Alur pemilihan trainer bersifat **plugin auto-discovery**:

```mermaid
flowchart LR
    JOB["TrainingJob.algorithm<br/>(mis. 'RFDETR')"] --> TF["TrainerFactory.get_trainer()"]
    TF -->|"1x load"| WALK["os.walk modules/algorithm/**<br/>cari file *Trainer.py"]
    WALK -->|import + register| REG["TrainerRegistry<br/>{key.lower(): class}"]
    TF -->|lookup by key| CLS["trainer_class"]
    CLS -->|instansiasi di subprocess| RUN["init → train → evaluate<br/>→ save_model → upload_model"]

    style RUN fill:#6E44FF,color:#fff
```

**Kontrak menambah algoritma baru:**
1. Buat folder `modules/algorithm/<Nama>/` dengan file `<Nama>Trainer.py`.
2. Kelas mewarisi `BaseTrainer` (`train`, `evaluate`, `save_model`, `load_model`).
3. Di akhir file: `TrainerRegistry.register_trainer('<Nama>', <Nama>Trainer)`.
4. `TrainerFactory._load_trainers()` men-scan `modules/algorithm/**` untuk file berakhiran `Trainer.py` dan mendaftarkannya otomatis (key = nama file, di-*lowercase*). Manager cukup mengirim `algorithm = "<Nama>"`.

**Detail RF-DETR (`RFDETRTrainer`):**
- **Effective batch** tetap `16`; *micro-batch* yang benar-benar masuk VRAM di-*auto-size* (`_auto_micro_batch`, heuristik ~4 GB/gambar) dan selisihnya ditutup dengan `grad_accum_steps`.
- **OOM guard:** bila `torch.cuda.OutOfMemoryError`, `empty_cache()` lalu retry di micro-batch 1 dengan grad-accum penuh. Error lain langsung dipropagasi (tanpa retry membabi buta).
- **lr** = `1e-4`. Checkpoint terbaik = `checkpoint_best_total.pth` (di-copy ke `artifacts/models/{job_id}/`, di-zip, di-upload ke `model/{job_id}.zip`).
- **Metrik per-epoch** (`RFDETRMetricsCallback.on_fit_epoch_end`): membaca `test_coco_eval_bbox` (12 nilai pycocotools) → `map_50_95` (idx 0), `map_50` (idx 1), `precision`, `recall`, hitung `f1 = 2PR/(P+R)`, dibulatkan 4 desimal, dikirim via `TrainerLogger.log_metric`.

**Isolasi proses:** setiap job berjalan di `multiprocessing.Process` terpisah dengan start-method `spawn` (wajib untuk CUDA). Ini mengisolasi crash/OOM per job dan memungkinkan terminate/kill bersih saat shutdown (`shutdown_handler`).

---

## 5. Tata Kelola & Kematangan

> **Untuk manajemen & auditor:** Bagian ini memetakan apa yang benar-benar ada di repo ke kerangka tata kelola TI standar. Pemetaan bersifat wajar berdasarkan fungsi, bukan klaim sertifikasi.

### COBIT 2019

| Objective | Bagaimana repo ini memenuhinya |
|---|---|
| **APO03** Managed Enterprise Architecture | Kontrak proto (`*.proto`) sebagai arsitektur antarmuka; pola trainer plugin terdokumentasi (§4) |
| **BAI03** Managed Solutions Build | Pipeline training terstruktur (init→train→evaluate→save→upload); artefak checkpoint & model.zip |
| **BAI06** Managed IT Changes | CI GitHub Actions pada `push`/`PR` ke `develop`/`main` (ruff + compileall); versi paket di `__init__.py` |
| **BAI07** Acceptance & Transitioning | `nedo-trainer doctor` — pemeriksaan pra-jalan (CUDA, GPU, disk, paket) sebelum menerima job |
| **DSS01** Managed Operations | `DataSenderWorker` + `SystemUsageManager` melaporkan CPU/RAM/GPU/latency ke manager secara berkala |
| **DSS05** Managed Security Services | Auth berbasis token; kredensial runtime via `GetConnectionInfo` (tidak persist); agen outbound-only |
| **MEA01** Performance Monitoring | Metrik per-epoch (mAP/P/R/F1) + status log + heartbeat sistem → dashboard IRIS |

### PMBOK — knowledge area

| Area | Deliverable konkret di repo |
|---|---|
| **Scope** | `TrainParams` mendefinisikan batas job (dataset, epoch, batch, split) |
| **Quality** | `mypy`/`black`/`isort`/`flake8` (config di `pyproject.toml`), CI ruff, metrik evaluasi objektif |
| **Risk** | OOM retry, reconnect gRPC (backoff eksponensial), reconnect RabbitMQ, cleanup artefak saat gagal |
| **Integration** | Kontrak gRPC + AMQP dengan manager; storage S3-compat |
| **Communications** | Log status/command real-time (`TrainerLogger`) ke manager → frontend |

### IT Maturity (CMMI-style)

**Level saat ini: 2 — Managed.**

Proses training dijalankan secara berulang dan terkendali: intake job terstruktur, penanganan error/OOM/reconnect eksplisit, monitoring resource, dan pelaporan metrik terstandar. Kematangan tertahan di Level 2 karena: **belum ada** test otomatis fungsional (CI hanya `compileall` + `ruff` non-blocking; `test.py` adalah skrip manual), **belum ada** Dockerfile/manifest deploy tereproduksi di repo (setup GPU manual), dan **tidak ada** versioning model/dataset formal (MLOps registry).

**Untuk naik ke Level 3 (Defined):** tambahkan (1) suite `pytest` untuk `COCODatasetHandler` & metrics callback, (2) Dockerfile CUDA + runbook deploy, (3) gate CI blocking, (4) pelacakan lineage dataset↔model.

---

## 6. Repository Structure

```
ai-vision-training-service/
├── nedo_vision_training/
│   ├── training_service.py          # 🚀 Orchestrator: init clients → start workers → loop
│   ├── cli.py                       #    Entrypoint `nedo-trainer` (subcommand run / doctor)
│   ├── doctor.py                    #    Pre-flight check: Python, CUDA, GPU, disk, paket
│   ├── exceptions.py                #    GrpcClientError, ConfigurationError, dll.
│   ├── initializer/AppInitializer.py#    Validasi UUID / host (no-op registration)
│   │
│   ├── client/                      # 🔌 gRPC & infra clients
│   │   ├── GrpcClientBase.py        #    Channel + retry + error-mapping gRPC
│   │   ├── ConnectionInfoClient.py  #    GetConnectionInfo → kredensial RabbitMQ+S3
│   │   ├── DatasetServiceClient.py  #    GetDataset(dataset_id)
│   │   ├── TrainingLoggerClient.py  #    Status/Command/Metrics/ModelPath logs
│   │   ├── TrainingAgentStatusClient.py # UpdateStatus(connected/disconnected)
│   │   ├── SystemUsageClient.py     #    SendSystemUsage (CPU/RAM/GPU/latency)
│   │   ├── RabbitMQClient.py        #    Wrapper pika: exchange/queue/consume
│   │   └── S3Client.py              #    boto3 download/upload/list/delete
│   │
│   ├── services/
│   │   ├── WorkerManager.py         #    Start/stop Trainer + DataSenderWorker
│   │   └── DataSenderWorker.py      #    Loop kirim system usage berkala
│   │
│   ├── modules/
│   │   ├── trainer/
│   │   │   ├── Trainer.py           # 🐇 Consumer nedo.train → spawn subprocess training
│   │   │   ├── TrainerFactory.py    #    Auto-discovery *Trainer.py
│   │   │   ├── TrainerRegistry.py   #    Registry {algorithm: class}
│   │   │   ├── BaseTrainer.py       #    ABC: train/evaluate/save_model/load_model
│   │   │   └── TrainerParams.py     #    TrainParams DTO
│   │   ├── algorithm/RFDETR/
│   │   │   ├── RFDETRTrainer.py     # 🧠 Pipeline RF-DETR (train/eval/save/upload)
│   │   │   └── RFDETRMetricsCallback.py # Ekstraksi mAP/P/R/F1 per epoch
│   │   ├── dataset/
│   │   │   ├── DatasetHandler.py    #    ABC: split, ensure dir, S3+gRPC clients
│   │   │   └── COCODatasetHandler.py#    Konversi anotasi → COCO json + download gambar
│   │   └── data_sync/SystemUsageManager.py # Monitor + latency thread
│   │
│   ├── logger/                      #    Logger + TrainerLogger (singleton gRPC)
│   ├── utils/                       #    system_monitor (NVML/Jetson), networking, hardware_id
│   └── protos/                      # 📐 3 .proto + stub _pb2 / _pb2_grpc generated
│       ├── TrainingAgentService.proto  # SendSystemUsage, UpdateStatus, GetConnectionInfo
│       ├── DatasetService.proto        # GetDataset
│       └── TrainingJobService.proto    # CreateStatusLog/CommandLog/MetricsLog, UpdateModelFilePath
│
├── .github/workflows/ci.yml         # 🔁 ruff (non-blocking) + compileall
├── requirements.txt                 #    Pin produksi GPU (torch 2.5.1+cu121)
├── pyproject.toml                   #    Metadata paket + tool config (mypy/black/isort)
├── setup.py                         #    Shim backward-compat
├── test.py                          #    Skrip uji manual (token contoh)
└── README_DEV.md                    #    One-liner regen stub proto
```

---

## 7. Konfigurasi & Environment

> **Untuk teknis:** Repo ini **tidak** memakai `.env` maupun Dockerfile. Konfigurasi runtime datang dari dua sumber: (a) argumen CLI yang di-supply operator, dan (b) `GetConnectionInfo` gRPC yang mengirim kredensial RabbitMQ + S3 dari manager. N/A untuk env-vars file.

### 7.1 Argumen CLI (`nedo-trainer run` / `nedo-training run`)

| Argumen | Wajib? | Default | Fungsi |
|---|---|---|---|
| `--token` | ✅ | — | Token autentikasi agen (identitas + kunci `GetConnectionInfo`) |
| `--server-host` | ⚠️ | `localhost` | Host gRPC manager |
| `--server-port` | ⚠️ | `50051` | Port gRPC manager |
| `--system-usage-interval` | ❌ | `30` (CLI) / `5` (lib) | Interval kirim usage (detik) |
| `--latency-check-interval` | ❌ | `10` | Interval ukur latency gRPC (detik) |

> Catatan: `training_service.py::main` (argparse langsung) menjadikan `--server-host`/`--server-port` **required**; CLI `cli.py` memberi default `localhost:50051`. Untuk produksi selalu set host/port eksplisit.

### 7.2 Field dari `GetConnectionInfo` (dikirim manager, tidak diset manual)

| Field | Fungsi |
|---|---|
| `rabbitmq_host` / `_port` / `_username` / `_password` | Koneksi AMQP untuk intake job |
| `s3_endpoint` / `s3_bucket` / `s3_region` / `s3_access_key` / `s3_secret_key` | Kredensial storage (opsional; S3/MinIO/GCS) |
| `id` | `agent_id` — routing key queue `nedo.train.queue.{agent_id}` |

### 7.3 Payload TrainingJob (JSON via RabbitMQ)

```json
{ "id": "<job_id>", "algorithm": "RFDETR", "dataset_id": "<uuid>",
  "epoch": 50, "batch_size": 0, "split_ratio": 0.8 }
```
`batch_size = 0` → auto-size micro-batch. `split_ratio` = porsi train (sisanya validasi; test = 0 pada RF-DETR path).

---

## 8. Local Development

### Prasyarat
```bash
python --version   # >= 3.10
nvidia-smi         # GPU NVIDIA + driver CUDA 12.1 (untuk training nyata)
# CPU-only tetap bisa jalan untuk uji alur (micro-batch = 1), tapi lambat
```

### Setup
```bash
git clone <repo-url> ai-vision-training-service
cd ai-vision-training-service

python -m venv .venv && source .venv/bin/activate

# Instal dependensi produksi (torch +cu121 dari index PyTorch)
pip install -r requirements.txt

# atau sebagai pustaka:
pip install -e .

# Cek kesiapan mesin (CUDA/GPU/disk/paket)
nedo-trainer doctor
```

### Regenerate stub proto (bila `.proto` berubah)
```bash
python -m grpc_tools.protoc --proto_path=. --python_out=. --grpc_python_out=. \
  nedo_vision_training/protos/*.proto
```

### Jalankan agen
```bash
nedo-trainer run \
  --token YOUR_AGENT_TOKEN \
  --server-host manager.internal \
  --server-port 50051 \
  --system-usage-interval 30
```
Agen akan: connect gRPC → `GetConnectionInfo` → set status `connected` → mulai listener `nedo.train` → menunggu job. Port terbuka: **tidak ada** (outbound-only).

---

## 9. Deployment & CI/CD

> **Untuk DevOps:** Tidak ada Dockerfile di repo (deploy GPU dilakukan manual di host / systemd). CI hanya melakukan lint & compile — bukan build image.

```mermaid
flowchart LR
    Push["git push / PR<br/>(develop, main)"] --> Setup["setup-python 3.10"]
    Setup --> Ruff["ruff check<br/>(continue-on-error)"]
    Setup --> Compile["python -m compileall<br/>(gate syntax)"]
    Ruff --> Done([selesai])
    Compile --> Done
    style Compile fill:#2e7d32,color:#fff
```

- **Workflow:** `.github/workflows/ci.yml`, trigger `push`/`pull_request` ke `develop` & `main`.
- **Langkah:** install `ruff` → `ruff check` (non-blocking) → `compileall` (menangkap syntax error tanpa memasang torch/CUDA).
- **Distribusi pustaka:** PyPI `nedo-vision-training` (`pip install`), upstream repo di GitLab Sindika.
- **Runtime deploy (aktual):** proses persisten di host GPU (mis. `systemd` / `screen` / `tmux`) menjalankan `nedo-trainer run --token …`. Skalakan horizontal dengan menjalankan beberapa agen, masing-masing `agent_id` unik.

---

## 10. Observability

| Sinyal | Mekanisme | Tujuan |
|---|---|---|
| **Status job** | gRPC `CreateStatusLog` (`initializing`→`running`→`evaluating`→`saving`→`completed`/`failed`) | Timeline job di manager/frontend |
| **Command log** | gRPC `CreateCommandLog` (pesan naratif tiap tahap) | Audit langkah |
| **Metrik training** | gRPC `CreateMetricsLog` per epoch (mAP50, mAP50-95, P, R, F1) | Grafik live (`nedo.training_job.metrics_log.update`) |
| **System usage** | gRPC `SendSystemUsage` tiap `system-usage-interval` (CPU/RAM/GPU util+mem+temp) | Heartbeat & dashboard resource |
| **Latency** | Thread `SystemUsageManager` ukur latency gRPC tiap `latency-check-interval` | Kesehatan koneksi |
| **Log lokal** | `Logger` / `TrainerLogger` (stdout, emoji-tagged) | Debug di host |

Tidak ada endpoint HTTP health/metrics (agen tidak melayani port); observability sepenuhnya *push* via gRPC ke manager.

---

## 11. Documentation Index

| Audiens | Dokumen | Lokasi |
|---|---|---|
| Awam / Manajemen | Executive summary, BPMN, tata kelola | README §1–2, §5 |
| Teknis (Engineer) | Arsitektur, config, dev setup | README §3, §7–8; `README_DEV.md` |
| Spesialis (ML) | Pola trainer plugin, RF-DETR, metrik COCO | README §4; `modules/algorithm/RFDETR/` |
| DevOps | CI, deploy, observability | README §9–10; `.github/workflows/ci.yml` |
| Kontrak antar-service | Definisi gRPC | `nedo_vision_training/protos/*.proto` |

---

## 12. Contact & License

- **Tech Lead (IRIS):** Yafi Anshori
- **Org GitHub:** [tekinfopg](https://github.com/tekinfopg)
- **Penulis pustaka (upstream):** Willy Achmat Fauzi — GitLab Sindika `research/nedo-vision`
- **Diagnostik cepat:** `nedo-trainer doctor`

**License.** Kode pustaka dirilis di bawah **MIT** (lihat `pyproject.toml`). Penggunaan operasional, model terlatih, dan dataset dalam ekosistem IRIS bersifat **internal & proprietary © PT Petrokimia Gresik** — tidak untuk distribusi publik.

---

<div align="center">

**Bagian dari IRIS — AI Vision Platform · PT Petrokimia Gresik**

*Melatih mata AI yang menjaga keselamatan kerja di pabrik.*

</div>
