Architecture · Diagram Design

Open data lake · End-to-end stack

Five functional zones from raw event ingestion to analytics consumption. MinIO is the focal centre — every other tool either lands data into it or reads out of it.

Open data lake · End-to-end stack Data lake architecture showing app servers and databases flowing through Apache NiFi into MinIO, then Trino, StarRocks, Superset, JupyterLab, and Python. SOURCES INGEST DATA LAKE QUERY CONSUME SQL WRITE EXT App servers Events · logs EXT Databases CDC · exports FLOW Apache NiFi Route · transform ORCH Airflow DAG scheduling STORE MinIO Object store · S3-API QUERY Trino SQL · any format OLAP StarRocks MPP · hot layer BI Superset Dashboards NB JupyterLab Exploration PROC Python Batch · ML LEGEND MinIO data lake (focal) Ingest · query tools BI · notebooks · pipelines Write-back path (batch) Primary query path

FOCAL · STORAGE

MinIO — data lake core

S3-compatible object storage. Every tool in the stack either lands data here or reads from it. All formats land as raw files; Iceberg table metadata sits alongside them. When in doubt, write to MinIO.

Ingest — NiFi + Airflow

  • NiFi: real-time routing, record-level transformation, CDC fan-out
  • Airflow: batch scheduling, cross-system DAGs, retry semantics
  • Two tools, two tempos — they complement rather than replace each other

Query — Trino + StarRocks

  • Trino: federated ad-hoc SQL across any file format, any catalog
  • StarRocks: sub-second MPP for hot partitions and dashboard queries
  • Different latency budgets, same underlying lake

Consume

Superset for dashboards, JupyterLab for exploration, Python for batch ML. All three can write processed results back to MinIO via the dashed write-back path.

open data lake · end-to-end stack example · diagram design