Architecture · Diagram Design

High-level architecture · End-to-end stack

Five functional phases from raw source ingestion to business visualization. Apache NiFi collects, MinIO stores, Trino virtualizes, notebooks model — all orchestrated by Airflow and secured by Active Directory, running on Kubernetes.

High-level architecture · End-to-end stack Architecture diagram showing data sources moving through Apache NiFi and MinIO into Trino, data modeling, and Superset dashboards, with Airflow orchestration. DATA SOURCES INGESTION STORAGE TRANSFORMATION & ANALYSIS VISUALIZATION SOURCES EXT WEB Census · Survey · Admin EXT FTP Census · Business EXT DB Connection CDC · SQL · API EXT Legacy System Trade data Orchestration / Workflow Automation (Apache Airflow) COLL Apache NiFi Collect · route · transform VIRT Trino SQL · data virtualization STORE MinIO / S3 Object store · S3-API ANLZ Data Modeling Notebook · Python · R DASH Dashboards Superset · reports Kubernetes Identity Manager (Active Directory) LEGEND External source · outside cluster Kubernetes boundary Focal component (MinIO) Primary data path Orchestration trigger

Ingestion · Apache NiFi

Data collection

  • Collects from WEB (HTTP/REST), FTP, database CDC, and legacy file exports
  • No-code dataflow routing with back-pressure and guaranteed delivery
  • Lands raw data into MinIO and exposes it to Trino simultaneously
  • Orchestrated by Airflow DAGs for scheduled batch and event-driven runs

Storage · MinIO / S3

Object store (focal)

  • S3-compatible API — single storage surface for all tools in the stack
  • Holds raw, cleansed, and curated layers (bronze / silver / gold paths)
  • Trino virtualizes directly over MinIO buckets — no ETL copy step needed
  • Python notebooks write model outputs back as Parquet for further querying

Query & Serve · Trino + Superset

Analysis & visualization

  • Trino federates SQL queries across MinIO, databases, and APIs in one query
  • JupyterLab notebooks run Python and R for statistical modeling and ML
  • Superset dashboards connect to Trino for live, governed report delivery
  • Active Directory enforces row- and column-level access across all surfaces
High-level architecture · Kubernetes-deployed open-source stack Sources → NiFi → MinIO → Trino → Notebooks → Superset