Architecture · Diagram Design · Parametric

High-level architecture · Vertical Orchestration + Security

Five horizontal phases (Sources → Ingestion → Storage → Transformation → Visualization) plus three cross-cutting verticals (Orchestration, Security, Observability) rendered as a reserved right strip. Two cross-cutting bars stack below the cluster — Identity and Monitoring — each paired 1:1 with a vertical chevron. The diagram is fully parametric: chevrons, sources, components, and connections drive every coordinate.

High-level architecture · Vertical Orchestration + Security Architecture diagram showing data sources moving through ingestion, storage, transformation, and visualization alongside orchestration, security, and observability. DATA SOURCES INGESTION STORAGE TRANSFORM VISUALIZATION ORCHESTRATION SECURITY OBSERVABILITY SOURCES EXT PostgreSQL CDC · SQL · API EXT SFTP drop Batch · landing EXT Web forms Survey · admin EXT Legacy system Trade · mainframe Orchestration / Workflow Automation (Apache Airflow) COLL Apache NiFi Collect · route · transform VIRT Trino SQL · data virtualization STORE MinIO / S3 Object store · S3-API ANLZ Notebooks Jupyter · Python · R DASH Superset Dashboards · reports Kubernetes Identity Manager (Keycloak · LDAP · OIDC) Monitoring & Observability (Prometheus · Grafana · Loki) LEGEND External · outside cluster Kubernetes boundary Focal (storage hub) Primary path Orchestration trigger Query read-back

Vertical · Orchestration

Workflow automation

  • Apache Airflow runs as a full-width bar at the top of the cluster — visible to every cluster node
  • Trigger drops are always dashed (style: trigger), originating from the bar's bottom edge
  • The vertical Orchestration chevron in the right strip labels the cross-cutting column
  • One bar component per vertical chevron — enforced by the pairing rule

Storage · MinIO / S3 (focal)

Object store hub

  • Single focal node — accent border, accent ink, accent connectors
  • Every edge touching the focal node is automatically style: primary (accent stroke)
  • Trino virtualizes MinIO buckets — read-back drawn as style: query (dashed dim)
  • Notebooks consume curated data; their output writes back as Parquet

Verticals · Security + Observability

Cross-cutting concerns

  • Multiple cross-cutting bars stack below the cluster, one row per concern (44 px each)
  • Each crosscut pairs 1:1 with a vertical chevron in the right strip (count constraint)
  • Bars emit no individual connectors — semantic is "applies to everything above"
  • Components carry an optional color override (Identity Manager rendered in rust-red here) — applies to border, icon, and name; connectors stay topology-driven