From 3 objects to 3,000.
Larger estates change the demands on dependency ordering, incremental state, scheduling and recovery.
What changes at each scale
-
3
Two spreadsheets and a Table.
Even a small project benefits from code-managed definitions, repeatable deployment and a Test. Ordinary files and code can define that work without a separate orchestration system.
-
30
The order stops fitting in your head.
Dependencies, load order and the impact of a change become difficult to keep correct through informal conventions. Keep them with the definitions instead of rebuilding them in schedules.
-
300
State has to outlast a run.
Incremental loads, bookmarks, fault tolerance, orchestration, SQL endpoint readiness, logging, validation and environment consistency are now recurring engineering work, not occasional fixes.
-
3,000
Scheduling and recovery are fundamental.
Independent work must run together; critical paths must move first. Unchanged runs should avoid unnecessary work, and a failure must leave enough recorded state to recover without starting the estate again.
Scheduling, readiness and recovery
Independent Spark and Warehouse work can run in parallel. Critical-path-aware scheduling gives priority to work that unlocks the rest of the graph; an invocation cap and per-resource limits keep that parallelism within the budget you choose.
SQL endpoints synchronise where selected consumers need them, rather than after every unrelated object. Per-consumer bookmarks keep incremental boundaries with the work that succeeded. Recorded failures and blocked dependants make recovery deliberate, while unchanged definitions and data avoid needless processing.
For development, Mirror starts with Views and OneLake shortcuts to existing data. Materialise changed work instead of copying the estate before the first edit.
What that looked like on F64
A synthetic estate contained 3,000 objects: 2,144 Tables, 816 Views and 40 Folders across one Lakehouse and two Warehouses.
First Load
About an hour
Approximately 4.05 billion row insertions across the processing layers.
Later measured incremental loads
About 13 minutes
The first incremental day took about 19 minutes; the later two were around 13.
Those insertions count work across layers, not distinct source records.
Operating the estate
Build, Load, Test and Health define the operating cycle; selection, concurrency and development controls change how that cycle is applied.