Real-time Streaming Lakehouse
THE PROBLEM
Ops and finance were running the day off dashboards built on nightly batch jobs, so every morning meeting argued about numbers that were already a day stale. The legacy full-refresh jobs could not stay: they re-read everything, they got slower as volume grew, and warehouse streaming pricing was not an option. I rebuilt the ingestion path as streaming on open table formats, and hit the classic wall early — thousands of small files from streaming writes turned queries into a crawl until compaction became part of the pipeline, not a cleanup script.
CONSTRAINTS
- Late and out-of-order events are the norm, not the exception
- Streaming writes create thousands of small files that murder reads
- dbt models must run against tables that never stop moving
- Storage and compute costs must stay flat as volume grows
DECISIONS & TRADEOFFS
FLINK over micro-batch streaming — because true event-time processing with watermarks handles late data honestly instead of faking it with window hacks
ICEBERG over proprietary warehouse tables — because open table format with snapshot isolation lets readers and stream writers coexist, and any engine can query the same data
SCHEDULED COMPACTION over write-optimized only — because I watched unbounded small files destroy read performance and object-store bills, so maintenance is part of the pipeline, not an afterthought
ARCHITECTURE
OUTCOME
Dashboards moved from daily batch to minutes-level streaming, so ops reacts to the current hour instead of yesterday, and downstream models query the same open tables instead of waiting on exports. Finance sees costs hold flat because compaction and partitioning are operationalized, not heroic. I run this on commodity object storage and the full-refresh jobs are gone.
PROOF / LINKS
AT 100× SCALE
At 100x this becomes a multi-engine data platform: the same Iceberg tables serve Flink, Spark, and ad-hoc SQL, with tiered storage and per-topic freshness SLAs instead of one-size-fits-all latency.