CASE FILE

CH.06DATA ENG

Data Reliability & FinOps

DATA CONTRACTSDBT TESTSANOMALY DETECTIONFINOPS
A torii gate stands in a river of data packets beside an abacus and coin scales, watched by a guardian holding a ledger.

THE PROBLEM

Executives were discovering broken pipelines by staring at wrong dashboards, and finance watched the warehouse bill grow faster than anyone could explain. The breaking point for me was an upstream producer silently renaming a column — every downstream model passed, every dashboard updated, and every number was wrong until a board meeting caught it. Reliability and cost were after-the-fact conversations; I moved both into the engineering loop.

CONSTRAINTS

  • Producers change schemas without telling consumers
  • Tests that only run in prod catch failures too late
  • Anomaly alerts without baselines are just noise
  • Cost is invisible until the invoice arrives

DECISIONS & TRADEOFFS

CONTRACTS IN CI over prod-only monitoring — because shift-left schema enforcement blocks breaking changes at the pull request, before any downstream job runs

ANOMALY DETECTION over static thresholds — because freshness and volume baselines learned from history catch the weird days static rules miss and stay quiet on normal ones

COST ATTRIBUTION PER MODEL over a single warehouse bill — because you cannot optimize what you cannot see, and per-model compute cost turns FinOps into an engineering metric

ARCHITECTURE

01DATA CONTRACTSschemas enforced atproducer02CI GATESbreaking changes fail thebuild03DBT TESTSfreshness, uniqueness,referential04ANOMALY MONITORSvolume and freshnessbaselines05LINEAGEimpact map for everyfailure06COST DASHBOARDSper-model computeattribution

OUTCOME

Breakage is caught in CI or by anomaly monitors before a dashboard shows it, and incidents come with lineage so the blast radius is known in minutes instead of discovered by executives. Finance gets warehouse spend attributed per model, so cost conversations happen at the pull request, where they can still change something, instead of at renewal time.

PROOF / LINKS

PROPRIETARY WORK — THE SHAPE OF IT IS PUBLIC HERE, THE SYSTEM ITSELF IS NOT.

AT 100× SCALE

At 100x this becomes platform policy: contracts as the only way to publish data, reliability SLIs per domain, and unit cost per pipeline reviewed alongside latency and uptime.