Change Data Capture, Exactly Once
THE PROBLEM
Analysts needed tables that mirror the operational database, but the nightly full-table syncs they inherited were hammering production at peak load and still leaving analytics a day behind. That sync could not stay — it locked tables, missed intraday changes, and had no way to rebuild history. I replaced it with log-based CDC and learned the three ways naive CDC lies to you: producer retries delivered duplicate events that doubled rows in finance tables until the sink went idempotent, source schemas evolved without asking, and replay had to be designed in from day one.
CONSTRAINTS
- Producer retries mean duplicate events are guaranteed — I have the doubled rows to prove it
- Source schemas evolve without asking permission
- Ordering matters for updates and deletes
- History must be replayable when logic changes
DECISIONS & TRADEOFFS
IDEMPOTENT CONSUMERS over trusting exactly-once delivery — because at-least-once is the honest guarantee, so upserts keyed on primary key plus log position make duplicates harmless
SCHEMA REGISTRY over schemaless JSON — because enforced Avro contracts with compatibility rules turn schema drift into a managed event instead of a pipeline fire
ICEBERG MERGE over append-only plus dedupe views — because updates and deletes land as real table state, not as query-time reconciliation everyone has to remember to do
ARCHITECTURE
OUTCOME
Analysts query tables that reflect production in near-real-time, and the nightly sync — and its load on the operational database — is retired. Duplicates, schema changes, and restarts are boring now instead of pageable. When transformation logic changes, history replays from a snapshot instead of demanding a new full dump from production.
PROOF / LINKS
AT 100× SCALE
At 100x this becomes the standard ingestion fabric for every operational system: self-serve connector onboarding, per-table freshness SLOs, and automated backfill orchestration when downstream models evolve.