ETL modernisation should preserve business meaning as well as move data. Inventory the old jobs and downstream dependencies, define versioned data contracts, then reconcile old and new outputs during a controlled dual run before the data owner approves promotion.
Composite reference architecture. A reference migration pattern; no production migration or performance gain is claimed.
The operating problem
Legacy pipelines contain undocumented business rules, manual recoveries and downstream timing assumptions. A clean rewrite can be technically elegant and operationally wrong.
How the workflow works
Document source schemas, transformation rules, cutoffs and recovery procedures before rebuilding a job. Run the replacement against matching input snapshots or a reconciled change stream. Compare record-level results and business aggregates, with explicit tolerances for late data and expected differences. A successful scheduler run does not establish that the output is complete or correct.
Where review and controls matter
Schema changes and backfills need bounded ranges and destination-aware retry behaviour. Preserve the old version and the last trusted output until the new pipeline has passed the agreed contract checks. Timezone rules and downstream availability deadlines should be tested explicitly, because a numerically correct dataset can still arrive too late for the business process that consumes it.
Orchestration reference: Apache Airflow core concepts.
What to measure
Use a fixed cohort and measurement window. Count numerator events only within the denominator's eligible cohort; sampled measures use the same audit sample for both terms. Record the source, exclusions and owner, and keep missing data visible. These definitions are proposed measurement contracts, not observed results.
| Measure | Definition and unit |
|---|---|
| Freshness | Elapsed time between the latest represented source event and the consumer's observation time; minutes or hours, with source-delay treatment. |
| Completeness | Expected records present in the destination divided by records expected from the reconciled source window; percent. |
| Reconciliation variance | Difference between old and new outputs under a declared record and aggregate comparison; count, currency or percentage as appropriate. |
| Failure recovery | Time from a detected pipeline failure to restored output meeting the data contract; median and p90 minutes. |
| Downstream SLA | Deliveries meeting the consumer's completeness and availability deadline divided by deliveries due; percent. |
A useful first pilot
Choose one pipeline with a known downstream consumer. Inventory its hidden dependencies, run representative historical and late-arriving data through both versions, and have the data owner approve a reconciliation report and rollback procedure.
Bring the workflow, its source records and the decision that needs an owner to a BergLabs pilot discussion.
