Run11 Observability · 1 of 3
Green jobs, wrong numbers
The route rests on one distinction. A team that cannot state it keeps buying dashboards of job status and keeps being told about bad numbers by the business.
Stand 29.07.2026
#The failure class nothing catches
WhatThe four signals, and what each catches in this warehouse
Monitoring watches the machinery. Observability watches the data. Monitoring answers did the job exit zero. Observability answers is what came out of it true. The interesting failures do not throw.
One run through this warehouse. The ECB publishes no rate on a Frankfurt bank holiday. The
bronze.fx_rate load finds no file, writes nothing, exits zero. silver.fx_rate has nothing to insert, exits zero. The gold build finds no rate for date_sk and leaves fx_rate — and therefore amount_eur — NULL on every CHF line.Four green jobs. Swiss revenue for the day is zero. Nobody notices until a controller asks why Switzerland collapsed on 1 August.
| Signal | Question | Cheap check | Catches |
|---|---|---|---|
| Freshness | is it current? | MAX(_tech_loaded_at) against a 26 h SLA | an SAP extract that stopped being scheduled |
| Volume | is all of it there? | daily count against a 7-day median | a CDC gap on sap_vbap; the webshop feed silently dry |
| Schema | is it the same shape? | information_schema diff against dwh_test | a retyped SAP field truncating amount_eur |
| Distribution | are the values sane? | share of NULL, CANCELLED, UNKNOWN per day | the missing fx_rate above; a broken customer join |
Four queries, no product. What they need is someone who owns each threshold.
Why
The failure this prevents is discovery by the business. Once a controller finds a wrong number before the data team does, every later number is suspect, and correct output does not buy that back quickly. Observability is how the team stays first to know.
It also sets triage speed. With these four signals, which day, which country, which source is answered from history you already have. Without them, the incident starts by writing the query that would have caught it.
Doing it
- Write the four checks as .sql files in the repoOn the same zero-rows-means-pass convention as the SQL test suite.
- Schedule them as Databricks SQL alerts on gold.fct_order_line and the silver tables that feed itNot as a notebook someone runs.
- Wrap each check in SELECT count(*) AS violations FROM ( … ) before it becomes an alertDatabricks alert conditions are an aggregation over a column against a threshold; there is no 'fire if any rows came back'.
- Put a named owner and a runbook link in every alert bodyAn alert with neither is a notification.
- Record each incident's detection path: did we find it, or did theyThat ratio is this route's only honest score.
Defaults
- Four signals on every gold table a report reads, before a fifth signal anywhere
- Checks live in the repo and deploy with the bundleNothing important is configured only in the UI.
- Zero rows means pass, so one file is a CI test and a production monitor
- Every alert names a person and a runbook. Never a shared mailbox
Gotchas
- Treating job-status dashboards as observabilityComplete, green, and silent about the fx_rate case — the pipeline did exactly what it was told, on data that was not there.
- Alerting on everything on day oneThe channel is muted within a fortnight, and a muted channel is worse than none because everyone believes it works.
- Checks that live only in the monitoring UIInvisible to code review, absent from the diff when someone changes the fact grain, gone when the workspace is rebuilt.
- Monitoring gold onlyThe fx_rate outage was visible one step earlier in silver.fx_rate, and nobody watched it because nothing reads it directly.
- Pointing an alert straight at a zero-rows-means-pass queryThere is no row-count trigger: the condition aggregates a column, and an empty result falls through to the separate Empty result state setting. The check is correct, the alert never fires, and that is discovered during the incident it was written for.