The warehouse-specific part of incident response is the severity ladder: wrong numbers outrank late numbers, and most teams have it backwards because only the late ones page.
Stand 29.07.2026
#Severity, blast radius, and the first ten minutes
What
Three severities, defined against this warehouse rather than a generic P1/P2/P3 ladder:
S1 — published numbers are wrong
gold.fct_order_line or gold.mv_sales is serving revenue that violates a business rule, and a dashboard has already shown it. Page at any hour.
S2 — data is stale past its SLA
gold.fct_order_line has a 26-hour freshness SLA and has breached it. Page in business hours: the numbers are right, just old, and the timestamp is visible.
S3 — a job failed, data correct and inside SLA
A retry handled it, or the next run will. Ticket, no page.
The ordering matters because the alerting is naturally the wrong way round: a failed job pages loudly, a wrong number pages not at all unless someone built the check. A team that alerts only on job failure has automated S2 and S3 and left S1 to whoever notices.
The first ten minutes of an S1 are the same three moves every time — stop publishing, measure, then diagnose:
Why
Wrong numbers cost trust, and trust does not restore with the table. A revenue figure that was wrong for six hours and got quoted in a meeting is fixed by telling the people who quoted it, not by correcting the table. That is why blast radius is a step and not an afterthought.
Pausing before diagnosing feels backwards and is the highest-value ten seconds available. Every minute of diagnosis with the refresh still scheduled adds another cohort holding a wrong number and no reason to doubt it.
How
sqlStop the bleeding, then find out how far it reached
-- 1. pause the downstream refresh FIRST. Diagnosing while a scheduled
-- BI refresh republishes the wrong number turns a one-hour incident
-- into a one-day one.
-- 2. how bad, and since when
DESCRIBE HISTORY dwh_prod.gold.fct_order_line LIMIT 10;
SELECT COUNT(*) AS bad_lines,
MIN(order_ts) AS first_bad,
MAX(order_ts) AS last_bad
FROM dwh_prod.gold.fct_order_line
WHERE status <> 'CANCELLED'
AND (amount_eur IS NULL OR amount_eur <= 0); -- business rule 5
-- 3. blast radius: who read it while it was wrong. Filter on event_date,
-- not event_time: the table is laid out by date, and event_time scans
-- the whole of it — a full scan you are paying for mid-incident.
SELECT DISTINCT entity_type, entity_id, target_table_full_name
FROM system.access.table_lineage
WHERE source_table_full_name = 'dwh_prod.gold.fct_order_line'
AND event_date >= current_date() - INTERVAL 7 DAYS;
-- a NULL target is a notebook, dashboard or ad-hoc read. Still a consumer.
-- 4. if a bad version was published, roll back — do not patch
RESTORE TABLE dwh_prod.gold.fct_order_line TO VERSION AS OF 412;
Lineage answers "who read it", not "who believed it", and it emits records only where lineage can be inferred — an empty result is not proof that nothing read the table. The limits, including the rolling one-year window on the system tables, are in Audit & lineage. Treat the result as a floor and notify wider.
Doing it
Define the three severities against real tables and real SLAsWrite down which one pages at 03:00.
Split the roles: one on the keyboard, one running the incident, one on commsBelow three people the lead is still a role — held by whoever is not typing.
Pause downstream refresh as move one, and record the pause so it gets undone
Prefer RESTORE to a corrective UPDATEA rollback is one auditable operation; a patch is a new transformation written under time pressure with no test.
Embedding it
Give the incident-lead role to a team member on the first real S2, with you in the room and silentTaking the keyboard once teaches them you are the escape hatch.
Have the team classify the last ten failures into S1/S2/S3 from run historyThe argument about which are S1 is the training.
Ask 'who is holding a wrong number right now?' in every incident until someone else asks it first
Practise the rollback in dwh_test until RESTORE is boringNobody runs an unfamiliar command on production mid-incident.
Defaults
Severity defined by consequence to the data, not by which component failed
Pause, measure, diagnose — in that order, every time
Rollback over patch; a corrective UPDATE gets the same review as any other transformation
Blast radius from lineage treated as a lower bound, never as a complete list
Gotchas
Diagnosing before pausing the refreshThe dashboard republishes the wrong number every fifteen minutes while you work, so the population who need telling afterwards grows for the whole duration of the incident.
Debugging in the incident channelStack traces and the narrative compete for one scroll, the business reader stops following, and starts asking the incident lead directly — which removes the incident lead.
Nobody declares the incident overThe paused refresh stays paused, the dashboard is stale for a week, and the next complaint is about the fix rather than the fault.
Assuming an empty lineage result means nothing consumed the tableLineage is emitted only where it can be inferred; a downstream extract nobody registered will not appear, and you will not notify its owner.
Three hours in dwh_test, with real code and real alert routing. You break things on purpose and do not touch the keyboard. Week 13 because it is a month past the point where the team's contribution overtakes yours — late enough that they built this, early enough that you are still here to fix what the day exposes.
The SAP extract never arrives — no file waiting for bronze.sap_kna1 when the chain starts at 03:00. Does anything alert, or does the pipeline succeed on yesterday's data?
The extract arrives twice: two batch ids for one logical day. Does anyone notice the missing SCD2 versions?
silver.fx_rate has no CHF row for one day, so amount_eur goes NULL for Swiss lines and breaks business rule 5. Which fires — the expectation, the freshness check, or nobody?
SAP adds a column to VBAP. Does the schema-drift check catch it, and does the pipeline keep running?
The gold MERGE is killed mid-run. Is gold.fct_order_line consistent, and does the runbook's re-run answer hold?
The pipeline's author is 'on holiday' and may not speak for three hours. This is the scenario that tests the runbooks.
Scenario six is the one people want to drop and the only one that measures anything. The rest test the alerting; the silent author tests whether the knowledge ever left their head.
Why
Runbooks decay invisibly: the alert text drifts, a warehouse is renamed, a permission is tightened. None of that fails a test. It produces a page that reads perfectly and does not work, and you find the stale sentence only by following it under pressure.
The day also answers a question no status report answers honestly — whether the team can operate this without you. Three hours watching them fail to find the event log is worth six weeks of nodding in a workshop.
Doing it
Three hours, dwh_test, alerts routed to the real rotaA muted workspace teaches nothing about routing.
Seed the failures from a fixtures script so the day is reproducible and can be repeated after handover
Time each scenario: time to alert, time to correct diagnosis, time to recoveryThose three numbers are the deliverable.
Every gap found becomes a pull request that week, with a named owner, before anyone goes home on Friday
Embedding it
Sit on your handsRescuing at minute ten converts a diagnostic into a demonstration of your value, which is the opposite of the goal.
Let the team choose two of the six scenariosWhat they pick shows where they feel exposed; what they avoid shows more.
Have the team run the debrief and write the actionsIf you write the actions, they are your actions.
Ask which scenario they would not survive alone at 03:00The honest answer is the plan for the next four weeks.
Defaults
Three hours, six scenarios, dwh_test, real alerts, consultant silent
Measure time-to-alert, time-to-diagnosis, time-to-recovery per scenario
Seeded from a script, so it repeats rather than being an event
One scenario always removes the author of the thing being broken
Gotchas
Running it where alerting is disabled 'to avoid noise'You then measure diagnosis and learn nothing about routing — and routing is what is broken in most teams, because it was configured once and never received a real page.
Announcing the scenarios in advanceThe team pre-reads the runbooks, every timer looks good, and the exercise certifies a state that will not exist at 03:00.
Choosing scenarios everyone already handlesA game day that finds nothing was designed to find nothing; a debrief with no actions means the scenarios were too kind.
Doing it in week 24Whatever it exposes is now a handover risk instead of a coaching opportunity, with no time to fix it with the team rather than for them.
The rule is mechanical: the fix and the runbook edit are the same pull request. Not a follow-up ticket, not an action item with a due date. As separate work it competes with feature work, and it loses every time.
Why
Post-mortems written a week later document a sequence that did not happen. People reconstruct a tidy order from memory, and the runbook then encodes a path nobody walked. The next responder follows it, it fails, and that is worse than no runbook — it cost them the time they spent trusting it.
The test for a closed loop is unforgiving: when the same failure recurs, does the second responder find the first responder's note without asking anyone? If not, the incident produced a document and no learning.
How
textWhat the incident of 03.07.2026 added to rb_silver_sales_order_valid_amount.md
2 FIRST QUERY (SQL warehouse "ops", catalog dwh_prod)
+ -- the rota needs SELECT on silver.fx_rate; PERMISSION_DENIED
+ -- here means you are not in grp_dwh_engineers.
SELECT l.order_ts::DATE AS d, l.currency, COUNT(*) AS lines
...
3 DECIDE
rows returned -> missing FX. Load the rate, then re-run.
zero rows -> not FX. Escalate, do NOT re-run.
+ ALL currencies missing for a date -> the ECB CSV was empty,
+ not late. Loading it again will not help. Escalate at once.
seen: 03.07.2026, 04:12-05:35. Detected by the expectation, not by
the freshness alert — that check runs at 06:05, so it would have
fired half an hour after the fix landed.
Two added lines and a timestamp. No root-cause essay, no five whys, no separate document. What earns its place is what the last responder did not know and the next one will need: the missing permission, the uncovered branch, and how long it really took.
Doing it
Edit the runbook in the same PR as the fixReviewers reject a fix whose incident revealed a gap and whose runbook is untouched.
Write it inside 48 hours, while the wrong turns are still rememberedThe wrong turns are the valuable part.
Record the detection path: which alert fired, which should have, and how long before a human saw it
Delete steps that turned out to be unnecessaryA runbook that only grows is unusable within a year.
Embedding it
Ask 'which line would have saved you ten minutes?' at the end of every incidentThat produces an edit; 'any learnings?' produces a paragraph.
Have the responder open the PR, not you, even when you know exactly what it should say
Track one number in the retro: incidents where the runbook was edited, over incidentsStop tracking it when it is always 100%.
Make a second occurrence a team question — why did the note not prevent it? — rather than a personal one
Defaults
Fix and runbook edit in one pull request, always
Written within 48 hours by the person who was paged
Detection path recorded: what fired, what should have, and the delay
Runbooks lose lines as well as gain them
Gotchas
Post-mortems in a separate document spaceWritten once, read never, and the next responder rediscovers the same missing permission from scratch — the incident repeats in exactly the same shape.
Action items with an owner and no date, or a date and no ownerBoth mean the item does not exist, and both look like follow-through in a retro.
Writing the note a week laterThe reconstructed sequence is tidier than the real one, so the runbook documents a path that does not work in the order given.
Adding and never removingPast two pages responders start skimming, the critical line drops below the fold, and the runbook now contains the answer while nobody reads it.