One runbook per alert, not per system. The alert names the runbook, the runbook names the first query, and the first query settles the only question the responder has: are the numbers wrong, or is this a late job that will be fine by breakfast?
Why
Runbooks written by the architect fail predictably: they omit the steps she performs without noticing. Which warehouse. Which catalog. Where the pipeline event log is, as opposed to the job run output. The responder stalls at step 2, and those twenty minutes are twenty more minutes of a wrong number being published.
Someone who was woken up writes a different page. They write the catalog name, because they typed it wrong once at 03:00. Being paged is the only reliable way to find out which knowledge was tacit.
How
textrb_silver_sales_order_valid_amount.md — the whole page, one alert
RUNBOOK silver_sales_order owner: grp_dwh_engineers
alert "expectation valid_amount: failed_records > 0"
page yes, 24/7 (wrong numbers, not late numbers)
1 WHAT BROKE
The Lakeflow pipeline dropped order lines violating business
rule 5: amount_eur > 0 for non-cancelled lines. Usual cause:
silver.fx_rate has no CHF row for the order date, so
amount_eur comes out NULL.
2 FIRST QUERY (SQL warehouse "ops", catalog dwh_prod)
SELECT l.order_ts::DATE AS d, l.currency, COUNT(*) AS lines
FROM dwh_prod.silver.sales_order_line l
LEFT JOIN dwh_prod.silver.fx_rate r
ON r.currency = l.currency
AND r.rate_date = l.order_ts::DATE
WHERE r.currency IS NULL
AND l.order_ts >= current_date() - INTERVAL 3 DAYS
GROUP BY 1, 2 ORDER BY 1;
3 DECIDE
rows returned -> missing FX. Load the rate, then re-run.
zero rows -> not FX. Escalate, do NOT re-run.
4 RE-RUN? safe blind. Full refresh: NO — see re-run table.
5 ESCALATE after 45 min to grp_dwh_engineers rota #2.
6 TELL #dwh-status, before you start, not after you finish.
Six blocks, one screen, no prose. The architecture, the rationale and the history of the pipeline belong elsewhere — a page that has to be scrolled is a page that gets skimmed.
Doing it
One runbook per alertA single pipeline can fail three ways that need three different first queries.
Put the runbook link in the alert payloadA runbook found by searching a wiki is a runbook read after the incident.
Start with a copy-pasteable query returning rows or zero rows, and say what each outcome means
Give the escalation a timer in minutes and a name'Escalate if needed' means nobody escalates.
Embedding it
Do not write the first runbookWait for the first real page, then have the person who took it write it while they are still annoyed.
Have someone who has never touched that pipeline follow the page literallyThey use no knowledge that is not written on it. Every place they stop is a missing line.
In review ask only 'could you run this at 03:00 with no context?' — checkable, unlike 'is it clear?'
Rotate the rota before handover, so gaps are found by four people rather than by the author
Defaults
One page, six blocks: what broke, first query, decide, re-run answer, escalate, tell
Runbook link in the alert payload; runbook file in the repo beside the pipeline
First step is always a query with a stated interpretation, never 'check the logs'
Written by the responder, reviewed by the pipeline's author — that direction, not the reverse
Gotchas
A step that says 'check the logs'Three logs could be meant — pipeline event log, job run output, driver log — and at 03:00 the responder opens the wrong one first. Name the log and the filter.
Runbooks in a wiki space nobody has bookmarkedThe symptom is an incident handled entirely from memory and Slack, followed by 'we have a runbook for that' the next morning.
One runbook per pipelineIt becomes a decision tree, the responder reads the wrong branch, and the first query answers a question they did not have.
A runbook never executed by anyone but its authorIt reads complete and is missing a catalog name or a permission the rota does not hold — which surfaces as access-denied, mid-incident.
Every scheduled job and pipeline carries one pre-agreed answer to a single question: is it safe to re-run this blind, or does it need intervention first? Three answers are allowed — yes, yes-if, no. It is decided at build time by whoever wrote the write logic, not at 03:00 by someone who did not.
The re-run table — one row per scheduled thing, kept in the repo
Job / pipeline
Write pattern
Re-run blind?
If you do it anyway
bronze.shop_order
Autoloader append, checkpointed
yes
Nothing — the checkpoint skips seen files. Full refresh is a separate question and the answer is no, for the opposite reason to the one people expect: it TRUNCATES the streaming table, deletes the checkpoints of every flow writing to it and re-reads the source from the beginning, so every file the Volume no longer holds is gone from bronze permanently. It also writes a non-append commit, which fails every downstream pipeline until they are fully refreshed too or read this table with skipChangeCommits.
bronze.sap_kna1
daily full snapshot, append-only
no — check first
A second batch lands for the same logical day. Snapshot comparison then diffs today against today, finds nothing, and writes no SCD2 version for customers that did change.
silver.customer
SCD2 merge
yes, if as_of is a parameter
With current_timestamp() the re-run closes the interval it opened four minutes ago and opens another. Two versions, one business change, _tech_is_current still true on exactly one row.
silver.sales_order_line
SCD1 full overwrite
yes
Nothing. Overwrite is idempotent by construction — the only row whose answer needs no caveat.
gold.fct_order_line
MERGE on order_line_sk
yes, if silver is complete
Against a half-written silver it upserts a partial day, and gold disagrees with silver with no error anywhere. Duplicate order_line_sk aborts the MERGE on multiple source matches — the loud, good case.
gold.agg_revenue_month
INSERT INTO … REPLACE WHERE month = :month
yes, if REPLACE WHERE is on the statement
Written as a bare INSERT OVERWRITE it replaces the WHOLE table with the one month in the SELECT — every other month gone in a single commit, no error. The table is liquid-clustered and unpartitioned, so partitionOverwriteMode=dynamic does not scope it either; REPLACE WHERE is the only mechanism that does. Scoped too widely, it re-states closed months from whatever gold.fct_order_line holds now, moving last quarter's published revenue after the quarter closed.
Three of those are 'yes' only because of a decision in the code. The SCD2 row is the sharpest: the clock arrives as a job parameter, or the answer flips to no.
Why
At 03:00 the responder will re-run something. The only question is whether they know the consequence or are hoping there is not one — and hope is the default when nobody wrote the answer down, because a re-run is the one action available to someone with no context, and it usually works.
'Usually' is the problem. Every failure in the table above is silent: a duplicate batch, a missing SCD2 version, a re-stated month. Nobody finds it that night — they find it in a quarterly comparison eleven weeks later, when the cause is no longer recoverable from memory.
How
pythonsrc/dwh/jobs/silver_customer.py — the injected clock is what makes the answer 'yes'
def run(spark, catalog: str, as_of: str) -> int:
"""Rebuild silver.customer as SCD2. Returns rows in the table after.
as_of is a JOB PARAMETER, never current_timestamp(). Re-running the
same logical day must produce identical intervals; a wall clock read
inside the job turns every re-run into a new version.
"""
target = f"{catalog}.silver.customer"
incoming = clean_customer(spark.read.table(f"{catalog}.bronze.sap_kna1"))
new_state = apply_scd2( # -> DataFrame, full new state
current=spark.read.table(target),
incoming=incoming,
keys=["customer_id"],
as_of=as_of,
)
# Delta refuses to overwrite a table that is also being read from, and
# new_state reads target. Stage, then swap. CREATE OR REPLACE keeps the
# history, grants, row filters and column masks; it drops the comment and
# the CLUSTER BY spec unless you restate them in the new DDL. INSERT
# OVERWRITE keeps all of it, which is why it is the safer swap here.
#
# BY NAME, not positional (DBR 13.3 LTS+). A column reordered inside
# apply_scd2 otherwise lands currency in status with no error.
stage = f"{catalog}.silver.stg_customer_scd2"
new_state.write.mode("overwrite").saveAsTable(stage)
spark.sql(f"INSERT OVERWRITE TABLE {target} BY NAME SELECT * FROM {stage}")
spark.sql(f"DROP TABLE {stage}")
return spark.table(target).count()
sqlThe 'check first' query for bronze.sap_kna1 — step 2 of its runbook
SELECT _tech_batch_id,
COUNT(*) AS rows_landed,
MAX(_tech_loaded_at) AS loaded_at
FROM dwh_prod.bronze.sap_kna1
WHERE _tech_loaded_at >= current_date()
GROUP BY _tech_batch_id
ORDER BY loaded_at DESC;
-- one batch id -> extract landed once, re-run is safe
-- two batch ids -> already re-run. Do NOT repeat; resume at silver.
Doing it
Add the re-run row in the same pull request as the jobRetro-fitting it means re-deriving intent from someone else's code.
Inject every clock: as_of, run_date and batch_id are parametersNever read from the wall clock inside a transform.
Give each yes-if row its check query verbatim in the runbookA condition that takes thinking to verify gets skipped.
Prove the 'yes' rows in CI: run twice, assert identical — see the warehouse-invariants route
Embedding it
Do not fill the table in for themAsk 'what happens if this runs twice?' in each job's review; the rows nobody can answer are the ones that will hurt.
Have them break one in dwh_test: load the snapshot twice, then find the missing SCD2 versionThey will remember the shape of a silent failure.
Make the re-run row a definition-of-done item, so it is refused at review rather than requested later
When someone marks a job 'no', make them write the intervention'No' alone just moves the 03:00 decision to a person with less information.
Defaults
Three answers only: yes, yes-if-<check>, no-with-procedure. No prose
The table lives in the repo and is reviewed with the code, not in an operations wiki
Re-run and full refresh answered separately for every pipeline
Idempotency proven by a test, never asserted in a table
Gotchas
current_timestamp() inside an SCD2 mergeEvery re-run opens a version for rows that did not change, _tech_is_current stays correct, nothing asserts. You find it when someone asks how often a customer changed segment and the answer is 'nine times, all on 14.03.2026'.
Treating full refresh as a stronger re-run, and fearing the wrong outcomeIt TRUNCATES the streaming table, removes the checkpoint data of every flow writing to it, and restarts them with new checkpoints — so the failure mode is DATA LOSS, not duplication. Anything the source no longer retains (a Kafka topic on 24-hour retention, a Volume with a lifecycle rule, an SAP extract directory that keeps a week) cannot be re-read, and bronze is now permanently missing rows nothing else holds. Duplication is the risk of the other manoeuvre — hand-resetting a checkpoint while leaving the data in place, which reprocesses the source into a table that still contains it.
Full-refreshing one table and leaving its downstreams aloneThe truncate is a non-append commit, so every streaming consumer fails on the next update until it is fully refreshed as well, or reads this table with skipChangeCommits — which is a decision about which downstream may silently keep the old rows, not a flag you set to make an error go away.
Marking a job safe because the last three re-runs were fineSafety is a property of the write pattern, not of the sample: an overwrite is safe, an append is not, however often it has been repeated without incident.
Re-running a downstream job while the upstream is half-writtengold.fct_order_line merges a partial silver day, nothing errors, and gold and silver now disagree behind a green run history.
Believing a WHERE clause scopes an INSERT OVERWRITEIt does not: the predicate filters the SELECT, the overwrite still replaces the whole table, so one re-run of gold.agg_revenue_month leaves exactly one month in it. On a liquid-clustered table there are no partitions for partitionOverwriteMode=dynamic to spare either. You notice when a year-on-year chart renders a single bar, and the fix is a backfill, not a rollback, if VACUUM already ran.