An alert has done its job when a named human acknowledges it inside the agreed time. Everything before that — the condition, the destination, the routing table — exists only to make that outcome likely.
Stand 29.07.2026
#Alert the chain; late and failed are different conditions
What
The nightly job is one promise: gold is correct and current by 06:00. Tasks are how the promise is kept, not what is promised. So notifications are declared once, at job level, and no task carries its own.
Why
Per-task notifications look thorough and are the fastest route to a muted channel. A twelve-task chain fails once and sends twelve messages, four of them for tasks merely cancelled when their upstream died. After the second incident someone writes a mail rule, and from then on the alerts arrive perfectly and are read by nobody.
Separating late from failed matters because the responses differ. Failed is a technical problem with a runbook. Late is often a business communication — the data will be there at 09:00, tell the controller before they open the report. Sending both down the same pipe in the same words guarantees one of them is handled wrongly.
How
yamlresources/jobs/dwh_daily.yml — one notification block for the whole chain
Failed means something ran and errored: the runbook decides re-run or intervene. Late means the deadline passed and the data is not there — which includes the case where the job never started, and therefore never failed, and therefore never alerted.
So the late condition is derived from the data and scheduled independently of the job it watches. The freshness SQL itself belongs to the observability route; what monitoring adds is that it must run on its own clock and route to the same rota under a different runbook.
sqlThe deadline check — scheduled 06:05, independent of dwh_daily
SELECT
MAX(_tech_loaded_at) AS last_load,
timestampdiff(MINUTE, MAX(_tech_loaded_at), current_timestamp()) AS minutes_stale
FROM dwh_prod.gold.fct_order_line;
-- Alert condition: minutes_stale > 185 -> LATE, not FAILED.
--
-- 185 is arithmetic, not taste: this query runs at 06:05 and the chain starts at
-- 03:00, so anything older than those 3h05 was written by LAST night's run.
-- Comparing two timestamps keeps the session time zone out of it. Re-derive the
-- number whenever the schedule moves.
Doing it
Declare notifications at job level in the bundle and delete every task-level block you inherit
Set no_alert_for_skipped_runs and no_alert_for_canceled_runs to true on any chain with conditional tasks
Schedule the deadline check as its own job, at deadline plus five minutesIt must not be a task of the thing it watches.
Give late its own one-page runbook with the business contact named in it
Embedding it
Ask the team what the 06:00 promise is, in one sentence, before touching any YAMLThree answers from three people means the alerting was never the problem.
Have them pause a schedule in test and wait for an alert that never comesTen silent minutes teaches the never-started failure mode better than any paragraph.
Let the team write the late runbook, including who they call in the businessNaming that person is where operations stops being an engineering-only activity.
Review muted channels together every quarter and ask what was muted and whyEvery mute is a design defect somewhere upstream of it.
Defaults
One notification block per chain, at job level
Failed and late are two conditions, two runbooks, one rota
The late check is data-driven and independently scheduled
Alert configuration lives in the bundle and is reviewed like code
Gotchas
Per-task notifications on a long chainOne failure, a dozen messages, and within two incidents the whole prefix is filtered into a folder — the alerting keeps working and stops being received, which is worse than having none because everyone believes it is covered.
Failure-only alertingThe run that never started emits nothing, so the outage with the longest detection time is precisely the one nobody is watching for.
Leaving no_alert_for_skipped_runs at its default on a chain with conditional tasksRoutine skips page the rota, skips are the most common non-event in a warehouse chain, and the rota learns to ignore the source.
Configuring the duration health rule but leaving on_duration_warning_threshold_exceeded without a destinationThe rule evaluates, nothing is sent, and the team reports that late alerting is covered because the YAML says so.
Deriving the late condition from the scheduler'The job succeeded' is not 'the data is current' — a chain can succeed against an empty source file and be perfectly on time with nothing in it.
A staleness threshold picked as a round numberThe chain starts 03:00 and normally lands gold around 04:15; a check that runs at 06:05 and alerts on 'older than 60 minutes' is true every single night. It fires on day one, is muted on day two, and the night the data really is missing looks identical to the forty before it. The threshold is the gap between the chain's start and the check's schedule, not a number that sounds strict.
Forgetting the one night the arithmetic breaksThe schedule is pinned to Europe/Berlin; when the clocks go back in October the real gap between a 03:00 start and a 06:05 check is 4h05, not 3h05, and a correctly configured check false-fires once a year. Suppress that date, or acknowledge it in the runbook — do not widen the threshold for the other 364 nights.
Notification destinations are configured once by a workspace admin — email, Slack, Teams, PagerDuty or a webhook — and referenced from job configuration by id. Jobs then never contain an address, which means changing who is paged is one edit in one place instead of a sweep through every job definition in the repository.
The routing table — agreed once, written down, reviewed quarterly
Condition
Goes to
Expected response
Runbook
Chain FAILED
on-call rota, paging destination
30 minutes
per-pipeline re-run runbook
Chain LATE at 06:05
on-call rota, paging destination
30 minutes, plus tell the business contact
late runbook
Duration ratio > 1.25
team channel
next sprint
none — it becomes a ticket
Data-quality anomaly
data steward for the domain
same working day
observability route
Cost anomaly > 25% week on week
platform owner
weekly review
storage & cost route
The rota is two names per week: the person on call, and the person who covers when the first one is at the dentist. It is published where the team looks, not in the consultant's handover deck.
What can be de-personalised is evaluation. Alerts v2 carry a run_as field that accepts a service principal, so the identity that runs the query and the identity that owns the object are two separate decisions. Set run_as to the DWH service principal and the alert keeps evaluating on the day someone leaves; ownership still has to be moved by hand, by a workspace admin, which is why it belongs on the leaver checklist rather than in an architecture diagram.
Why
The common failure mode is precise: alerts go to dwh-team@client.example. Everyone receives it, so nobody is responsible for it. The first three are read, the tenth becomes an Outlook rule, and the alerting system is now a machine that converts incidents into unread mail. Diffusion of responsibility has a name and it is a distribution list.
The second is the consultant's own address in the configuration. It works beautifully for four months. Then the engagement ends, the address bounces, and the alert nobody is receiving is by definition the alert nobody knows they are not receiving. The alerting system fails silently, which is the one thing an alerting system may never do.
How
bashAlerts v2 — set the evaluation identity; ownership is a separate, manual act
# The EVALUATION identity may be a service principal.
curl -sS -X PATCH \
"https://${DATABRICKS_HOST}/api/2.0/alerts/${ALERT_ID}?update_mask=run_as" \
-H "Authorization: Bearer ${DATABRICKS_TOKEN}" \
-H "Content-Type: application/json" \
-d '{ "run_as": { "service_principal_name": "d1e2f3a4-5b6c-7d8e-9f0a-1b2c3d4e5f60" } }'
# There is no equivalent call for OWNERSHIP. No payload makes a service principal
# or a group the owner of an alert. Transferring it is a workspace admin moving it
# to another USER, and it is the line on the leaver checklist that gets forgotten:
#
# [ ] reassign every alert and scheduled query owned by <leaver> owner: <admin>
#
# Scheduled queries follow the same rule; the run_as / owner split is the same.
yamldatabricks.yml — destinations are per-target variables, never literals
variables:
oncall_destination:
description: Notification destination id for the DWH on-call rota.
dev_null_destination:
description: Dead-letter destination. Dev and test page nobody, on purpose.
default: 11112222-3333-4444-5555-666677778888
targets:
dev:
variables:
oncall_destination: ${var.dev_null_destination}
test:
variables:
oncall_destination: ${var.dev_null_destination}
prod:
variables:
oncall_destination: 0a1b2c3d-4e5f-6789-abcd-ef0123456789
# Both variables are DECLARED above. A ${var.x} that was never declared is not a
# runtime surprise — bundle validate rejects it, which is why validate is the
# first step in CI.
Doing it
Create the destinations at workspace level and reference them by id through a bundle variable per targetDev and test point at a dead-letter channel so a broken branch cannot page the rota.
Set run_as on every alert and scheduled query to the DWH service principalEvaluation then survives a leaver.
Park alert ownership on one long-lived admin account, never on the engineer who happened to create itA service principal cannot hold it, and an owner removed from the workspace leaves the alert with no owner at all.
Add a line to the offboarding checklist: reassign every alert and scheduled query the leaver ownsPut a named workspace admin against it. Only an admin can do it, and only by hand.
Grep the bundle for external mail domains before handoverOne line to check, and never empty the first time.
Test the alert path monthlyFail a canary job in test on purpose and confirm a human acknowledged it inside the agreed response time.
Embedding it
Put a team member's name in the rota from week oneSit beside them for the first page rather than taking it yourself.
Hand the monthly alert test to the team and let them run it without you presentThe first time the canary fires and nobody notices is the finding you want them to own.
Ask in every weekly: who was paged last, and what did they do?An unanswerable question means alerts are landing where nobody reads them.
Agree at kick-off the date on which you come off the rota, and put it in the planAn unstated end date becomes a permanent dependency.
Walk the client's workspace admin through one alert-ownership transfer while you watchLet them write the checklist line themselves. It is the only step in this topic that no configuration can automate away, so it has to live in someone's process rather than in your bundle.
Defaults
Destinations by id, resolved per bundle target. Never an address in a job definition
run_as = service principal; owner = one long-lived admin user, because an alert can be owned by nothing else
Alert ownership transfer is a named line on the leaver checklist, with a named admin against it
Two names on the rota every week, published where the team looks
A monthly canary that proves a human is at the other end
Gotchas
A shared mailbox as the destinationEveryone is notified, nobody is accountable, and the mail rule that hides it is written after the second incident rather than the twentieth.
The consultant's address surviving handoverThe bounce goes to a mail server, not to the client, so the failure of the alerting system is itself unalerted — and it is discovered by the incident it was supposed to catch.
One destination id across dev, test and prodDevelopers page the rota with broken branches, the rota mutes the channel within a month, and the production alert that follows is muted along with it.
Owning alerts with the account of whoever built themWhen that person is removed from the workspace the alert has no owner at all, nothing errors anywhere, and only a workspace admin can put it right — by hand, on an alert nobody is now looking at.
Trying to fix that by giving the alert to a service principalIt cannot be done: service principals and groups may not own an alert, so the design that works everywhere else in this warehouse is unavailable here. Teams that assume it worked stop tracking alert owners entirely, and the leaver checklist never gets the line it needs.
Reading run_as as ownershipSetting run_as to a service principal keeps the alert evaluating after a leaver — it does not change who owns it, and it is discovered in an incident, when the rota finds it cannot edit the alert that just fired.
A routing table agreed in a workshop and never rehearsedThe first real page is then also the first time anyone discovers the paging destination was never configured.