A pipeline has two halves that fail differently. The transform is ordinary Python and is tested on a laptop. The declarative half — which datasets exist, which expectations guard them, and how often each one fired — only exists inside a running update, and the way you test it is to seed a known defect, run the pipeline, and read the event log. This route decides what belongs in each half and what the assertion actually looks like.
Stand 29.07.2026
What can be tested without a pipeline
Most of what teams believe is untestable is untestable only because it sits behind a decorator. Move two things out and the interesting half becomes ordinary pytest.
This is the whole pipeline file for silver.sales_order_line. The transform and the rules come from elsewhere; the file is glue.
pythonsrc/dwh/pipelines/silver_sales_order.py
from pyspark import pipelines as dp
from dwh.pipelines.transforms.silver_sales_order import EXPECTATIONS, to_silver_sales_order
# CI points this at dwh_test.run_<job_run_id>; prod points it at dwh_prod.bronze.
SOURCE = spark.conf.get("dwh.source_schema")
@dp.materialized_view(name="sales_order_line")
@dp.expect_all_or_drop(EXPECTATIONS)
def sales_order_line():
return to_silver_sales_order(
spark.read.table(f"{SOURCE}.sap_vbap"),
spark.read.table(f"{SOURCE}.fx_rate"),
)
Now try to import it from a test. There are four walls, and teams usually stop at the first one and conclude the platform is at fault.
textFour reasons this function is not the unit under test
1. spark is injected by the pipeline runtime as a global.
Outside an update: NameError: name 'spark' is not defined
— raised at IMPORT time, before any test body runs.
2. The decorators register a dataset into a pipeline graph.
With no graph to register into, decoration itself fails.
3. @dp.materialized_view does not hand your function back.
sales_order_line is a registered dataset, not a callable you can assert on.
4. Even if it were callable, it takes no arguments and names two
tables — so calling it needs a live catalog with data in it.
The fifth reason is the one that matters most and is the easiest to miss: the data quality rules are not in the function body at all. They are decorator arguments. A test that somehow executed sales_order_line() would still test none of them.
So two things move out of the pipeline file. The transform becomes a pure function — that is the standards route's first testability rule. The rules become a plain dict in the same transform module, imported by the pipeline and by the tests, so they cannot drift apart:
pythonsrc/dwh/pipelines/transforms/silver_sales_order.py — business rules 1, 3, 5 and 6 as data
EXPECTATIONS = {
"amount_positive": "amount_eur > 0 OR status = 'CANCELLED'",
"known_country": "country IN ('DE', 'AT', 'CH', 'NL')",
"order_not_ancient": "order_ts >= '2020-01-01'",
"status_domain": "status IN ('OPEN', 'SHIPPED', 'CANCELLED')",
}
Business rule 2 — amount_eur = ROUND(quantity * unit_price * fx_rate, 2) — is deliberately absent. An expectation asserts a predicate over the row it is handed, and a row whose fx_rate was silently defaulted satisfies that arithmetic exactly. Rule 2 is enforced by the reconciliation assertions in the SQL-testing route instead, and the gotcha at the end of the next topic is the reason.
Why here
The failure this prevents is a conclusion, not a bug: our pipelines can only be tested by running them. A team that believes this checks its rules by starting a pipeline, which costs ninety seconds and a warehouse, so the check moves to a nightly job, so nobody edits a rule with confidence and nobody deletes one at all.
With EXPECTATIONS as an importable dict, changing known_country is a one-second test run. Without it the rule text is typed twice — once in the decorator, once in the test — and after the first edit the test guards a rule production no longer runs, while staying green.
Doing it
Move every constraint out of the decorator call into a module-level EXPECTATIONS dict in the transform module. Mechanical commit first, logic changes second.
Read the source schema from spark.conf, set by the pipeline's configuration block — never a literal catalog in the pipeline file.
Keep the pipeline file under ten lines. If it grows a branch, that branch has escaped the tested half.
Point the test suite at transforms/ only. Nothing under pipelines/ is ever imported by a test.
Embedding it
Ask someone to write a unit test for the decorated function without telling them it is impossible. Ten minutes of hitting the four walls teaches the layout rule permanently; being told it takes and leaves nothing.
In review, ask one question of every pipeline file: 'which line here could be wrong?' If the honest answer is none, the glue is thin enough.
Have the team delete an expectation on purpose and find out which test catches it. If none does, that is today's work — see the event-log assertion below.
When someone proposes testing a pipeline by running it in dev and eyeballing the table, agree — and then ask what they will check tomorrow, and who will remember.
Defaults
Pipeline file = imports, one conf read, decorators. Nothing else.
EXPECTATIONS is a dict in the transform module, imported by the pipeline, the tests and the CI assertion.
@dp.table creates a streaming table; @dp.materialized_view creates a materialized view. sales_order_line is a batch rebuild, so it is a materialized view.
Source schema arrives through pipeline configuration, so the same file runs against seeded fixtures and against production.
Gotchas
Copying an old DLT example that uses @dlt.table for everything. Under Lakeflow pipelines @table now means STREAMING TABLE, so a batch rebuild silently becomes a streaming table with a checkpoint and different restart semantics — and the first symptom is a full refresh that processes zero rows.
Constraints written inline in the decorator. The test re-types the same SQL string, the two copies drift on the first edit, and both keep passing while guarding different rules.
A hardcoded source table in the pipeline file. The CI pipeline then reads production bronze, the seeded bad row never enters, every assertion passes, and the harness proves nothing at all.
Importing anything from pipelines/ into a test 'just to check it parses'. The import raises before collection and the whole test session errors out, which reads like a broken conftest and costs an hour.
An expectation is a filter with a counter attached. expect_all_or_drop removes the rows that fail and records how many. Nothing turns red. That inverts the usual risk: the dangerous defect is not a rule that fails, it is a rule that succeeds more often than it should, quietly deleting valid data.
Two tests, both pure pytest, both sub-second. The first is a corpus of rows that are legitimate business data and must survive every rule:
pythontests/test_expectations.py — no rule may drop a valid row
import pytest
from dwh.pipelines.transforms.silver_sales_order import EXPECTATIONS
BASE = dict(
order_id="4711", line_number=1, currency="EUR", country="DE",
status="SHIPPED", quantity=2.0, unit_price=125.00,
order_ts="2026-07-01 09:00:00", amount_eur=250.00,
)
MUST_PASS = [
("cancelled line, zero value", {**BASE, "status": "CANCELLED", "amount_eur": 0.00}),
("swiss order converted", {**BASE, "country": "CH", "currency": "CHF",
"quantity": 10.0, "unit_price": 12.00,
"amount_eur": 112.80}), # 10 x 12.00 x 0.94
("oldest permitted order", {**BASE, "order_ts": "2020-01-01 00:00:00"}),
("dutch order still open", {**BASE, "country": "NL", "status": "OPEN"}),
]
@pytest.mark.parametrize("name,predicate", sorted(EXPECTATIONS.items()))
@pytest.mark.parametrize("case,row", MUST_PASS, ids=[c for c, _ in MUST_PASS])
def test_no_expectation_drops_a_valid_row(spark, name, predicate, case, row):
kept = spark.createDataFrame([row]).where(predicate).count()
assert kept == 1, f"{name} drops a valid row ({case}); predicate: {predicate}"
The second is the mirror. A rule that rejects nothing is not protection, it is decoration — and it is indistinguishable from protection until the day it is needed.
pythontests/test_expectations.py — every rule must reject something
MUST_FAIL = {
"amount_positive": {**BASE, "status": "SHIPPED", "amount_eur": 0.00},
"known_country": {**BASE, "country": "FR"},
"order_not_ancient": {**BASE, "order_ts": "2019-12-31 23:59:59"},
"status_domain": {**BASE, "status": "RETURNED"},
}
def test_every_rule_has_a_counterexample():
assert set(MUST_FAIL) == set(EXPECTATIONS), "a rule was added without a counterexample"
@pytest.mark.parametrize("name,row", sorted(MUST_FAIL.items()))
def test_expectation_rejects_its_counterexample(spark, name, row):
kept = spark.createDataFrame([row]).where(EXPECTATIONS[name]).count()
assert kept == 0, f"{name} no longer rejects the row it exists to reject"
Why here
Take the concrete case. Someone writes country IN ('DE', 'AT', 'NL') — Switzerland omitted, because CH was added to the business after the list was typed. Every Swiss order line is dropped at silver. No job fails. No alert fires.
The event log recorded it faithfully, as known_country: failed_records = 1284, in a table nobody had queried since the pipeline was built. What the business saw was Swiss revenue at zero for six days, and the first person to notice was a sales lead, not an engineer. The MUST_PASS corpus above turns that into a red build in under a second, because country: "CH" is in it.
Doing it
Add a MUST_PASS row for every value the business actually has: all four countries, all three statuses, the cancelled-with-zero case, and the exact date boundary.
Add the counterexample in the same commit as the rule. A rule without one is not reviewable — nobody can tell what it is meant to catch.
Write NULL handling explicitly. amount_eur > 0 drops every NULL silently, because in SQL three-valued logic the predicate is neither true nor false and the row is not retained.
Grep the event log for rules with zero failed records over ninety days and interrogate each one. Either the data is perfect or the rule is pointed at the wrong column.
Embedding it
Give the team the Swiss story with the numbers and let them find which rule caused it. Five minutes, and the class of bug is understood for good.
Have each engineer add one MUST_PASS row for a case they have personally seen in the source. Their cases are better than yours — they have read the data.
Make 'where is the counterexample?' the standard review comment on any pull request touching EXPECTATIONS, and stop reviewing the rule text yourself once they are asking it.
Run a five-minute kata: break one rule, run pytest, read which assertion caught it. Repeat with a rule that no test covers — that gap is the next ticket.
Defaults
Every rule has at least one row that must pass and exactly one row that must fail.
The MUST_PASS corpus covers the real domain: DE, AT, CH, NL; OPEN, SHIPPED, CANCELLED; the 2020-01-01 boundary.
expect_all warns and keeps; expect_all_or_drop removes; expect_all_or_fail stops the update. Pick per rule and write down why.
Rules that can silently delete data are drop rules and get a MUST_PASS row. Rules that are advisory are warn rules and do not.
Gotchas
NULLs. amount_eur > 0 OR status = 'CANCELLED' evaluates to NULL when amount_eur is NULL, and a NULL predicate keeps nothing — so a nullable column turns a value rule into a completeness rule nobody wrote. You notice it as a row count that drops the week the source starts sending nulls, with every rule reporting a failure you did not intend to write. Decide which one you meant and say so: amount_eur IS NULL OR amount_eur > 0, or the opposite.
Using expect_all when you meant expect_all_or_drop. The rows are counted and then written anyway, so the event log shows failures while the bad rows sit in silver — and everyone reading the dashboard believes the failures were dropped.
A rule with zero failed records for a year. It is either redundant or wrong, and it costs nothing to keep, which is why nobody checks. status_domain over a column the source can only emit three values into guards a door that has no wall.
The blind spot no expectation covers: F.coalesce(F.col('fx_rate'), F.lit(1.0)) in the transform turns a missing FX rate into a rate of 1.0. amount_eur is still positive, so every rule passes, and CHF revenue is overstated by roughly seven percent. Catch it with a seeded row whose currency has no rate that day and an assertion on the VALUE, not on the rule.
Seed, run, assert
Unit tests prove the rules are right. They cannot prove the deployed pipeline is running them. That takes a real update against known data, and one query afterwards.
One job, three tasks strictly ordered — write known data, run the pipeline over it, assert what happened — plus a teardown carrying run_if: ALL_DONE so the run schema is dropped whether the assertion passed or not. The middle task is a pipeline_task with full_refresh: true, and the reason for that flag is in the gotchas — it is the difference between a real test and a vacuous one.
yamlresources/ci.yml — the pipeline gets its own schema, the job gets three tasks and a teardown
resources:
pipelines:
silver:
name: dwh-silver-${bundle.target}
catalog: ${var.catalog}
schema: ${var.schema} # PUBLISHED into. CI: run_<job_run_id>. prod: silver
edition: ADVANCED # expectations require it
continuous: false # triggered: the update has to END
configuration:
# READ from. A separate variable, not ${var.schema}: in prod the
# pipeline publishes to silver and reads bronze. They coincide only in
# CI, where the seed writes into the same run schema.
dwh.source_schema: ${var.catalog}.${var.source_schema}
libraries:
- glob:
include: ../src/dwh/pipelines/**
jobs:
ci_assert:
name: dwh-ci-assert-${bundle.target}
max_concurrent_runs: 1
tasks:
- task_key: seed
notebook_task:
notebook_path: ../src/dwh/ci/seed.py
source: WORKSPACE
base_parameters:
catalog: ${var.catalog}
schema: ${var.source_schema} # the seed writes the SOURCE tables
- task_key: pipeline
depends_on:
- task_key: seed
pipeline_task:
pipeline_id: ${resources.pipelines.silver.id}
full_refresh: true
- task_key: assert
depends_on:
- task_key: pipeline
notebook_task:
notebook_path: ../src/dwh/ci/assert_expectations.py
source: WORKSPACE
base_parameters:
pipeline_id: ${resources.pipelines.silver.id}
- task_key: teardown
depends_on:
- task_key: assert
run_if: ALL_DONE # the point: also runs when assert failed
notebook_task:
notebook_path: ../src/dwh/ci/teardown.py
source: WORKSPACE
base_parameters:
catalog: ${var.catalog}
schema: ${var.schema}
The seed is seven rows: six that are legitimate and exactly one that violates exactly one rule. That discipline is what lets the assertion be an equality rather than a greater-than.
pythonsrc/dwh/ci/seed.py — six good lines and one FR
catalog = dbutils.widgets.get("catalog")
schema = dbutils.widgets.get("schema")
spark.sql(f"CREATE SCHEMA IF NOT EXISTS {catalog}.{schema}")
lines = [
# order_id, line_number, currency, country, status, quantity, unit_price, order_ts
("4711", 1, "EUR", "DE", "SHIPPED", 2.0, 125.00, "2026-07-01 09:00:00"),
("4711", 2, "EUR", "DE", "OPEN", 1.0, 49.50, "2026-07-01 09:00:00"),
("4712", 1, "CHF", "CH", "SHIPPED", 10.0, 12.00, "2026-07-01 11:15:00"),
("4713", 1, "EUR", "NL", "CANCELLED", 3.0, 80.00, "2026-07-02 08:00:00"),
("4714", 1, "EUR", "AT", "SHIPPED", 1.0, 999.00, "2026-07-02 16:40:00"),
("4715", 1, "EUR", "DE", "SHIPPED", 4.0, 25.00, "2020-01-01 00:00:00"), # boundary: must PASS
("4716", 1, "EUR", "FR", "SHIPPED", 1.0, 10.00, "2026-07-03 10:00:00"), # the one bad row
]
# Declare the temporal and decimal columns as STRING/DOUBLE and CAST.
# createDataFrame refuses a str for a TimestampType or DateType field and a
# float for a DecimalType field — it raises TypeError before the seed writes
# anything, and the message reads like a Spark bug rather than a schema typo.
cols = "order_id string, line_number int, currency string, country string, \
status string, quantity double, unit_price double, order_ts string"
spark.createDataFrame(lines, cols) \
.selectExpr("order_id", "line_number", "currency", "country", "status",
"quantity", "unit_price",
"CAST(order_ts AS TIMESTAMP) AS order_ts") \
.write.mode("overwrite").saveAsTable(f"{catalog}.{schema}.sap_vbap")
rates = [("EUR", "2026-07-01", 1.000000), ("CHF", "2026-07-01", 0.940000),
("EUR", "2026-07-02", 1.000000), ("EUR", "2026-07-03", 1.000000),
("EUR", "2020-01-01", 1.000000)]
spark.createDataFrame(rates, "currency string, rate_date string, fx_rate double") \
.selectExpr("currency",
"CAST(rate_date AS DATE) AS rate_date",
"CAST(fx_rate AS DECIMAL(18,6)) AS fx_rate") \
.write.mode("overwrite").saveAsTable(f"{catalog}.{schema}.fx_rate")
Why here
Unit tests cannot see deployment. They cannot catch a pipeline deployed on the wrong product edition, a libraries glob that stopped matching after a file moved, a configuration key renamed on one side only, or an expectation that is present in the code and absent from the running graph because the module was never loaded. Those are all real, all cheap to cause, and all invisible until an update runs.
Product editions are CORE, PRO and ADVANCED, and expectations require ADVANCED. Selecting a cheaper edition on a pipeline that declares expectations fails with an explanatory error rather than silently skipping the checks — the good outcome, but only if someone reads it. Under deadline pressure the temptation is to delete the expectations rather than change one line of YAML.
Doing it
Deploy the CI bundle with run-specific schema variables — databricks bundle deploy -t test --var schema=run_$GITHUB_RUN_ID --var source_schema=run_$GITHUB_RUN_ID — so each run owns its pipeline, its seed and its output. The two are the same string only in CI; keep them two variables so prod cannot inherit the collapse.
Seed exactly one violation of exactly one rule. Two violations and every assertion becomes 'at least', which passes for the wrong reason.
Include the boundary rows: the 2020-01-01 order, the CANCELLED line at zero value, one row per country.
Add a teardown task with run_if: ALL_DONE so it fires on failure too, dropping the run schema. A leaked schema per CI run is visible within a week.
Embedding it
Have the team add the seventh row — the bad one — themselves, then watch the assert task go red. Owning the failure is what makes the harness theirs.
Ask them to break the harness deliberately: point the pipeline at production bronze and see that everything still passes. That is the lesson about parameterised sources, delivered by the harness rather than by you.
Give the teardown step to whoever complains first about test-schema clutter. It is fifteen minutes of work and it converts an irritation into ownership.
Let the CI job stay red for one afternoon rather than fixing it yourself. Who fixes it tells you who has adopted it.
Defaults
Triggered pipeline, full_refresh: true, three tasks, strict depends_on.
One schema per CI run; teardown runs even when the tests failed.
edition: ADVANCED declared explicitly rather than relied on as the default, so a copied pipeline cannot arrive on PRO unnoticed.
The seed is code in the repository, not a fixture table someone loaded once by hand.
Gotchas
Omitting full_refresh. A materialized view is updated incrementally and a streaming table remembers its checkpoint, so re-running against the same seed processes zero rows — every rule reports zero passed and zero failed, and the assertion passes vacuously. Guard it: assert passed + failed > 0 per rule.
Two CI runs sharing one pipeline. The second update overwrites the first's tables, the assert task reads the other branch's numbers, and the developer debugs their own correct code for an afternoon.
A seed row that violates two rules. The assertion for the second rule now expects 1 for reasons nobody wrote down, and when someone edits the seed a year later two tests fail with no explanation of what the row was for.
Teardown as a plain last task. Without run_if: ALL_DONE it only runs on success, so the schemas that leak are precisely the ones from failing runs, which are the ones you wanted to inspect and now cannot find among the hundred others.
Pointing dwh.source_schema at the schema the pipeline publishes into. In CI the seed writes into the run schema and the pipeline reads it back, so it works and the build is green. In production the pipeline publishes to silver and would read silver.sap_vbap, which does not exist — a TABLE_OR_VIEW_NOT_FOUND on the first prod deploy, weeks after CI proved nothing about it. Read schema and publish schema are two variables.
Expectation results are events, not table rows. Every update writes flow_progress events whose details column carries a JSON blob; the data quality metrics live under data_quality.expectations as an array of {name, dataset, passed_records, failed_records}. Unpacking it needs an explicit from_json schema — there is nothing to infer from.
sqlsrc/dwh/ci/metrics_sql.py — METRICS_SQL: one update's expectation metrics, per rule
WITH latest AS (
SELECT origin.update_id AS update_id
FROM event_log('<pipeline_id>')
WHERE event_type = 'create_update'
ORDER BY timestamp DESC
LIMIT 1
),
unpacked AS (
SELECT explode(
from_json(
details:flow_progress.data_quality.expectations,
'array<struct<name: string, dataset: string,
passed_records: bigint, failed_records: bigint>>'
)
) AS e
FROM event_log('<pipeline_id>')
WHERE event_type = 'flow_progress'
AND origin.update_id = (SELECT update_id FROM latest)
)
SELECT e.dataset,
e.name AS rule,
SUM(e.passed_records) AS passed,
SUM(e.failed_records) AS failed
FROM unpacked
GROUP BY e.dataset, e.name
ORDER BY e.dataset, e.name
The assertion is then arithmetic. known_country must have dropped exactly the one seeded row; every other rule must have dropped nothing; and every rule declared in EXPECTATIONS must appear, which is what catches a rule commented out during debugging and never restored.
pythonsrc/dwh/ci/assert_expectations.py
from dwh.ci.metrics_sql import METRICS_SQL
from dwh.pipelines.transforms.silver_sales_order import EXPECTATIONS
pipeline_id = dbutils.widgets.get("pipeline_id")
DATASET = "sales_order_line"
EXPECTED_FAILURES = {"known_country": 1} # the single seeded FR row
rows = spark.sql(METRICS_SQL.replace("<pipeline_id>", pipeline_id)).collect()
# Key by (dataset, name), then narrow. Expectation names are unique per dataset,
# not per pipeline: a dict keyed by name alone loses a rule silently the day a
# second dataset declares its own known_country.
by_key = {(r["dataset"], r["rule"]): r for r in rows}
by_rule = {rule: row for (dataset, rule), row in by_key.items() if dataset == DATASET}
assert by_rule, f"no expectation metrics for {DATASET} — did the pipeline actually run?"
assert set(by_rule) == set(EXPECTATIONS), (
f"declared {sorted(EXPECTATIONS)} but the update evaluated {sorted(by_rule)}"
)
for rule, row in sorted(by_rule.items()):
evaluated = row["passed"] + row["failed"]
assert evaluated > 0, f"{rule} saw zero rows — full_refresh did not happen"
expected = EXPECTED_FAILURES.get(rule, 0)
assert row["failed"] == expected, (
f"{rule}: expected {expected} failed record(s), got {row['failed']} "
f"of {evaluated} evaluated"
)
Why here
Without this query, 'the pipeline is green' means the update did not crash. It says nothing about whether the rules ran, how many rows each removed, or whether a rule exists at all — the three facts that separate a pipeline that works from a pipeline you can hand over.
It also gives the drop a direction. failed_records trending upward on known_country means the source started sending a new country: a business event, arriving through a data quality metric, days before anyone files a ticket about missing revenue.
Doing it
Assert equality on failed_records, never a threshold. A threshold passes when the number is wrong in the direction you did not think about.
Assert that the set of rules in the log equals the set of keys in EXPECTATIONS. This is the only check that catches a deleted rule.
Key the metrics by (dataset, name), not by name alone — expectation names are unique per dataset, and the second table with a known_country rule silently overwrites the first in a dict keyed by name.
In production, chart failed_records per rule per day and alert on the change, not on the level. The level is a policy decision; the change is an event.
Publish the production event log to Unity Catalog and grant SELECT to the on-call group. The default log is queryable by the pipeline owner only, which during an incident is the person on holiday.
Embedding it
Have the team write the from_json schema themselves against a real update. Ten minutes with the raw details column teaches more about the pipeline runtime than any diagram.
Ask them to predict the six numbers this query will return before running it. Wrong predictions are exactly where their mental model of the pipeline is wrong.
Hand the production event-log dashboard to the team in week one and stop citing it yourself. If they are reading it without you by the mid-point, the observability handover is done.
When an expectation fires in production, let the team run the query and explain the number to the business. That is the moment data quality stops being an engineering concern.
Defaults
Exact-equality assertions, one seeded violation, one expected count.
Set equality between declared rules and evaluated rules, every run.
A passed + failed > 0 guard on every rule, so a no-op update cannot pass.
Production event log published to Unity Catalog, granted to a group, never dropped.
Metrics keyed by dataset and name together, from the first pipeline onward.
Gotchas
Taking 'the latest update' without pinning the update id. Two runs in flight and the assertion reads the other one's metrics — a flaky test whose flakiness is caused by the test, not the code, which is the hardest kind to diagnose.
Inferring the JSON schema instead of declaring it. from_json with a wrong or partial schema returns NULLs rather than an error, SUM over NULLs returns NULL, and NULL == 0 is NULL — so the assertion neither passes nor fails in the way you expect and the build goes green on nothing.
Dropping the event log to reclaim storage. It is small and it is the only history you have; the day you need it is the day the auditor asks how long the number has been wrong.
Asserting failed_records >= 1. It passes when the rule dropped four hundred rows instead of one, which is precisely the over-strict failure the whole route exists to catch.
Failure semantics
Whether an update ends is not a scheduling preference. It decides whether a failing rule produces a red build, a stale table, or a retry loop nobody is watching.
A triggered pipeline refreshes its datasets once and stops. A continuous pipeline keeps running and processes data as it arrives. The scheduling difference is obvious; the failure difference is not, and it is what decides whether any of this route applies.
What happens when an expectation with a fail action is violated
Triggered
Continuous
The update
fails and terminates
the flow fails and is retried automatically
The job task
fails — the build goes red
there is no completion for a downstream task to wait for
Downstream tables
keep yesterday's data: stale, but internally consistent
partially updated, and the boundary moves while you look at it
What you notice
an alert, within minutes
rising cost and a pipeline that has been RUNNING for a suspiciously long time
Testable this way
yes — seed, run, assert
no — nothing ends, so nothing can be asserted after it
A deterministic data defect in continuous mode is the bad case: the record that violates the rule is still there on the retry, so it fails again, and the loop is paid for by the hour. The failure is not hidden — it is in the event log every time — but nothing pages anyone, because from the outside the pipeline is running.
Why here
Teams choose continuous for freshness and inherit an operational model they did not choose: every retry costs compute, no retry produces an alert, and the SLA breach is found by whoever opens the dashboard rather than by monitoring.
For this warehouse the freshness requirement is a daily SAP extract and a webshop feed. Triggered, on a schedule, with a 26-hour freshness SLA asserted separately — see the observability route — is both cheaper and more testable. Continuous is a choice to be argued for on a named requirement, not a default to drift into.
Doing it
Declare continuous: false in the bundle for every pipeline in this warehouse, and require a written requirement to change it.
Use fail actions on rules where processing wrong data is worse than processing none — currency conversion and the country domain. Use drop where a bad row is a nuisance and stale totals are worse.
If a pipeline must be continuous, alert on flow retry counts from the event log, because the pipeline's own state will read RUNNING throughout.
Alert on the job, not on the task — a triggered pipeline that fails at task two of five should page once, with the chain's name on it.
Embedding it
Ask the team what happens to yesterday's dashboard when tonight's update fails. If they do not know, that is the runbook's first paragraph and they should write it.
Run the failure once in test: seed a row that violates a fail rule, watch the update stop, and check what the gold tables show. Everyone then has a first-hand answer to the question the business will ask.
When someone proposes continuous, ask which decision gets made faster because the data is minutes old rather than hours. If nobody can name one, the requirement does not exist yet.
Give the retry-cost query to whoever owns the budget. Cost is the fastest teacher of failure semantics.
Defaults
Triggered by default; continuous only against a written latency requirement with a named owner.
continuous: false stated explicitly in the bundle, not inherited from a default.
Fail actions on correctness rules, drop actions on hygiene rules, warn actions on anything you are still calibrating.
Validation datasets live in a separate pipeline from the ones that feed gold, so a failing check does not stop the warehouse — orchestrate the two with a job.
Gotchas
Flipping a pipeline to continuous for a latency experiment and leaving it. The CI job's assert task now has nothing to wait against, and the harness degrades to 'the pipeline started' without anybody editing a test.
A fail expectation on a deterministic defect in a continuous pipeline. It retries forever, the pipeline reports RUNNING, the bill grows, and the only symptom anyone sees is a table that stopped moving.
Assuming a triggered failure leaves gold half-written. It does not — the update stops and the previous version stands — which is good, and which means the real risk is silent staleness. That is a freshness check's job, not an expectation's.
Deploying the pipeline on CORE or PRO. Expectations require ADVANCED, and the deploy fails with an explanatory error — which is the right behaviour, and which someone under deadline pressure resolves by deleting the expectations rather than by fixing one line of YAML.
The generic platform material lives in the official docs and is not restated here. These are the pages worth having open while you work through this route.