Four are transcriptions of something already declared — the key, the grain, the foreign keys, the country list. Four are the seven business rules made executable. Between them they cover every way this warehouse has actually been wrong.
The first two are one query applied to two tuples: the SCD2 key of gold.dim_customer above, and the declared grain of gold.fct_order_line.
Why
These four make a number wrong without making anything fail. A duplicated fact line double-counts revenue and no job goes red. An orphaned line drops out of every dashboard that joins to dim_customer, so the total comes back smaller — unnoticed, because nobody knows what it should be.
They are also the four you can generate. Key, grain and foreign keys are already declared on the table; the domain is already in the business rules. The assertion is a transcription, not an invention — which is why a team handed this is productive the same afternoon.
How
sqltests/sql/fct_order_line_grain.sql — grain is 'one row per order line'
SELECT
order_id,
line_number,
COUNT(*) AS rows_at_grain,
COUNT(DISTINCT order_line_sk) AS distinct_sks
FROM gold.fct_order_line
GROUP BY order_id, line_number
HAVING COUNT(*) > 1
distinct_sks is not decoration. If it is 1, the load ran twice and inserted the same surrogate key twice — an idempotency bug. If it equals rows_at_grain, the hash inputs differ, so the business key is not what the model says it is. Two causes, told apart by one column in the failure output.
The orphan check is the one written wrong most often. UC foreign keys are informational, so nothing prevents a fact row pointing at a customer version that does not exist. Rule 7 says every line resolves to a current customer or to the unknown member — and since customer_sk derives from customer_id alone, resolving it means joining on the interval:
sqltests/sql/fct_order_line_orphans.sql
SELECT
f.order_line_sk,
f.customer_sk,
f.order_ts
FROM gold.fct_order_line f
LEFT JOIN gold.dim_customer d
ON d.customer_sk = f.customer_sk
AND f.order_ts >= d._tech_valid_from
AND (d._tech_valid_to IS NULL OR f.order_ts < d._tech_valid_to)
-- IS DISTINCT FROM, not <>: a NULL customer_sk must be REPORTED, and
-- f.customer_sk <> 'UNKNOWN' is NULL on exactly those rows.
WHERE f.customer_sk IS DISTINCT FROM 'UNKNOWN'
AND d.customer_sk IS NULL
Without the two interval predicates this passes on a late-arriving dimension — the customer exists, just not yet at order_ts — while the report that joins on the interval silently loses the row. The assertion has to join the way the report joins.
SELECT
country,
COUNT(*) AS affected_rows,
MIN(customer_id) AS example_customer
FROM gold.dim_customer
WHERE country IS NULL
OR country NOT IN ('DE', 'AT', 'CH', 'NL')
GROUP BY country
Doing it
Generate the uniqueness, grain and orphan assertions from the declared PK/FK metadataHand-write only the interval predicates.
Put the UNKNOWN-member exclusion in the WHERE clause explicitlyThe reader then sees that unresolved rows are a policy and not a leak.
Add the diagnostic column — distinct_sks, example_customer — to every grouped assertion
Assert the domain against the same literal list the transform usesIf the list appears twice in the repository it will disagree with itself within a quarter.
Embedding it
Give them the grain assertion and ask them to break it two waysProducing both the double-load and the wrong-hash failure teaches the diagnostic column without being told.
In review, ask what happens to this assertion when a dimension arrives lateFastest way to find out whether the author has internalised the interval join.
Let the team write the generator over the PK/FK metadataAn afternoon, and the suite stops being something you gave them.
Defaults
Uniqueness, grain and orphans are generated from declarations, not typed per table
Every orphan check joins on the validity interval, with an exclusive end
Domain lists live in one place, referenced by both transform and assertion
The UNKNOWN member is excluded explicitly, never by a NOT NULL on the key
Gotchas
Trusting a declared PRIMARY KEYUC constraints are informational — declare one, insert a duplicate, it succeeds. With RELY the optimizer believes the declaration and can eliminate a join, so a violation returns wrong results faster than before you declared it.
An orphan check joining on customer_sk aloneIt stays green while the report that joins on the interval quietly drops every late-arriving row — so assertion and dashboard disagree, and the assertion is the one that is wrong.
Checking COUNT(*) on the surrogate key instead of the business keyA key that normalises differently — '4711' versus ' 4711' — hashes to two different sks, and the check is blind to the case it was written for.
A grain assertion without the distinct_sks columnYou learn there are duplicates, then spend an hour deciding whether the load is not idempotent or the key is wrong. The answer was one column away.
SCD2 correctness is two assertions and neither alone suffices. The first says no customer has two versions covering the same instant; the second says exactly one version is open, flagged, and the same row.
Why
These four are the ones a stakeholder can read. 'Revenue in silver and gold agree to the cent, every month, and no line was valued without a rate or counted without its order header' is a sentence a finance lead can sign off. 'The FK constraint is satisfied' is not.
They also close the loop the layer contract opened. Silver is source-shaped, gold is modelled, and between them sits a transform someone wrote. The reconciliation makes 'gold is derived from silver' a checked claim rather than an architectural intention.
How
sqltests/sql/dim_customer_scd2_no_overlap.sql
-- Intervals are [_tech_valid_from, _tech_valid_to) — end is EXCLUSIVE.
-- The asymmetric join reports each offending pair once, not twice.
SELECT
a.customer_id,
a._tech_valid_from AS earlier_from,
a._tech_valid_to AS earlier_to,
b._tech_valid_from AS later_from
FROM gold.dim_customer a
JOIN gold.dim_customer b
ON b.customer_id = a.customer_id
AND b._tech_valid_from > a._tech_valid_from
WHERE a._tech_valid_to IS NULL
OR b._tech_valid_from < a._tech_valid_to
sqltests/sql/dim_customer_scd2_one_current.sql
SELECT
customer_id,
COUNT_IF(_tech_is_current) AS flagged_current,
COUNT_IF(_tech_valid_to IS NULL) AS open_intervals,
COUNT_IF(_tech_is_current AND _tech_valid_to IS NULL) AS both_on_same_row
FROM gold.dim_customer
GROUP BY customer_id
HAVING COUNT_IF(_tech_is_current) <> 1
OR COUNT_IF(_tech_valid_to IS NULL) <> 1
OR COUNT_IF(_tech_is_current AND _tech_valid_to IS NULL) <> 1
The third HAVING arm earns its keep. A partially applied update leaves the flag on the old row and the open interval on the new one — one of each, so the first two arms pass, while queries filtering on _tech_is_current return the superseded version and queries filtering on _tech_valid_to IS NULL return the right one. Two dashboards, two answers, no error anywhere.
Currency conversion is checked row by row against rules 2 and 5:
sqltests/sql/fct_order_line_amount_eur.sql
SELECT
order_line_sk,
currency,
fx_rate,
quantity,
unit_price,
amount_eur,
status,
ROUND(quantity * unit_price * fx_rate, 2) AS expected_eur
FROM gold.fct_order_line
WHERE amount_eur IS DISTINCT FROM ROUND(quantity * unit_price * fx_rate, 2)
OR (currency = 'EUR' AND fx_rate IS DISTINCT FROM 1)
OR (status IS DISTINCT FROM 'CANCELLED' AND COALESCE(amount_eur, 0) <= 0)
OR currency IS NULL
OR currency NOT IN ('EUR', 'CHF')
IS DISTINCT FROM rather than <> throughout: a NULL amount_eur compared with <> yields NULL, the row is not returned, and the assertion passes on the rows where conversion failed hardest.
The fourth recomputes from silver.fx_rate — (currency, rate_date, fx_rate), one row per currency per day — rather than reusing the fx_rate gold already stored. Reusing gold's own rate would make the test tautological: it would prove that multiplication works.
sqltests/sql/reconcile_revenue_silver_gold.sql — rules 1 and 2
WITH line AS (
-- LEFT JOIN to the header, for the same reason as the rate join below: an
-- order line whose header has not landed yet (late CDC on bronze.sap_vbak)
-- would be deleted from the SILVER side by an inner join, and if gold's build
-- joins the same way it is deleted from BOTH sides equally — delta 0,00 on a
-- month that is short a whole order.
SELECT
l.order_id,
l.line_number,
l.quantity,
l.unit_price,
l.currency,
l.status,
-- the header carries the authoritative order date; fall back to the line's
-- own timestamp so a headerless line still lands in a month. If it has
-- neither, its month is NULL — a bucket the FULL OUTER JOIN below keeps
-- visible instead of deleting.
COALESCE(o.order_ts, l.order_ts) AS order_ts,
o.order_id AS header_order_id
FROM silver.sales_order_line l
LEFT JOIN silver.sales_order o ON o.order_id = l.order_id
),
silver_month AS (
SELECT
DATE_TRUNC('MONTH', x.order_ts) AS month,
-- EUR is the base currency and is absent from the ECB file, so it is 1 by
-- definition. Rule 2 rounds PER LINE; rounding the sum drifts by cents.
SUM(ROUND(x.quantity * x.unit_price *
CASE WHEN x.currency = 'EUR' THEN CAST(1 AS DECIMAL(18,6))
ELSE r.fx_rate END, 2)) AS silver_eur,
-- a missing rate makes BOTH sides smaller by the same amount, so the delta
-- stays zero. This column is the only place that defect is visible.
COUNT_IF(x.currency IS DISTINCT FROM 'EUR'
AND r.fx_rate IS NULL) AS lines_without_rate,
-- and the same argument for the header. Same shape, same silence.
COUNT_IF(x.header_order_id IS NULL) AS lines_without_header
FROM line x
LEFT JOIN silver.fx_rate r ON r.currency = x.currency
AND r.rate_date = DATE(x.order_ts)
WHERE x.status IS DISTINCT FROM 'CANCELLED'
GROUP BY 1
),
gold_month AS (
SELECT
DATE_TRUNC('MONTH', order_ts) AS month,
SUM(amount_eur) AS gold_eur
FROM gold.fct_order_line
WHERE status IS DISTINCT FROM 'CANCELLED'
GROUP BY 1
)
SELECT
COALESCE(s.month, g.month) AS month,
s.silver_eur,
g.gold_eur,
COALESCE(g.gold_eur, 0) - COALESCE(s.silver_eur, 0) AS delta_eur,
COALESCE(s.lines_without_rate, 0) AS lines_without_rate,
COALESCE(s.lines_without_header, 0) AS lines_without_header
FROM silver_month s
FULL OUTER JOIN gold_month g ON g.month = s.month
WHERE ABS(COALESCE(g.gold_eur, 0) - COALESCE(s.silver_eur, 0)) > 0.01
OR COALESCE(s.lines_without_rate, 0) > 0
OR COALESCE(s.lines_without_header, 0) > 0
Doing it
Assert both SCD2 properties, including the both-on-same-row armThe pair is the test; either alone is a false sense of coverage.
Use IS DISTINCT FROM in every predicate, without deciding case by case whether a column can be NULL
Recompute the reconciliation from silver.fx_rate, never from the rate the fact already storedLEFT JOIN it so an absent rate becomes a counted column rather than a deleted row.
LEFT JOIN every lookup the reconciliation walks through — the rate and the order header both — and count the missesAny inner join on the silver side deletes rows that gold's build also deletes, and symmetric deletion is a delta of zero.
Reconcile at month grain and also for the current month to dateA monthly-only reconciliation finds yesterday's break in four weeks.
Embedding it
Seed a customer whose flag and open interval disagree, then ask the team why two dashboards give two answersNobody forgets the third HAVING arm afterwards.
Have whoever wrote the transform explain why the reconciliation must not reuse gold's fx_rateIf they cannot, they will write the tautological version next time.
Ask the finance lead to state the tolerance they can live with, in eurosIt turns a parameter into an agreement.
Hand them a green reconciliation with an INNER JOIN and a deliberately missing monthLet them find it.
Then delete one row from silver.sales_order in the test catalog and re-run the INNER JOIN versionIt stays green, and the team discovers for themselves that a symmetric deletion is the one defect a delta can never show.
Defaults
Exclusive interval ends everywhere; assertion and transform share the convention or both are wrong
IS DISTINCT FROM by default in assertions
Reconciliations are FULL OUTER and independent of the value they check
Every lookup on the reconciliation's own side is a LEFT JOIN with a counted-miss column beside itThat column is surfaced in the zero-rows predicate.
Rules 1–7 map one-to-one onto assertionsA rule with no assertion is a comment.
Gotchas
An inclusive interval endIf _tech_valid_to equals the next version's _tech_valid_from and the join uses <=, every boundary instant matches two versions and every temporal join double-counts — small enough to look like rounding, large enough to fail an audit.
The SCD2 flag and the open interval drifting apartBoth single-property assertions pass, and the warehouse serves two different 'current' customers depending on which predicate the query used.
Rounding at the wrong pointRule 2 rounds per line; summing unrounded values and rounding the total drifts by cents per thousand lines — just inside a naive tolerance and just outside a finance reconciliation.
A percentage toleranceIt scales with the error it is meant to catch, so a conversion bug costing 0.3 % of revenue is invisible at every volume.
Reconciling against gold's own stored fx_rateGreen by construction, through a completely wrong rate table.
An INNER JOIN from silver.sales_order_line to silver.fx_rateThe ECB file has no EUR row — EUR is the base currency — so every euro line vanishes from the silver side, and a reconciliation between DE/AT/NL revenue and total revenue reports a delta of exactly the euro business. It looks like a catastrophic transform bug and is a join.
A missing rate for a real currencyIt removes the same lines from silver and from gold, the delta stays zero, and the test is green on a month that is short by one currency for one day. Only the lines_without_rate column sees it.
An INNER JOIN from silver.sales_order_line to silver.sales_orderExactly the same defect one join earlier, and the easier one to miss because a header always exists eventually: a line whose header is still in flight on sap_vbak drops out of the silver side, gold's build drops it too, and the reconciliation reports 0,00 for a month that is missing an entire order. lines_without_header is the only witness — and with the LEFT JOIN in place, a line with no header and no timestamp of its own lands in a NULL month rather than in the wrong one, which is why the FULL OUTER JOIN has to survive a NULL month key.