Test data is one chapter of the anonymisation concept and the one most likely to be broken by a well-meaning engineer at 17:00 on a Thursday. It gets its own page for that reason.
The rule is one sentence: no production data in dwh_dev or dwh_test — not sampled, not hashed, not 'only the one table'. The rest of the chapter is the machinery that lets the rule survive contact with a real bug.
What replaces it is the factory the PySpark tests already use, scaled up and run as a seed job into dwh_test.fixtures:
Why
The request always arrives in the same shape — we cannot reproduce this without real data, just this once, just this table — and it is a reasonable request from a good engineer, which is exactly why it succeeds. A copy taken once exists forever, in a catalog with development grants, inside whatever backup schedule dev happens to have.
The second argument persuades engineers rather than lawyers: production data is poor test data. It contains no CANCELLED line with a negative amount, no customer whose segment changed twice on one day, no order dated 2019. Those are precisely the rows the assertions need, and a factory can be told to produce them.
How
pythontests/factories.py — the customer factory, next to the order_line_rows the PySpark route already defines
from typing import Any
from pyspark.sql.types import StringType, StructField, StructType
# RFC 2606 reserves example.com, so a stray test mail can never reach an inbox.
# The national regulator reserves a number block per area code for drama and
# testing. Both live in one constant each: that is what lets the privacy proof
# below stay two LIKEs instead of a regex nobody maintains.
TEST_DOMAIN = "example.com"
DRAMA_BLOCK = "+49 30 23125"
CUSTOMER_SCHEMA = StructType([
StructField("customer_id", StringType(), False),
StructField("customer_name", StringType(), False),
StructField("email", StringType(), True),
StructField("phone", StringType(), True),
StructField("street", StringType(), True),
StructField("postal_code", StringType(), True),
StructField("city", StringType(), True),
StructField("country", StringType(), False),
StructField("customer_segment", StringType(), False),
])
# A boring customer: country in scope, segment set, contact details
# unreachable by construction. Each argument to the factory is a delta from it.
_CUSTOMER: dict[str, Any] = {
"customer_id": "C-0001",
"customer_name": "Muster Handel GmbH",
"email": f"kunde0001@{TEST_DOMAIN}",
"phone": f"{DRAMA_BLOCK}001",
"street": "Teststrasse 1",
"postal_code": "10115",
"city": "Musterstadt",
"country": "DE",
"customer_segment": "B",
}
def customer_rows(*overrides: dict[str, Any]) -> list[dict[str, Any]]:
"""One row per argument; each argument is the delta from a valid customer."""
return [{**_CUSTOMER, **o} for o in (overrides or ({},))]
pythontests/seed_fixtures.py — the same factory, scaled up; the catalog is a parameter, never a literal
import random
from tests.factories import CUSTOMER_SCHEMA, DRAMA_BLOCK, TEST_DOMAIN, customer_rows
COUNTRIES = ["DE", "AT", "CH", "NL"] # business rule 3
SEGMENTS = ["A", "B", "C"]
def seed_customers(spark, catalog: str, n: int, seed: int = 42) -> int:
"""Seed <catalog>.fixtures.customer from the unit tests' own factory.
Returns rows written. Only ever pointed at dwh_test.
"""
rnd = random.Random(seed)
rows = customer_rows(*(
{
"customer_id": f"C-{i:05d}",
"customer_name": f"Muster Handel {i:05d} GmbH",
"email": f"kunde{i:05d}@{TEST_DOMAIN}",
"phone": f"{DRAMA_BLOCK}{i % 1000:03d}",
"street": f"Teststrasse {i % 200 + 1}",
"postal_code": f"{rnd.randrange(10_000, 99_999)}",
"country": rnd.choice(COUNTRIES),
"customer_segment": rnd.choice(SEGMENTS),
}
for i in range(n)
))
# Explicit schema, as everywhere else: inference is how a test ends up
# exercising a type the warehouse does not have.
df = spark.createDataFrame(rows, CUSTOMER_SCHEMA)
df.write.mode("overwrite").saveAsTable(f"{catalog}.fixtures.customer")
return df.count()
sqltests/privacy/no_production_data_outside_prod.sql — zero rows means pass
-- Every address in a non-production catalog must sit on the reserved test
-- domain and every number in the regulator's drama block. Anything else is a
-- production copy or a hand-typed real customer, and both are the same finding.
--
-- NULL is a finding too, deliberately. The factory always fills both columns,
-- so a NULL here means either a row nobody generated or a column mask standing
-- between this principal and the data. A mask that returns NULL would
-- otherwise empty the result and let the check pass having read nothing.
--
-- The two literals below repeat TEST_DOMAIN and DRAMA_BLOCK from
-- tests/factories.py. Change one and this goes red, which is the intent.
SELECT 'dwh_dev' AS catalog_name, customer_id, email, phone
FROM dwh_dev.gold.dim_customer
WHERE email IS NULL OR email NOT LIKE '%@example.com'
OR phone IS NULL OR phone NOT LIKE '+49 30 23125%'
UNION ALL
SELECT 'dwh_test' AS catalog_name, customer_id, email, phone
FROM dwh_test.gold.dim_customer
WHERE email IS NULL OR email NOT LIKE '%@example.com'
OR phone IS NULL OR phone NOT LIKE '+49 30 23125%'
Doing it
Seed dwh_test.fixtures from the factories before the suite runsDrop the per-run schema in a step that executes even when the tests failed.
Give every generated address the reserved domain and every generated number the regulator's drama blockThe proof then stays two LIKEs rather than a regex nobody maintains.
When a bug genuinely needs production characteristics, copy the shape and not the rowsProfile row counts, cardinalities and null rates, then reproduce that distribution synthetically.
One factory module, imported by both the unit tests and the seed jobTwo generators means two sets of fixtures, and the suite then green-lights a shape the seed job cannot actually write.
Give each row-level business rule a named overrideA country outside DE/AT/CH/NL (rule 3), a zero unit_price on a non-cancelled line (rule 5), an order dated 2019 (rule 6), a customer_id no dimension row resolves (rule 7) — so no assertion is written without a row that fails it.
Embedding it
The first time someone asks for a production copy, do not refusePair for thirty minutes and build the failing row synthetically. They are faster on the second bug and stop asking by the fourth.
Have the team add every production incident's shape to the factory as a named builderThe factory becomes the team's museum of things that actually went wrong.
Ask in review where the test data came fromIt is a one-word answer and a very strong signal.
Defaults
One factory module, seeded per run, dropped per run
Reserved test domain on every address and the regulator's drama block on every numberThe check then stays two LIKEs.
Edge cases generated deliberately, one per business rule
The privacy proof runs nightly against every non-production catalog, not only in CI
Gotchas
'Just one table, and it's hashed'Hashing is pseudonymisation: it is still production personal data in a catalog with development grants, and the anonymisation concept has just been made false in writing.
The one-off copy taken during go-live weekendNothing about it is one-off — it is found two years later by a classification scan, in a schema whose owner has left, and it has been in every dev backup since.
Synthetic data with only the happy pathThe suite is green, and the first CANCELLED line with a negative amount reaches gold in production because no test ever saw one.
Factories that drift from the schemaThe generator emits a column the table dropped, the seed job dies at 03:00, and the failure reads as a pipeline bug for the first hour. Assert the factory's keys against the target table's columns in a three-line test and it fails at merge instead.