Crosshire
Written concepts
Engagement20 Written concepts · 3 of 3

The chapter that gets violated

Test data is one chapter of the anonymisation concept and the one most likely to be broken by a well-meaning engineer at 17:00 on a Thursday. It gets its own page for that reason.

Stand 29.07.2026

#Never a production copy in dev

What
The rule is one sentence: no production data in dwh_dev or dwh_test — not sampled, not hashed, not 'only the one table'. The rest of the chapter is the machinery that lets the rule survive contact with a real bug.
What replaces it is the factory the PySpark tests already use, scaled up and run as a seed job into dwh_test.fixtures:
Why
The request always arrives in the same shape — we cannot reproduce this without real data, just this once, just this table — and it is a reasonable request from a good engineer, which is exactly why it succeeds. A copy taken once exists forever, in a catalog with development grants, inside whatever backup schedule dev happens to have.
The second argument persuades engineers rather than lawyers: production data is poor test data. It contains no CANCELLED line with a negative amount, no customer whose segment changed twice on one day, no order dated 2019. Those are precisely the rows the assertions need, and a factory can be told to produce them.
How
python
tests/factories.py — the customer factory, next to the order_line_rows the PySpark route already defines
from typing import Any

from pyspark.sql.types import StringType, StructField, StructType

# RFC 2606 reserves example.com, so a stray test mail can never reach an inbox.
# The national regulator reserves a number block per area code for drama and
# testing. Both live in one constant each: that is what lets the privacy proof
# below stay two LIKEs instead of a regex nobody maintains.
TEST_DOMAIN = "example.com"
DRAMA_BLOCK = "+49 30 23125"

CUSTOMER_SCHEMA = StructType([
    StructField("customer_id",      StringType(), False),
    StructField("customer_name",    StringType(), False),
    StructField("email",            StringType(), True),
    StructField("phone",            StringType(), True),
    StructField("street",           StringType(), True),
    StructField("postal_code",      StringType(), True),
    StructField("city",             StringType(), True),
    StructField("country",          StringType(), False),
    StructField("customer_segment", StringType(), False),
])

# A boring customer: country in scope, segment set, contact details
# unreachable by construction. Each argument to the factory is a delta from it.
_CUSTOMER: dict[str, Any] = {
    "customer_id":      "C-0001",
    "customer_name":    "Muster Handel GmbH",
    "email":            f"kunde0001@{TEST_DOMAIN}",
    "phone":            f"{DRAMA_BLOCK}001",
    "street":           "Teststrasse 1",
    "postal_code":      "10115",
    "city":             "Musterstadt",
    "country":          "DE",
    "customer_segment": "B",
}


def customer_rows(*overrides: dict[str, Any]) -> list[dict[str, Any]]:
    """One row per argument; each argument is the delta from a valid customer."""
    return [{**_CUSTOMER, **o} for o in (overrides or ({},))]
python
tests/seed_fixtures.py — the same factory, scaled up; the catalog is a parameter, never a literal
import random

from tests.factories import CUSTOMER_SCHEMA, DRAMA_BLOCK, TEST_DOMAIN, customer_rows

COUNTRIES = ["DE", "AT", "CH", "NL"]     # business rule 3
SEGMENTS = ["A", "B", "C"]


def seed_customers(spark, catalog: str, n: int, seed: int = 42) -> int:
    """Seed <catalog>.fixtures.customer from the unit tests' own factory.

    Returns rows written. Only ever pointed at dwh_test.
    """
    rnd = random.Random(seed)
    rows = customer_rows(*(
        {
            "customer_id":      f"C-{i:05d}",
            "customer_name":    f"Muster Handel {i:05d} GmbH",
            "email":            f"kunde{i:05d}@{TEST_DOMAIN}",
            "phone":            f"{DRAMA_BLOCK}{i % 1000:03d}",
            "street":           f"Teststrasse {i % 200 + 1}",
            "postal_code":      f"{rnd.randrange(10_000, 99_999)}",
            "country":          rnd.choice(COUNTRIES),
            "customer_segment": rnd.choice(SEGMENTS),
        }
        for i in range(n)
    ))

    # Explicit schema, as everywhere else: inference is how a test ends up
    # exercising a type the warehouse does not have.
    df = spark.createDataFrame(rows, CUSTOMER_SCHEMA)
    df.write.mode("overwrite").saveAsTable(f"{catalog}.fixtures.customer")
    return df.count()
sql
tests/privacy/no_production_data_outside_prod.sql — zero rows means pass
-- Every address in a non-production catalog must sit on the reserved test
-- domain and every number in the regulator's drama block. Anything else is a
-- production copy or a hand-typed real customer, and both are the same finding.
--
-- NULL is a finding too, deliberately. The factory always fills both columns,
-- so a NULL here means either a row nobody generated or a column mask standing
-- between this principal and the data. A mask that returns NULL would
-- otherwise empty the result and let the check pass having read nothing.
--
-- The two literals below repeat TEST_DOMAIN and DRAMA_BLOCK from
-- tests/factories.py. Change one and this goes red, which is the intent.
SELECT 'dwh_dev' AS catalog_name, customer_id, email, phone
FROM dwh_dev.gold.dim_customer
WHERE email IS NULL OR email NOT LIKE '%@example.com'
   OR phone IS NULL OR phone NOT LIKE '+49 30 23125%'
UNION ALL
SELECT 'dwh_test' AS catalog_name, customer_id, email, phone
FROM dwh_test.gold.dim_customer
WHERE email IS NULL OR email NOT LIKE '%@example.com'
   OR phone IS NULL OR phone NOT LIKE '+49 30 23125%'
Doing it
  • Seed dwh_test.fixtures from the factories before the suite runsDrop the per-run schema in a step that executes even when the tests failed.
  • Give every generated address the reserved domain and every generated number the regulator's drama blockThe proof then stays two LIKEs rather than a regex nobody maintains.
  • When a bug genuinely needs production characteristics, copy the shape and not the rowsProfile row counts, cardinalities and null rates, then reproduce that distribution synthetically.
  • One factory module, imported by both the unit tests and the seed jobTwo generators means two sets of fixtures, and the suite then green-lights a shape the seed job cannot actually write.
  • Give each row-level business rule a named overrideA country outside DE/AT/CH/NL (rule 3), a zero unit_price on a non-cancelled line (rule 5), an order dated 2019 (rule 6), a customer_id no dimension row resolves (rule 7) — so no assertion is written without a row that fails it.
Embedding it
  • The first time someone asks for a production copy, do not refusePair for thirty minutes and build the failing row synthetically. They are faster on the second bug and stop asking by the fourth.
  • Have the team add every production incident's shape to the factory as a named builderThe factory becomes the team's museum of things that actually went wrong.
  • Ask in review where the test data came fromIt is a one-word answer and a very strong signal.
Defaults
  • One factory module, seeded per run, dropped per run
  • Reserved test domain on every address and the regulator's drama block on every numberThe check then stays two LIKEs.
  • Edge cases generated deliberately, one per business rule
  • The privacy proof runs nightly against every non-production catalog, not only in CI
Gotchas
  • 'Just one table, and it's hashed'Hashing is pseudonymisation: it is still production personal data in a catalog with development grants, and the anonymisation concept has just been made false in writing.
  • The one-off copy taken during go-live weekendNothing about it is one-off — it is found two years later by a classification scan, in a schema whose owner has left, and it has been in every dev backup since.
  • Synthetic data with only the happy pathThe suite is green, and the first CANCELLED line with a negative amount reaches gold in production because no test ever saw one.
  • Factories that drift from the schemaThe generator emits a column the table dropped, the seed job dies at 03:00, and the failure reads as a pipeline bug for the first hour. Assert the factory's keys against the target table's columns in a three-line test and it fails at merge instead.
Platform facts on this page verified 29.07.2026 against the official documentation. Volatile claims are anchored to the currency register. This is section 3 of 3 in 20 Written concepts.