Foundation01 Architecture · 3 of 3

Environments and catalogs

Three environments, three catalogs, and one binding rule that makes 'could dev read production?' answerable with a yes or a no rather than with a discussion.

Stand 29.07.2026

#Catalog per environment

What

Each environment is its own Unity Catalog catalog. The layers are schemas inside it, so a fully qualified name differs between environments in exactly one place — the first segment.

Why

The alternative most teams reach for first is one catalog with the environment in the schema name — dwh.prod_gold, dwh.dev_gold. It looks equivalent and is not, for one reason: grants are given at catalog and schema level. In the single-catalog design, every grant is one careless wildcard away from spanning environments, and there is no structural boundary to stop it.

With a catalog per environment you get workspace-catalog binding: the development workspace can be bound so that dwh_prod is not merely ungranted but not reachable. 'Could someone in dev read production?' becomes a configuration you can show an auditor, rather than a review of every grant ever issued.

How

text

The topology

dwh_dev  /  dwh_test  /  dwh_prod
  |-- bronze      raw, append-only, technical metadata added
  |-- silver      cleansed, conformed, historised
  |-- gold        star schema + aggregates + metric views
  |-- fixtures    seeded test data          (dwh_test only)
  '-- ci          Volume for JUnit reports  (dwh_test only)

per-CI-run isolation:  dwh_test.run_<job_run_id>

Because only the first segment varies, the catalog is a deployment parameter and never a literal in code:

python

src/dwh/jobs/silver_customer.py — the catalog arrives from configuration

def run(spark, catalog: str) -> int:
    """Rebuild silver.customer from bronze. Returns rows written."""
    source = f"{catalog}.bronze.sap_kna1"
    target = f"{catalog}.silver.customer"

    df = clean_customer(spark.read.table(source))
    df.write.mode("overwrite").saveAsTable(target)
    return df.count()

Every CI run gets its own schema — dwh_test.run_<job_run_id> — so two pipelines running concurrently on two branches cannot see each other's tables. Without that, the second developer to push gets a failure caused by the first developer's data, and spends the afternoon debugging their own correct code.

Doing it

Create three catalogs before any table existsRetrofitting this means moving every table and re-pointing every report.
Bind each workspace to the catalogs it may see; do not rely on grants alone
Pass the catalog in as a bundle variable per targetNever a literal, never an environment sniff at runtime.
Give each CI run its own schema and drop it at the end of the runThe drop belongs in a step that runs even when the tests failed.

Embedding it

Ask the team to demonstrate that dev cannot read prod, rather than to assert itThe demonstration is the artefact — keep it, it is your access-review evidence later.
Have them find the hardcoded catalog names themselves with a grepThere are always some, and finding them is more instructive than being told.
Make the CI-schema cleanup step the team's, and let it fail onceA leaked schema per run is visible within a week and teaches the lesson permanently.

Defaults

Catalog per environment; layers as schemas; one metastore per region
Workspace-catalog binding on top of grants, not instead of them
Catalog as a deployment parameter, resolved at deploy time
Production objects owned by groups, never by individuals — covered in the Unity Catalog route
CI schemas are per-run and dropped in an always-runs step

Gotchas

Hardcoded catalog names that work in every environment the author testedThey fail in the one they did not — and the failure mode is not an error, it is a dev job quietly writing into dwh_prod because the string was right there in the notebook.
Retrofitting catalog-per-environment after go-liveEvery table moves, every grant is reissued, every report is re-pointed, and there is no partial rollout. Cost this at the start, when it is free.
CI runs sharing one schemaTwo branches collide, the failure looks like a data bug, and the developer debugs their own correct transformation for an afternoon before anyone suspects the harness.
Assuming binding is inheritedBinding is configured per workspace; a new workspace created later starts unbound, which is the moment the boundary you demonstrated in month one silently stops being true.