Each has a table of contents you can copy. Fixing the contents in advance means the argument about scope happens once, in week one, instead of in every review for the rest of the project.
The developer concept answers one question: how does work get built here. It is the shortest of the three, because most chapters link into this handbook rather than restating it.
Why
It stops the same three arguments recurring: where does this table belong, what is it called, does it need a test. Twenty minutes each, in every review, for the life of the project. A written answer turns an argument into a lookup a junior reviewer can perform on a senior engineer's pull request.
The second payoff is onboarding. A new engineer with this document and the canonical model contributes within a week; without it the tacit rules are learned one rejected pull request at a time, and each rejection costs two people a day.
How
textEntwicklerkonzept — full table of contents
owner: lead engineer
reviewers: two engineers + platform/architecture
timebox: 3 days initial, half a day per quarter
proof: tests/standards/
1 Scope and non-goals
2 Architecture
2.1 Layer contract: what may enter and leave bronze / silver / gold
2.2 Topology: dwh_dev / dwh_test / dwh_prod, layers as schemas
3 Modelling
3.1 Star schema in gold; silver stays source-shaped
3.2 Surrogate keys: sha2 over normalised business keys
3.3 SCD2 interval semantics, exclusive end, the UNKNOWN member
4 Naming
4.1 Prefixes: dim_ fct_ agg_ brg_ stg_ mv_ _tech_
4.2 ^[a-z][a-z0-9_]*$, ASCII only; reserved words (sales_order, not order)
5 Code structure
5.1 The transform is a pure function; IO at the edges
5.2 Module layout under src/dwh/
5.3 Workspace files and the sys.path bootstrap — no wheel
6 Testing
6.1 The four kinds: PySpark unit, SQL assertion, pipeline, warehouse invariant
6.2 What every merge runs; what runs nightly
7 Delivery
7.1 Bundle targets dev / test / prod; bundle validate first in CI
7.2 Branching, review rules, OAuth service principals
8 Definition of done
9 Not testable: on-call rota, escalation, team topology
Chapters 2 to 7 are one to two pages each. Chapter 8 is the one that actually gets read, so it is written as conditions a reviewer can check without judgement:
Grain sentence in the table COMMENT, deployed in DDL.
A uniqueness assertion and an orphan assertion exist in tests/sql/ for every new gold table.
The transform is importable and tested with no pipeline decorator in the call path.
bundle validate passes on the test target.
Identifiers match the regex; no new reserved word used unquoted.
The runbook entry answers the re-run question: safe blind, or intervention first.
Doing it
Draft it in week one, before there are tables to argue about, and keep it near fifteen pages
Link out to docs.databricks.com for anything genericThe concept records decisions, not platform behaviour.
Write chapter 8 so that at least one recent pull request would have failed itA definition of done nothing has ever failed is a description, not a gate.
Land each rule and its CI check in the same pull request, so no rule is merged unenforced
Embedding it
Have two engineers draft chapters 3 and 4 and defend them to the rest of the teamThe defence is where the modelling disagreements surface.
Ask the newest joiner to list what they still had to ask a colleague after reading itThat list is the next revision.
When a review comment repeats itself for the third time, stop commenting and make the team add it to chapter 8
Defaults
Fifteen pages: links out, decisions in
Every rule in chapters 4 to 7 has a CI check, or is marked as a convention
Chapter 8 is checkable by someone who was not in the room when the rule was made
Concept and code live in one repository and move in one pull request
Gotchas
A developer concept that restates platform documentationIt reaches sixty pages, nobody reads past chapter 2, and the project's own decisions are buried behind material that is a search away and better maintained.
A definition of done written so it cannot fail'Code is tested' passes everything; 'every new gold table has a grain comment and a uniqueness assertion in tests/sql/' fails something this week.
Writing chapter 5 without chapter 6Structure decisions not made for testability produce a document that mandates pure functions over a codebase where every transform sits behind a decorator — untrue on the day it ships.
Reusing the last client's document with a find-and-replace on the nameIt survives until someone asks why the concept describes a Data Vault this warehouse does not have, and after that no chapter of it is trusted.
#Berechtigungskonzept — access as policy, not a spreadsheet
What
This is the document a German client asks for by name, often before the first table exists, and usually the one that stalls — because it gets written as a matrix of individual grants that is out of date the day it is signed.
Why
The failure mode is the grant spreadsheet: 140 rows of principal × object, correct on the day of sign-off, decaying from the first new table onward. Two quarters later nobody can say whether it describes production, so the access review consists of reading the spreadsheet — reviewing the document against itself.
Written as attributes, the same document is one page and stays true. Four groups, four tags, three policies, and every table created afterwards is covered the moment it is tagged — including the tables nobody has designed yet.
How
textBerechtigungskonzept — full table of contents
owner: data owner (Vertrieb) - a named person, drafted by the lead engineer
reviewers: Security/IT, Datenschutzbeauftragter
timebox: 2 days initial, reviewed on every new schema
proof: tests/access/
1 Scope: which catalogs, and what is explicitly out of scope
2 Principals
2.1 Groups only - no grant to a named user, ever
2.2 One service principal per job; OAuth, never personal access tokens
2.3 Group membership: source of truth in the IdP, who approves a join
3 Role matrix
3.1 grp_dwh_engineer / grp_dwh_analyst / grp_dwh_bi_reader / grp_dwh_owner
3.2 Grant level per group: gold as a whole to engineer and analyst,
per object to bi_reader; silver by exception; bronze engineers only
3.3 USE CATALOG / USE SCHEMA, listed - the proof checks all three levels
4 Attribute-based access
4.1 Governed tags: pii, classification, domain, environment
4.2 Column mask policies attached to tags, not to tables
4.3 Row filter on country, driven by the group's region attribute
5 Environment isolation
5.1 Workspace-catalog binding: a dev workspace cannot reach dwh_prod
5.2 Who may create external locations (a very short list)
6 Ownership: production objects owned by groups, never individuals
7 Review
7.1 Quarterly access review over pii-tagged tables - the query, and who signs
7.2 Alerting on permission changes in dwh_prod
8 Exceptions register: who, what, and until when
Chapter 4 is what keeps the document short, and it is deployed rather than described:
sqlThe concept's chapter 4.1, executed as DDL
-- "email, phone and street are personal data" - that sentence, as three
-- statements. The mask policy hangs off the tag, so a column added to any
-- table next quarter is protected the moment it is tagged.
--
-- Precondition: pii must already exist as a GOVERNED tag defined at account
-- level. Run these before that and they still succeed - they just create an
-- ordinary free-form tag, which no ABAC policy can attach to. The columns
-- then look tagged in Catalog Explorer and are masked by nothing.
ALTER TABLE dwh_prod.gold.dim_customer ALTER COLUMN email SET TAGS ('pii' = 'true');
ALTER TABLE dwh_prod.gold.dim_customer ALTER COLUMN phone SET TAGS ('pii' = 'true');
ALTER TABLE dwh_prod.gold.dim_customer ALTER COLUMN street SET TAGS ('pii' = 'true');
Doing it
Write the role matrix at the level of four groups and gold as a wholeGrant object-by-object only where a real exception exists.
Define governed tags at account level before writing chapter 4A tag invented inside one schema cannot carry a policy across the catalog.
Attach masks to tags; never write a per-table mask you must remember to repeatTag dim_customer.email, .phone and .street.
Give the exceptions register an expiry column and let entries expireAn exception with no end date is a rule change nobody reviewed.
Embedding it
Have the data owner, not the engineer, present chapter 3 to SecurityThe concept is theirs the moment they have defended it.
Let a new table go live untagged once in dwh_test and have them find it with the proof queryThe gap between 'we tag things' and 'tagging is enforced' becomes concrete in one run.
Put the quarterly access review in the team's calendar with a named signerBefore you leave, not in the handover deck.
Defaults
Groups only — a grant to a person is a finding, not a shortcut
Policy on tags; per-table masks only where a tag genuinely cannot express the rule
Binding on top of grants, so isolation is structural rather than a grant nobody has issued yet
One page of matrix, one page of policy, one register of exceptions with dates
Gotchas
The grant spreadsheetYou notice it has failed when SHOW GRANTS and the sheet disagree and nobody can say which is authoritative — usually during the audit that asked for both.
'Temporary' grants to individual usersThey outlive the role and often the person. The tell is a grantee in information_schema.table_privileges that looks like an email address rather than a group name: one query, run monthly, and every hit is a real finding.
Believing a masked column is a denied columnAn excluded user can still CLONE the table and TIME TRAVEL it, so the concept must distinguish what is denied from what is merely hidden — otherwise it promises something the platform does not do.
Signing the concept before workspace-catalog binding existsChapter 5 is then aspirational, and it is discovered by the first person who attaches a development cluster to dwh_prod because nothing stopped them.
#Anonymisierungskonzept — and the word that is usually wrong
What
Which data is personal, what is done to it, and what happens when a person asks to be erased. Owned by the DPO, drafted by the lead engineer — that split is the point, not bureaucracy.
Why
Two failures, both expensive. The first is a development catalog seeded from a production export because 'the emails are hashed' — an incident no test will surface, found instead by a classification scan or by a subject access request that has to be answered honestly.
The second is the erasure request arriving in month nine, when nobody has decided what erasure means against an SCD2 dimension holding eight versions of one customer and a fact table pointing at all of them. That decision takes an afternoon in week two and a fortnight in month nine, against a statutory clock measured in days.
How
textAnonymisierungskonzept — full table of contents
owner: Datenschutzbeauftragter
reviewers: lead engineer, legal
timebox: 2 days initial, reviewed on every new PII column
proof: tests/privacy/
1 Legal basis and scope: which processing, which catalogs
2 Inventory of personal data
2.1 gold.dim_customer.email / .phone / .street - directly identifying
2.2 country + postal_code + customer_segment - quasi-identifiers
2.3 Keeping the inventory current: automated classification + governed tags
3 Techniques and what each one legally achieves
3.1 Pseudonymisation: hashing, tokenisation, masking - personal data
for us, who hold the mapping; say for whom, not "still personal"
3.2 Anonymisation: aggregation above a k-threshold, generalisation, suppression
3.3 Where each is used in this warehouse, and where neither is
4 Test data
4.1 No production copy in dwh_dev or dwh_test, in any form
4.2 Synthetic generation from the factories, referential integrity preserved
4.3 The check that proves it
5 Access: masks, row filters, and the one group that sees unmasked values
6 Erasure requests
6.1 Finding every SCD2 version of a subject in gold.dim_customer
6.2 What is erased, and what is retained under a statutory obligation
6.3 Deletion vectors, VACUUM, and the moment the bytes actually go
7 Deletion from bronze - the hard part, because bronze is append-only by contract
8 Sharing: Delta Sharing, never a CSV export
The direct consequence here: gold.dim_customer.email hashed into dwh_dev is a copy of production personal data wearing a costume. The relativity argument buys nothing in this case — dwh_dev sits in the same account, under the same controller, with the mapping one catalog away. Not out of scope, not exempt from erasure, and not lawful because it looks like hexadecimal.
Doing it
Generate chapter 2 from the governed tags rather than typing itA hand-maintained inventory is wrong within a quarter, and wrong in the direction that matters.
Decide erasure semantics against gold.dim_customer while it holds eight rows, not eight millionWhich versions are overwritten, which columns survive, what the fact rows point at afterwards.
Name the retention obligation that overrides erasure and the columns it coversSource it from the client's legal team — never from an engineer's memory of a paragraph number.
Route every external hand-off through Delta SharingA CSV leaves the platform and takes its masks, its lineage and its erasure reachability with it.
Embedding it
Have the engineer who believes hashing anonymises demonstrate the oppositeHash a thousand plausible addresses, join to the hashed column, count the matches. Four minutes, and the argument never comes back.
Make the DPO the owner and mean it — the engineer drafts, the DPO signsA privacy document owned by engineering is an engineering opinion with a stamp on it.
Run one erasure request as a timed drill on a seeded subject in dwh_test before a real one arrivesThe number you get is what the business is told.
Treat 'can we copy prod to reproduce this bug' as the standing test of whether the concept is realIt is asked about once a quarter.
Defaults
The inventory is generated from tags, never typed
Write pseudonymisation when you mean pseudonymisationNever write 'anonymised' about a hash.
Name the holder in every scope sentence: personal data for us, who hold the mappingAn unqualified claim is the one that gets corrected in the review that mattered.
Erasure semantics decided in week two and drilled once before they are needed
Delta Sharing out — no CSV, no ad-hoc extract, no 'just this once'
Gotchas
'It's hashed, so it's anonymised'It is pseudonymised: in your hands personal, in scope and erasable, because you hold the mapping — and a hashed email is directly reversible against a candidate list, so the claim is wrong in practice as well as in law.
Over-correcting into 'a hash is always personal data, for everyone'It is not: the test is relative to the holder and the means reasonably likely to be used. State it flat in front of a lawyer and you lose the argument you were right about, and with it the chapters that were not being questioned. Write for whom the data is personal, and the sentence survives review.
Erasing from gold and forgetting bronzeBronze is append-only by contract and holds every historical extract of sap_kna1, so the subject is still there in full and the erasure confirmation you signed is false. Decide bronze retention in the concept or the concept does not close.
Running DELETE and considering it doneDelta writes a deletion vector; the rows stay in the underlying files and stay readable via TIME TRAVEL until VACUUM passes the retention window. 'When did the data actually go' is the question you will be asked — answer it in chapter 6.3.
Ignoring quasi-identifiersOn a B2B distributor, country + postal_code + customer_segment isolates a large share of rows by itself, so a table with the name column dropped is not therefore anonymous. Chapter 2.2 exists to say so before someone exports one.