22Reference
Two things a handbook owes its reader. First, honesty about which of its claims are perishable — that is the currency register above, rendered from data so it cannot drift from the pages that depend on it. Second, the vocabulary, in both languages, so nobody nods along to a word they have not pinned down.
Currency register
Every volatile platform claim in this handbook is anchored to an entry below rather than asserted loose in prose, so re-verifying is one pass over one file. All 18 entries were last checked on 29.07.2026 against docs.databricks.com and learn.microsoft.com.
- verified
bundles-renameStand 29.07.2026Databricks Asset Bundles are now Declarative Automation Bundles. The rename is non-breaking: the bundle CLI command and all existing configuration are unchanged.
Renamed 16.03.2026. 'Asset' was dropped because the word already meant more than one thing in Databricks. You will still see the old name in blog posts, job adverts and most of the internet.
- changed
ldp-renameStand 29.07.2026Delta Live Tables is now Lakeflow pipelines, running Spark Declarative Pipelines. Existing DLT code keeps working with no migration; `import dlt` becomes `from pyspark import pipelines as dp`.
Two corrections to the commonly repeated version. (1) The open-source Apache Spark Declarative Pipelines API begins in Apache Spark 4.1, not 4.0. (2) The decorators split: @table now creates a STREAMING TABLE and the new @materialized_view creates a materialized view — code that used @table for everything needs reading before it is copied.
- verified
dqm-renameStand 29.07.2026Lakehouse Monitoring is now Data Quality Monitoring: anomaly detection at catalog/schema level plus data profiling at table level.
The old name maps to what is now only the data-profiling half. Anomaly detection is the newer part: it scans every table in a schema, prioritises important tables, and judges freshness and completeness from historical patterns rather than from thresholds you write.
- changed
abac-gaStand 29.07.2026Unity Catalog ABAC — row filter and column mask policies — plus governed tags and automated data classification are GENERALLY AVAILABLE.
This is the single biggest change against the brief, which had ABAC at Public Preview with a documented fallback. It went GA in May 2026. The consequence for a warehouse project is direct: tag-driven policies are now the default recommendation over per-table row filters and column masks, and you no longer need a preview-risk paragraph in the permissions concept.
- check first
abac-grant-policiesStand 29.07.2026ABAC GRANT policies — dynamic privilege grants — are Beta and currently scoped to EXECUTE on models.
Not a warehouse feature yet. Do not plan table access around it.
- check first
governance-hubStand 29.07.2026Governance Hub is in BETA. It is an account-level console with three pages — Data, AI and Cost. The Data page covers object access, asset usage, governed tags, data classification and data quality monitoring. An ACCOUNT ADMIN ENABLES IT THEMSELVES from the account console Previews page; access is not gated by Databricks.
Corrected 29.07.2026 after expert review. The DAIS 2026 announcement blog called it private preview; the documentation (updated 27.07.2026) says Beta and self-enable, and the docs win. The practical consequence is the opposite of the usual preview advice: the client can switch it on today, so 'we cannot get access' is not a reason to defer. Data may take up to a day to load after enabling. It introduces no new permissions — views respect existing UC grants, with account admin seeing all three pages, metastore admin the Data page, workspace admin Cost and AI. Still Beta, so nothing in a signed permissions concept should depend on it.
Governance Hub →Used by: Access & ABAC - verified
metric-views-gaStand 29.07.2026Unity Catalog metric views are GA, and Unity Catalog Business Semantics is being open-sourced. Measures are read with the MEASURE() function.
Definitions are portable across AI/BI dashboards, Genie, notebooks, SQL and third-party BI tools connected to Databricks — which is the whole argument for defining revenue once in the warehouse instead of once per BI tool.
- check first
metrics-materializationStand 29.07.2026Metric-view materialization is Public Preview; metric import from third-party tools is Beta.
Relevant if a dashboard over mv_sales is too slow. Do not assume it in a sizing estimate.
- verified
external-lineage-gaStand 29.07.2026External lineage is generally available.
Matters for this warehouse specifically: it is what lets lineage span the SAP source and the Power BI report, rather than stopping at the edges of Databricks.
- check first
uc-domainsStand 29.07.2026Unity Catalog Domains are in Public Preview.
Worth knowing if the client is heading toward a hub-and-spoke or data-mesh operating model.
- verified
lineage-retentionStand 29.07.2026Lineage system tables keep a rolling 1-year window. Catalog Explorer and the lineage API retain lineage captured after 01.09.2024 indefinitely.
Two separate retention regimes for the same underlying feature. If an audit question spans more than a year, system.access.table_lineage will not answer it and Catalog Explorer will.
- verified
liquid-clusteringStand 29.07.2026Liquid clustering replaces partitioning and ZORDER. Databricks recommends it over Z-ordering for all new tables, and CLUSTER BY AUTO lets predictive optimization choose the keys.
Clustering is NOT compatible with Z-ordering — it is one or the other. Most Delta tables under 100 TB need no partitioning at all, which for this warehouse means none of our tables should be partitioned.
- verified
predictive-optimizationStand 29.07.2026Predictive optimization runs OPTIMIZE, VACUUM and ANALYZE automatically on Unity Catalog managed tables. It is on by default for accounts created on or after 11.11.2024.
The rollout to older accounts is expected to complete in August 2026 — i.e. right now. Check whether it is on before writing a maintenance job, because on an account mid-rollout the honest answer to 'do we need scheduled OPTIMIZE?' changed recently. Settable at account, catalog, schema or table level.
- verified
pipeline-editionsStand 29.07.2026Pipeline product editions are CORE, PRO and ADVANCED. Expectations require ADVANCED. ADVANCED is the default.
CORE is streaming ingest only; PRO adds CDC; ADVANCED adds expectations. Selecting a cheaper edition on a pipeline that declares expectations fails with an explanatory error rather than silently skipping the checks.
- verified
syspath-executorsStand 29.07.2026From Databricks Runtime 13.3 LTS, directories added to sys.path (or structured as Python packages) are automatically distributed to all executors.
On 12.2 LTS and below they must be installed on executors explicitly — which is why the same code passes on the driver and fails inside a UDF on an older runtime. Separately, DBR 14.0+ changed the default working directory to the directory of the notebook being run; relative paths written against 13.3 can break on upgrade.
- verified
bundles-pythonStand 29.07.2026Python configuration for bundles (the databricks-bundles package) left experimental in Databricks CLI 0.275.0, which is also the minimum version required to use it.
Pin the CLI version in CI. A developer on an older CLI gets a confusing failure rather than a version error.
- check first
bundle-target-excludeStand 29.07.2026Excluding a top-level bundle resource for a single target is not supported.
NOT re-confirmed in the 29.07.2026 pass — carried forward from the project brief and still tracked as an open CLI issue. Treat as unverified. The practical workaround is to define the resource once as a YAML anchor and reference it only from the targets that should have it, rather than defining it top-level and trying to subtract it.
No official source — treat as unverified.Used by: Standards & CI/CD - check first
serverless-environment-keyStand 29.07.2026Serverless notebook tasks are reported to reject environment_key.
UNVERIFIED. Community-reported, not confirmed in official documentation, and not reproduced here. Included because it costs an afternoon if you hit it, but do not quote it to a client as fact — reproduce it on their workspace first.
No official source — treat as unverified.Used by: Standards & CI/CD, Testing · SQL
System tables
Databricks writes operational metadata into read-only Delta tables under the system catalog. They are the evidence base for cost, access, lineage and performance questions — no agent to install, no export to build.
| Table | One row is | Answers |
|---|---|---|
system.billing.usage | a metered usage record | where the money goes, per workspace, SKU and — if you tagged things — per domain |
system.billing.list_prices | a price for a SKU over a validity interval | converting DBUs into currency without hardcoding a rate |
system.access.audit | an audited action | who read, changed or granted what, and when |
system.access.table_lineage | a read/write between a table and a job or query | what feeds this table, and what breaks if it changes |
system.access.column_lineage | a column-level read/write | which source column a PII column derives from |
system.query.history | a completed query | which queries are expensive, and which are merely frequent |
system.compute.clusters | a cluster definition version | what compute exists and how it is configured |
system.compute.warehouse_events | a SQL warehouse state change | whether auto-stop is doing its job |
system.lakeflow.jobs | a job definition | the inventory of what is scheduled |
system.lakeflow.job_run_timeline | a job run | success, duration and the run-duration trend |
system.information_schema.* | a catalog object | schema drift, grants, constraints, comments |
#Reading system tables without being misled by them
SELECT
u.custom_tags['domain'] AS domain,
u.custom_tags['environment'] AS environment,
ROUND(SUM(u.usage_quantity * p.pricing.default), 2) AS cost_usd
FROM system.billing.usage AS u
LEFT JOIN system.billing.list_prices AS p
ON u.sku_name = p.sku_name
AND u.usage_end_time >= p.price_start_time
AND (p.price_end_time IS NULL OR u.usage_end_time < p.price_end_time)
WHERE u.usage_date >= current_date() - INTERVAL 30 DAYS
GROUP BY ALL
ORDER BY cost_usd DESC;NULL. That is not a broken query — it is the finding, and it is the argument for the tag taxonomy in the storage and cost route.Doing it
- Check when each schema was enabled before quoting any historical number from it.
- Join usage to list_prices on the validity interval rather than hardcoding a rate — prices change and the interval join keeps history correct.
- State the window on every figure you report. 'EUR 4,200' means nothing; 'EUR 4,200 over the last 30 days' means something.
- Treat an empty result as a finding to explain, never as a clean bill of health.
Defaults
- Enable the system schemas on day one; it is retroactive for nothing and free for everything.
- Never hardcode a DBU price — join list_prices on the validity interval.
- Every reported figure carries its time window and the date it was run.
- Audit data is itself sensitive; grant it deliberately, as covered in the audit and lineage route.
Gotchas
- Quoting a historical figure from a schema enabled last week. It is not retroactive, so the number is real but the period is wrong — and it is wrong in the flattering direction, because the missing history is missing spend.
- Reading an empty lineage result as 'nothing depends on this table'. Both lineage tables are a subset: rows are emitted only when lineage can be inferred, so absence is not evidence of absence. Dropping a table on that basis is how a report dies silently.
- Column lineage misses inserts that write explicit literal values, so a column populated by a constant appears to have no upstream at all — exactly the case you most want to spot in a PII trace.
- Assuming system tables are consistent across clouds. Availability and column sets differ between AWS, Azure and GCP; check on the client's cloud before promising a query works.
Glossary
Platform vocabulary, plus the German terms a DACH client will use in the room. If you are coaching in English on a German project, the right-hand column is the one that prevents an hour of talking past each other.
| Term | Means | German |
|---|---|---|
| Bronze / Silver / Gold | Refinement stages. Not a data model. | Rohschicht / Bereinigt / Auswertung |
| Grain | What exactly one row of a table represents. | Granularität |
| Fact table | Measurements at a declared grain. | Faktentabelle |
| Dimension | The descriptive context you group and filter by. | Dimension |
| Surrogate key | A system-generated key standing in for the business key. | Surrogatschlüssel |
| Business key | The identifier the source system uses. | Geschäftsschlüssel |
| SCD1 | Overwrite on change; no history kept. | Historisierung Typ 1 |
| SCD2 | A new row per change, with validity intervals. | Historisierung Typ 2 |
| Conformed dimension | One dimension shared by several facts. | Konforme Dimension |
| Unknown member | The placeholder row a fact points at when the dimension is missing. | Unbekannt-Satz |
| Data Vault | Hub / Link / Satellite modelling. Common in German enterprises. | Data Vault |
| Hub / Link / Satellite | Business key / relationship / descriptive history. | Hub / Link / Satellit |
| Idempotent | Running it twice leaves the same result as running it once. | Idempotent |
| CDC | Change data capture — a stream of source changes. | Änderungserfassung |
| Full extract | The whole source, re-delivered each time. | Vollabzug |
| Delta / incremental load | Only what changed since last time. | Deltaladung |
| Snapshot comparison | Diffing today against yesterday when the source has no change timestamp. | Snapshot-Vergleich |
| Autoloader | Incremental file ingestion with schema evolution. | Autoloader |
| Expectation | A declared data-quality rule on a pipeline: warn, drop or fail. | Erwartung / Qualitätsregel |
| Lakeflow pipeline | Declarative pipeline; formerly Delta Live Tables. | Lakeflow-Pipeline |
| Streaming table | A table incrementally appended from a stream. | Streaming-Tabelle |
| Materialized view | A precomputed, refreshable query result. Not a metric view. | Materialisierte Sicht |
| Metric view | A governed semantic definition of measures and dimensions. | Kennzahlen-Sicht |
| MEASURE() | The SQL function that reads a measure from a metric view. | — |
| Unity Catalog | The governance control plane: metastore, catalogs, grants, lineage. | Unity Catalog |
| Metastore | The top-level UC container, one per region. | Metastore |
| Governed tag | An account-level key-value applied to securables, used in policies. | Verwalteter Tag |
| ABAC | Access decided by attributes (tags) rather than per-table rules. | Attributbasierte Zugriffskontrolle |
| Row filter / column mask | Policies restricting which rows and columns a principal sees. | Zeilenfilter / Spaltenmaske |
| Liquid clustering | Adaptive data layout replacing partitioning and Z-order. | Liquid Clustering |
| Predictive optimization | Automatic OPTIMIZE, VACUUM and ANALYZE on managed tables. | Prädiktive Optimierung |
| Managed table | A table whose lifecycle and storage UC owns. | Verwaltete Tabelle |
| Delta Sharing | Governed sharing without copying data out. | Delta Sharing |
| Lineage | The recorded derivation path between tables and columns. | Datenherkunft |
| Pseudonymisation | Reversible de-identification. Still personal data under GDPR. | Pseudonymisierung |
| Anonymisation | Irreversible; out of GDPR scope. Hashing is NOT this. | Anonymisierung |
| Personal data | Data relating to an identifiable person. | Personenbezogene Daten |
| Definition of done | The checkable list a change satisfies before merge. | Definition of Done |
| Runbook | One page telling whoever was woken what to do. | Runbook / Betriebshandbuch |
| Characterization test | A test pinning current behaviour before a refactor. | Charakterisierungstest |
Dating conventions
#Why every page carries a date
facts array on the topic. routesForFact(key) then answers the maintenance question directly: change one fact, and it lists every page that needs revisiting. No grep, no memory.Doing it
- Anchor any platform claim that could move to a currency key rather than asserting it in prose.
- Re-verify the register against the official docs before reusing this material in a client deliverable.
- Prefer docs.databricks.com and learn.microsoft.com over blogs — blogs lag renames badly.
- Mark anything you could not confirm as unverified rather than dropping it; a flagged risk is more useful than a silent gap.
Defaults
- Every platform claim carries a Stand marker or a register cross-reference.
- Preview and beta features are never load-bearing in a design a client signs.
- Unverified claims stay, marked unverified, with what you would need to confirm them.
- The register is data, so the page rendering it cannot drift from the pages depending on it.
Gotchas
- Copying platform claims out of a blog post. Blogs are not updated on rename, so half the internet still says Delta Live Tables and Databricks Asset Bundles — names that are correct history and wrong instruction.
- Letting a preview feature become load-bearing in a signed design. When the preview shape changes, the redesign is not billable and the conversation is not pleasant.
- Dating the page but not the claim. A page stamped July with a claim from March reads as verified-in-July, and nobody can tell which sentence is the stale one.
- Assuming GA on one cloud means GA on all three. Feature availability diverges across AWS, Azure and GCP more often than the marketing implies.
Straight to the documentation
The generic platform material lives in the official docs and is not restated here. These are the pages worth having open while you work through this route.
- System tables reference — the authoritative list, including which schemas exist and how to enable them
- Databricks release notes — where renames and status changes appear first — the input to your next re-verification pass
- Lineage system tables — the retention rules, stated precisely
- Databricks SQL language reference — dialect questions, reserved words, function signatures