05BI & semantics
This route decides where the definition of revenue lives, and therefore which number is right when two reports disagree. It is the last foundation decision and the one most often made by accident — by a report author, in a hurry, in DAX. The technical content is one YAML file; the expensive part is deciding that there is exactly one of them.
Defining revenue once
A metric view is a Unity Catalog object carrying dimensions and measures but no grain. It aggregates at whatever grain the query asks for, which is why one definition serves a monthly board pack and a per-customer drill-down.
#gold.mv_sales, in full
gold.fct_order_line and is deployed as DDL from the repository, like every other object. Business rule 1 — revenue excludes cancelled lines — is enforced once, in the filter, rather than in each of the eleven reports that would otherwise each have to remember it.CREATE OR REPLACE VIEW ${catalog}.gold.mv_sales
WITH METRICS
LANGUAGE YAML
COMMENT 'Sales metrics over gold.fct_order_line. No grain: measures aggregate at query time.'
AS $$
version: 1.1
source: ${catalog}.gold.fct_order_line
filter: status <> 'CANCELLED' # business rule 1, enforced once
joins:
- name: customer
source: ${catalog}.gold.dim_customer
on: source.customer_sk = customer.customer_sk AND customer._tech_is_current
cardinality: many_to_one
rely:
at_most_one_match: true
fields:
- name: country
expr: customer.country
display_name: Land
comment: 'DE | AT | CH | NL'
- name: customer_segment
expr: customer.customer_segment
display_name: Kundengruppe
comment: 'A | B | C'
- name: region
expr: source.region_sk
display_name: Vertriebsregion
comment: 'surrogate key on the fact - the label lives in gold.dim_sales_region and is deliberately not joined'
- name: order_date
expr: CAST(order_ts AS DATE)
display_name: Auftragsdatum
- name: month
expr: DATE_TRUNC('MONTH', order_ts)
display_name: Monat
measures:
- name: revenue
expr: SUM(amount_eur)
display_name: Umsatz (EUR)
- name: order_count
expr: COUNT(DISTINCT order_id)
display_name: Auftraege
- name: average_order_value
expr: SUM(amount_eur) / NULLIF(COUNT(DISTINCT order_id), 0)
display_name: Durchschnittlicher Auftragswert
- name: active_customers
expr: COUNT(DISTINCT CASE WHEN source.customer_sk <> 'UNKNOWN'
THEN source.customer_sk END)
display_name: Aktive Kunden
$$;order_count is COUNT(DISTINCT order_id), not COUNT(1), because the source grain is the order line. active_customers excludes the UNKNOWN member that business rule 7 guarantees will be present. And region_sk and customer_sk exist on both the fact and the dimension, so they carry the source namespace — unqualified they are ambiguous.display_name. One model, two vocabularies, no umlauts in a column name.-- monthly revenue by country, from a workspace bound to one catalog
SELECT
country,
month,
MEASURE(revenue) AS revenue,
MEASURE(order_count) AS orders,
MEASURE(average_order_value) AS aov
FROM gold.mv_sales
WHERE month >= '2026-01-01'
GROUP BY country, month
ORDER BY country, month;
-- same definitions, different grain, no second object
SELECT customer_segment, MEASURE(active_customers) AS customers
FROM gold.mv_sales
GROUP BY customer_segment;SUM(amount_eur) over everything including the cancellations. Natural-language questions are where an undefined metric does most damage, because nobody reviews the SQL.Doing it
- Deploy the metric view as DDL from the repo, in the same bundle as the tables it reads — one edited in the UI is invisible to code review.
- Write business rule 1 into the filter, then delete the equivalent WHERE clause from every dashboard query that carries one.
- Add the metric view, not gold.fct_order_line, as the Genie data source, so 'what was revenue in March' cannot be answered by an ad-hoc SUM.
- Point one existing report at MEASURE(revenue) and diff it against today's number. That difference is your first real conversation with the business.
Defaults
- One metric view per business process, not per report. Sales is one; procurement is another.
- Measures always through MEASURE(); never a materialised copy of a measure in a table.
- Business rules live in the filter, once, with the rule number in a comment.
- display_name carries the local-language label; the identifier stays ASCII.
Gotchas
- Joining the fact to an SCD2 dimension without pinning to the current version. customer_sk is sha2(customer_id), so it repeats once per validity interval in gold.dim_customer. Drop the _tech_is_current predicate and every line fans out to that customer's version count — revenue does not look broken, it looks good, and it grows every month as more customers acquire a second version.
- rely: at_most_one_match: true is a promise to the optimizer, not a check. If the join does match more than once you get no error, only the wrong number faster.
- COUNT(1) where you meant COUNT(DISTINCT order_id). gold.fct_order_line is one row per line, so the two differ by the average lines per order — 'orders' silently becomes 'lines' in every KPI tile downstream, and because the ratio is stable month to month the number looks plausible rather than broken. You catch it by counting rows in silver.sales_order and comparing, not by looking at the tile.
- Counting the UNKNOWN member as a customer. Rule 7 sends every unresolvable line to customer_sk = 'UNKNOWN', so active_customers gains one phantom customer permanently and the drill-down shows a customer with no name.
- NULL in the filter column. status <> 'CANCELLED' is three-valued: a row whose status is NULL is neither kept nor reported as excluded, it silently leaves revenue altogether. If status is nullable, write status IS DISTINCT FROM 'CANCELLED' — and reconcile MEASURE(revenue) against a plain SUM(amount_eur) over the fact, because the gap is the only place the dropped rows show up.
- Grouping by region and getting surrogate keys. region is source.region_sk: the fact carries the key, not the name, and this view deliberately does not join gold.dim_sales_region for the label. A slicer built on it lists surrogate keys and the business assumes the report is broken. Either denormalise the region name onto gold.fct_order_line in gold, or let the report join the dimension — what you must not do is add a second join here, which is what closes the BI-compatibility path below.
- Editing the YAML in Catalog Explorer at 17:00. It works, so nobody redeploys — and either the next deployment silently reverts it, or it does not and the repo no longer describes production.
#What a metric view will not do
- Databricks-only. The definition is evaluated by the Databricks SQL engine, so any consumer reading a copy of the data — an Excel extract, a Tableau extract, a Fabric model over a shortcut — is outside the semantic layer for good.
- Join limits. Joins are LEFT OUTER and many-to-one. A single aggregate expression may only reference columns from one source, so a measure that multiplies source.quantity by a column of the joined gold.dim_customer cannot be written at all — that arithmetic has to happen in gold.fct_order_line first. Joined tables cannot carry MAP columns.
- Every time grain is explicit. There is no free date hierarchy: month exists because it is written above, quarter and week do not exist until someone writes them, and filtering on month filters the truncated value, not order_ts.
- Measures are not columns. You cannot GROUP BY one, put one in a slicer, or reference one without MEASURE(); a measure in a GROUP BY returns METRIC_VIEW_MEASURE_IN_GROUP_BY.
average_order_value is non-additive — the AOV of Germany plus the AOV of Austria is not the AOV of both. Because it is a measure and not a column, nothing can sum it: the engine re-evaluates SUM(amount_eur) / COUNT(DISTINCT order_id) at whatever grain was asked for. Materialise that figure into a table and the protection is gone.week was never defined — at which point the semantic layer looks unfinished rather than deliberate.Doing it
- Define every time grain the business asks for by name, and refuse to add one that has no named consumer.
- Keep the join count as low as the fact allows; region_sk is denormalised onto gold.fct_order_line precisely so the common case needs none.
- Establish what the actual reporting surface is before committing to metric views — if it is an extract, write that into the concept rather than discovering it at UAT.
- Benchmark the slowest realistic dashboard query before promising response times; materialization is a preview and cannot be assumed.
Defaults
- Non-additive figures — ratios, averages, distinct counts — exist only as measures, never as columns in a table.
- Time grains are declared explicitly and reviewed once a quarter against what reports actually use.
- Joins in a metric view stay under three; beyond that, conform in gold instead.
Gotchas
- Materialising a ratio into gold.agg_revenue_month so a dashboard is faster. Someone shows it across four regions and the tile sums four averages: plausible, always wrong, and wrong by more the more rows are on screen.
- Assuming a date hierarchy exists. A BI tool that offers drill-down on a date column offers nothing on a metric view field called month — there is no year above it unless you defined one.
- Adding a join to reach one label. Every join narrows which BI paths can consume the view later, and a metric view carrying joins is rejected outright by BI compatibility mode with METRIC_VIEW_JOIN_NOT_SUPPORTED.
- Quoting materialization in a sizing estimate. It is Public Preview; if it is not enabled on the client's workspace, the dashboard you promised at two seconds runs at twelve.
Where the semantics live
Two credible places to define a measure, and one popular third option that is not a compromise but a defect.
#Metric view, BI model, or both
| UC metric view | Power BI semantic model | Both | |
|---|---|---|---|
| Definition lives in | gold.mv_sales — YAML in the repo | DAX, in a model file or workspace | two places |
| Changed by | a pull request | whoever opens the file | whoever gets there first |
| Reaches | SQL, notebooks, AI/BI dashboards, Genie, any tool that sends SQL to a warehouse | Power BI reports and Excel pivots on that model | everything, inconsistently |
| Tested by | a SQL oracle in CI | nothing, in practice | the half that has a test |
| Cost of a new measure | one PR, live for every consumer at once | one file, live for that model's consumers | twice — and the second one is forgotten |
| Breaks when | a tool cannot emit MEASURE() | a second tool needs the same number | the two numbers differ and nothing can arbitrate |
Doing it
- Make the decision in writing, in the developer concept, with a date and a named owner.
- If metric views win: state that the BI model defines no measures, and check it — an unexpected DAX measure is a review finding, not a preference.
- If the BI model wins: say so honestly, stop building metric views, and accept that Genie and notebooks will not see those definitions.
- Count the measures already living in DAX before choosing. The number decides whether this is a decision or a migration.
Defaults
- One definition per measure, in one place, owned by a named person.
- If both layers exist, one of them defines nothing or is generated from the other.
- The choice is written down with its date — a decision nobody can find gets remade quarterly.
Gotchas
- 'We will keep them in sync.' Nobody has ever kept two semantic layers in sync, because divergence happens through an exception added under time pressure in whichever layer's owner is in the room.
- A Power BI model that starts as a passthrough and adds 'just one' local measure. That measure is the fork, and it is the one nobody documents, because it was meant to be temporary.
- Choosing metric views while the business's real reporting surface is an Excel pivot on an Import model. The semantics are perfect and reach nobody, and you find out at UAT.
- Leaving the decision to the BI team by not raising it. It gets made in DAX by default within two sprints, and reversing it is a reconciliation project rather than a refactor.
Power BI in practice
Two settings decide most of what a Power BI deployment on Databricks costs and most of what it leaks: the storage mode, and what triggers the refresh.
#Import vs DirectQuery
| Import | DirectQuery | |
|---|---|---|
| Where the data sits | cached inside Power BI | stays in Unity Catalog |
| Freshness | as of the last refresh | as of the query |
| UC row filters and column masks | applied once, to the identity that ran the refresh — every reader then sees that snapshot | applied per query, per reader |
| Warehouse cost | one load per refresh | every page render and every slicer click |
| Metric views | only as a fixed Native query at one grain, which discards the point of the metric view | the documented path: Native query with MEASURE() |
| Fits | a stable monthly report over a small model | a governed model where who-sees-what matters |
email from support staff is applied at refresh time to the refreshing principal and never again — whatever lands in the cache is visible to anyone who can open the report. DirectQuery keeps every read inside UC, where the policy is evaluated per query. Dual mode covers the middle: with DirectQuery as the default, small dimensions like gold.dim_date come from cache while the fact stays live.SUM. All aggregation functions are rewritten to the measure definition, so SUM, MIN and MAX return the same correct value, while AVG, distinct count, median, variance and standard deviation produce patterns the rewrite cannot handle.gold.mv_sales as written, and it is worth saying out loud rather than discovering in a Tableau workbook: BI compatibility mode does not accept a metric view carrying a join, and this one joins gold.dim_customer to reach country and segment. Tableau and Sigma get these numbers through a plain SQL query with MEASURE(), or country and segment get conformed onto gold.fct_order_line so the join disappears. Which of those you pick is a modelling decision, not a connector setting.Doing it
- Default to DirectQuery for anything carrying personal data from gold.dim_customer, and justify each Import model in writing.
- Mark small dimensions Dual once the model is DirectQuery; leave gold.fct_order_line live.
- Inventory the reports still using BI compatibility mode and re-point them at Native query rather than waiting for a connector upgrade to break them.
- Run one realistic report session and read the cost off system.billing.usage before agreeing a storage mode.
Defaults
- DirectQuery by default where UC policies must hold; Import by exception, in writing.
- Dual for small dimensions, never for facts.
- Native query plus MEASURE(), in DirectQuery, is the metric-view path from Power BI to design for — write it down before someone rebuilds the measures in DAX out of frustration.
Gotchas
- Import mode as a governance hole: masks and row filters are evaluated at refresh time under the refreshing identity, so the cached model holds data the reader was never allowed to query. Nothing errors, nothing is logged as a violation, and it surfaces during an access review.
- Assuming the Power BI connector still offers BI compatibility mode. Where it has gone the report does not degrade, it stops returning rows, and the connector error names the query rather than the missing option — so it reads as a permissions problem and gets escalated to the wrong team. Check the connector version in the client's own report before promising a date.
- Setting a measure column's aggregation to anything but SUM, MIN or MAX in a BI-compat tool. The tool rewrites the aggregation to the measure definition, so AVG, distinct count, median, variance and standard deviation return a number that is not the thing they name, and nothing errors. You notice it only by reading the same measure through MEASURE() in SQL and finding a different figure.
- Dragging a measure onto an axis, legend or slicer. The query fails with METRIC_VIEW_MEASURE_IN_GROUP_BY, which is correct behaviour and reads as a broken connector to whoever hit it.
- DirectQuery over an unclustered fact table. Every slicer click is a full scan and the report gets blamed; the fix is CLUSTER BY on gold.fct_order_line.
#Refresh on pipeline completion, not on the clock
resources:
jobs:
dwh_daily:
name: dwh_daily
schedule:
quartz_cron_expression: "0 0 4 * * ?"
timezone_id: Europe/Zurich
tasks:
- task_key: silver_gold
pipeline_task:
pipeline_id: ${resources.pipelines.dwh_core.id}
- task_key: assert_gold
depends_on:
- task_key: silver_gold
sql_task:
warehouse_id: ${var.warehouse_id}
file:
path: ../../tests/sql/revenue_reconciliation.sql
- task_key: refresh_power_bi
depends_on:
- task_key: assert_gold # green tests, not merely a green pipeline
power_bi_task:
connection_resource_name: pbi_prod
power_bi_model:
workspace_name: Sales
model_name: dwh_sales
storage_mode: DIRECT_QUERY
refresh_after_update: false
tables:
- name: fct_order_line
catalog: ${var.catalog}
schema: goldrefresh_after_update. It needs a Power BI connection in Unity Catalog and USE CONNECTION on it, plus rights on the tables and the warehouse.Doing it
- Delete the scheduled refresh in the Power BI service once the job task exists. With two triggers, the one you forgot about is the one that runs during an incident.
- Point the Power BI task at the assertion task, not at the pipeline task.
- Create the Power BI connection in Unity Catalog and grant USE CONNECTION to the job's service principal in the bundle, not by hand in prod.
- Alert when the timestamp on the report page disagrees with the latest _tech_loaded_at on gold.fct_order_line.
Defaults
- One job owns ingest, transform, assert and publish. Downstream refreshes are tasks, never separate schedules.
- Publish on assertions passing, not on the pipeline succeeding.
- Connections and grants are bundle resources like anything else.
- The report shows the source-data timestamp, not the refresh timestamp — different facts, and only one of them matters.
Gotchas
- A leftover scheduled refresh in the Power BI service alongside the job task. It fires mid-pipeline and publishes a half-loaded model; the job's own refresh corrects it twenty minutes later, so the incident is unreproducible.
- Editing the semantic model in the Power BI service while the task is updating it. The model sticks on 'Pending changes' and the next run fails with an error pointing at the model rather than at the person editing it.
- Refreshing after a green pipeline instead of after green tests. Expectations that only warn do not fail a pipeline — the run is green, the data is wrong, and you have just published it.
- Missing USE CONNECTION in prod only, because the connection was created by hand in dev. It works everywhere it was tested and fails on the first production run.
- An Import refresh timed against a warehouse with aggressive auto-stop. It pays the cold start every run and the report is late by exactly that much, every day, blamed on the network.
Straight to the documentation
The generic platform material lives in the official docs and is not restated here. These are the pages worth having open while you work through this route.
- Unity Catalog metric views — the overview and the current status of the Business Semantics area — read before promising a client anything about portability
- Metric view YAML syntax reference — field-by-field spec; check the version key and the rely semantics before copying any example, including this one
- Query metric views in Power BI — the Native-query-in-DirectQuery path and the list of things that break — check it before every Power BI estimate
- Query metric views with BI compatibility mode — for Tableau and Sigma: how to enable it, and why every measure must stay on SUM
- Power BI task for jobs — storage modes, refresh_after_update, and the Unity Catalog connection the task needs