Run11 Observability · 3 of 3

What the platform does for you

Data Quality Monitoring covers the profiling and the statistical half. It does not replace the four checks — it makes them the exception rather than the workload. Half of it is still Public Preview, and that decides how much of it a signed design may rest on.

Stand 29.07.2026

#Data Quality Monitoring

What

Lakehouse Monitoring is now Data Quality Monitoring, and the rename split it in two halves with different maturity. The old product is the data profiling half, per table, and it is the GA part. The new half is anomaly detection at catalog or schema level, judging freshness and completeness from each table's own history rather than from thresholds you write — and it is Public Preview.

Profile types — pick by what the table is, not by preference

Profile	For	Here
Snapshot	current state; metrics over the whole table each run	gold.dim_customer, gold.dim_product
Time series	a timestamp column defines windows; metrics per window	gold.fct_order_line on order_ts
Inference	model input/output logs, prediction drift	not used by this warehouse

Why

Hand-written checks scale linearly with tables. Forty tables is forty thresholds, and the fortieth is written badly by someone in a hurry. Anomaly detection covers the long tail nobody would have got round to instrumenting; the hand-written four cover the tables where you know exactly what wrong looks like.

The division of labour is the decision this topic makes: the platform learns what normal looks like; you assert what correct means. No amount of profiling will discover business rule 2 — that amount_eur must equal ROUND(quantity * unit_price * fx_rate, 2). That is an assertion, and it belongs in the SQL test suite.

How

python

A time-series monitor on the fact table

from databricks.sdk import WorkspaceClient
from databricks.sdk.service.catalog import MonitorTimeSeries

w = WorkspaceClient()

w.quality_monitors.create(
    table_name="dwh_prod.gold.fct_order_line",
    output_schema_name="dwh_prod.gold",
    assets_dir="/Workspace/Shared/dwh/monitors/fct_order_line",
    time_series=MonitorTimeSeries(
        timestamp_col="order_ts",
        granularities=["1 day"],
    ),
    # slicing is evaluated on the monitored table, so only its own columns
    slicing_exprs=["status", "currency", "region_sk"],
)

The profiling monitor writes ordinary Delta tables into the output schema you nominate: <table>_profile_metrics and <table>_drift_metrics. That is why it is worth adopting. Alerting, dashboards and your own reconciliation queries are all SQL over Delta, so the platform's metrics and your four checks land in one alerting path instead of two consoles.

Anomaly detection adds intelligent scanning: it sweeps every table in the schema but prioritises the important ones, and judges freshness and completeness from each table's own history rather than from thresholds you write. That is what keeps a schema-wide sweep affordable. When it fires, the alert is an entry point into Unity Catalog lineage — upstream to what fed the anomaly, downstream to who is about to read it. External lineage is GA, so that chain reaches the SAP source and the Power BI report instead of stopping at the platform edge.

Its output is queryable on the same terms, and this is the part most write-ups omit: anomaly detection writes to system.data_quality_monitoring.table_results — one row per table per evaluation, carrying the check that ran, its verdict, and the downstream impact it worked out from lineage. That system table is Public Preview too. It is what makes route the platform's alerts into the same channel as the four checks a real instruction rather than a wish: the anomaly half is a SQL query over a system table, the profiling half is a SQL query over the two metrics tables in your output schema, and the four checks are SQL files in the repo. Three sources, one alerting path.

Anomaly detection is Public Preview — use it, do not sign it

Profiling is GA. Anomaly detection is Public Preview, and this handbook's own rule, stated in Reference, is that a preview feature is never load-bearing in a design a client signs. Enable it, put the long tail of tables behind it, and let it find what nobody would have got round to instrumenting — but the observability concept the client signs rests on the four committed checks, not on this. Three consequences that bite in the deployment window rather than in the design review: enabling requires MANAGE on the schema or catalog, so a service principal holding USE SCHEMA and SELECT cannot turn it on; it bills as DATA_QUALITY_MONITORING DBUs, which is a line on the invoice and not a free checkbox; and the Percent Null check is itself Beta inside the Preview.

Doing it

Enable anomaly detection at schema level on goldIt needs MANAGE on the schema or catalog, which the deployment principal usually does not have. Write into the concept that it is Public Preview and supplements the four checks rather than replacing one.
Time series on gold.fct_order_line keyed on order_ts; snapshot on the dimensionsThe dimensions are the tables small enough for the 4 TB snapshot limit to be irrelevant.
Schedule a query over system.data_quality_monitoring.table_resultsRoute it into the same channel as the four checks, instead of expecting anyone to open a console page daily.
Check the SDK surface against your installed version before copying the snippetThe API names predate the rename and lag it.

Embedding it

Have the team read the drift-metrics table as SQL in week oneOnce they see it is a Delta table it stops being a black box and starts being data they can join.
Make the team own the list of business rules the platform cannot coverWriting that list is the exercise.
Walk one real alert through lineage to a named report owner in front of them, and let them do the second one

Defaults

Anomaly detection schema-wide; profiling on the tables you can name a consumer for
Metrics tables treated as first-class data — queried, joined, alerted on in SQLsystem.data_quality_monitoring.table_results for the anomaly half, _profile_metrics and _drift_metrics for the profiling half.
GA carries the commitmentA Preview feature earns its place by covering the long tail, never by appearing in the signed concept as the thing that catches a named failure.
Platform detects drift; the repo asserts rules. Never expect the first to do the second
One alerting channel for platform and hand-written checks alike

Gotchas

Expecting anomaly detection to catch a wrong business ruleIt learns what this table normally looks like, so a conversion that has been wrong since go-live is normal and will never be flagged.
Profiling every table because it is one checkboxThe metrics tables and the compute are real cost, and alert volume from tables nobody reads is what teaches people to ignore the channel.
Monitoring a table whose grain is about to changeYour own deployment invalidates the learned baseline, the monitor alarms on a correct change, and the reflex is to disable it rather than reset it.
Assuming lineage completeness during root causeRecords are emitted only where lineage can be inferred, so a missing edge is not evidence that nothing reads the table.
Writing a Public Preview feature into the signed observability conceptWhen the auditor asks what catches a stale SAP extract, the answer has to be a committed SQL file under version control — not a checkbox whose behaviour, billing and system-table schema can move between the design workshop and go-live.
Meeting the MANAGE prerequisite for the first time on deployment dayAnomaly detection needs MANAGE on the schema or catalog to enable; the bundle's service principal usually has neither, so the deploy goes green and the platform half of the observability design silently does not exist.
Pointing a snapshot profile at the fact tableSnapshot recomputes over the whole table on every run and is capped at 4 TB; time series on order_ts is the profile type fct_order_line is shaped for, and the cost difference shows up long before the cap does.

Depends on