14Audit & health check
This route decides what happens before you have an opinion worth charging for. The first-week sweep is read-only, runs entirely on system.* tables, and produces the thing that makes a later recommendation land: numbers from the client's own account. It is deliberately short — the SQL lives in the audit library and this page hands off to it.
Why the sweep comes first
Week one is a room full of confident, contradictory statements about one account. The sweep replaces them with a table.
#Opinion into evidence
gold.fct_order_line, serverless for the nightly load, a tag taxonomy — all right, and all indistinguishable from the deck the last vendor left. It also prevents the opposite: a team says the loads are slow, and the sweep shows the nightly job sharing an all-purpose cluster with four notebooks, its duration tracking contention rather than data volume.Doing it
- Run it in their workspace with them watching — the questions they ask while it runs are half the value.
- Save the exact SQL per finding, so the handover sweep is a re-run and not a re-derivation.
- Record every result the same day, healthy ones included. An unrecorded green area gets re-litigated in month three.
Defaults
- One day, five areas, one question each. Depth only where the sweep pointed.
- Baseline before any change; identical SQL at handover.
- Measurement and proposal are separate documents and separate sessions.
Gotchas
- Sweeping and recommending in one breath. The client argues about the fix and stops reading the number, and the number was the hard part to get.
- Sweeping from your laptop against an export. Nothing is reproducible, so the first challenged figure becomes a credibility problem instead of a re-run.
- Letting the sweep grow to 100 queries because the library has 100. You notice it in the findings sheet: forty rows, no Owner column filled, because forty findings compete with each other and none of them is anybody's. Week one answers five questions; the rest are for whatever week one flagged.
#Read-only, and nothing to install
SELECT against the system catalog — billing, compute, lakeflow, query, storage, access and information_schema. No agent, no collector, no scheduled notebook, no data leaving the account. The commercial consequence: the sweep needs a grant, not a security review, and a grant takes an afternoon.SELECT table_schema, COUNT(*) AS tables
FROM system.information_schema.tables
WHERE table_catalog = 'system'
GROUP BY table_schema
ORDER BY table_schema;system.billing.usage corrects the past by appending RETRACTION and RESTATEMENT rows rather than editing history, so recent dollars are still moving. Retention: the lineage system tables keep a rolling one-year window, while Catalog Explorer and the lineage API retain lineage captured after 01.09.2024 indefinitely — a question reaching further back is answerable in the UI, not in SQL.system.information_schema.column_tags' produce the same empty result, and only one of them is true. The same trap sits under lineage: both lineage tables are a subset, emitted only where lineage could be inferred, so a missing edge means not captured — never not happening.Doing it
- Ask for SELECT on the system schemas in the kickoff mail. It is the only access the sweep needs and the only thing that can delay it.
- Run query zero first; record the missing schemas alongside the findings.
- Anchor cost findings on a complete 30-day window ending two days back, and run the library's restatement trust query before any of them — a window whose restated share is large makes every dollar figure computed over it provisional, and you want to know that before the readout, not after.
Defaults
- SELECT on system schemas, nothing else. A sweep query needing write access is not a sweep query.
- Query zero before every sweep, including the handover re-run — enablement changes.
- State the retention boundary rather than silently answering a shorter question than the one asked.
Gotchas
- A disabled system schema returning zero rows, read as a healthy answer. It is the easiest way to hand a client a confidently wrong governance finding in week one.
- Quoting a saving from a period that later restates. The correction lands after your readout, and the first number you ever gave them turns out to be wrong.
- Treating a missing lineage edge as proof a table is unused. Dropping it takes down a report nobody could see.
- Comparing a partial day to a full one. Ingestion latency puts a phantom cliff at the right edge of every chart you make today.
The first-week sweep
Five areas, one question each. You are not looking for a score but for a bad answer — the shape of result that decides where day two goes.
#Cost, compute, jobs, storage, governance
| Area | The one question | A bad answer looks like | Source |
|---|---|---|---|
| Cost | Where does the bill go, and can you attribute it? | a large share of DBUs with no domain or environment tag | system.billing.usage, list_prices |
| Compute | What runs when nothing is being asked of it? | all-purpose clusters with no auto-termination; long warehouse idle tails | system.compute.clusters, warehouse_events |
| Jobs | What fails, what runs long, what runs on the wrong compute? | job tasks on all-purpose compute; repeat failures still burning DBUs | system.lakeflow.job_run_timeline, job_tasks |
| Storage | Is maintenance happening, and who is doing it? | no maintenance at all — or two mechanisms on one table | system.storage.predictive_optimization_operations_history |
| Governance | Who can read personal data, and who did? | untagged PII columns; production objects owned by a named person | system.information_schema.*, system.access.audit |
SELECT
SUM(usage_quantity) AS dbus,
ROUND(100 * SUM(CASE WHEN custom_tags['domain'] IS NULL
THEN usage_quantity ELSE 0 END)
/ NULLIF(SUM(usage_quantity), 0), 1) AS pct_no_domain,
ROUND(100 * SUM(CASE WHEN custom_tags['environment'] IS NULL
THEN usage_quantity ELSE 0 END)
/ NULLIF(SUM(usage_quantity), 0), 1) AS pct_no_environment
FROM system.billing.usage
-- 30 complete days, ending two days back so ingestion latency cannot clip the
-- right edge. current_date() - 31 .. - 2 is 30 days inclusive.
WHERE usage_date BETWEEN current_date() - INTERVAL 31 DAYS
AND current_date() - INTERVAL 2 DAYS
-- ORIGINAL only: this is a SHARE, and counting RETRACTION/RESTATEMENT rows
-- alongside their originals double-counts the denominator. A DOLLAR total needs
-- the netted form instead — see the library's restatement trust query.
AND record_type = 'ORIGINAL';pct_no_domain is the bad answer, and it is unfixable backwards: tags attach to compute when it starts, so untagged history stays untagged. You are measuring how much of last quarter can never be explained.system.storage.predictive_optimization_operations_history for recent OPTIMIZE and VACUUM on the gold tables, then check whether a scheduled maintenance job also exists. Neither running means small files accumulating and query times drifting up with no event to point at. Both running is rarer and worse: a hand-written OPTIMIZE competing with predictive optimization pays twice for the same work and produces conflicting commits.dwh_prod owned by a group rather than a named individual, and does system.access.audit show anyone actually reading those columns. The first is the one that gates everything else, so it is the one to run.SELECT c.column_name, t.tag_name, t.tag_value
FROM system.information_schema.columns AS c
LEFT JOIN system.information_schema.column_tags AS t
ON t.catalog_name = c.table_catalog
AND t.schema_name = c.table_schema
AND t.table_name = c.table_name
AND t.column_name = c.column_name
WHERE c.table_catalog = 'dwh_prod'
AND c.table_schema = 'gold'
AND c.table_name = 'dim_customer'
AND c.column_name IN ('email', 'phone', 'street')
ORDER BY c.column_name;tag_name is the bad answer, and it is bad specifically: with no tags there is nothing for a tag-driven ABAC policy to attach to, so every access decision stays a per-table grant someone has to remember to write. A production catalog owned by an individual is one leaving date away from an object nobody can alter.Doing it
- Join usage to system.billing.list_prices for a dollar figure, and label it 'at list' every time it appears.
- Split job DBUs by compute type before ranking — an expensive job on the right compute and a cheap one on all-purpose need opposite responses.
- Pull p50 and p90 duration per job over 30 days, not only failures. The trend is the finding; the failures are already known.
- Run one access-review query over the tagged columns for the last 30 days even if the answer is nobody. That null result is the baseline.
Defaults
- Dollars at list price, labelled, unless the client hands you their contract.
- Rank by recoverable DBUs, not total DBUs — the biggest line is often correct spend.
- One maintenance mechanism per table: predictive optimization or a scheduled job, never both.
- PII tags derived from the data model, so classification and model are one statement.
Gotchas
- Reading an idle warehouse as pure waste. Part of that tail is a deliberate warm window; check who queries it first, or the complaint after your change is a dashboard that now takes a minute to wake up.
- Chasing failure counts and ignoring duration drift. A job that went from four minutes to eleven has never failed, and it is the one that pages someone at 03:00 next quarter.
- Answering 'is predictive optimization on?' at the wrong level, in either direction. It is on by default for accounts created from 11.11.2024, the rollout to older accounts was due to finish in August 2026 — i.e. during this sweep — and it is settable at account, catalog, schema and table level. So the account answer is not the catalog answer and neither is the table answer, and a note written last quarter is already wrong. Read it per table on the gold tables before you either write a maintenance job or tell them theirs is redundant.
- Presenting the ownership finding as a security problem in front of the person who is the owner. It is an operational risk about leaving dates; framing it otherwise buys an enemy on the platform team in week one.
What you do with the answers
The sweep produces a sheet, not a slide. And the SQL for everything past week one already exists — this handbook does not restate it.
#The findings sheet, and where the SQL lives
| Column | Contains | Bad if |
|---|---|---|
| Area | cost / compute / jobs / storage / governance | a sixth category appears — the sweep grew |
| Finding | one sentence, past tense, no recommendation | it contains the word 'should' |
| Number | the measured value and its window | it is a rating rather than a measurement |
| Query | the exact SQL, or its slug in the library | 'from the audit' — nobody can re-run it |
| So what | the consequence in the client's terms | it restates the number in other words |
| Owner | a named person on the client side | a team name — or you |
| Effort | config change / one sprint / phase two | blank, so everything looks equally urgent |
Doing it
- Write the sheet during the sweep, not afterwards from notes. A finding written three days later has lost the query and gained an opinion.
- Put the library slug in the Query column so any row can be re-run by someone who was not in the room.
- Assign every row to a named client-side owner in the readout itself, while everyone is present.
Defaults
- One row per finding, with a number and a re-runnable query in every row.
- Library slugs as the citation format, so the sheet stays short and reproducible.
- The same sheet re-run at handover and diffed against the baseline.
Gotchas
- A findings sheet with no Query column. Six weeks later a number is challenged, nobody can reproduce it, and the whole sheet loses authority over one row.
- Owners assigned afterwards by email. Everything ends up owned by the platform lead, who has no time, and the sheet stops moving in week three.
- Presenting the sweep and never re-running it. Without the handover comparison nobody can tell whether the engagement improved anything or merely coincided with it.
Straight to the documentation
The generic platform material lives in the official docs and is not restated here. These are the pages worth having open while you work through this route.
- Monitor account activity with system tables — which schemas exist, which are enabled by default, and how an account admin enables the rest
- Billable usage system table reference — the record_type, custom_tags and usage_metadata columns every cost figure here depends on
- Jobs system table reference — job_run_timeline and job_task_run_timeline — the duration-drift and wrong-compute findings
- Query history system table reference — read the latency and retention notes before quoting anything from the last 24 hours
- Lineage system tables reference — the one-year rolling window, and why a missing edge is not proof of an unused table