Crosshire
← Handbook
RunFirst week · system tables

14Audit & health check

This route decides what happens before you have an opinion worth charging for. The first-week sweep is read-only, runs entirely on system.* tables, and produces the thing that makes a later recommendation land: numbers from the client's own account. It is deliberately short — the SQL lives in the audit library and this page hands off to it.

Stand 29.07.2026

Why the sweep comes first

Week one is a room full of confident, contradictory statements about one account. The sweep replaces them with a table.

#Opinion into evidence

What
The platform lead says cost is under control. The engineers say the jobs are fine, it is the cluster policy. Finance says the bill doubled in April and nobody could explain it. Three people describing one account, none of whom has run a query.
The audit converts those statements into evidence before you propose anything. Five areas, one question each, a recorded decision per finding. Not a code review and not a health score — a measurement of the account as it is today, taken before you touch it, and therefore a baseline. Sweep on day three, re-run the identical queries at handover. 'Untagged spend went from 38% to 4%' is a sentence you can only write if you measured on day three.
Why here
This prevents the correct recommendation that gets rejected. Liquid clustering on gold.fct_order_line, serverless for the nightly load, a tag taxonomy — all right, and all indistinguishable from the deck the last vendor left. It also prevents the opposite: a team says the loads are slow, and the sweep shows the nightly job sharing an all-purpose cluster with four notebooks, its duration tracking contention rather than data volume.
Doing it
  • Run it in their workspace with them watching — the questions they ask while it runs are half the value.
  • Save the exact SQL per finding, so the handover sweep is a re-run and not a re-derivation.
  • Record every result the same day, healthy ones included. An unrecorded green area gets re-litigated in month three.
Embedding it
  • Ask them to predict each number before you run the query. Being wrong about your own account is memorable in a way a slide is not.
  • Put a client engineer on the keyboard while you read the queries out. They keep the muscle memory; you keep the day.
  • When a result is disputed, open the query with them rather than defending it. A number they can reproduce stops being your number.
Defaults
  • One day, five areas, one question each. Depth only where the sweep pointed.
  • Baseline before any change; identical SQL at handover.
  • Measurement and proposal are separate documents and separate sessions.
Gotchas
  • Sweeping and recommending in one breath. The client argues about the fix and stops reading the number, and the number was the hard part to get.
  • Sweeping from your laptop against an export. Nothing is reproducible, so the first challenged figure becomes a credibility problem instead of a re-run.
  • Letting the sweep grow to 100 queries because the library has 100. You notice it in the findings sheet: forty rows, no Owner column filled, because forty findings compete with each other and none of them is anybody's. Week one answers five questions; the rest are for whatever week one flagged.

#Read-only, and nothing to install

What
Every query is a SELECT against the system catalog — billing, compute, lakeflow, query, storage, access and information_schema. No agent, no collector, no scheduled notebook, no data leaving the account. The commercial consequence: the sweep needs a grant, not a security review, and a grant takes an afternoon.
sql
Query zero — which schemas can this identity actually see?
SELECT table_schema, COUNT(*) AS tables
FROM   system.information_schema.tables
WHERE  table_catalog = 'system'
GROUP BY table_schema
ORDER BY table_schema;
Three limits bound what the sweep can claim. Latency: usage and query-history rows arrive hours late, so a sweep run at 09:00 about yesterday reads a partial day. Restatement: system.billing.usage corrects the past by appending RETRACTION and RESTATEMENT rows rather than editing history, so recent dollars are still moving. Retention: the lineage system tables keep a rolling one-year window, while Catalog Explorer and the lineage API retain lineage captured after 01.09.2024 indefinitely — a question reaching further back is answerable in the UI, not in SQL.
Why here
This prevents reporting an absence as a finding. 'No PII columns are tagged' and 'I cannot read system.information_schema.column_tags' produce the same empty result, and only one of them is true. The same trap sits under lineage: both lineage tables are a subset, emitted only where lineage could be inferred, so a missing edge means not captured — never not happening.
Doing it
  • Ask for SELECT on the system schemas in the kickoff mail. It is the only access the sweep needs and the only thing that can delay it.
  • Run query zero first; record the missing schemas alongside the findings.
  • Anchor cost findings on a complete 30-day window ending two days back, and run the library's restatement trust query before any of them — a window whose restated share is large makes every dollar figure computed over it provisional, and you want to know that before the readout, not after.
Embedding it
  • Tell them up front that your queries appear under your identity in system.query.history and system.access.audit, and show them where. Volunteering your own audit trail buys more trust than any assurance about being read-only.
  • Have them run query zero on their own identities. Analysts and engineers see different schemas, and discovering that beats being told.
  • Give the missing-schema list to the account owner and let an employee raise it internally, not you.
Defaults
  • SELECT on system schemas, nothing else. A sweep query needing write access is not a sweep query.
  • Query zero before every sweep, including the handover re-run — enablement changes.
  • State the retention boundary rather than silently answering a shorter question than the one asked.
Gotchas
  • A disabled system schema returning zero rows, read as a healthy answer. It is the easiest way to hand a client a confidently wrong governance finding in week one.
  • Quoting a saving from a period that later restates. The correction lands after your readout, and the first number you ever gave them turns out to be wrong.
  • Treating a missing lineage edge as proof a table is unused. Dropping it takes down a report nobody could see.
  • Comparing a partial day to a full one. Ingestion latency puts a phantom cliff at the right edge of every chart you make today.

The first-week sweep

Five areas, one question each. You are not looking for a score but for a bad answer — the shape of result that decides where day two goes.

#Cost, compute, jobs, storage, governance

What
One question per area, and the answer that means keep digging
AreaThe one questionA bad answer looks likeSource
CostWhere does the bill go, and can you attribute it?a large share of DBUs with no domain or environment tagsystem.billing.usage, list_prices
ComputeWhat runs when nothing is being asked of it?all-purpose clusters with no auto-termination; long warehouse idle tailssystem.compute.clusters, warehouse_events
JobsWhat fails, what runs long, what runs on the wrong compute?job tasks on all-purpose compute; repeat failures still burning DBUssystem.lakeflow.job_run_timeline, job_tasks
StorageIs maintenance happening, and who is doing it?no maintenance at all — or two mechanisms on one tablesystem.storage.predictive_optimization_operations_history
GovernanceWho can read personal data, and who did?untagged PII columns; production objects owned by a named personsystem.information_schema.*, system.access.audit
Cost. Can spend be attributed to a domain and an environment at all? An unattributable bill cannot be argued about — only feared.
sql
Is the bill attributable? — tag taxonomy from the storage & cost route
SELECT
  SUM(usage_quantity)                                AS dbus,
  ROUND(100 * SUM(CASE WHEN custom_tags['domain'] IS NULL
                       THEN usage_quantity ELSE 0 END)
            / NULLIF(SUM(usage_quantity), 0), 1)     AS pct_no_domain,
  ROUND(100 * SUM(CASE WHEN custom_tags['environment'] IS NULL
                       THEN usage_quantity ELSE 0 END)
            / NULLIF(SUM(usage_quantity), 0), 1)     AS pct_no_environment
FROM system.billing.usage
-- 30 complete days, ending two days back so ingestion latency cannot clip the
-- right edge. current_date() - 31 .. - 2 is 30 days inclusive.
WHERE usage_date BETWEEN current_date() - INTERVAL 31 DAYS
                     AND current_date() - INTERVAL 2 DAYS
-- ORIGINAL only: this is a SHARE, and counting RETRACTION/RESTATEMENT rows
-- alongside their originals double-counts the denominator. A DOLLAR total needs
-- the netted form instead — see the library's restatement trust query.
  AND record_type = 'ORIGINAL';
A double-digit pct_no_domain is the bad answer, and it is unfixable backwards: tags attach to compute when it starts, so untagged history stays untagged. You are measuring how much of last quarter can never be explained.
Compute. All-purpose clusters with auto-termination unset or generous, warehouses whose idle tail between the last query and the stop event is a large share of running time, clusters whose owner has left. A few long-lived all-purpose clusters carrying a disproportionate share of the DBUs above is usually the largest recoverable number in the sweep, and it needs no code change.
Jobs. Three bad answers, in increasing order of usefulness. Repeat failures still consuming DBUs mean nobody reads the alerts. Job tasks pinned to all-purpose compute mean the job was built in a notebook and never promoted. A p50 duration drifting upward over 30 days is the one people miss: it never fails, appears in no alert, and it is what breaks the SLA next quarter.
Storage. Check system.storage.predictive_optimization_operations_history for recent OPTIMIZE and VACUUM on the gold tables, then check whether a scheduled maintenance job also exists. Neither running means small files accumulating and query times drifting up with no event to point at. Both running is rarer and worse: a hand-written OPTIMIZE competing with predictive optimization pays twice for the same work and produces conflicting commits.
Governance. Three questions, in this order: are the personal-data columns tagged at all, is dwh_prod owned by a group rather than a named individual, and does system.access.audit show anyone actually reading those columns. The first is the one that gates everything else, so it is the one to run.
sql
Are the personal-data columns on dim_customer classified?
SELECT c.column_name, t.tag_name, t.tag_value
FROM      system.information_schema.columns     AS c
LEFT JOIN system.information_schema.column_tags AS t
       ON t.catalog_name = c.table_catalog
      AND t.schema_name  = c.table_schema
      AND t.table_name   = c.table_name
      AND t.column_name  = c.column_name
WHERE c.table_catalog = 'dwh_prod'
  AND c.table_schema  = 'gold'
  AND c.table_name    = 'dim_customer'
  AND c.column_name  IN ('email', 'phone', 'street')
ORDER BY c.column_name;
Three rows with a NULL tag_name is the bad answer, and it is bad specifically: with no tags there is nothing for a tag-driven ABAC policy to attach to, so every access decision stays a per-table grant someone has to remember to write. A production catalog owned by an individual is one leaving date away from an object nobody can alter.
Why here
The order is deliberate. Cost, compute and jobs are where the money is and where the fix is configuration, so the sweep can pay for the engagement before anyone argues about the data model. Storage and governance are last in the sweep and first in the risk register: neither produces a number finance cares about, and both produce the finding that ends an engagement badly if it surfaces in month five.
Doing it
  • Join usage to system.billing.list_prices for a dollar figure, and label it 'at list' every time it appears.
  • Split job DBUs by compute type before ranking — an expensive job on the right compute and a cheap one on all-purpose need opposite responses.
  • Pull p50 and p90 duration per job over 30 days, not only failures. The trend is the finding; the failures are already known.
  • Run one access-review query over the tagged columns for the last 30 days even if the answer is nobody. That null result is the baseline.
Embedding it
  • Ask them to guess the untagged percentage before you run it. The guess is always low, and the gap turns the tag taxonomy from a chore into a priority.
  • Have the engineer who owns the nightly job find its compute type themselves, in front of everyone. Nobody builds the next job on all-purpose compute after that.
  • Ask who owns dwh_prod before you run the query, then run it. Room answer and catalog answer differ often enough to be worth the theatre.
  • When they propose hashing the email column to close the finding, send them the classification route rather than correcting it in the room — pseudonymisation is a rule they have to arrive at.
Defaults
  • Dollars at list price, labelled, unless the client hands you their contract.
  • Rank by recoverable DBUs, not total DBUs — the biggest line is often correct spend.
  • One maintenance mechanism per table: predictive optimization or a scheduled job, never both.
  • PII tags derived from the data model, so classification and model are one statement.
Gotchas
  • Reading an idle warehouse as pure waste. Part of that tail is a deliberate warm window; check who queries it first, or the complaint after your change is a dashboard that now takes a minute to wake up.
  • Chasing failure counts and ignoring duration drift. A job that went from four minutes to eleven has never failed, and it is the one that pages someone at 03:00 next quarter.
  • Answering 'is predictive optimization on?' at the wrong level, in either direction. It is on by default for accounts created from 11.11.2024, the rollout to older accounts was due to finish in August 2026 — i.e. during this sweep — and it is settable at account, catalog, schema and table level. So the account answer is not the catalog answer and neither is the table answer, and a note written last quarter is already wrong. Read it per table on the gold tables before you either write a maintenance job or tell them theirs is redundant.
  • Presenting the ownership finding as a security problem in front of the person who is the owner. It is an operational risk about leaving dates; framing it otherwise buys an enemy on the platform team in week one.

What you do with the answers

The sweep produces a sheet, not a slide. And the SQL for everything past week one already exists — this handbook does not restate it.

#The findings sheet, and where the SQL lives

What
One row per finding, no row without a number. These columns are the whole format: it fits in a spreadsheet, it survives being forwarded, and it is what the next phase gets scoped from.
The findings sheet
ColumnContainsBad if
Areacost / compute / jobs / storage / governancea sixth category appears — the sweep grew
Findingone sentence, past tense, no recommendationit contains the word 'should'
Numberthe measured value and its windowit is a rating rather than a measurement
Querythe exact SQL, or its slug in the library'from the audit' — nobody can re-run it
So whatthe consequence in the client's termsit restates the number in other words
Ownera named person on the client sidea team name — or you
Effortconfig change / one sprint / phase twoblank, so everything looks equally urgent
This route says what to look at in week one and why those five questions; the library says how to ask them. Pruning ratios, spill, per-recipient sharing, network egress — all library, none of it repeated here.
Why here
A sheet with a query slug in every row is re-runnable after you leave, which is the difference between an audit and an audit capability. The handover version of this engagement is not a document; it is the team running the same ten queries without you. The format also protects the sweep from how it usually degrades: a findings sheet that fills with recommendations becomes a proposal, and a proposal gets negotiated. Numbers do not.
Doing it
  • Write the sheet during the sweep, not afterwards from notes. A finding written three days later has lost the query and gained an opinion.
  • Put the library slug in the Query column so any row can be re-run by someone who was not in the room.
  • Assign every row to a named client-side owner in the readout itself, while everyone is present.
Embedding it
  • Have the team fill the last two areas themselves while you watch. They will write a recommendation into the Finding column and correct themselves once, which teaches the distinction better than the rule does.
  • Book the handover re-run in week one with a client name against it, not yours.
  • At handover, check that one finding was closed by someone who was not in the original sweep. That is the evidence the capability transferred.
Defaults
  • One row per finding, with a number and a re-runnable query in every row.
  • Library slugs as the citation format, so the sheet stays short and reproducible.
  • The same sheet re-run at handover and diffed against the baseline.
Gotchas
  • A findings sheet with no Query column. Six weeks later a number is challenged, nobody can reproduce it, and the whole sheet loses authority over one row.
  • Owners assigned afterwards by email. Everything ends up owned by the platform lead, who has no time, and the sheet stops moving in week three.
  • Presenting the sweep and never re-running it. Without the handover comparison nobody can tell whether the engagement improved anything or merely coincided with it.

Straight to the documentation

The generic platform material lives in the official docs and is not restated here. These are the pages worth having open while you work through this route.

Platform facts on this page verified 29.07.2026 against the official documentation. Volatile claims are anchored to the currency register.