Crosshire
← Databricks Audit
— Crosshire / Learn · Tech · Databricks · Audit

War stories

Three findings told the way you actually meet them: a symptom, a wrong first guess, and the one query that cracked it. Each ends in a fix and a measured effect.

Synthetic scenarios — illustrative. The patterns and mechanisms are real and recur across accounts; the customers, tables and numbers are invented. No real account data appears anywhere on this page.

The nightly job that paid the interactive rate

A jobs bill up a third month-over-month, and nobody added a job.

Symptom
The jobs product line jumped ~30% month-over-month. Run counts were flat, data volumes were flat, and no new workflows shipped. Finance wanted the number explained by Friday.
First hypothesis (wrong)
“Data must be growing, or a job got bigger.” The team spent a day profiling input sizes and shuffle — and found nothing that explained a third of the bill. The workloads were the same size they had always been.
The query
Instead of chasing volume, they asked where the tasks ran. lakeflow_jobs_on_all_purpose counts task-runs per job per cluster and tags each cluster's cluster_source, so a scheduled task pinned to a UI- or API-created (all-purpose) cluster stands out immediately.
Output excerpt
job_idcompute_idcluster_sourcetask_runsnet_dbusest_usd_list
8844220715-091234-ab12cd34UI61214,2007,100.00
7755330715-090011-jobclu99JOB1,44030,50015,250.00
Synthetic scenario — illustrative
Mechanism
Six weeks earlier an engineer had cloned a job by attaching it to an existing interactive cluster “just to test.” The schedule stuck. All-purpose compute bills at a materially higher DBU SKU than jobs compute for identical work — so every one of those 612 nightly task-runs paid the interactive rate for a batch job.
The fix
Point the job at a job cluster (or serverless). One config change, no code. The JOB-source row beside it in the output was the control: same shop, correct placement.
Measured effect
The all-purpose line for that job — ~$7,100 est · at list for the window — collapsed to the jobs-SKU equivalent, a recurring saving every night going forward. Total time to find it once the right query was run: under an hour.

“We're 100% tagged” — and nobody could be billed

A perfect tagging-coverage dashboard that produced zero real chargeback.

Symptom
FinOps reported 100% tag coverage and a green compliance tile. Yet when they tried to bill spend back to teams, every path led to one shared bucket, and no team recognised its own number.
First hypothesis (wrong)
“Coverage is 100%, so attribution is solved — the chargeback model just needs wiring up.” The coverage metric was technically true and completely useless.
The query
cost_chargeback_by_tag keeps untagged usage as NULL-key rows (so the untagged denominator is real) and lets you count distinct tag_values per tag_key. That second number is what the coverage tile never showed.
Output excerpt
billing_origin_producttag_keytag_valuenet_usage_quantityrecord_count
JOBScost_centershared-platform9,850.00412
DLTNULLNULL1,560.2088
Synthetic scenario — illustrative
Mechanism
A cluster policy auto-stamped cost_center = shared-platform on almost everything. Coverage looked like 100% because nearly every row carried the key — but the key resolved to a single value, so it partitioned nothing. And a DLT slice slipped the policy entirely, carrying no tag at all: real untagged spend the coverage tile counted as covered by rounding.
The fix
Retire the umbrella value and require a real per-team / cost-center dimension at cluster-policy time; treat the NULL-key slice as a hard backlog with named owners, not a rounding error.
Measured effect
“100% coverage” resolved to its honest figure: one umbrella value covering the bulk of DBUs plus a real untagged slice (here ~1,560 DBUs of DLT). The measured win is not a dollar saving — it is that six- and seven-figure spend became attributable, so it could finally be billed back instead of cross-subsidised.
Governance, Access & Securityaccess_classified_unmasked

Locked-down grants, wide-open PII

The access review passed. The data was still in the clear.

Symptom
A quarterly access review signed off: grants were tight, only the analytics group could SELECT the customer tables. An external audit then flagged customer email addresses returned in plaintext — from those same “locked-down” tables.
First hypothesis (wrong)
“Grants are restricted, so the PII is protected.” The review had checked who could query the tables and stopped there.
The query
access_classified_unmasked joins HIGH-confidence auto-classified columns to the column-mask inventory and flags the ones with no mask — the columns that are sensitive and unprotected, independent of who holds the grant.
Output excerpt
catalog_nameschema_nametable_namecolumn_nameclass_tagMASK_NAMEis_unmasked
prodcustomersaccountsemailemail_addressNULLtrue
prodcustomersaccountsssnus_ssnmask_ssn_defaultfalse
Synthetic scenario — illustrative
Mechanism
Grants control who can run a query; column masks control what those people see. SSN had a mask, so it read as false — protected. Email never got one, so every one of the dozens of analysts with a legitimate SELECT grant saw raw addresses. A tight grant list is not a mask.
The fix
Apply a column mask to each is_unmasked = true HIGH-confidence column, then re-run to confirm the row flips to false. (Verify the mask genuinely doesn't exist first — a true can also mean the collecting principal simply couldn't see the mask table.)
Measured effect
The unmasked HIGH-confidence PII columns were masked and the audit finding closed. There is no est · at list here — the measured effect is exposure removed, not dollars saved: a real breach or GDPR/CCPA penalty is the cost this avoids.

Want to find these on your own account? Start with the first-audit runbook, and keep the reference & glossary open for the terms.