Crosshire
← Databricks Audit
— Crosshire / Learn · Tech · Databricks · Audit

Investigation trails

Four guided reading paths through related queries — each starts at a first-audit query and follows the queries’ own where-to-go-next links as an ordered forward path. The numbered steps are the spine; the branches under each step are the “or, if…” alternatives, and links back to an earlier step are shown as see also.

These trails are derived from the manifest graph, not hand-drawn: the step order and the branch conditions come straight from each query’s next edges. The one exception is the cost trail, which is seeded with the bill-trust check as step 1 so it opens where the rest of the site does. Every threshold you meet on the way is a labelled field heuristic you tune per account.

Trail

Cost overspend: is the bill trustworthy, and where does it go?

Open with the bill-trust check (net-corrected totals), then turn DBUs into money and see whether your negotiated discount is actually landing and what price basis the numbers rest on.

  1. One row = a cloud's usage totals for the window, split by record_type. The columns that matter are net_usage_quantity (the true net, corrections already applied) versus original_usage_quantity/retracted_abs_quantity/restatement_usage_quantity (how much of that net was corrected after the fact) - this is a trust/QA check on the billing data itself, not a cost or waste signal.

  2. One row = a day + cloud + SKU + product + usage type/unit's usage, priced at list. The columns that matter are net_usage_quantity (the native-unit volume) and net_list_cost (usage_quantity x the list rate) - this is a pre-discount estimate, not what you actually pay.

  3. realization ratio at/above :warn_realization_ratio (WARN) or :crit_realization_ratio (CRITICAL) - you are paying close to list price on that SKU; or either cost column is NULL (NOT_ASSESSED - a priced-coverage gap, not $0).

  4. One row = a SKU's price at a point in time. The columns that matter are pricing_default (the list rate, cast to STRING because pricing.default is a high-precision decimal) and price_end_time (NULL means this is the currently active price for that SKU/cloud/currency).

  5. One row = a SKU's price at a point in time, with the full pricing struct preserved as JSON. The columns that matter are pricing_default (the list rate) and price_end_time (NULL means this is the currently active price) - pricing_effective_list_json and pricing_promotional_json carry the raw structs for you to parse yourself.

Trail

Job & pipeline waste: all-purpose misuse, failures, retries

Follow the jobs that run on the wrong compute into the DBUs burned by failures, retries and repairs — the recoverable pipeline spend.

  1. cluster_source IN ('UI','API') (WARN), especially with est_usd_list_share at/above :crit_share_usd (CRITICAL) - a job pinned to an always-on interactive cluster (field heuristic). The trailing NOT_ASSESSED summary row counts runs dropped for null compute_ids; do not read those as "clean".

  2. net_usage_quantity at/above :warn_job_dbus_per_day (WARN) or :crit_job_dbus_per_day (CRITICAL) DBUs/day - field heuristic; a job that is consistently in-band and just large may be fine, a job whose per-run cost keeps climbing is the one to open first.

  3. failed_run_rows at/above :warn_failed_run_rows (WARN) or :crit_failed_run_rows (CRITICAL) for a single termination_code - field heuristic; one code dominating points at a systemic cause, not one-off flakiness.

  4. One row = a (workspace, termination_code) combination in the window. Use run_rows / distinct_runs to see which termination reasons dominate before drilling into the specific failing jobs.

  5. failed_runs at/above :warn_failed_runs or wasted_dbus_proxy at/above :warn_wasted_dbus (WARN); either at/above :crit_failed_runs / :crit_wasted_dbus (CRITICAL) - field heuristic.

  6. total_retries at/above :warn_total_retries (WARN) or :crit_total_retries (CRITICAL) - field heuristic; a job where runs_with_retry is close to distinct_runs is retrying on (almost) every run, which usually points at a systemic cause.

Trail

Governance & access risk: run-as escalation to grant sprawl

Trace privilege from run-as escalation and admin-role changes down into who actually holds which grants across the metastore.

  1. status = WARN at/above :warn_runas_events, CRITICAL at/above :crit_runas_events - field heuristic; also treat a ZERO-ROW result with suspicion rather than relief - see caveats.

  2. status = WARN at/above :warn_admin_events, CRITICAL at/above :crit_admin_events - field heuristic; also worth a look whenever distinct_source_ips is unusually high for one actor.

  3. Who Holds Which Grantsaccess_grants_inventoryGovernance, Access & Security

    One row = an object scope (TABLE or CATALOG) x privilege type x grantee, rolled up. The columns that matter are grant_count (how many individual grants) and distinct_objects (how many different tables/catalogs that grantee touches).

  4. One row = an object scope (SCHEMA, CONNECTION, CREDENTIAL, or EXTERNAL_LOCATION) x privilege type x grantee, with a grant count. Use it to see who holds broad integration-level access beyond plain table/catalog grants.

Trail

Performance & spillage: the heaviest shapes, pruning and spill

Rank the heaviest statements, roll them up by shape, then chase the ones that over-scan or spill to disk.

  1. pct_of_warehouse_exec_ms at/above :warn_wh_share (WARN) or :crit_wh_share (CRITICAL) - one statement shape eating a large share of a warehouse's execution time (field heuristic).

  2. total_exec_ms at/above :warn_total_hours (WARN) or :crit_total_hours (CRITICAL) cumulative hours - a hot repeated shape (field heuristic). High runs + low avg_exec_ms means fix the caller (frequency / caching); high avg_exec_ms means tune the statement.

  3. pruning ratio below :warn_prune_ratio (WARN) or below :crit_prune_ratio (CRITICAL) - field heuristic; a stable low ratio on the same table/warehouse combination over several days is the real signal, not one bad day.

  4. shuffle_read_bytes_sum at/above :warn_shuffle_gb GB (WARN) or :crit_shuffle_gb GB (CRITICAL), OR average written-file size below :warn_avg_file_mb MB (WARN) or :crit_avg_file_mb MB (CRITICAL) - field heuristics; either signal alone is enough to flag the row.

  5. spilled_local_bytes_sum at/above :warn_spill_gb GB/day (WARN) or :crit_spill_gb GB/day (CRITICAL) - field heuristic; sustained daily spill on the same warehouse is the real signal.

Prefer a checklist? The first-audit runbook walks the ★10 in order; the printable ★10 checklist is the one-page version. Keep the reference & glossary open for the terms.