Crosshire
← Databricks Audit
— Crosshire / Learn · Tech · Databricks · Audit

System-table map

An entity-relationship diagram of the 50 system.* tables the audit library reads, across 10 schemas. Where the query lineage shows how the queries connect, this shows how the tables connect — the data model underneath. Start with the schema overview, then drop into a subject area for its tables and join keys.

This is grounded, not hand-drawn: the tables are exactly what the queries read, and every line is a join the SQL actually makes — the usage_metadata foreign keys and the JOIN … ON clauses mined from the query bodies. Read each line as child → parent: is the many end, 1 the one end, and the label is the join key. PK / FK / K badges mark the primary key, foreign keys and business keys on each table.

  • Primary key
  • Foreign key
  • Business / join key
  • One-to-many

How the schemas connect

Six subject areas. system.billing.usage is the hub: its usage_metadata struct is the foreign key that attributes every DBU out to compute, jobs and serving, and workspace_id ties it to the access schema. Governance hangs off the securables that lineage points at. The label on each link is the joining column.

Each subject area, in detail

One ER diagram per subject area: its tables, their primary and foreign keys, and the join that links them. Foreign keys that reach into another subject area leave as a “joins out” caption. Wide diagrams scroll sideways.

Billing & cost5 tables

system.billing.usage is the account's spine: one row per SKU per hour. It prices against list_prices / account_prices on a time-windowed join, and its usage_metadata struct is the foreign key that attributes every DBU to a cluster, job, warehouse, pipeline or endpoint.

The library's dominant join (used by 22 queries): each usage row matches the list price whose [price_start_time, price_end_time) window covers usage_end_time. est · at list — DBU-only, pre-discount.Same time-windowed match, but against the account's negotiated prices — only when a query explicitly opts into account_prices instead of list_prices.attributed_usage is the pre-attributed rollup of the same consumption — same record grain, keyed back to usage.Cloud VM / infra cost sits alongside the DBU bill; aligned by workspace, cloud and day (not a PK↔FK — a day-grain roll-up).
billingusage
hub · 1 row / SKU / hour
  • record_id
  • sku_name · cloud
  • workspace_id
  • usage_metadata.*
  • usage_quantity (DBU)
billinglist_prices
est · at list
  • sku_name · cloud
  • usage_unit
  • price_start · end_time
  • pricing.default
billingaccount_prices
negotiated
  • sku_name · cloud
  • price_start · end_time
  • price (discounted)
billingcloud_infra_cost
VM / infra $
  • workspace_id · cloud
  • usage_date
  • cost
billingattributed_usage
attributed view
  • record_id
  • billing_origin_product
  • usage_quantity
1sku · cloud · window1sku · cloud · windowrecord_idws · cloud · day

Compute8 tables

Clusters, SQL warehouses and instance pools — plus the per-minute node timeline and the node-type catalogue that size them. query.history bridges here: every statement records the warehouse or cluster it ran on.

One node-timeline row per node per minute rolls up to its cluster.Each cluster's driver and worker node types resolve to the instance-type catalogue (cores, memory, GPU).A cluster can draw its driver / workers from an instance pool.Pool up/down-scale events belong to a pool.Warehouse start / stop / scaling events belong to a warehouse.Every SQL statement records the warehouse (or cluster) it executed on.Statements on all-purpose / job compute record a cluster_id instead of a warehouse_id.
computeclusters
latest snapshot / cluster
  • cluster_id
  • workspace_id
  • driver / worker_node_type
  • driver / worker_instance_pool_id
  • owned_by
computenode_timeline
per-minute utilisation
  • cluster_id
  • instance_id · start_time
  • cpu / mem_utilization
computenode_types
instance-type catalogue
  • node_type
  • core_count · memory_mb
  • gpu_count
computeinstance_pools
  • instance_pool_id
  • workspace_id
  • node_type_id
computeinstance_events
  • instance_pool_id
  • event_type · event_time
computewarehouses
SQL warehouse
  • warehouse_id
  • workspace_id
  • warehouse_name · size
computewarehouse_events
  • warehouse_id
  • event_type · cluster_count
queryhistory
one row / statement
  • statement_id
  • compute.warehouse_id
  • executed_by · read_bytes
1cluster_id1node_type1instance_pool_id1instance_pool_id1warehouse_id1warehouse_idcluster_id

Jobs & pipelines (lakeflow)6 tables

The lakeflow schema is a clean parent→child hierarchy: a job has tasks and runs; a run has task-runs; a DLT pipeline has updates. Job / pipeline ids are how billing.usage attributes spend to a workflow.

A job's task graph.One timeline row per job run.Task-runs belong to a job run.Each task-run is an execution of a defined task.One row per DLT pipeline update.
lakeflowjobs
latest snapshot / job
  • job_id
  • workspace_id
  • name · run_as · creator_id
lakeflowjob_tasks
  • job_id
  • task_key
  • depends_on
lakeflowjob_run_timeline
one row / run
  • job_id
  • run_id
  • result_state · period
lakeflowjob_task_run_timeline
one row / task-run
  • job_id · run_id
  • task_key
  • task_run_id
lakeflowpipelines
DLT pipeline
  • pipeline_id
  • workspace_id
  • name · created_by
lakeflowpipeline_update_timeline
  • pipeline_id
  • update_id
  • state
1job_id1job_id1job_id · run_id1job_id · task_key1pipeline_id

Serving & AI gateway3 tables

A served entity (model / external endpoint) is the parent; per-request usage and AI-gateway logs hang off its endpoint id. billing.usage attributes serving DBUs by the same endpoint id.

One usage row per inference request against a served entity.AI-gateway request logs join to the endpoint by name (gateway tracking must be on, else empty).
servingserved_entities
model / external endpoint
  • served_entity_id
  • endpoint_id · endpoint_name
  • workspace_id
  • entity_type
servingendpoint_usage
per request
  • served_entity_id
  • endpoint_id
  • request_time · tokens
ai_gatewayusage
gateway request log
  • endpoint_name
  • workspace_id
  • request / response tokens
1endpoint_idendpoint_name

Access, audit & lineage6 tables

workspaces_latest is the workspace dimension nearly every system table joins to by workspace_id. The audit log is the raw event stream; table / column lineage records what read or wrote what, keyed to the job run or statement that did it.

Every audit event names its workspace; workspaces_latest resolves the id to a name.Inbound network access events, per workspace.Outbound (serverless egress) events, per workspace.Column lineage is the same edges at column grain.
accessworkspaces_latest
workspace dimension
  • workspace_id
  • workspace_name · url
  • status
accessaudit
raw event log
  • workspace_id
  • user_identity.email
  • service · action_name
  • request_params · response
accessinbound_network
  • workspace_id
  • source_ip · event_time
accessoutbound_network
  • workspace_id
  • destination · event_time
accesstable_lineage
who read / wrote what
  • source / target_table_full_name
  • entity_type · entity_run_id
  • event_time
accesscolumn_lineage
column grain
  • source / target_table_full_name
  • source / target_column_name
1workspace_id1workspace_id1workspace_idtable · column

Catalog & governance3 tables

The information_schema securables — tables, views, volumes — are the hub. Tags, grants, masking / row-filter policies, Delta-Sharing usage, PII classification and storage optimisation all attach to a securable by its catalog · schema · table name. Related tables are grouped into families to keep the map readable; each row is a real system table.

Views are securables in the same namespace as tables.table / column / schema / volume tags attach to their securable by name.One grant row per (grantee, privilege, securable) across tables, schemas, catalogs, connections, credentials and external locations.Column masks and row filters bind a UDF to a securable column (manual UC policies only — ABAC is invisible here).PII classification results and storage optimisation / metrics history are keyed to the table they scanned — column_masks ↔ data_classification.results is a real catalog·schema·table·column join.Which securables are in which Delta share, and who may read them.volume_tags key to volumes rather than tables.
information_schematables
securable hub
  • table_catalog · schema · name
  • table_type · table_owner
information_schemaviews
  • table_catalog · schema · name
  • view_definition
information_schemavolumes
  • volume_catalog · schema · name
  • volume_type
tags (by securable · tag_name)
  • table_tags
  • column_tags
  • schema_tags
  • volume_tags
*_privileges (grantor · grantee)
  • table_privileges
  • schema · catalog_privileges
  • connection · credential_privileges
  • external_location_privileges
access policies (by column)
  • column_masks
  • row_filters
classification & storage
  • data_classification.results
  • storage.predictive_optimization…
  • storage.table_metrics_history
Delta Sharing
  • shares
  • share_recipient_privileges
  • table · schema_share_usage
1catalog·schema·name1securable · tagsecurable1securable · column1catalog·schema·table1share → tablevolume name

Now that you know how the tables connect, see how the queries connect, walk the investigation trails, or check which queries can come back empty on the coverage page.