Crosshire
Audit & lineage
Govern18 Audit & lineage · 1 of 3

The audit log

Every Unity Catalog action is already being recorded. The work is not collecting it — it is turning it into three specific answers before someone urgently needs one of them.

Stand 29.07.2026

#What system.access.audit records

What
Unity Catalog writes access and administration events to system.access.audit with no agent and no exporter. Every securable resolution, permission change, table creation and drop lands there, attributed to a principal, an IP address and a request id.
Three columns carry most of the weight: action_name, user_identity.email, and request_params — a MAP<STRING, STRING> whose keys differ per action, and which is the single most common reason one of these queries silently returns nothing. getTable is documented with full_name_arg; generateTemporaryTableCredential is documented with full_name and widely reported as table_full_name; updatePermissions carries full_name plus changes. Do not pick one key and hope — coalesce the candidates, and confirm the actual key set on the client's workspace before any of this ships.
Why
This prevents discovering, mid-incident, that you have opinions instead of records. Somebody exported a customer list; the question is who, and the honest answer is a shrug and a fortnight of interviews.
The slower failure is more common in DACH projects: a data protection officer asks for evidence that access to gold.dim_customer.email is controlled, and the team sends a screenshot of the grants. A screenshot proves today. The question was about six months of yesterdays.
How
sql
Run this FIRST — which keys does each action actually carry here?
SELECT action_name,
       array_sort(map_keys(request_params)) AS param_keys,
       count(*)                             AS events
FROM   system.access.audit
WHERE  service_name = 'unityCatalog'
  AND  action_name IN ('getTable', 'generateTemporaryTableCredential',
                       'updatePermissions',
                       'createEntityTagAssignment',
                       'deleteEntityTagAssignment')
  AND  event_date >= current_date() - INTERVAL 7 DAYS
GROUP  BY action_name, array_sort(map_keys(request_params))
ORDER  BY action_name, events DESC;
sql
Orientation query — what has this principal been doing in dwh_prod
SELECT event_time,
       action_name,
       -- One key per action, and the names are not interchangeable:
       -- full_name_arg (getTable) / table_full_name / full_name.
       -- Verify against the probe query above before shipping.
       coalesce(request_params.full_name_arg,
                request_params.table_full_name,
                request_params.full_name)     AS securable,
       response.status_code                   AS status,
       source_ip_address
FROM   system.access.audit
WHERE  service_name        = 'unityCatalog'
  AND  user_identity.email = 'sp-dwh-prod-loader'
  AND  event_date >= current_date() - INTERVAL 7 DAYS
ORDER  BY event_time DESC
LIMIT  200;
Three uses, three shapes, three different people asking
UseThe questionQuery shapeWho asks
Access reviewWho could read personal data last quarter, and who did?90 days, grouped by principal and tabledata protection officer, internal audit
Incident forensicsBetween 14:00 and 16:00 that day, who touched dim_customer?narrow window, full detail, no aggregationwhoever is running the incident
GDPR evidenceShow that access to email and phone is restricted and reviewed.the review query, plus the fact that it runs on a schedulethe customer's lawyer, eventually
Doing it
  • Have a metastore admin enable the system schemas on day one, before there are tables to audit
  • Run the map_keys probe on the client's own metastore before writing anything elseBuild every later query from the key set you saw there rather than from a query library.
  • Write the three queries once, save them as a dashboard, and hand over the dashboard — not SQL in a chat message
  • Measure delivery latency on the client's own workspaceEvents arrive in minutes, not seconds, and an alert assuming real time fires late and gets distrusted.
  • Enable verbose audit logging if interactive notebook commands matter to their compliance story
Embedding it
  • Give the team the incident question, not the query'Someone exported customer data at 15:00 last Tuesday — find them.' Let them discover which action_name they need.
  • Have whoever will field the DPO's request write the review queryThe person under time pressure later has then already built the tool.
  • When they hand you a query with no response.status_code, ask what a denied attempt looks like in their result setDenials are the interesting rows.
Defaults
  • Enable system schemas at metastore setup, not when the first request arrives
  • Filter on event_date, not event_timeThe table is laid out by date and the difference is a full scan.
  • Always carry response.status_code; counting only successes hides every attempt correctly refused
  • Coalesce the request_params name keys instead of hard-coding oneRe-run the probe whenever a query stops returning what it used to.
  • Say 'Public Preview' out loud in the deliverable that rests on this table, once, in writing
Gotchas
  • Jobs are attributed to the service principal that runs them, not the human who triggered the runA review over user_identity.email shows sp-dwh-prod-loader touching every PII table daily and no person anywhere — check identity_metadata before concluding no human read anything.
  • Audit retention is a rolling window — 365 days at the time of writingStand 29.07.2026; re-read the system-table reference before putting a number in a concept document. A question spanning 'since the project started' hits that edge silently: rows come back, just not all of them, and nothing in the result says so.
  • Filtering only on action_name = 'getTable' and calling it 'who read the data'getTable is metadata resolution; credential-based reads appear under generateTemporaryTableCredential. Miss the second and the review understates reality.
  • Hard-coding request_params.full_name_arg for every actionIt is the documented key for getTable; the credential-vending event is documented with full_name and reported in the field as table_full_name. Read the wrong key and the column is NULL on every data-access row — which then drops out of any join — so you get exactly the understated review the previous gotcha warns about, from a query that looks right and runs clean.
  • Looking here for query textIt is not here — statement text lives in query history (itself Public Preview), with different retention and, because WHERE clauses carry literals, arguably more sensitive content than the audit log itself.
  • Quoting a Preview table as a permanent control in a signed conceptWhen the schema moves, the correction is yours to issue, and it lands on the one deliverable the client shows their regulator.

#The standing access review

What
A review that lists tables is theatre. The one worth running answers a narrower question: for every column classified as personal data, which principals actually touched the table carrying it, and when did they stop? The tags are already in Unity Catalog, so this is one join rather than a spreadsheet.
Why
Grants accumulate. Somebody needed gold.dim_customer for a one-off analysis in March, it was issued in a hurry, and it is still there in November because no process removes anything. Every one is an extra row in the answer to 'who could have seen this data'.
Driving the review off classification tags rather than a table list is what makes it survive: a new table carrying an email column joins the review the moment it is tagged, with nobody remembering anything. Same argument as tag-driven ABAC, applied to evidence instead of enforcement.
How
sql
90-day access review over PII-tagged tables in dwh_prod
-- information_schema is PER CATALOG: this must be dwh_prod's, not system's.
WITH pii_tables AS (
  SELECT DISTINCT
         catalog_name || '.' || schema_name || '.' || table_name AS full_name
  FROM   dwh_prod.information_schema.column_tags
  -- The governed tag is 'pii' with values email | phone | address (see the
  -- Access & ABAC route). Match on the KEY, not on a value -- a value filter
  -- silently drops any class you add later, and this query failing open looks
  -- exactly like a clean review.
  WHERE  tag_name = 'pii'
),
touches AS (
  SELECT user_identity.email  AS principal,
         -- getTable is documented with full_name_arg; the credential-vending
         -- event is NOT -- it is documented with full_name and reported as
         -- table_full_name. Pick one key and every data-access row joins to
         -- NULL and disappears. Confirm the keys with the map_keys probe in
         -- 'What system.access.audit records' before you trust this output.
         coalesce(request_params.full_name_arg,
                  request_params.table_full_name,
                  request_params.full_name) AS full_name,
         event_time,
         response.status_code  AS status
  FROM   system.access.audit
  WHERE  service_name = 'unityCatalog'
    AND  action_name IN ('getTable', 'generateTemporaryTableCredential')
    AND  event_date >= current_date() - INTERVAL 90 DAYS
)
SELECT t.principal,
       t.full_name,
       count_if(t.status =  200) AS allowed,
       count_if(t.status <> 200) AS refused,
       max(t.event_time)         AS last_touch
FROM   touches t
JOIN   pii_tables p ON p.full_name = t.full_name
GROUP  BY t.principal, t.full_name
ORDER  BY allowed DESC;
Two columns make this a review rather than a report. refused shows people trying doors that are correctly locked — usually a broken workflow, occasionally something worse. last_touch is the revocation list: a principal whose last touch was four months ago holds a grant nobody needs.
Doing it
  • Run it quarterly on a schedule and store the outputThe record that a review happened is itself the deliverable.
  • Work last_touch first: everything past 90 days is a revocation candidateRevoking is the fastest measurable win in the governance workstream.
  • Split service principals from humans before presentingA DPO reading a list where sp-dwh-prod-loader has 40,000 events will not find the two rows that matter.
  • Cross-check against the grantsA principal in the grants but never in the log is dead access; one in the log but not the grants means your tag coverage is wrong.
Embedding it
  • Hand the quarterly run to the client with a calendar entry and a named deputyA review that happens only when the consultant is on site stops at handover.
  • Have them present the first review to their own DPO with you in the room but silentWhat the DPO pushes back on is the real requirement.
  • Ask what a person who left the company should look like in this outputThen let them check whether that is true today. It usually is not.
Defaults
  • Drive the review off classification tags, never a hand-maintained table list
  • Report allowed and refused separately; refusals are the signal
  • Every review ends with a revocation list somebody executes that week
Gotchas
  • Reading the tags from system.information_schemaIt exists, the query runs, and it describes the system catalog only — information_schema is per catalog, so pii_tables comes back empty, the JOIN removes every row, and the review reports that nobody touched personal data. An empty access review is the one result nobody questions.
  • Reviewing tables while classifying columnsA table joins the review only if a column on it is tagged, so an untagged PII column is invisible to the review AND to any ABAC policy — one missing tag removes a table from both enforcement and evidence, silently.
  • Filtering the review on a tag key or value that nothing actually carriesWrite tag_name = 'data_classification' when the governed tag is pii and pii_tables returns nothing, the join eliminates every row, and the review reports that no one touched personal data. It is the same failure as the empty review above, with a cause that survives code review because the SQL is faultless. Governed tags with a declared value list are what stop the drift; a free-text tag will not.
  • Taking the securable name from a single request_params keyThe credential-vending rows come back with NULL, the JOIN drops them, and the review reports metadata resolutions as if they were the whole story — a third empty-by-construction result, and the one that survives review because the SQL reads perfectly.
  • Presenting event counts as 'how much data was read'One event can be a full table scan and one a BI tool refreshing its schema cache. The count answers who and when, never how much.
  • Reviewing dwh_prod onlyDev usually holds a copy of production data that nobody classified, and it is the least-governed catalog in the estate.

#Alerting on permission changes

What
A quarterly review catches an over-broad grant three months late; an alert catches it the same afternoon. Production permission changes are rare enough that every one can be read by a human, which makes this one of the few alerts in the warehouse that can safely fire on any row at all.
Two things belong in the alert that are usually left out:
Tag changes
Under ABAC a column mask applies through a governed tag, so UNSET TAG on dwh_prod.gold.dim_customer.email switches the mask off with no grant and no policy changing. An alert watching only updatePermissions reports a quiet afternoon.
A coalesce over the name keys
For the same reason as every other query in this route: updatePermissions is documented with full_name, older query libraries use securable_full_name, and the tag events carry the entity under their own key.
Why
The realistic scenario is not malice. It is a Friday afternoon, an urgent report, and someone with metastore admin granting SELECT on dwh_prod.gold to account users to unblock a colleague. It works, the report ships, nobody mentions it, and the permissions concept you wrote is now fiction.
Caught the same afternoon that is a two-minute conversation. Caught at the quarterly review it is a documented three-month exposure of personal data — a materially different conversation with a regulator.
How
sql
Permission AND tag changes on dwh_prod — alert when this returns any row
SELECT event_time,
       user_identity.email            AS changed_by,
       action_name,
       request_params.securable_type  AS securable_type,
       -- Name keys differ per action; verify with the map_keys probe from
       -- 'What system.access.audit records' before this alert goes live.
       coalesce(request_params.securable_full_name,
                request_params.full_name,
                request_params.entity_name)  AS securable,
       request_params.changes               AS changes,
       source_ip_address
FROM   system.access.audit
WHERE  service_name = 'unityCatalog'
  AND  action_name IN ('updatePermissions',
                       'createCatalog', 'deleteCatalog',
                       -- tagging IS a permission boundary under ABAC
                       'createEntityTagAssignment',
                       'deleteEntityTagAssignment')
  AND  event_date >= current_date() - INTERVAL 1 DAY
  AND  coalesce(request_params.securable_full_name,
                request_params.full_name,
                request_params.entity_name, '') LIKE 'dwh_prod%'
ORDER  BY event_time DESC;
The changes parameter carries added and removed privileges as JSON, so the alert body can say what was granted to whom rather than that something happened. That decides whether the alert is read or filtered into a folder.
Doing it
  • Schedule it hourly as a Databricks SQL alert on 'returns more than zero rows'Route it to the rota that takes pipeline failures.
  • Derive the verb list and the name keys from the client's own metastore firstSELECT DISTINCT action_name plus the map_keys probe — then write the alert.
  • Test the tag half separately: set and unset a governed tag on a non-production columnConfirm the alert fires. A grant test does not exercise that path.
  • Put the changes payload in the alert bodyAn alert saying only 'permissions changed' gets acknowledged without being read.
  • Suppress the deployment service principal by name, not by patternCI grants are then quiet and human grants are loud.
  • Test it by making a harmless grant in production and watching the alert arriveAn untested alert is a belief.
Embedding it
  • Let the team make the test grant and time the alert themselvesThey learn the latency and trust the control because they watched it work.
  • Agree the response in advance — revert, or ratify and write it downThe first real alert is then not a debate about whose job it is.
  • Ask who is allowed to grant on dwh_prod at allThe honest answer is usually 'about six people', and narrowing that beats any alert.
  • Ask which is worse: granting SELECT on gold, or removing the pii tag from dim_customer.emailThey will pick the second once they think it through — and it is the one nobody was watching.
Defaults
  • Any row means alertIf that is intolerable, the problem is the change rate, not the alert.
  • Grants and tag assignments in the same alert — under ABAC they are the same boundary
  • CI grants suppressed by principal name; human grants always alert
  • Route to a rota — never a shared mailbox, never the consultant's address
  • Close every alert with a revert or a written ratificationNothing closes silently.
Gotchas
  • Watching grant verbs and ignoring tag verbsUnder tag-driven ABAC the column mask on email hangs off a governed tag, so removing the tag disables the mask without a single grant changing — the alert reports a quiet week while personal data is readable in the clear, and the access review a quarter later shows nothing either, because nothing was granted.
  • Building the verb list by copying a query libraryupdateSecurable is the standard example: it is not in the documented Unity Catalog event list, it matches nothing, and it fails silently — so the alert appears healthy while covering fewer verbs than the concept document claims.
  • Alerting on the whole metastore instead of dwh_prodDev and test grants change constantly, the channel is noise within a week, and the alert everyone muted is the one that would have caught the production grant.
  • Assuming securable names always start with the catalogMetastore-level and external-location changes carry no catalog prefix, so LIKE 'dwh_prod%' drops exactly the broadest changes anyone can make — run a second, unfiltered query for those.
  • Treating audit latency as zero on a five-minute scheduleEvents land minutes after the fact, so a tight window plus latency silently skips the change that happened on the boundary.
  • Alerting without agreeing the responseThe first alert produces a thread about ownership instead of a revert, and the second gets ignored because the first did nothing.
Platform facts on this page verified 29.07.2026 against the official documentation. Volatile claims are anchored to the currency register. This is section 1 of 3 in 18 Audit & lineage.