Engagement23 The engagement, end to end · 1 of 3
The arc
Five phases. The dates are illustrative for a roughly one-quarter engagement; the ordering is not, because each phase produces the input the next one needs.
Stand 29.07.2026
| Phase | Weeks | Produces | Routes |
|---|---|---|---|
| Assess | 1–2 | Evidence about the platform as it is, not opinion about it. A written intake for what the warehouse must answer. | Audit & health check · Requirements |
| Decide | 2–4 | The expensive-to-reverse choices, written down: layer contract, model, naming, environment topology, ingestion pattern per source, semantic layer ownership. | Architecture · Data model · Ingestion · BI & semantics |
| Build | 3–10 | Working pipelines, plus the standards and CI that keep the tenth one as good as the first. | Standards & CI/CD · Storage & cost · Unity Catalog · Access & ABAC |
| Prove | 5–12 | Tests that fail for the right reasons, monitoring that pages a person, and observability that catches wrong numbers behind green jobs. | The four testing routes · Observability · Monitoring |
| Hand over | 10–13 | Named owners, runbooks written by the people who get woken, a survived game day, and the consultant off the rota. | Operations · Enablement · Written concepts |
#The audit is the opening move
What
Week one is a read-only sweep of the account using
system.* tables: where the money goes, what compute exists, which jobs fail, what is granted to whom, what is governed and what is not.Nothing is installed and nothing is changed. The output is a short document of findings with the query beside each one, so any of it can be re-run by the client the day after you leave.
Why
It converts opinion into evidence before anyone proposes changes, which changes the politics of the whole engagement. "Your clusters are oversized" is an argument with the person who sized them. "Here is the query, here is the number, here is the same query you can run next month" is not.
It also protects you. The findings are dated and reproducible, so improvements later in the engagement have a baseline that nobody has to take on trust.
Doing it
- Get the system schemas enabled first — it is the one prerequisite that can delay week one by a weekIt costs nothing and it needs an account admin. What you must not do is tell the client the earlier history is definitively gone: whether enabling backfills the window is not documented either way, and most system tables carry 365 days of Databricks-side retention regardless. Enable, then run MIN() over the date column of each schema you intend to quote, and let that answer define the period.
- Run the sweep read-only and say so explicitlyIt lowers the barrier to being given access at all.
- Report findings with the query attached, never as bare assertions
- Separate 'no evidence' from 'evidence of none' in every findingThey are different sentences and only one is defensible.
Defaults
- Read-only, evidence-first, in week one
- Every finding carries its query and its date
- Baseline at the start, re-run at the end, show the delta
- Findings are ranked by cost or risk, not by how interesting they were to find
Gotchas
- Leading with the audit findings in a kickoff before you have any relationshipIt reads as an indictment of the incumbent team, who then spend the engagement defending rather than learning. Share it with the team first, then jointly with management.
- Quoting a quarter from a schema enabled last week, without asking the table what its earliest row isWhether enabling backfills the window is undocumented, so the covered period is a MIN() query and never an assumption — and the error runs one way: a short window hides spend rather than inventing it, so the flattering number is the one that survives to the readout.
- Turning the audit into the engagementIt is a two-week input, not a deliverable to polish — the value is in what changes afterwards.
Depends on
- verifiedLineage system tables keep a rolling 1-year window. Catalog Explorer and the lineage API retain lineage captured after 01.09.2024 indefinitely.Stand 29.07.2026
- check firstSystem schemas must be enabled per metastore, and this handbook states in several places that enabling is not retroactive.Stand 29.07.2026