One bundle, three targets, one command per CI step. The interesting parts are the test target and the single bundle limitation that has a workaround worth knowing.
The bundle carries the whole deployable unit: jobs, pipelines and the src/ workspace files. The catalog is a variable per target, which is what lets the same code run against dwh_dev, dwh_test and dwh_prod without a single environment check at runtime.
Why
A test target is what turns 'the tests pass' into 'the tests pass against a deployed pipeline in a real workspace'. Unit tests on a laptop cannot catch an expectation that never fires because the pipeline edition is CORE, or a task that fails because a notebook path is wrong in the bundle. Those are deployment defects and only a deployment finds them.
The variable-per-target design removes the most damaging incident class in this kind of project: a job that wrote to the wrong catalog because the catalog was a literal in a notebook someone copied.
How
yamldatabricks.yml — targets, and the CI job that exists only in test
bundle:
name: dwh
include:
- resources/*.yml
variables:
catalog:
description: Unity Catalog catalog this target writes to.
deploy_sp:
description: Application ID of the OAuth service principal that owns prod runs.
# Declared, required by prod.run_as, and deliberately given no default:
# the application ID differs per customer workspace and does not belong
# in the repository. It has to arrive as --var deploy_sp=<app-id> or as
# BUNDLE_VAR_deploy_sp in the release job's environment. Leave it unset
# and "bundle validate -t prod" fails with
# no value assigned to required variable deploy_sp
targets:
dev:
mode: development
default: true
variables:
catalog: dwh_dev
test:
variables:
catalog: dwh_test
resources:
jobs:
ci_assert: &ci_assert
name: dwh-ci-assert-${bundle.target}
tasks:
- task_key: seed
notebook_task:
# ./ and NOT ../ — paths resolve relative to the file that
# declares them, and this file is the sync root. Both CI
# files start with "# Databricks notebook source".
notebook_path: ./src/dwh/ci/seed.py
source: WORKSPACE
base_parameters:
source_root: ${workspace.file_path}/src
catalog: ${var.catalog}
- task_key: pipeline
depends_on:
- task_key: seed
pipeline_task:
pipeline_id: ${resources.pipelines.silver.id}
full_refresh: true
- task_key: assert
depends_on:
- task_key: pipeline
notebook_task:
notebook_path: ./src/dwh/ci/assert_expectations.py
source: WORKSPACE
base_parameters:
source_root: ${workspace.file_path}/src
catalog: ${var.catalog}
sandbox:
variables:
catalog: dwh_dev
resources:
jobs:
ci_assert: *ci_assert # referenced, not redefined
prod:
mode: production
variables:
catalog: dwh_prod
run_as:
service_principal_name: ${var.deploy_sp}
# ci_assert is simply ABSENT here. That is the workaround.
Two details in that file bite before anything is deployed. First, every relative path resolves against the file that declares it, and it has to stay inside the sync root. databricks.ymlis the sync root, so a task declared there says ./src/…; the identical task declared in resources/jobs.yml says ../src/…. Write ../ in databricks.yml and the path points at the parent of the repository, which the CLI rejects with path … is not contained in sync root path. The corollary matters more than the rule: moving a resource between databricks.yml and resources/*.yml — exactly what happens when an anchored job graduates to a top-level one — means rewriting every path inside it.
Second, deploy_sp is declared but never assigned. That is deliberate, and it is also the reader's first red validate: a variable with no default that no target assigns makes databricks bundle validate -t prod fail with no value assigned to required variable deploy_sp. Pass it as --var deploy_sp=<application-id> or export BUNDLE_VAR_deploy_sp in the release job. Decide the same question for every workspace-specific variable you add later — default, target assignment, or caller-supplied — and write the decision next to the declaration, because the CLI only tells you at the moment you are trying to release.
You cannot define ci_assert as a top-level resource and then exclude it from prod — excluding a top-level bundle resource for a single target is not supported. So the seed/run/assert job is defined inside the first target that needs it, anchored with &ci_assert and referenced with *ci_assert from any other target that wants it. Targets that should not have it say nothing, which is the only reliable way to subtract.
Doing it
Define the catalog as a bundle variable resolved per targetNo literal catalog names anywhere in src/.
Anchor any resource that belongs to some targets but not others, and define it in the first target that uses itKeep anchors in databricks.yml itself — they do not cross include: file boundaries, and remember that a resource moved into databricks.yml needs its ../ paths rewritten to ./.
Pin the Databricks CLI version in CIPython bundle configuration requires 0.275.0 or newer, and an older CLI fails confusingly rather than with a version error.
Run databricks bundle validate -t prod -o json locally before every release and read the resolved configurationPlain bundle validate prints only a summary — name, target, workspace, user, path — plus diagnostics and 'Validation OK!'; the variable-substituted configuration comes out only under -o json, and that is where a wrong catalog is visible before anything is deployed.
Give every required variable a home before the first releaseA default, a target assignment, or a documented --var / BUNDLE_VAR_ in the release job. Validate prod with the same invocation the release uses, or you are validating a different bundle from the one you ship.
Embedding it
Have the team add the fourth target themselvesTwenty minutes, and it converts the bundle from your artefact into their configuration.
Ask them to predict what bundle validate -o json will say before running it, then run itTwo rounds of that and they stop deploying blind — and they learn that the default output confirms nothing about which catalog they are pointing at.
When someone proposes an environment check inside the code, ask which target it is compensating forThe answer is always a missing variable.
Defaults
One bundle, one repository, one databricks.yml, resources split into resources/*.yml — anchors excepted
Paths are relative to the declaring file: ./src/… in databricks.yml, ../src/… in resources/*.ymlRewrite them whenever a resource moves between the two.
Catalog per target as a variable; mode: production on prod so the CLI enforces production guardrails
Subtract by omission and YAML anchors, never by trying to exclude a top-level resource
Every declared variable has a default, a target assignment, or a documented --var / BUNDLE_VAR_None is left to be discovered at release time.
bundle validate -o json is the release check; the bare command is only a smoke test
Pin the CLI version in CI and in the developer setup instructionsProd runs as a service principal via run_as.
Gotchas
../src/… in databricks.ymlThat file is the sync root, so ../ resolves to the parent of the repository and the CLI rejects it with 'path ... is not contained in sync root path'. The same string is correct in resources/*.yml, which is why copying a task between the two files breaks it and the diff looks innocent.
Reading plain bundle validate output expecting the resolved configurationIt prints a summary and 'Validation OK!' — the substituted variables are only in -o json, so a target pointed at the wrong catalog validates green and you find out after the deploy.
A variable declared with no default that no target assigns — deploy_sp is the one in this bundlebundle validate -t prod fails with 'no value assigned to required variable deploy_sp', and it fails for the first time on release day because nobody validates prod until then.
Trying to exclude a top-level resource for one targetIt is not supported, so the CI job you wrote for test also lands in prod, where it seeds fixture data into the production catalog.
An anchor defined in resources/jobs.yml and referenced in databricks.ymlAnchors are per-document, so it never resolves and the message points at YAML syntax rather than at the include boundary.
An anchor referenced before it is definedYAML resolves in document order, so reordering targets breaks a deploy that worked yesterday with no code change.
environment_key on a serverless notebook task is reported to be rejectedUnverified — but if a serverless task fails to start for no visible reason, this is the first thing to try removing.
databricks bundle validate is the first step, before the linter and before pytest. It is the only step that catches a malformed bundle and it takes two seconds — running it after a ninety-second test suite wastes ninety seconds every time the YAML is wrong.
Why
The audit row decides it. If CI runs as a person, system.access.audit cannot separate what a human did from what a pipeline did — so the standing access review has no signal, and 'who read the PII table' is unanswerable for the exact identity that touches everything.
The step ordering is a smaller point with a daily payoff: validate, then a fast static check, then unit tests, then anything that touches a workspace. Every step needing the network runs after every step that does not.
How
yaml.github/workflows/pr.yml — cheapest and most decisive step first
name: dwh
on:
pull_request:
branches: [main]
jobs:
verify:
runs-on: ubuntu-latest
env:
DATABRICKS_HOST: ${{ secrets.DATABRICKS_HOST }}
DATABRICKS_CLIENT_ID: ${{ secrets.DATABRICKS_CI_CLIENT_ID }}
DATABRICKS_CLIENT_SECRET: ${{ secrets.DATABRICKS_CI_CLIENT_SECRET }}
steps:
- uses: actions/checkout@v4
- uses: databricks/setup-cli@main
with:
version: 0.275.0 # pinned; see the bundles-python fact
- name: bundle validate
run: databricks bundle validate -t test
# Every DDL form that INTRODUCES or CHANGES an object name, not only
# CREATE TABLE. First grep finds the candidate lines; the second keeps
# only those whose name does NOT match the rule, so the step fails on
# exactly the violations. Keywords are matched uppercase, which is the
# house style, so a COMMENT holding a German term is never touched.
# Not covered: column names inside a column list — those are the
# reviewer's row in the definition of done, and the message below says
# only what this step actually enforces.
- name: identifier rule
run: |
DDL='(CREATE( OR REPLACE)?( TEMPORARY)? (TABLE|VIEW|MATERIALIZED VIEW|STREAMING TABLE)( IF NOT EXISTS)?|ALTER TABLE [^ ]+ RENAME TO) +'
OK='([a-z][a-z0-9_]*)(\.[a-z][a-z0-9_]*)*([ (;]|$)'
if grep -rEn "$DDL" src/ | grep -Ev "$DDL$OK"; then
echo "object names: every dot-separated part must match ^[a-z][a-z0-9_]*$ (ASCII, no leading digit or underscore, no uppercase)" >&2
exit 1
fi
- name: unit tests
run: pytest tests/unit --junitxml=reports/unit.xml
- name: deploy to test
run: databricks bundle deploy -t test
- name: seed, run, assert
run: databricks bundle run -t test ci_assert
The identifier step is worth reading twice, because the naive version of it is worse than none. Matching bad characters inside a name is the wrong shape: it accepts 1_orders and _orders, which the stated rule ^[a-z][a-z0-9_]*$ forbids — so the build stays green while the message on the wall claims something stricter than the build enforces. Inverting it — find the DDL lines, then keep the ones that do not match a legal name — enforces exactly the rule as written. And a check must claim only what it does: this one covers CREATE TABLE, CREATE VIEW, CREATE MATERIALIZED VIEW, CREATE STREAMING TABLE and ALTER TABLE … RENAME TO. Column names are not in it, which is why the definition of done keeps a reviewer row for them.
CI authenticates as an OAuth service principal using machine-to-machine credentials: DATABRICKS_CLIENT_ID and DATABRICKS_CLIENT_SECRET, which the CLI exchanges for a short-lived token on every invocation. Personal access tokens do not appear in this repository, this pipeline, or anyone's .databrickscfg on a shared machine.
Why the identity choice is not a preference
Personal access token
OAuth service principal
Belongs to
a person
the pipeline
When that person leaves
prod deploys stop
nothing happens
In system.access.audit
their name, on every automated run
the service principal, distinguishable from humans
Lifetime
long-lived secret at rest
short-lived token minted per call
Scope
everything the person can do
exactly the grants the SP was given
Doing it
Create one service principal per environment, grant it only what its target needsStore its secret in the CI secret store — never in the repository, never as a bundle variable default.
Revoke every personal access token used for deployment in week oneConfirm with the token API that none remain.
Make bundle validate the first step and let it fail the buildA warning nobody reads is not a check.
Encode the identifier and read-direction rules from the architecture route as grep steps hereEmit JUnit XML so failures surface in the pull request rather than in a log nobody opens.
Test every grep check against a deliberate violation of each form it claims to coverA view, a materialized view, a rename, a leading digit — break each one before you trust the check. A check nobody has seen fail is a check nobody knows the scope of.
Embedding it
Have the team add the next CI check themselvesThe read-direction grep is the natural second one, and it is ten lines.
Ask them to run the access-review query and point at their own CI service principal in the outputSeeing it as a distinct identity is what makes the rule stick.
Rotate the service principal secret once during the engagement, with the team drivingThe runbook is then written by someone who has done it.
Defaults
bundle validate first, static checks second, unit tests third, workspace steps last
OAuth M2M service principals for every automated identity; zero personal access tokens
One service principal per environment, granted per target, secret in the CI secret store
Pinned CLI version, JUnit XML published to the pull request, and every agreed standard that can be a grep is a grep
Gotchas
CI running as a personal access token belonging to whoever set it upIt works until that person leaves or the token expires, and then production deployment is blocked by an account nobody can access.
bundle validate after the test suiteEvery YAML typo costs a full test run before you learn about it, which teaches people to push and wait rather than validate locally.
A service principal granted ALL PRIVILEGES on the metastore 'to unblock CI'It never gets narrowed, and the identity with the widest grants in the warehouse is the one whose secret sits in a CI configuration.
Deploying to test from a branch without an isolated schemaTwo pull requests deploy the same job name, the second overwrites the first, and both builds report failures caused by each other.
An identifier check that only looks at CREATE TABLECREATE VIEW, CREATE MATERIALIZED VIEW, CREATE STREAMING TABLE and ALTER TABLE ... RENAME TO all walk past it, so the convention holds for tables and quietly rots everywhere else — and views are exactly what the BI layer reads.
A check whose failure message states a stricter rule than its regex enforcesThe message says ^[a-z][a-z0-9_]*$, the regex accepts a leading digit, and the team believes a rule the build has never once applied. Whatever the message claims, prove it by breaking it.