Classic, Connect, and the difference that returns wrong numbers
Local pytest runs Spark Classic in one JVM. Serverless compute, standard-access clusters and databricks-connect all run Spark Connect. Same API, different evaluation model — and two of the differences change a number instead of raising.
In Classic, a DataFrame holds an analysed plan built in the same process as the engine. In Connect, it holds an unresolved protobuf plan that travels to a remote server and is resolved there, when an action runs. Almost everything follows from that one sentence.
The four that matter
Spark Classic — local pytest
Spark Connect — serverless, standard access
How it reaches you
Analysis
eager: df.select('nope') raises at that line
lazy: the plan leaves unresolved; the error appears at the action
a pytest.raises around a select passes locally and fails at night
Temp views
the plan captures the view as analysed at that moment
the view NAME is resolved at execution time
replacing a view changes an existing DataFrame's result, silently
Python UDFs
captured eagerly, at UDF creation: the UDF is created and its closure serialized on the spot
lazy: serialization and registration are deferred to execution
a value changed between those two lines gives different numbers
RDD API
spark.sparkContext available
absent
a test helper using parallelize dies with a clear error — the easy case
The temp-view row is the one to internalise, because it returns a number rather than an error:
Why
The dangerous class is not fails on Databricks. CI catches that on the first deployment, it costs an afternoon, and everyone learns something. The dangerous class is passes in both places and returns different numbers, because nothing in the process is looking for it.
The exposure here is concrete: currency conversion, SCD2 clock injection and the surrogate-key build are all places where a captured-versus-resolved-later difference changes a value rather than raising. A discrepancy found by a business user six weeks after go-live costs more credibility than the whole suite bought.
How
pythonSame code, same data, two different answers
spark.createDataFrame(
order_line_rows({"status": "SHIPPED"}), ORDER_LINE_SCHEMA
).createOrReplaceTempView("lines")
shipped = spark.table("lines").where("status = 'SHIPPED'")
# ... 200 lines later, another helper reuses the name
spark.createDataFrame(
order_line_rows({"status": "CANCELLED"}, {"status": "CANCELLED"}),
ORDER_LINE_SCHEMA,
).createOrReplaceTempView("lines")
shipped.count()
# Spark Classic -> 1 the view was resolved when `shipped` was built
# Spark Connect -> 0 the name is resolved now, and nothing is SHIPPED
No exception, no warning, no log line. A test that seeds through temp views is green locally and wrong on the platform.
The UDF row runs the same way round, and for the same reason. Classic creates the UDF and serializes its closure the moment udf(...) is called; under Connect, Python UDFs are lazy — serialization and registration are deferred until execution. So a value that changes between those two lines is read differently by each engine, and again Connect is the one that sees the later value:
pythonUDF closures are captured at different moments
from decimal import Decimal
from pyspark.sql.functions import udf
from pyspark.sql.types import DecimalType
rate = Decimal("1.00") # EUR
to_eur = udf(lambda amount: amount * rate, DecimalType(18, 2))
rate = Decimal("0.94") # CHF, loaded afterwards
df.select(to_eur("unit_price")).collect()
# Spark Classic -> uses 1.00 the closure is captured when the UDF is defined
# Spark Connect -> uses 0.94 serialization is deferred to execution time
The documented mitigation is a wrapper factory. Build the UDF inside a function that takes the value as a parameter, and the rebinding outside can no longer reach it — the parameter lives in a fresh scope that nothing else writes to, so both engines agree on 1.00:
pythonmake_udf: freeze the value in a scope nobody else can rebind
Use the factory whenever a UDF is unavoidable, and treat it as a containment measure rather than a fix. It makes the two engines agree; it does not make the captured value correct, and it hides the smell — a transform that depends on a Python variable's value at some particular line is one refactor away from being wrong on both engines at once.
Both behaviours are defensible; neither is what the author meant. The fix is not a better UDF — rule 2 is a join to silver.fx_rate on currency and date, which is deterministic, testable in SQL, and free.
Doing it
Seed with DataFrames from factories, not with temp viewsThe difference disappears if the name never exists.
Avoid Python UDFs in transformsBuilt-in functions and joins have identical semantics on both engines and are an order of magnitude faster.
Where a UDF is unavoidable and it closes over a value, build it from a factory functionThe factory takes the value as a parameter, so eager and deferred capture see the same thing.
Import exceptions from pyspark.errors, never from pyspark.sql.utilsAn except clause then matches under both engines.
Mark the handful of genuinely Classic-only tests with a marker whose reason carries an expiry date
Embedding it
Run the temp-view example in front of the team, locally and then on the workspaceTwo numbers on one screen ends the discussion permanently.
Ask in every review where a test seeds its dataTemp views in a test are the tell; the team learns to spot it faster than you can.
When a Connect difference bites, make the team write the regression test before you explain the cause
Defaults
Factories and DataFrames for seeding; temp views only where the view itself is under test
No Python UDFs in the warehouse pathIf one is unavoidable, isolate it and test it under both engines.
Exceptions imported from pyspark.errors; client and runtime versions pinned and upgraded together
Gotchas
A Python UDF that imports a module from workspace files works on the driver and fails inside the UDFOn an old runtime, that is. From Databricks Runtime 13.3 LTS, directories added to sys.path, or structured as Python packages, are distributed to all executors automatically; on 12.2 LTS and below they must be installed on executors explicitly. Locally the driver and the executor are the same process, so the suite never sees it.
Testing for a raised AnalysisException around a selectUnder Connect nothing is raised until an action, so the pytest.raises block exits without an exception and the test fails with a message about the exception not being raised — which reads like the code got better.
spark.sparkContext in a test helperIt is a clean, loud failure under Connect, which makes it the best of these problems, but it is usually buried in a shared helper that a hundred tests import.
A databricks-connect client whose version does not match the runtime major.minor it points atIt produces protocol and serialization errors that read like bugs in your query plan. Engineers debug the transformation for hours before checking pip show databricks-connect.
A UDF that closes over a module-level variable which something later rebindsA fixture, a second notebook cell, a config reload. Classic froze it at udf(...); Connect reads it at execution. Both return a number, neither raises, and the local suite is green, so the difference is found by a reconciliation against SAP rather than by CI. The wrapper factory removes the ambiguity; removing the UDF removes the problem.
One suite, two engines, two schedules. Pull requests run local Spark and finish in minutes. A scheduled job runs the identical tests against the workspace overnight, and its only job is to disagree with the local run.
Why
The divergence class in the previous topic is invisible to a local suite by construction. No amount of local test writing finds it, because the local engine is the thing being wrong about. The only mitigation is to run the same assertions on the real engine.
That converts we find out at go-live into we find out within twenty-four hours, for one serverless run a night. Cheapest insurance in the build phase, and the first thing teams skip.
How
bashThe two invocations
# every pull request — local Spark, no workspace, minutes
SPARK_TEST_MODE=local pytest -n 4 --dist loadfile -q
# nightly — the same tests, against serverless.
# RUN_ID comes from the job parameter below; there is no ambient env var for it.
SPARK_TEST_MODE=connect pytest -n 0 -q \
--junitxml=/Volumes/dwh_test/ci/reports/parity-${RUN_ID}.xml
-n 0 disables xdist for the parity run deliberately. Each worker would open its own session to the workspace, multiplying serverless cost by the worker count and running into concurrency limits that surface as flaky, unrepeatable failures. It overrides the -n 4 in addopts because pytest prepends addopts to the command line, so the later flag wins — worth knowing, because the same precedence is what lets a stray addopts entry silently undo a flag you thought you set.
yamlresources/parity.job.yml — defined in the test target only
resources:
jobs:
parity:
name: "parity: pytest under Spark Connect"
schedule:
quartz_cron_expression: "0 30 2 * * ?"
timezone_id: Europe/Zurich
pause_status: UNPAUSED
email_notifications:
on_failure: [dwh-oncall@example.com] # a rota, never a person
tasks:
- task_key: pytest_connect
notebook_task:
notebook_path: ../tests/run_parity
base_parameters:
spark_test_mode: connect
run_id: "{{job.run_id}}"
target_schema: "run_{{job.run_id}}"
The JUnit XML lands in the ci Volume of dwh_test, so a failure leaves an artefact somebody can open six weeks later without re-running anything.
Doing it
Run the identical suiteResist a separate connect-only test set: the moment the two diverge, the parity run stops being a parity run.
Schedule it on main, not on a long-lived feature branch, or it reports on code nobody is shipping
Give it a per-run schema in dwh_test named from the job run idUse the same pattern CI uses, and drop it in a step that runs even when the tests failed.
Route failures to the same rota as production jobs, with the same expectations
Embedding it
In week two, have the team add the temp-view example as a real testIt goes green locally and red at night. That single demonstration is worth the whole route.
Put the last green parity date on the handover checklist, next to the named ownersA stale date is the honest status of the practice.
When it fails, let the team triage it without you onceWatching them do it is the acceptance test for the coaching.
Defaults
One suite, two modes, one switch. Never two suites
Nightly, on main, against the compute production uses
Failures page a rotaA nightly nobody reads is a cron job producing a tick.
Never against dwh_prodThe parity run writes.
Gotchas
A parity run that emails a shared mailboxIt is read for three weeks and then filtered, and the first anyone hears of the divergence is from a business user.
Running parity per pull request because it feels more rigorousIt is ten to fifty times slower, developers start pushing without waiting, and within a month the team has learned to ignore a red build.
Pointing the parity run at dwh_prod for realistic dataIt creates tables, and eventually one of them shares a name with something real. Realistic data is a fixtures problem, not a catalog problem.
Letting the connect suite quietly skipIf credentials expire, pytest collects zero tests and exits 0, which is indistinguishable from success in the job UI. Assert a minimum collected-test count in the run script.