distributed-execution/documents/user-guide/readiness-checklist.md

Distributed Execution: Readiness Checklist

Pre-run validation for a workflow's execution target. Companion to the user guide; this document is the validation checklist required by SIMPL-30451 AC3 and SIMPL-30787 AC3. Section 1 holds the checks common to both setups, section 2 the tightly coupled checks and section 3 the loosely coupled ones; every item states the evidence that proves it.

Run these checks before the first execution of a workflow, and again after any change to the run launcher, the executor, the target namespace or the code location image.

Both Kubernetes paths are cluster-verified. L4–L10 were cleared on sandbox-cat-dat on 2026-08-31. The tightly coupled k8s_job_executor path reached RUN_SUCCESS through the platform launcher on 2026-09-01. Checks L1–L3 and L11, which exercise the payload contract, message parsing and image isolation, are also verified locally. See section 2.1.

---

1. Common checks — both setups

#	Check	How to verify	Expected evidence
C1	Code location loads	Dagster UI → Deployment → Code locations	Location distributed-execution shows status Loaded, with a recent load timestamp and no error banner
C2	Jobs are registered	Dagster UI → Jobs	The jobs listed in the guide's section 5.5 appear under the code location
C3	Image tag matches the intended release	kubectl -n dagster get deploy -l dagster/code-location=distributed-execution -o jsonpath='{.items[*].spec.template.spec.containers[*].image}'	Tag equals the version in pipeline.variables.sh; never latest
C4	Image architecture matches the nodes	docker manifest inspect <image>	Includes linux/amd64; a manifest with only linux/arm64 produces no match for platform at pull time
C5	Run launcher type is as intended	kubectl -n <namespace> get cm dagster-instance -o yaml	run_launcher block shows K8sRunLauncher. Note that job_namespace here is only a default — a code location's DAGSTER_CONTAINER_CONTEXT overrides it, so read that too before concluding where run pods land
C6	Target namespace exists and is schedulable	kubectl get ns <namespace> and kubectl -n <namespace> get resourcequota	Namespace is Active; remaining quota exceeds the job's aggregate requests. Also confirm the launcher's instance_config_map, postgres_password_secret and any PVC volumes exist in the namespace run pods actually use — these references do not cross namespaces (see section 2.1)

2. Tightly coupled checks

#	Check	How to verify	Expected evidence
T1	Run pod reaches the metadata database	Launch tightly_coupled_in_process_job	Run reaches SUCCESS; report_execution_target metadata lists DAGSTER_PG_PASSWORD under env_vars_present without exposing its value, and env_vars_missing is empty. Successful step startup is the connectivity proof because the step loads run state from Postgres before executing the op
T2	Vault injection works	Same run; inspect the run pod	kubectl -n dagster describe pod <run-pod> shows the vault-env init container completed; no vault: literal remains in the process environment
T3	Object storage is reachable	Same run, if the workflow uses S3	No EndpointConnectionError in run logs; S3_ENDPOINT_URL present in env_vars_present
T4	Multiprocess fan-out actually fans out	Launch tightly_coupled_local_job	Run succeeds; summarise_results metadata shows one entry per unit in contributing_workers and a single entry in contributing_hosts — separate processes, same machine
T5	RBAC permits step Jobs	kubectl -n dagster auth can-i create jobs --as=system:serviceaccount:dagster:dagster-dev	Returns yes; required only for k8s_job_executor
T6	Step pods are actually created	Launch tightly_coupled_k8s_job, then kubectl -n dagster get jobs -l dagster/run-id=<run-id>	One Job per mapped unit; contributing_hosts now shows one entry per unit, not one
T7	Step pod egress is permitted	Same run	Steps do not hang in STARTING; run logs contain no connection timeouts to port 5432
T8	Failure surfaces as a pod failure	Force a step failure in a scratch namespace	failPodOnRunFailure: true is set, and the step pod reports Failed rather than Completed

2.1 Sandbox state, 2026-08-31

The T checks were assessed against sandbox-cat-dat by reading the live dagster-instance ConfigMap in dataprovider01 and by running T1, T4, T6 and T7 from pods there.

What the configuration shows:

#	Finding
C5	run_launcher is K8sRunLauncher; job_namespace: dagster is the default, overridden per code location to dataprovider01
T2	Vault injection is configured — pod_template_spec_metadata carries the banzaicloud annotations with role sandbox-cat-dat-role
T3	S3_ENDPOINT_URL is https://s3.sandbox-cat-dat.simpl-europe.eu, with access keys injected from Vault
T5	Passes. dagster-role, bound to dagster-svc-account, grants batch/jobs with create, delete, get, list, patch, update and watch
T8	fail_pod_on_run_failure: true is set

T1 and T4 were executed on 2026-08-31 from a pod in dataprovider01 (yaml/sandbox/probe-tightly-coupled-sandbox.yaml), using a local SQLite instance rather than the platform's Postgres.

#	Result
T1	Reachability passes — check_tcp_reachable('pg-cluster.common01.svc.cluster.local', 5432) returned passed: True. The env-var half of the check does not hold; see below
T4	Passes. tightly_coupled_local_job reached RUN_SUCCESS with contributing_hosts: ['distexec-tc-probe-hbdfl'] — one entry — and four distinct PIDs in contributing_workers. Separate processes, same machine, exactly as the row predicts

tightly_coupled_in_process_job also succeeded, reporting a single worker.

T1's env-var check and platform contract

The run emitted:

Orchestration runtime env vars not visible to this process: DAGSTER_POSTGRES_HOST, DAGSTER_POSTGRES_USER, DAGSTER_POSTGRES_DB. Expected for a loosely coupled target; a misconfiguration for a tightly coupled one.

That warning was a false positive on a correctly configured platform run pod. The chart does not set those variables: it injects DAGSTER_PG_PASSWORD from dagster-postgresql-secret and stores host, user and database in dagster.yaml. The reference check now tests for DAGSTER_PG_PASSWORD and reports only its name, never its value. Database reachability is evidenced by successful step startup: a tightly coupled step must load its run state from the configured Dagster instance before report_execution_target can execute.

T6 and T7, executed 2026-08-31

Running them needed three things the earlier attempt lacked: dagster pinned to 1.12.8 to match the control plane, so writing to the live dataprovider01_dagster could not trigger a migration; step outputs moved onto the RWX dagster-shared-pvc, because the default I/O manager writes them under $DAGSTER_HOME/storage and each step pod otherwise gets its own empty volume; and automount_service_account_token: true in step_k8s_config, because dagster-svc-account disables it by default. The probe is yaml/sandbox/probe-k8s-executor-sandbox.yaml; the run is d05c736a-8ec7-4e59-8cb0-9cb238db40dc.

#	Result
T6	Passes. Six Jobs labelled dagster/job=tightly_coupled_k8s_job were created, one per step, each in its own pod
T7	Passes. No step hung in STARTING and no step reported a connection timeout to port 5432. Every step pod opened the platform Postgres itself — the outputs and inputs below were all recorded through it

Four steps executed and succeeded: generate_work_units (four dynamic outputs), report_execution_target, and process_work_unit[unit_0] and [unit_1], each loading its input from the shared volume that a different pod wrote.

The run then failed, and summarise_results never started. The cause is a property of the probe, not of the code or the platform:

Detected run worker status UNKNOWN: DagsterK8sUnrecoverableAPIError … jobs.batch "dagster-run-d05c736a-…" is forbidden: User "system:serviceaccount:dataprovider01: dagster-svc-account" cannot get resource "jobs/status" … in the namespace "dagster"

Run monitoring polls for a run worker Job named dagster-run-<run-id>. The probe starts the run with dagster job execute from a hand-rolled Job, so no such object exists; and with no launcher-supplied container context the monitor looked in the launcher's default namespace, dagster, where dagster-svc-account holds no rights. The 403 turned absent into unknown, and the monitor failed the run at its first poll. process_work_unit[unit_2] and [unit_3] had their pods started already; both logged Skipping step execution … since the run is in status DagsterRunStatus.FAILURE and exited 0 — which is why all six Jobs read Complete while only four steps ran.

A run submitted the normal way, through the webserver to K8sRunLauncher, gets a dagster-run-<run-id> Job in dataprovider01 and is not exposed to this. Closing T6/T7 end to end, including summarise_results, needs the code location registered on the platform Dagster.

T6 and T7 closed end to end, 2026-09-01

With the code location registered on the platform Dagster, tightly_coupled_k8s_job was launched from the webserver and reached RUN_SUCCESS as run 1d8cb167-9fb8-4a34-a036-789eac381e13. K8sRunLauncher created dagster-run-1d8cb167-… in dataprovider01, K8sStepHandler created one dagster-step-… Job per step, and every step — including summarise_results — completed. No hand-rolled probe, no run-monitoring 403.

The last thing in the way was where step outputs are written. Two earlier attempts failed on FileNotFoundError: /tmp/dagster/storage/<run-id>/generate_work_units/result/unit_0, with every step pod using its own container filesystem. Mounting dagster-shared-pvc into the step pods was necessary but not sufficient: setting DAGSTER_HOME through step_k8s_config does not take effect, because the Dagster chart already injects DAGSTER_HOME=/tmp/dagster and the appended second entry does not displace the first. The fix is to bypass DAGSTER_HOME and pin the I/O manager instead — fs_io_manager.configured({"base_dir": SHARED_IO_BASE_DIR}) on the job, with SHARED_IO_BASE_DIR under the mounted PVC. The successful run logs confirm it:

Handled output "result" using IO manager "io_manager" — /dagster/shared/distributed-execution/storage/1d8cb167-…/generate_work_units/result/unit_0

The platform run exposed the false-positive DAGSTER_POSTGRES_* warning described under T1 above. The reference now checks the chart's actual DAGSTER_PG_PASSWORD injection instead.

Note: report_execution_target reported namespace: <not-in-kubernetes> on a step pod that plainly was in Kubernetes. The check reads the service account namespace file, which is absent when the token is not automounted. It is a property of the probe, not of the run.

C6 passes, contrary to what job_namespace alone suggests. The launcher's job_namespace: dagster is only a default. Each code location ships a DAGSTER_CONTAINER_CONTEXT environment variable — generated by the chart — whose k8s.namespace is dataprovider01, and the code location's context is merged over the launcher's, so it wins (dagster_k8s/container_context.py: namespace = other.namespace if other.namespace else self.namespace). Run pods therefore land in dataprovider01, where dagster-instance, dagster-postgresql-secret and dagster-shared-pvc all exist.

Both deployed code locations carry it, so both would launch correctly:

Code location	DAGSTER_CONTAINER_CONTEXT k8s.namespace
template-code-location	dataprovider01
semaphore-ui	dataprovider01

The successful run confirms that the code-location container context places run and step pods in dataprovider01 with the required namespace-local dependencies.

The residual risk is narrower than a broken namespace, but real: a code location that reaches the launcher without a container context namespace inherits dagster and would fail on three missing objects. The chart always sets it, so this bites only hand-rolled pods — the reason the C6 row asks you to check the launcher's namespace-local dependencies rather than just the namespace.

An earlier revision of this section claimed C6 failed and that this explained the absence of runs. That was wrong: it read job_namespace without accounting for the container-context override.

3. Loosely coupled checks

#	Check	How to verify	Expected evidence
L1	Payload contract is intact	uv run pytest tests/test_loosely_coupled.py	test_payload_does_not_import_dagster passes — the payload imports dagster_pipes only
L2	Pipes round trip works	Launch loosely_coupled_subprocess_job	Run reaches SUCCESS; run logs contain the payload's External payload started on … line, proving pipes.log crossed the channel
L3	Silence is treated as failure	Same test module	test_silent_message_path_is_treated_as_failure passes — an empty message list raises rather than yielding an empty result
L4	Payload image is pullable by the target cluster	kubectl -n <payload-ns> run pull-probe --image=<payload-image> --restart=Never --command -- true	Pod reaches Completed; no ImagePullBackOff. Both Gitea images pull anonymously — a bare registry GET returns 401, but that is the start of the Docker token handshake, not a refusal
L5	Dispatcher can create Jobs in the payload namespace	kubectl -n <payload-ns> auth can-i create jobs --as=system:serviceaccount:dagster:dagster-dev	Returns yes
L6	Dispatcher can read pod logs — the message channel	kubectl -n <payload-ns> auth can-i get pods/log --as=system:serviceaccount:dagster:dagster-dev	Returns yes. A no here breaks reporting without failing the workload
L7	Payload Job is actually created	Launch loosely_coupled_k8s_job, then kubectl -n <payload-ns> get jobs -l app.kubernetes.io/name=distributed-execution-payload	One Job per dispatch, labelled dagster/execution-target=loosely-coupled
L8	Payload has no orchestration connectivity	Same run; inspect run logs	No Payload could see orchestration runtime credentials warning. This is a positive check — absence of errors is not sufficient
L9	Work ran off-platform	Same run	summarise_results metadata shows contributing_hosts containing the payload pod names, not the run worker's hostname
L10	One workload dispatched per unit	Same run, or uv run pytest -k dispatched_to_its_own locally	contributing_workers has one entry per unit; the local test asserts four distinct external workers
L11	Payload image carries no orchestration dependency	docker run --rm <payload-image> python -c "import importlib.util; print(importlib.util.find_spec('dagster') is not None)"	Prints False. L1 proves the source does not import dagster; this proves the shipped image does not contain it either

Checks L4–L9 require a cluster. L1–L3, L10 and L11 run on a laptop and should gate every change to the payload or the dispatching op.

yaml/loosely-coupled/probe-pipes-k8s.yaml clears L4–L9 in a single run from a throwaway namespace, without deploying a code location or a Dagster control plane. Prefer it over assembling the cluster checks by hand: the RBAC it grants is exactly the set L5 and L6 ask about, so a failure localises immediately.

L4–L10 were cleared on sandbox-cat-dat on 2026-08-31 using the sandbox variant yaml/sandbox/probe-pipes-k8s-sandbox.yaml. Run cff9b348-bfc3-4ac1-ab51-a94892b8e3a0 reached RUN_SUCCESS in dataprovider01: four payload Jobs, four distinct payload pod hostnames in contributing_hosts, and the payload's External payload started on … lines in the dispatcher's log, which is the pod log stream doing its job as the message channel. No credentials were needed anywhere.

One trap the run exposed. The platform's dagster-svc-account sets automountServiceAccountToken: false, and the Dagster chart overrides it to true on every pod it manages. A hand-written pod that does not is a plausible future failure: the pipes client selects in-cluster authentication correctly and then fails on Service token file does not exist, which points at Kubernetes rather than at the omission.

---

4. Common misconfiguration symptoms

4.1 Both setups

Symptom	Likely cause	Correction
Code location stuck in Loading, then errors	Entry point path in codeServerArgs does not match the image layout	Confirm --python-file matches workspace.yaml; both must be src/distributed_execution/repository.py
ImagePullBackOff with no match for platform	Image published for a single non-matching architecture	Rebuild multi-arch with docker buildx, and pin a version tag rather than latest
Run stays in QUEUED indefinitely	Run coordinator concurrency limit reached, or no schedulable node	Check max_concurrent_runs and tag concurrency limits; check node capacity and resource quota
Run fails immediately with a serialisation error	Code location image and Dagster control-plane versions diverge	Align the dagster version in pyproject.toml with the chart's version and rebuild

4.2 Tightly coupled

Symptom	Likely cause	Correction
Run pod starts, then fails with a connection timeout to port 5432	Execution namespace NetworkPolicy does not permit egress to Postgres	Add an egress rule for the metadata database, or move run pods to an already-approved namespace via jobNamespace
env_vars_missing is non-empty in report_execution_target metadata	Env vars are set on the code location deployment but not on the run pod	Add them under runLauncher.config.k8sRunLauncher.runK8sConfig.containerConfig.env — code location env is not inherited by run pods
A literal vault:... string appears as a value at runtime	Vault mutating webhook did not process the pod	Verify the vault.security.banzaicloud.io/* annotations are on the run pod template, not only the code location pod
Steps hang in STARTING with k8s_job_executor	Service account lacks Job create/watch permission	Apply yaml/tightly-coupled/rbac-step-executor.yaml and confirm with kubectl auth can-i
FileNotFoundError loading a step input from /tmp/dagster/storage/...	Step pods each use their own container filesystem for I/O manager output	Mount an RWX volume into step pods via step_k8s_config and pin fs_io_manager's base_dir under it. Setting DAGSTER_HOME in step_k8s_config does not work — the chart's DAGSTER_HOME=/tmp/dagster is injected first and wins
contributing_hosts shows one host when k8s_job_executor is configured	Run tags or Launchpad config overrode the executor, or the image predates the change	Confirm the code location reloaded after the image bump; check the run's Config tab for an execution: override
Step pods OOMKilled under fan-out	Per-step memory limit applied per pod, aggregate exceeded quota	Raise step_k8s_config limits or lower step concurrency; the two multiply
Postgres refuses connections once fan-out grows	Each step pod is an independent DB client	Reduce step concurrency, raise the Postgres connection limit, or move the fan-out step to a loosely coupled target

4.3 Loosely coupled

Symptom	Likely cause	Correction
Op fails with No pipes messages received from the external payload	The message path is broken, not the workload	Work through L6 then L4. The payload very likely ran and succeeded; only its reporting was lost
Payload pod Completed, but Dagster shows no payload log lines	A log shipper is intercepting or truncating stdout	Exclude the payload namespace from the shipper, or switch to an object-storage message reader
Op hangs until pod_wait_timeout (default 24 h)	Payload Job never scheduled — quota, node selector or image pull	Check kubectl -n <payload-ns> describe job <name>; lower pod_wait_timeout so the failure surfaces quickly
403 Forbidden creating the Job	Dispatcher service account lacks Job create permission	Apply yaml/loosely-coupled/rbac-pipes-dispatch.yaml in the payload namespace
Warning: Payload could see orchestration runtime credentials	Payload pod inherited run-pod env or a Vault annotation	Remove the inherited env; the payload should receive only what extras and explicit env pass it
Payload exits non-zero but the run reports success	Exit status not being checked, or messages read before failure	Confirm the dispatching op returns through _result_from_pipes; do not swallow PipesClientCompletedInvocation errors
Payload receives no units	extras key mismatch between dispatcher and pipes.get_extra()	Both sides must use the same key; a typo yields a KeyError inside the payload
Run cancelled in the UI, payload pod keeps running	Cancellation is not propagated to dispatched workloads automatically	delete_pod_on_completion handles the normal path; for cancellation, verify orphaned Jobs and add a cleanup sensor

Partly observed. The pipes rows were exercised on sandbox-cat-dat on 2026-08-31. The tightly coupled k8s_job_executor rows were closed on 2026-09-01 by run 1d8cb167-9fb8-4a34-a036-789eac381e13; the remaining rows in the table above still follow from the dagster-k8s API rather than from observation.

One symptom the cluster run added, absent from the table above: a service account with automountServiceAccountToken: false — which the platform's dagster-svc-account uses — makes the pipes client fail with ConfigException: Service token file does not exist. It reads as a Kubernetes fault; the fix is automountServiceAccountToken: true on the pod.

---

5. Evidence retention

For each workflow's first execution, attach to the workflow's repository or change record:

1. The run ID and its final status.
2. The report_execution_target output metadata block (pod identity, namespace, env var presence).
3. The summarise_results metadata block (contributing_hosts and contributing_workers), which proves which execution target was actually used and that the fan-out reached it.
4. For k8s_job_executor, the output of kubectl get jobs -l dagster/run-id=<run-id>.
5. For the loosely coupled target, the payload image digest and the run log line emitted by pipes.log — together they prove which payload version ran and that the message channel was open.

Items 2 and 3 together are sufficient to demonstrate that the configured execution target is the one that ran — which is the point of the checklist.