distributed-execution/documents/user-guide/readiness-checklist.md

Distributed Execution: Readiness Checklist

Pre-run validation for a workflow's execution target. Companion to the user guide; this document covers AC3.

Run these checks before the first execution of a workflow, and again after any change to the run launcher, the executor, the target namespace or the code location image.

L4–L10 were cleared on sandbox-cat-dat on 2026-08-31. Checks L1–L3 and L11, which exercise the payload contract, the message-parsing path and image-level isolation, are verified locally. The tightly coupled Kubernetes path has been assessed against the same cluster but not executed — T2, T3, T5 and T8 are confirmed from the live instance configuration and C6 fails there. See section 2.1.

---

1. Common checks — both setups

#	Check	How to verify	Expected evidence
C1	Code location loads	Dagster UI → Deployment → Code locations	Location distributed-execution shows status Loaded, with a recent load timestamp and no error banner
C2	Jobs are registered	Dagster UI → Jobs	The jobs listed in the guide's section 5.5 appear under the code location
C3	Image tag matches the intended release	kubectl -n dagster get deploy -l dagster/code-location=distributed-execution -o jsonpath='{.items[*].spec.template.spec.containers[*].image}'	Tag equals the version in pipeline.variables.sh; never latest
C4	Image architecture matches the nodes	docker manifest inspect <image>	Includes linux/amd64; a manifest with only linux/arm64 produces no match for platform at pull time
C5	Run launcher type is as intended	kubectl -n <namespace> get cm dagster-instance -o yaml	run_launcher block shows K8sRunLauncher and the expected job_namespace
C6	Target namespace exists and is schedulable	kubectl get ns <namespace> and kubectl -n <namespace> get resourcequota	Namespace is Active; remaining quota exceeds the job's aggregate requests. Also confirm the launcher's instance_config_map, postgres_password_secret and any PVC volumes exist in that same namespace — these references do not cross namespaces, and on sandbox-cat-dat they do not match job_namespace (see section 2.1)

2. Tightly coupled checks

#	Check	How to verify	Expected evidence
T1	Run pod reaches the metadata database	Launch tightly_coupled_in_process_job	Run reaches SUCCESS; report_execution_target output metadata lists DAGSTER_POSTGRES_HOST, DAGSTER_POSTGRES_USER and DAGSTER_POSTGRES_DB under env_vars_present, and env_vars_missing is empty
T2	Vault injection works	Same run; inspect the run pod	kubectl -n dagster describe pod <run-pod> shows the vault-env init container completed; no vault: literal remains in the process environment
T3	Object storage is reachable	Same run, if the workflow uses S3	No EndpointConnectionError in run logs; S3_ENDPOINT_URL present in env_vars_present
T4	Multiprocess fan-out actually fans out	Launch tightly_coupled_local_job	Run succeeds; summarise_results metadata shows one entry per unit in contributing_workers and a single entry in contributing_hosts — separate processes, same machine
T5	RBAC permits step Jobs	kubectl -n dagster auth can-i create jobs --as=system:serviceaccount:dagster:dagster-dev	Returns yes; required only for k8s_job_executor
T6	Step pods are actually created	Launch tightly_coupled_k8s_job, then kubectl -n dagster get jobs -l dagster/run-id=<run-id>	One Job per mapped unit; contributing_hosts now shows one entry per unit, not one
T7	Step pod egress is permitted	Same run	Steps do not hang in STARTING; run logs contain no connection timeouts to port 5432
T8	Failure surfaces as a pod failure	Force a step failure in a scratch namespace	failPodOnRunFailure: true is set, and the step pod reports Failed rather than Completed

2.1 Sandbox state, 2026-08-31

The T checks were assessed against sandbox-cat-dat by reading the live dagster-instance ConfigMap in dataprovider01. None of them could be executed: the tightly coupled Kubernetes path needs this code location registered on the platform Dagster, which is blocked on the control plane skew, and unlike the pipes transport it cannot be exercised from a standalone pod — every step pod connects to the metadata database on its own, so they must share instance storage rather than a local SQLite file.

What the configuration shows:

#	Finding
C5	run_launcher is K8sRunLauncher with job_namespace: dagster
T2	Vault injection is configured — pod_template_spec_metadata carries the banzaicloud annotations with role sandbox-cat-dat-role
T3	S3_ENDPOINT_URL is https://s3.sandbox-cat-dat.simpl-europe.eu, with access keys injected from Vault
T5	Passes. dagster-role, bound to dagster-svc-account, grants batch/jobs with create, delete, get, list, patch, update and watch
T8	fail_pod_on_run_failure: true is set

C6 fails, and it is the reason nothing has ever run here. The launcher sends run pods to namespace dagster, but every namespace-local dependency it names — instance_config_map: dagster-instance, postgres_password_secret: dagster-postgresql-secret, and the dagster-shared-pvc volume — exists in dataprovider01. A ConfigMap, Secret or PVC reference resolves only within the pod's own namespace, so a run pod placed in dagster would reference three objects that are not there. Consistent with that, dataprovider01 holds no pods labelled dagster/run-id and the daemon log shows no launch activity.

One caveat on the evidence: project-scoped access means kubectl get ns dagster returns Forbidden rather than NotFound, so whether that namespace exists is unconfirmed. It does not change the conclusion — the dependencies are in the wrong namespace either way.

This is a platform configuration defect, not a defect in this service. It is exactly what C6 exists to catch.

3. Loosely coupled checks

#	Check	How to verify	Expected evidence
L1	Payload contract is intact	uv run pytest tests/test_loosely_coupled.py	test_payload_does_not_import_dagster passes — the payload imports dagster_pipes only
L2	Pipes round trip works	Launch loosely_coupled_subprocess_job	Run reaches SUCCESS; run logs contain the payload's External payload started on … line, proving pipes.log crossed the channel
L3	Silence is treated as failure	Same test module	test_silent_message_path_is_treated_as_failure passes — an empty message list raises rather than yielding an empty result
L4	Payload image is pullable by the target cluster	kubectl -n <payload-ns> run pull-probe --image=<payload-image> --restart=Never --command -- true	Pod reaches Completed; no ImagePullBackOff. Both Gitea images pull anonymously — a bare registry GET returns 401, but that is the start of the Docker token handshake, not a refusal
L5	Dispatcher can create Jobs in the payload namespace	kubectl -n <payload-ns> auth can-i create jobs --as=system:serviceaccount:dagster:dagster-dev	Returns yes
L6	Dispatcher can read pod logs — the message channel	kubectl -n <payload-ns> auth can-i get pods/log --as=system:serviceaccount:dagster:dagster-dev	Returns yes. A no here breaks reporting without failing the workload
L7	Payload Job is actually created	Launch loosely_coupled_k8s_job, then kubectl -n <payload-ns> get jobs -l app.kubernetes.io/name=distributed-execution-payload	One Job per dispatch, labelled dagster/execution-target=loosely-coupled
L8	Payload has no orchestration connectivity	Same run; inspect run logs	No Payload could see orchestration runtime credentials warning. This is a positive check — absence of errors is not sufficient
L9	Work ran off-platform	Same run	summarise_results metadata shows contributing_hosts containing the payload pod names, not the run worker's hostname
L10	One workload dispatched per unit	Same run, or uv run pytest -k dispatched_to_its_own locally	contributing_workers has one entry per unit; the local test asserts four distinct external workers
L11	Payload image carries no orchestration dependency	docker run --rm <payload-image> python -c "import importlib.util; print(importlib.util.find_spec('dagster') is not None)"	Prints False. L1 proves the source does not import dagster; this proves the shipped image does not contain it either

Checks L4–L9 require a cluster. L1–L3, L10 and L11 run on a laptop and should gate every change to the payload or the dispatching op.

yaml/loosely-coupled/probe-pipes-k8s.yaml clears L4–L9 in a single run from a throwaway namespace, without deploying a code location or a Dagster control plane. Prefer it over assembling the cluster checks by hand: the RBAC it grants is exactly the set L5 and L6 ask about, so a failure localises immediately.

L4–L10 were cleared on sandbox-cat-dat on 2026-08-31 using the sandbox variant yaml/sandbox/probe-pipes-k8s-sandbox.yaml. Run cff9b348-bfc3-4ac1-ab51-a94892b8e3a0 reached RUN_SUCCESS in dataprovider01: four payload Jobs, four distinct payload pod hostnames in contributing_hosts, and the payload's External payload started on … lines in the dispatcher's log, which is the pod log stream doing its job as the message channel. No credentials were needed anywhere.

One trap the run exposed. The platform's dagster-svc-account sets automountServiceAccountToken: false, and the Dagster chart overrides it to true on every pod it manages. A hand-written pod that does not is a plausible future failure: the pipes client selects in-cluster authentication correctly and then fails on Service token file does not exist, which points at Kubernetes rather than at the omission.

---

4. Common misconfiguration symptoms

4.1 Both setups

Symptom	Likely cause	Correction
Code location stuck in Loading, then errors	Entry point path in codeServerArgs does not match the image layout	Confirm --python-file matches workspace.yaml; both must be src/distributed_execution/repository.py
ImagePullBackOff with no match for platform	Image published for a single non-matching architecture	Rebuild multi-arch with docker buildx, and pin a version tag rather than latest
Run stays in QUEUED indefinitely	Run coordinator concurrency limit reached, or no schedulable node	Check max_concurrent_runs and tag concurrency limits; check node capacity and resource quota
Run fails immediately with a serialisation error	Code location image and Dagster control-plane versions diverge	Align the dagster version in pyproject.toml with the chart's version and rebuild

4.2 Tightly coupled

Symptom	Likely cause	Correction
Run pod starts, then fails with a connection timeout to port 5432	Execution namespace NetworkPolicy does not permit egress to Postgres	Add an egress rule for the metadata database, or move run pods to an already-approved namespace via jobNamespace
env_vars_missing is non-empty in report_execution_target metadata	Env vars are set on the code location deployment but not on the run pod	Add them under runLauncher.config.k8sRunLauncher.runK8sConfig.containerConfig.env — code location env is not inherited by run pods
A literal vault:... string appears as a value at runtime	Vault mutating webhook did not process the pod	Verify the vault.security.banzaicloud.io/* annotations are on the run pod template, not only the code location pod
Steps hang in STARTING with k8s_job_executor	Service account lacks Job create/watch permission	Apply yaml/tightly-coupled/rbac-step-executor.yaml and confirm with kubectl auth can-i
contributing_hosts shows one host when k8s_job_executor is configured	Run tags or Launchpad config overrode the executor, or the image predates the change	Confirm the code location reloaded after the image bump; check the run's Config tab for an execution: override
Step pods OOMKilled under fan-out	Per-step memory limit applied per pod, aggregate exceeded quota	Raise step_k8s_config limits or lower step concurrency; the two multiply
Postgres refuses connections once fan-out grows	Each step pod is an independent DB client	Reduce step concurrency, raise the Postgres connection limit, or move the fan-out step to a loosely coupled target

4.3 Loosely coupled

Symptom	Likely cause	Correction
Op fails with No pipes messages received from the external payload	The message path is broken, not the workload	Work through L6 then L4. The payload very likely ran and succeeded; only its reporting was lost
Payload pod Completed, but Dagster shows no payload log lines	A log shipper is intercepting or truncating stdout	Exclude the payload namespace from the shipper, or switch to an object-storage message reader
Op hangs until pod_wait_timeout (default 24 h)	Payload Job never scheduled — quota, node selector or image pull	Check kubectl -n <payload-ns> describe job <name>; lower pod_wait_timeout so the failure surfaces quickly
403 Forbidden creating the Job	Dispatcher service account lacks Job create permission	Apply yaml/loosely-coupled/rbac-pipes-dispatch.yaml in the payload namespace
Warning: Payload could see orchestration runtime credentials	Payload pod inherited run-pod env or a Vault annotation	Remove the inherited env; the payload should receive only what extras and explicit env pass it
Payload exits non-zero but the run reports success	Exit status not being checked, or messages read before failure	Confirm the dispatching op returns through _result_from_pipes; do not swallow PipesClientCompletedInvocation errors
Payload receives no units	extras key mismatch between dispatcher and pipes.get_extra()	Both sides must use the same key; a typo yields a KeyError inside the payload
Run cancelled in the UI, payload pod keeps running	Cancellation is not propagated to dispatched workloads automatically	delete_pod_on_completion handles the normal path; for cancellation, verify orphaned Jobs and add a cleanup sensor

Partly observed. The pipes rows were exercised on sandbox-cat-dat on 2026-08-31. Rows describing the tightly coupled k8s_job_executor still follow from the dagster-k8s API rather than from observation.

One symptom the cluster run added, absent from the table above: a service account with automountServiceAccountToken: false — which the platform's dagster-svc-account uses — makes the pipes client fail with ConfigException: Service token file does not exist. It reads as a Kubernetes fault; the fix is automountServiceAccountToken: true on the pod.

---

5. Evidence retention

For each workflow's first execution, attach to the workflow's repository or change record:

1. The run ID and its final status.
2. The report_execution_target output metadata block (pod identity, namespace, env var presence).
3. The summarise_results metadata block (contributing_hosts and contributing_workers), which proves which execution target was actually used and that the fan-out reached it.
4. For k8s_job_executor, the output of kubectl get jobs -l dagster/run-id=<run-id>.
5. For the loosely coupled target, the payload image digest and the run log line emitted by pipes.log — together they prove which payload version ran and that the message channel was open.

Items 2 and 3 together are sufficient to demonstrate that the configured execution target is the one that ran — which is the point of the checklist.