[SIMPL-30451] Close the distributed execution reference service

Check the chart's actual DAGSTER_PG_PASSWORD injection in the tightly coupled preflight instead of DAGSTER_POSTGRES_*, which the Simpl chart never sets; the old check warned about a misconfiguration on a correctly configured run pod. Evidence reports the variable name only, never its value.

Record the platform k8s_job_executor run as cluster-verified in the guide and readiness checklist, add the Dagster UI screenshots covering job list, graph, code-configured tags and a successful run.

Changelog: fixed

CHANGELOG.md

@@ -9,10 +9,25 @@
   (`PipesSubprocessClient`, `PipesK8sClient`) with a standalone external payload image.
 - Working example configuration and RBAC for both execution targets.
 
+### verified
+
+- `tightly_coupled_k8s_job` executed end to end on sandbox-cat-dat through the
+  platform `K8sRunLauncher` (run `1d8cb167-9fb8-4a34-a036-789eac381e13`), closing
+  readiness rows T6 and T7. Step outputs are written to the shared
+  `dagster-shared-pvc` by pinning `fs_io_manager`'s `base_dir`; `DAGSTER_HOME` set
+  through `step_k8s_config` is ignored, because the Dagster chart injects
+  `DAGSTER_HOME=/tmp/dagster` ahead of it.
+- `loosely_coupled_k8s_job` reached `RUN_SUCCESS` on sandbox-cat-dat through the
+  standalone probe (run `cff9b348-bfc3-4ac1-ab51-a94892b8e3a0`), with four
+  external payload pods reporting over the pod-log message channel.
+- Tightly coupled preflight evidence now checks the chart's actual
+  `DAGSTER_PG_PASSWORD` injection without exposing its value.
+- Dagster UI screenshots cover the registered jobs, graph, code-configured
+  execution-target tags and a successful reference run.
+- The platform deployment diagram includes the service and both integration
+  patterns.
+
 ### pending
 
-- Publish the payload image to the container registry.
+- Extend the project-level GitLab pipeline to publish the payload image as well
-- End-to-end run of `loosely_coupled_k8s_job` against a cluster, and confirmation
+  as the code-location image.
-  of the Kubernetes rows in the readiness checklist.
-- Screenshots of UI surfaces and integration-pattern diagrams.
-- Platform architecture document update.

README.md

@@ -14,7 +14,7 @@ actually run, and how does runtime information get back to the control plane?*
 | User guide | [documents/user-guide/distributed-execution-guide.md](documents/user-guide/distributed-execution-guide.md) | Complete |
 | Readiness checklist | [documents/user-guide/readiness-checklist.md](documents/user-guide/readiness-checklist.md) | Complete |
 | Tightly coupled reference | [src/distributed_execution/tightly_coupled/jobs.py](src/distributed_execution/tightly_coupled/jobs.py) | Runnable |
-| Loosely coupled reference | [src/distributed_execution/loosely_coupled/jobs.py](src/distributed_execution/loosely_coupled/jobs.py) | Runnable (subprocess verified) |
+| Loosely coupled reference | [src/distributed_execution/loosely_coupled/jobs.py](src/distributed_execution/loosely_coupled/jobs.py) | Runnable and cluster-verified |
 | External payload | [payload/work.py](payload/work.py) | Runnable |
 | Example configuration | [yaml/](yaml/) | Complete |
 
@@ -88,10 +88,10 @@ pipeline still has to learn.
 
 ## Status
 
-Both execution targets are implemented. The tightly coupled jobs and the loosely
+Both execution targets are implemented. Local variants are verified by the test
-coupled **subprocess** transport are verified by the test suite. The loosely
+suite, `tightly_coupled_k8s_job` has run end to end through the platform launcher,
-coupled **Kubernetes** transport is implemented but has not yet been run against
+and `loosely_coupled_k8s_job` has run end to end through the standalone cluster
-a cluster; see the guide's *Outstanding work* section.
+probe. See the guide and readiness checklist for the recorded evidence.
 
 ## Licence
 

documents/user-guide/distributed-execution-guide.md

@@ -5,14 +5,10 @@
 integration pattern for a Dagster workflow on the Simpl orchestration platform.
 
 > **Document status.** Both setups are backed by runnable reference
-> implementations in this repository. The tightly coupled setup and the loosely
+> implementations and verified end to end. The loosely coupled Kubernetes path
-> coupled **subprocess** transport are verified end to end, including runs
+> reached `RUN_SUCCESS` on sandbox-cat-dat on 2026-08-31; the tightly coupled
-> launched from the Dagster UI — see [section 5.6](#56-what-a-verified-run-actually-produced).
+> `k8s_job_executor` path reached `RUN_SUCCESS` through the platform launcher on
-> The loosely coupled **Kubernetes** transport was run on sandbox-cat-dat on
+> 2026-09-01. See the readiness checklist for the recorded evidence.
-> 2026-08-31 and reached `RUN_SUCCESS`; see the readiness checklist for the
-> evidence. What remains unproven on a cluster is the *tightly* coupled
-> Kubernetes path — `k8s_job_executor` — whose rows stay marked
-> **NOT CLUSTER-VERIFIED**. See [Outstanding work](#8-outstanding-work).
 
 ---
 
@@ -233,12 +229,18 @@ tightly_coupled_local_job = distributed_execution_reference.to_job(
 ```python
 tightly_coupled_k8s_job = distributed_execution_reference.to_job(
     name="tightly_coupled_k8s_job",
-    executor_def=k8s_job_executor.configured(
+    executor_def=k8s_job_executor,
-        {
+    resource_defs={
-            "image_pull_policy": "IfNotPresent",
+        "io_manager": fs_io_manager.configured({"base_dir": SHARED_IO_BASE_DIR})
-            "step_k8s_config": STEP_K8S_CONFIG,
+    },
+    config={
+        "execution": {
+            "config": {
+                "image_pull_policy": "IfNotPresent",
+                "step_k8s_config": STEP_K8S_CONFIG,
+            }
         }
-    ),
+    },
     tags={**COMMON_TAGS, "executor": "k8s_job"},
 )
 ```
@@ -253,6 +255,22 @@ operationally:
   [yaml/tightly-coupled/rbac-step-executor.yaml](../../yaml/tightly-coupled/rbac-step-executor.yaml).
 - Per-step resource requests apply per pod, so the aggregate request for a
   fan-out step is the per-step request multiplied by concurrency.
+- **Step outputs must land on storage every step pod can read.** With the default
+  filesystem I/O manager each pod writes to its own container filesystem, so a
+  downstream step opens a path that does not exist there. `STEP_K8S_CONFIG` mounts
+  the RWX `dagster-shared-pvc` at `/dagster/shared`, and the job pins
+  `fs_io_manager`'s `base_dir` underneath it.
+- Pin `base_dir` directly rather than relying on `DAGSTER_HOME`. The default
+  `base_dir` is `$DAGSTER_HOME/storage`, but the Dagster chart already injects
+  `DAGSTER_HOME=/tmp/dagster` into the step container; a second entry appended by
+  `step_k8s_config` does not displace it, so the override is silently ignored and
+  outputs keep going to `/tmp/dagster/storage`.
+
+The executor's defaults go through `config=` rather than
+`k8s_job_executor.configured(...)`. `.configured()` collapses the executor's
+config schema to `Any`, which makes a run-config `execution:` block accepted and
+then discarded, and makes `job_image` and `job_namespace` impossible to supply at
+launch time.
 
 The evidence that the switch took effect is in `summarise_results` output
 metadata, and the two fields say different things:
@@ -485,6 +503,32 @@ on a cluster it is what distinguishes `k8s_job_executor` from
 holds the namespace the run pod landed in, which is the placement half of the
 execution target.
 
+### 5.7 Dagster UI reference
+
+The screenshots below were captured from the local reference service on
+2026-09-02. They show the surfaces participants use to inspect job linkage and
+run evidence. Cluster placement is evidenced separately by the sandbox run IDs
+in the readiness checklist.
+
+All five reference jobs are registered under one code location:
+
+![Distributed execution job list](images/distributed-execution-job-list.png)
+
+The job overview exposes the graph whose executor or dispatch op defines the
+integration pattern:
+
+![Tightly coupled reference graph](images/distributed-execution-job-graph.png)
+
+The Launchpad shows that `execution_target` and `executor` are job tags persisted
+by code configuration, not a separate runtime-only selector:
+
+![Execution-target tags in Launchpad](images/distributed-execution-launchpad.png)
+
+A successful local run displays the step events and final status used alongside
+output metadata as retained evidence:
+
+![Successful tightly coupled reference run](images/distributed-execution-successful-run.png)
+
 ---
 
 ## 6. Upstream documentation
@@ -512,8 +556,8 @@ Tracked under SIMPL-30451.
 |---|---|---|
 | Execution-target choices, prerequisites, workflow-level documentation | AC1 | Complete |
 | Comparison, environment-fit indicators, trade-offs | AC2 | Complete |
-| Readiness checklist and evidence mapping | AC3 | Complete; tightly coupled K8s executor rows not cluster-verified |
+| Readiness checklist and evidence mapping | AC3 | Complete; both Kubernetes paths cluster-verified |
-| Misconfiguration symptoms and corrections | AC3 | Complete; tightly coupled K8s symptoms not cluster-verified |
+| Misconfiguration symptoms and corrections | AC3 | Complete; corrected with observed cluster failure modes |
 | Configuration constructs and persistence | AC4 | Complete |
 | Before-and-after example, tightly coupled | AC4 | Complete |
 | Before-and-after example, switching to loosely coupled | AC4 | Complete |
@@ -524,6 +568,6 @@ Tracked under SIMPL-30451.
 | Payload image published to a container registry | Tech details | Complete on the sandbox Gitea registry; the GitLab registry still pending |
 | GitLab pipeline builds both images | Tech details | **Pending** — the shared `ds.gitlab-ci.yml` template builds one image from the root Dockerfile |
 | End-to-end run of `loosely_coupled_k8s_job` on a cluster | Tech details | Complete — sandbox-cat-dat, 2026-08-31, run `cff9b348…`; see the readiness checklist |
-| `distributed-execution` registered as a code location on a platform Dagster | Tech details | **Pending** — blocked on the 1.13.19 vs 1.12.8 control plane skew |
+| `distributed-execution` registered as a code location on a platform Dagster | Tech details | Complete — tightly coupled run `1d8cb167…` launched through the platform webserver |
-| Screenshots of UI surfaces | Tech details | **Pending** — needs a deployed platform instance, not a local dev server |
+| Screenshots of UI surfaces | Tech details | Complete — job list, graph, Launchpad linkage and successful run captured in section 5.7 |
-| Platform architecture document update | Tech details | **Pending** |
+| Platform architecture document update | Tech details | Complete — root `deployment_diagram.md` includes the service and both runtime-information paths |

documents/user-guide/readiness-checklist.md

@@ -7,12 +7,11 @@ Run these checks **before the first execution** of a workflow, and again after a
 change to the run launcher, the executor, the target namespace or the code
 location image.
 
-> **L4–L10 were cleared on sandbox-cat-dat on 2026-08-31.** Checks L1–L3 and L11,
+> **Both Kubernetes paths are cluster-verified.** L4–L10 were cleared on
-> which exercise the payload contract, the message-parsing path and image-level
+> sandbox-cat-dat on 2026-08-31. The tightly coupled `k8s_job_executor` path
-> isolation, are verified locally. The **tightly coupled** Kubernetes path has
+> reached `RUN_SUCCESS` through the platform launcher on 2026-09-01. Checks
-> been assessed against the same cluster but not executed — C5, C6, T2, T3, T5
+> L1–L3 and L11, which exercise the payload contract, message parsing and image
-> and T8 are confirmed from the live instance configuration and the deployed code
+> isolation, are also verified locally. See [section 2.1](#21-sandbox-state-2026-08-31).
-> locations. See [section 2.1](#21-sandbox-state-2026-08-31).
 
 ---
 
@@ -31,7 +30,7 @@ location image.
 
 | # | Check | How to verify | Expected evidence |
 |---|---|---|---|
-| T1 | Run pod reaches the metadata database | Launch `tightly_coupled_in_process_job` | Run reaches `SUCCESS`; `report_execution_target` output metadata lists `DAGSTER_POSTGRES_HOST`, `DAGSTER_POSTGRES_USER` and `DAGSTER_POSTGRES_DB` under `env_vars_present`, and `env_vars_missing` is empty |
+| T1 | Run pod reaches the metadata database | Launch `tightly_coupled_in_process_job` | Run reaches `SUCCESS`; `report_execution_target` metadata lists `DAGSTER_PG_PASSWORD` under `env_vars_present` without exposing its value, and `env_vars_missing` is empty. Successful step startup is the connectivity proof because the step loads run state from Postgres before executing the op |
 | T2 | Vault injection works | Same run; inspect the run pod | `kubectl -n dagster describe pod <run-pod>` shows the `vault-env` init container completed; no `vault:` literal remains in the process environment |
 | T3 | Object storage is reachable | Same run, if the workflow uses S3 | No `EndpointConnectionError` in run logs; `S3_ENDPOINT_URL` present in `env_vars_present` |
 | T4 | Multiprocess fan-out actually fans out | Launch `tightly_coupled_local_job` | Run succeeds; `summarise_results` metadata shows one entry per unit in `contributing_workers` and a single entry in `contributing_hosts` — separate processes, same machine |
@@ -67,7 +66,7 @@ instance rather than the platform's Postgres.
 
 `tightly_coupled_in_process_job` also succeeded, reporting a single worker.
 
-### T1's env-var check is wrong for this platform
+### T1's env-var check and platform contract
 
 The run emitted:
 
@@ -75,19 +74,20 @@ The run emitted:
 > `DAGSTER_POSTGRES_HOST`, `DAGSTER_POSTGRES_USER`, `DAGSTER_POSTGRES_DB`.
 > Expected for a loosely coupled target; a misconfiguration for a tightly coupled one.
 
-That warning would fire on a **correctly configured** platform run pod. The chart
+That warning was a false positive on a **correctly configured** platform run
-does not set those variables: it injects `DAGSTER_PG_PASSWORD` from
+pod. The chart does not set those variables: it injects `DAGSTER_PG_PASSWORD`
-`dagster-postgresql-secret` and bakes host, user and database into `dagster.yaml`.
+from `dagster-postgresql-secret` and stores host, user and database in
-So `TIGHTLY_COUPLED_ENV_VARS` in `preflight.py` names variables Simpl does not
+`dagster.yaml`. The reference check now tests for `DAGSTER_PG_PASSWORD` and
-use, and the check reports a misconfiguration that is not one. The reachability
+reports only its name, never its value. Database reachability is evidenced by
-probe above is the part that carries real evidence.
+successful step startup: a tightly coupled step must load its run state from
+the configured Dagster instance before `report_execution_target` can execute.
 
 ### T6 and T7, executed 2026-08-31
 
 Running them needed three things the earlier attempt lacked: dagster pinned to
 1.12.8 to match the control plane, so writing to the live `dataprovider01_dagster`
-could not trigger a migration; `DAGSTER_HOME` moved onto the RWX
+could not trigger a migration; step outputs moved onto the RWX
-`dagster-shared-pvc`, because the default I/O manager writes step outputs under
+`dagster-shared-pvc`, because the default I/O manager writes them under
 `$DAGSTER_HOME/storage` and each step pod otherwise gets its own empty volume;
 and `automount_service_account_token: true` in `step_k8s_config`, because
 `dagster-svc-account` disables it by default. The probe is
@@ -125,6 +125,39 @@ A run submitted the normal way, through the webserver to `K8sRunLauncher`, gets
 T6/T7 end to end, including `summarise_results`, needs the code location
 registered on the platform Dagster.
 
+### T6 and T7 closed end to end, 2026-09-01
+
+With the code location registered on the platform Dagster,
+`tightly_coupled_k8s_job` was launched from the webserver and reached
+`RUN_SUCCESS` as run `1d8cb167-9fb8-4a34-a036-789eac381e13`. `K8sRunLauncher`
+created `dagster-run-1d8cb167-…` in `dataprovider01`, `K8sStepHandler` created one
+`dagster-step-…` Job per step, and every step — including `summarise_results` —
+completed. No hand-rolled probe, no run-monitoring 403.
+
+**The last thing in the way was where step outputs are written.** Two earlier
+attempts failed on
+`FileNotFoundError: /tmp/dagster/storage/<run-id>/generate_work_units/result/unit_0`,
+with every step pod using its own container filesystem. Mounting
+`dagster-shared-pvc` into the step pods was necessary but not sufficient: setting
+`DAGSTER_HOME` through `step_k8s_config` does not take effect, because the Dagster
+chart already injects `DAGSTER_HOME=/tmp/dagster` and the appended second entry
+does not displace the first. The fix is to bypass `DAGSTER_HOME` and pin the I/O
+manager instead — `fs_io_manager.configured({"base_dir": SHARED_IO_BASE_DIR})` on
+the job, with `SHARED_IO_BASE_DIR` under the mounted PVC. The successful run logs
+confirm it:
+
+> Handled output "result" using IO manager "io_manager" —
+> `/dagster/shared/distributed-execution/storage/1d8cb167-…/generate_work_units/result/unit_0`
+
+The platform run exposed the false-positive `DAGSTER_POSTGRES_*` warning
+described under T1 above. The reference now checks the chart's actual
+`DAGSTER_PG_PASSWORD` injection instead.
+
+> Note: `report_execution_target` reported `namespace: <not-in-kubernetes>` on a
+> step pod that plainly was in Kubernetes. The check reads the service account
+> namespace file, which is absent when the token is not automounted. It is a
+> property of the probe, not of the run.
+
 **C6 passes, contrary to what `job_namespace` alone suggests.** The launcher's
 `job_namespace: dagster` is only a default. Each code location ships a
 `DAGSTER_CONTAINER_CONTEXT` environment variable — generated by the chart — whose
@@ -141,10 +174,8 @@ Both deployed code locations carry it, so both would launch correctly:
 | `template-code-location` | `dataprovider01` |
 | `semaphore-ui` | `dataprovider01` |
 
-What is still true is that **no run has ever executed on this cluster** —
+The successful run confirms that the code-location container context places run
-`dataprovider01` holds no pods labelled `dagster/run-id` and the daemon log shows
+and step pods in `dataprovider01` with the required namespace-local dependencies.
-no launch activity. That appears to be because nothing has been launched, not
-because anything is broken.
 
 The residual risk is narrower than a broken namespace, but real: a code location
 that reaches the launcher *without* a container context namespace inherits
@@ -216,6 +247,7 @@ rather than at the omission.
 | `env_vars_missing` is non-empty in `report_execution_target` metadata | Env vars are set on the code location deployment but not on the run pod | Add them under `runLauncher.config.k8sRunLauncher.runK8sConfig.containerConfig.env` — code location env is **not** inherited by run pods |
 | A literal `vault:...` string appears as a value at runtime | Vault mutating webhook did not process the pod | Verify the `vault.security.banzaicloud.io/*` annotations are on the **run pod** template, not only the code location pod |
 | Steps hang in `STARTING` with `k8s_job_executor` | Service account lacks Job create/watch permission | Apply `yaml/tightly-coupled/rbac-step-executor.yaml` and confirm with `kubectl auth can-i` |
+| `FileNotFoundError` loading a step input from `/tmp/dagster/storage/...` | Step pods each use their own container filesystem for I/O manager output | Mount an RWX volume into step pods via `step_k8s_config` **and** pin `fs_io_manager`'s `base_dir` under it. Setting `DAGSTER_HOME` in `step_k8s_config` does not work — the chart's `DAGSTER_HOME=/tmp/dagster` is injected first and wins |
 | `contributing_hosts` shows one host when `k8s_job_executor` is configured | Run tags or Launchpad config overrode the executor, or the image predates the change | Confirm the code location reloaded after the image bump; check the run's *Config* tab for an `execution:` override |
 | Step pods `OOMKilled` under fan-out | Per-step memory limit applied per pod, aggregate exceeded quota | Raise `step_k8s_config` limits or lower step concurrency; the two multiply |
 | Postgres refuses connections once fan-out grows | Each step pod is an independent DB client | Reduce step concurrency, raise the Postgres connection limit, or move the fan-out step to a loosely coupled target |
@@ -234,8 +266,10 @@ rather than at the omission.
 | Run cancelled in the UI, payload pod keeps running | Cancellation is not propagated to dispatched workloads automatically | `delete_pod_on_completion` handles the normal path; for cancellation, verify orphaned Jobs and add a cleanup sensor |
 
 > **Partly observed.** The pipes rows were exercised on sandbox-cat-dat on
-> 2026-08-31. Rows describing the tightly coupled `k8s_job_executor` still follow
+> 2026-08-31. The tightly coupled `k8s_job_executor` rows were closed on
-> from the dagster-k8s API rather than from observation.
+> 2026-09-01 by run `1d8cb167-9fb8-4a34-a036-789eac381e13`; the remaining rows in
+> the table above still follow from the dagster-k8s API rather than from
+> observation.
 >
 > One symptom the cluster run added, absent from the table above: a service
 > account with `automountServiceAccountToken: false` — which the platform's

src/distributed_execution/preflight.py

@@ -12,9 +12,7 @@ from urllib.parse import urlparse
 
 # Runtime dependencies a tightly coupled execution pod must be able to resolve.
 TIGHTLY_COUPLED_ENV_VARS = (
-    "DAGSTER_POSTGRES_HOST",
+    "DAGSTER_PG_PASSWORD",
-    "DAGSTER_POSTGRES_USER",
-    "DAGSTER_POSTGRES_DB",
 )
 
 

tests/test_loosely_coupled.py

@@ -65,9 +65,7 @@ def test_each_unit_is_dispatched_to_its_own_external_worker():
 
 
 def test_payload_reports_no_orchestration_credentials(monkeypatch):
-    monkeypatch.delenv("DAGSTER_POSTGRES_HOST", raising=False)
+    monkeypatch.delenv("DAGSTER_PG_PASSWORD", raising=False)
-    monkeypatch.delenv("DAGSTER_POSTGRES_USER", raising=False)
-    monkeypatch.delenv("DAGSTER_POSTGRES_DB", raising=False)
 
     result = loosely_coupled_subprocess_job.execute_in_process()
     mapped = result.output_for_node("dispatch_external_work_subprocess")

tests/test_tightly_coupled.py

@@ -81,6 +81,16 @@ def test_env_var_check_reports_missing_names():
     assert result["missing"] == ["DEFINITELY_NOT_SET_12345"]
 
 
+def test_env_var_check_reports_present_names(monkeypatch):
+    monkeypatch.setenv("DAGSTER_PG_PASSWORD", "not-exposed-in-evidence")
+
+    result = check_env_vars(("DAGSTER_PG_PASSWORD",))
+
+    assert result["passed"] is True
+    assert result["present"] == ["DAGSTER_PG_PASSWORD"]
+    assert "not-exposed-in-evidence" not in repr(result)
+
+
 def test_pod_identity_falls_back_outside_kubernetes(monkeypatch):
     monkeypatch.delenv("DAGSTER_K8S_PIPELINE_RUN_NAMESPACE", raising=False)