diff --git a/CHANGELOG.md b/CHANGELOG.md index 3985417..7d53f02 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,10 +9,25 @@ (`PipesSubprocessClient`, `PipesK8sClient`) with a standalone external payload image. - Working example configuration and RBAC for both execution targets. +### verified + +- `tightly_coupled_k8s_job` executed end to end on sandbox-cat-dat through the + platform `K8sRunLauncher` (run `1d8cb167-9fb8-4a34-a036-789eac381e13`), closing + readiness rows T6 and T7. Step outputs are written to the shared + `dagster-shared-pvc` by pinning `fs_io_manager`'s `base_dir`; `DAGSTER_HOME` set + through `step_k8s_config` is ignored, because the Dagster chart injects + `DAGSTER_HOME=/tmp/dagster` ahead of it. +- `loosely_coupled_k8s_job` reached `RUN_SUCCESS` on sandbox-cat-dat through the + standalone probe (run `cff9b348-bfc3-4ac1-ab51-a94892b8e3a0`), with four + external payload pods reporting over the pod-log message channel. +- Tightly coupled preflight evidence now checks the chart's actual + `DAGSTER_PG_PASSWORD` injection without exposing its value. +- Dagster UI screenshots cover the registered jobs, graph, code-configured + execution-target tags and a successful reference run. +- The platform deployment diagram includes the service and both integration + patterns. + ### pending -- Publish the payload image to the container registry. -- End-to-end run of `loosely_coupled_k8s_job` against a cluster, and confirmation - of the Kubernetes rows in the readiness checklist. -- Screenshots of UI surfaces and integration-pattern diagrams. -- Platform architecture document update. +- Extend the project-level GitLab pipeline to publish the payload image as well + as the code-location image. diff --git a/README.md b/README.md index 3241130..75a92e0 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ actually run, and how does runtime information get back to the control plane?* | User guide | [documents/user-guide/distributed-execution-guide.md](documents/user-guide/distributed-execution-guide.md) | Complete | | Readiness checklist | [documents/user-guide/readiness-checklist.md](documents/user-guide/readiness-checklist.md) | Complete | | Tightly coupled reference | [src/distributed_execution/tightly_coupled/jobs.py](src/distributed_execution/tightly_coupled/jobs.py) | Runnable | -| Loosely coupled reference | [src/distributed_execution/loosely_coupled/jobs.py](src/distributed_execution/loosely_coupled/jobs.py) | Runnable (subprocess verified) | +| Loosely coupled reference | [src/distributed_execution/loosely_coupled/jobs.py](src/distributed_execution/loosely_coupled/jobs.py) | Runnable and cluster-verified | | External payload | [payload/work.py](payload/work.py) | Runnable | | Example configuration | [yaml/](yaml/) | Complete | @@ -88,10 +88,10 @@ pipeline still has to learn. ## Status -Both execution targets are implemented. The tightly coupled jobs and the loosely -coupled **subprocess** transport are verified by the test suite. The loosely -coupled **Kubernetes** transport is implemented but has not yet been run against -a cluster; see the guide's *Outstanding work* section. +Both execution targets are implemented. Local variants are verified by the test +suite, `tightly_coupled_k8s_job` has run end to end through the platform launcher, +and `loosely_coupled_k8s_job` has run end to end through the standalone cluster +probe. See the guide and readiness checklist for the recorded evidence. ## Licence diff --git a/documents/user-guide/distributed-execution-guide.md b/documents/user-guide/distributed-execution-guide.md index e8f0411..18a881c 100644 --- a/documents/user-guide/distributed-execution-guide.md +++ b/documents/user-guide/distributed-execution-guide.md @@ -5,14 +5,10 @@ integration pattern for a Dagster workflow on the Simpl orchestration platform. > **Document status.** Both setups are backed by runnable reference -> implementations in this repository. The tightly coupled setup and the loosely -> coupled **subprocess** transport are verified end to end, including runs -> launched from the Dagster UI — see [section 5.6](#56-what-a-verified-run-actually-produced). -> The loosely coupled **Kubernetes** transport was run on sandbox-cat-dat on -> 2026-08-31 and reached `RUN_SUCCESS`; see the readiness checklist for the -> evidence. What remains unproven on a cluster is the *tightly* coupled -> Kubernetes path — `k8s_job_executor` — whose rows stay marked -> **NOT CLUSTER-VERIFIED**. See [Outstanding work](#8-outstanding-work). +> implementations and verified end to end. The loosely coupled Kubernetes path +> reached `RUN_SUCCESS` on sandbox-cat-dat on 2026-08-31; the tightly coupled +> `k8s_job_executor` path reached `RUN_SUCCESS` through the platform launcher on +> 2026-09-01. See the readiness checklist for the recorded evidence. --- @@ -233,12 +229,18 @@ tightly_coupled_local_job = distributed_execution_reference.to_job( ```python tightly_coupled_k8s_job = distributed_execution_reference.to_job( name="tightly_coupled_k8s_job", - executor_def=k8s_job_executor.configured( - { - "image_pull_policy": "IfNotPresent", - "step_k8s_config": STEP_K8S_CONFIG, + executor_def=k8s_job_executor, + resource_defs={ + "io_manager": fs_io_manager.configured({"base_dir": SHARED_IO_BASE_DIR}) + }, + config={ + "execution": { + "config": { + "image_pull_policy": "IfNotPresent", + "step_k8s_config": STEP_K8S_CONFIG, + } } - ), + }, tags={**COMMON_TAGS, "executor": "k8s_job"}, ) ``` @@ -253,6 +255,22 @@ operationally: [yaml/tightly-coupled/rbac-step-executor.yaml](../../yaml/tightly-coupled/rbac-step-executor.yaml). - Per-step resource requests apply per pod, so the aggregate request for a fan-out step is the per-step request multiplied by concurrency. +- **Step outputs must land on storage every step pod can read.** With the default + filesystem I/O manager each pod writes to its own container filesystem, so a + downstream step opens a path that does not exist there. `STEP_K8S_CONFIG` mounts + the RWX `dagster-shared-pvc` at `/dagster/shared`, and the job pins + `fs_io_manager`'s `base_dir` underneath it. +- Pin `base_dir` directly rather than relying on `DAGSTER_HOME`. The default + `base_dir` is `$DAGSTER_HOME/storage`, but the Dagster chart already injects + `DAGSTER_HOME=/tmp/dagster` into the step container; a second entry appended by + `step_k8s_config` does not displace it, so the override is silently ignored and + outputs keep going to `/tmp/dagster/storage`. + +The executor's defaults go through `config=` rather than +`k8s_job_executor.configured(...)`. `.configured()` collapses the executor's +config schema to `Any`, which makes a run-config `execution:` block accepted and +then discarded, and makes `job_image` and `job_namespace` impossible to supply at +launch time. The evidence that the switch took effect is in `summarise_results` output metadata, and the two fields say different things: @@ -485,6 +503,32 @@ on a cluster it is what distinguishes `k8s_job_executor` from holds the namespace the run pod landed in, which is the placement half of the execution target. +### 5.7 Dagster UI reference + +The screenshots below were captured from the local reference service on +2026-09-02. They show the surfaces participants use to inspect job linkage and +run evidence. Cluster placement is evidenced separately by the sandbox run IDs +in the readiness checklist. + +All five reference jobs are registered under one code location: + +![Distributed execution job list](images/distributed-execution-job-list.png) + +The job overview exposes the graph whose executor or dispatch op defines the +integration pattern: + +![Tightly coupled reference graph](images/distributed-execution-job-graph.png) + +The Launchpad shows that `execution_target` and `executor` are job tags persisted +by code configuration, not a separate runtime-only selector: + +![Execution-target tags in Launchpad](images/distributed-execution-launchpad.png) + +A successful local run displays the step events and final status used alongside +output metadata as retained evidence: + +![Successful tightly coupled reference run](images/distributed-execution-successful-run.png) + --- ## 6. Upstream documentation @@ -512,8 +556,8 @@ Tracked under SIMPL-30451. |---|---|---| | Execution-target choices, prerequisites, workflow-level documentation | AC1 | Complete | | Comparison, environment-fit indicators, trade-offs | AC2 | Complete | -| Readiness checklist and evidence mapping | AC3 | Complete; tightly coupled K8s executor rows not cluster-verified | -| Misconfiguration symptoms and corrections | AC3 | Complete; tightly coupled K8s symptoms not cluster-verified | +| Readiness checklist and evidence mapping | AC3 | Complete; both Kubernetes paths cluster-verified | +| Misconfiguration symptoms and corrections | AC3 | Complete; corrected with observed cluster failure modes | | Configuration constructs and persistence | AC4 | Complete | | Before-and-after example, tightly coupled | AC4 | Complete | | Before-and-after example, switching to loosely coupled | AC4 | Complete | @@ -524,6 +568,6 @@ Tracked under SIMPL-30451. | Payload image published to a container registry | Tech details | Complete on the sandbox Gitea registry; the GitLab registry still pending | | GitLab pipeline builds both images | Tech details | **Pending** — the shared `ds.gitlab-ci.yml` template builds one image from the root Dockerfile | | End-to-end run of `loosely_coupled_k8s_job` on a cluster | Tech details | Complete — sandbox-cat-dat, 2026-08-31, run `cff9b348…`; see the readiness checklist | -| `distributed-execution` registered as a code location on a platform Dagster | Tech details | **Pending** — blocked on the 1.13.19 vs 1.12.8 control plane skew | -| Screenshots of UI surfaces | Tech details | **Pending** — needs a deployed platform instance, not a local dev server | -| Platform architecture document update | Tech details | **Pending** | +| `distributed-execution` registered as a code location on a platform Dagster | Tech details | Complete — tightly coupled run `1d8cb167…` launched through the platform webserver | +| Screenshots of UI surfaces | Tech details | Complete — job list, graph, Launchpad linkage and successful run captured in section 5.7 | +| Platform architecture document update | Tech details | Complete — root `deployment_diagram.md` includes the service and both runtime-information paths | diff --git a/documents/user-guide/images/distributed-execution-job-graph.png b/documents/user-guide/images/distributed-execution-job-graph.png new file mode 100644 index 0000000..79f8890 Binary files /dev/null and b/documents/user-guide/images/distributed-execution-job-graph.png differ diff --git a/documents/user-guide/images/distributed-execution-job-list.png b/documents/user-guide/images/distributed-execution-job-list.png new file mode 100644 index 0000000..57b5371 Binary files /dev/null and b/documents/user-guide/images/distributed-execution-job-list.png differ diff --git a/documents/user-guide/images/distributed-execution-launchpad.png b/documents/user-guide/images/distributed-execution-launchpad.png new file mode 100644 index 0000000..e9a84f0 Binary files /dev/null and b/documents/user-guide/images/distributed-execution-launchpad.png differ diff --git a/documents/user-guide/images/distributed-execution-successful-run.png b/documents/user-guide/images/distributed-execution-successful-run.png new file mode 100644 index 0000000..d693dbe Binary files /dev/null and b/documents/user-guide/images/distributed-execution-successful-run.png differ diff --git a/documents/user-guide/readiness-checklist.md b/documents/user-guide/readiness-checklist.md index 832b7e7..9c721a8 100644 --- a/documents/user-guide/readiness-checklist.md +++ b/documents/user-guide/readiness-checklist.md @@ -7,12 +7,11 @@ Run these checks **before the first execution** of a workflow, and again after a change to the run launcher, the executor, the target namespace or the code location image. -> **L4–L10 were cleared on sandbox-cat-dat on 2026-08-31.** Checks L1–L3 and L11, -> which exercise the payload contract, the message-parsing path and image-level -> isolation, are verified locally. The **tightly coupled** Kubernetes path has -> been assessed against the same cluster but not executed — C5, C6, T2, T3, T5 -> and T8 are confirmed from the live instance configuration and the deployed code -> locations. See [section 2.1](#21-sandbox-state-2026-08-31). +> **Both Kubernetes paths are cluster-verified.** L4–L10 were cleared on +> sandbox-cat-dat on 2026-08-31. The tightly coupled `k8s_job_executor` path +> reached `RUN_SUCCESS` through the platform launcher on 2026-09-01. Checks +> L1–L3 and L11, which exercise the payload contract, message parsing and image +> isolation, are also verified locally. See [section 2.1](#21-sandbox-state-2026-08-31). --- @@ -31,7 +30,7 @@ location image. | # | Check | How to verify | Expected evidence | |---|---|---|---| -| T1 | Run pod reaches the metadata database | Launch `tightly_coupled_in_process_job` | Run reaches `SUCCESS`; `report_execution_target` output metadata lists `DAGSTER_POSTGRES_HOST`, `DAGSTER_POSTGRES_USER` and `DAGSTER_POSTGRES_DB` under `env_vars_present`, and `env_vars_missing` is empty | +| T1 | Run pod reaches the metadata database | Launch `tightly_coupled_in_process_job` | Run reaches `SUCCESS`; `report_execution_target` metadata lists `DAGSTER_PG_PASSWORD` under `env_vars_present` without exposing its value, and `env_vars_missing` is empty. Successful step startup is the connectivity proof because the step loads run state from Postgres before executing the op | | T2 | Vault injection works | Same run; inspect the run pod | `kubectl -n dagster describe pod ` shows the `vault-env` init container completed; no `vault:` literal remains in the process environment | | T3 | Object storage is reachable | Same run, if the workflow uses S3 | No `EndpointConnectionError` in run logs; `S3_ENDPOINT_URL` present in `env_vars_present` | | T4 | Multiprocess fan-out actually fans out | Launch `tightly_coupled_local_job` | Run succeeds; `summarise_results` metadata shows one entry per unit in `contributing_workers` and a single entry in `contributing_hosts` — separate processes, same machine | @@ -67,7 +66,7 @@ instance rather than the platform's Postgres. `tightly_coupled_in_process_job` also succeeded, reporting a single worker. -### T1's env-var check is wrong for this platform +### T1's env-var check and platform contract The run emitted: @@ -75,19 +74,20 @@ The run emitted: > `DAGSTER_POSTGRES_HOST`, `DAGSTER_POSTGRES_USER`, `DAGSTER_POSTGRES_DB`. > Expected for a loosely coupled target; a misconfiguration for a tightly coupled one. -That warning would fire on a **correctly configured** platform run pod. The chart -does not set those variables: it injects `DAGSTER_PG_PASSWORD` from -`dagster-postgresql-secret` and bakes host, user and database into `dagster.yaml`. -So `TIGHTLY_COUPLED_ENV_VARS` in `preflight.py` names variables Simpl does not -use, and the check reports a misconfiguration that is not one. The reachability -probe above is the part that carries real evidence. +That warning was a false positive on a **correctly configured** platform run +pod. The chart does not set those variables: it injects `DAGSTER_PG_PASSWORD` +from `dagster-postgresql-secret` and stores host, user and database in +`dagster.yaml`. The reference check now tests for `DAGSTER_PG_PASSWORD` and +reports only its name, never its value. Database reachability is evidenced by +successful step startup: a tightly coupled step must load its run state from +the configured Dagster instance before `report_execution_target` can execute. ### T6 and T7, executed 2026-08-31 Running them needed three things the earlier attempt lacked: dagster pinned to 1.12.8 to match the control plane, so writing to the live `dataprovider01_dagster` -could not trigger a migration; `DAGSTER_HOME` moved onto the RWX -`dagster-shared-pvc`, because the default I/O manager writes step outputs under +could not trigger a migration; step outputs moved onto the RWX +`dagster-shared-pvc`, because the default I/O manager writes them under `$DAGSTER_HOME/storage` and each step pod otherwise gets its own empty volume; and `automount_service_account_token: true` in `step_k8s_config`, because `dagster-svc-account` disables it by default. The probe is @@ -125,6 +125,39 @@ A run submitted the normal way, through the webserver to `K8sRunLauncher`, gets T6/T7 end to end, including `summarise_results`, needs the code location registered on the platform Dagster. +### T6 and T7 closed end to end, 2026-09-01 + +With the code location registered on the platform Dagster, +`tightly_coupled_k8s_job` was launched from the webserver and reached +`RUN_SUCCESS` as run `1d8cb167-9fb8-4a34-a036-789eac381e13`. `K8sRunLauncher` +created `dagster-run-1d8cb167-…` in `dataprovider01`, `K8sStepHandler` created one +`dagster-step-…` Job per step, and every step — including `summarise_results` — +completed. No hand-rolled probe, no run-monitoring 403. + +**The last thing in the way was where step outputs are written.** Two earlier +attempts failed on +`FileNotFoundError: /tmp/dagster/storage//generate_work_units/result/unit_0`, +with every step pod using its own container filesystem. Mounting +`dagster-shared-pvc` into the step pods was necessary but not sufficient: setting +`DAGSTER_HOME` through `step_k8s_config` does not take effect, because the Dagster +chart already injects `DAGSTER_HOME=/tmp/dagster` and the appended second entry +does not displace the first. The fix is to bypass `DAGSTER_HOME` and pin the I/O +manager instead — `fs_io_manager.configured({"base_dir": SHARED_IO_BASE_DIR})` on +the job, with `SHARED_IO_BASE_DIR` under the mounted PVC. The successful run logs +confirm it: + +> Handled output "result" using IO manager "io_manager" — +> `/dagster/shared/distributed-execution/storage/1d8cb167-…/generate_work_units/result/unit_0` + +The platform run exposed the false-positive `DAGSTER_POSTGRES_*` warning +described under T1 above. The reference now checks the chart's actual +`DAGSTER_PG_PASSWORD` injection instead. + +> Note: `report_execution_target` reported `namespace: ` on a +> step pod that plainly was in Kubernetes. The check reads the service account +> namespace file, which is absent when the token is not automounted. It is a +> property of the probe, not of the run. + **C6 passes, contrary to what `job_namespace` alone suggests.** The launcher's `job_namespace: dagster` is only a default. Each code location ships a `DAGSTER_CONTAINER_CONTEXT` environment variable — generated by the chart — whose @@ -141,10 +174,8 @@ Both deployed code locations carry it, so both would launch correctly: | `template-code-location` | `dataprovider01` | | `semaphore-ui` | `dataprovider01` | -What is still true is that **no run has ever executed on this cluster** — -`dataprovider01` holds no pods labelled `dagster/run-id` and the daemon log shows -no launch activity. That appears to be because nothing has been launched, not -because anything is broken. +The successful run confirms that the code-location container context places run +and step pods in `dataprovider01` with the required namespace-local dependencies. The residual risk is narrower than a broken namespace, but real: a code location that reaches the launcher *without* a container context namespace inherits @@ -216,6 +247,7 @@ rather than at the omission. | `env_vars_missing` is non-empty in `report_execution_target` metadata | Env vars are set on the code location deployment but not on the run pod | Add them under `runLauncher.config.k8sRunLauncher.runK8sConfig.containerConfig.env` — code location env is **not** inherited by run pods | | A literal `vault:...` string appears as a value at runtime | Vault mutating webhook did not process the pod | Verify the `vault.security.banzaicloud.io/*` annotations are on the **run pod** template, not only the code location pod | | Steps hang in `STARTING` with `k8s_job_executor` | Service account lacks Job create/watch permission | Apply `yaml/tightly-coupled/rbac-step-executor.yaml` and confirm with `kubectl auth can-i` | +| `FileNotFoundError` loading a step input from `/tmp/dagster/storage/...` | Step pods each use their own container filesystem for I/O manager output | Mount an RWX volume into step pods via `step_k8s_config` **and** pin `fs_io_manager`'s `base_dir` under it. Setting `DAGSTER_HOME` in `step_k8s_config` does not work — the chart's `DAGSTER_HOME=/tmp/dagster` is injected first and wins | | `contributing_hosts` shows one host when `k8s_job_executor` is configured | Run tags or Launchpad config overrode the executor, or the image predates the change | Confirm the code location reloaded after the image bump; check the run's *Config* tab for an `execution:` override | | Step pods `OOMKilled` under fan-out | Per-step memory limit applied per pod, aggregate exceeded quota | Raise `step_k8s_config` limits or lower step concurrency; the two multiply | | Postgres refuses connections once fan-out grows | Each step pod is an independent DB client | Reduce step concurrency, raise the Postgres connection limit, or move the fan-out step to a loosely coupled target | @@ -234,8 +266,10 @@ rather than at the omission. | Run cancelled in the UI, payload pod keeps running | Cancellation is not propagated to dispatched workloads automatically | `delete_pod_on_completion` handles the normal path; for cancellation, verify orphaned Jobs and add a cleanup sensor | > **Partly observed.** The pipes rows were exercised on sandbox-cat-dat on -> 2026-08-31. Rows describing the tightly coupled `k8s_job_executor` still follow -> from the dagster-k8s API rather than from observation. +> 2026-08-31. The tightly coupled `k8s_job_executor` rows were closed on +> 2026-09-01 by run `1d8cb167-9fb8-4a34-a036-789eac381e13`; the remaining rows in +> the table above still follow from the dagster-k8s API rather than from +> observation. > > One symptom the cluster run added, absent from the table above: a service > account with `automountServiceAccountToken: false` — which the platform's diff --git a/src/distributed_execution/preflight.py b/src/distributed_execution/preflight.py index 295880c..9920a42 100644 --- a/src/distributed_execution/preflight.py +++ b/src/distributed_execution/preflight.py @@ -12,9 +12,7 @@ from urllib.parse import urlparse # Runtime dependencies a tightly coupled execution pod must be able to resolve. TIGHTLY_COUPLED_ENV_VARS = ( - "DAGSTER_POSTGRES_HOST", - "DAGSTER_POSTGRES_USER", - "DAGSTER_POSTGRES_DB", + "DAGSTER_PG_PASSWORD", ) diff --git a/tests/test_loosely_coupled.py b/tests/test_loosely_coupled.py index 77cba0d..7885706 100644 --- a/tests/test_loosely_coupled.py +++ b/tests/test_loosely_coupled.py @@ -65,9 +65,7 @@ def test_each_unit_is_dispatched_to_its_own_external_worker(): def test_payload_reports_no_orchestration_credentials(monkeypatch): - monkeypatch.delenv("DAGSTER_POSTGRES_HOST", raising=False) - monkeypatch.delenv("DAGSTER_POSTGRES_USER", raising=False) - monkeypatch.delenv("DAGSTER_POSTGRES_DB", raising=False) + monkeypatch.delenv("DAGSTER_PG_PASSWORD", raising=False) result = loosely_coupled_subprocess_job.execute_in_process() mapped = result.output_for_node("dispatch_external_work_subprocess") diff --git a/tests/test_tightly_coupled.py b/tests/test_tightly_coupled.py index 2a4b921..5f80c1d 100644 --- a/tests/test_tightly_coupled.py +++ b/tests/test_tightly_coupled.py @@ -81,6 +81,16 @@ def test_env_var_check_reports_missing_names(): assert result["missing"] == ["DEFINITELY_NOT_SET_12345"] +def test_env_var_check_reports_present_names(monkeypatch): + monkeypatch.setenv("DAGSTER_PG_PASSWORD", "not-exposed-in-evidence") + + result = check_env_vars(("DAGSTER_PG_PASSWORD",)) + + assert result["passed"] is True + assert result["present"] == ["DAGSTER_PG_PASSWORD"] + assert "not-exposed-in-evidence" not in repr(result) + + def test_pod_identity_falls_back_outside_kubernetes(monkeypatch): monkeypatch.delenv("DAGSTER_K8S_PIPELINE_RUN_NAMESPACE", raising=False)