diff --git a/CHANGELOG.md b/CHANGELOG.md index 3985417..a34a2fd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,15 +4,33 @@ - Repository skeleton for the `distributed-execution` supporting data service. - User guide covering execution-target selection, decision support and readiness checks. +- Cluster-configuration input reference: mandatory versus optional inputs per setup, + a reusable minimum input set and worked examples in both setups' own formats. +- Requirement traceability mapping the guide's sections onto SIMPL-30451 and SIMPL-30787. - Runnable tightly coupled reference implementations (`in_process`, `multiprocess`, `k8s_job_executor`). - Runnable loosely coupled reference implementations via dagster-pipes (`PipesSubprocessClient`, `PipesK8sClient`) with a standalone external payload image. - Working example configuration and RBAC for both execution targets. +### verified + +- `tightly_coupled_k8s_job` executed end to end on sandbox-cat-dat through the + platform `K8sRunLauncher` (run `1d8cb167-9fb8-4a34-a036-789eac381e13`), closing + readiness rows T6 and T7. Step outputs are written to the shared + `dagster-shared-pvc` by pinning `fs_io_manager`'s `base_dir`; `DAGSTER_HOME` set + through `step_k8s_config` is ignored, because the Dagster chart injects + `DAGSTER_HOME=/tmp/dagster` ahead of it. +- `loosely_coupled_k8s_job` reached `RUN_SUCCESS` on sandbox-cat-dat through the + standalone probe (run `cff9b348-bfc3-4ac1-ab51-a94892b8e3a0`), with four + external payload pods reporting over the pod-log message channel. +- Tightly coupled preflight evidence now checks the chart's actual + `DAGSTER_PG_PASSWORD` injection without exposing its value. +- Dagster UI screenshots cover the registered jobs, graph, code-configured + execution-target tags and a successful reference run. +- The platform deployment diagram includes the service and both integration + patterns. + ### pending -- Publish the payload image to the container registry. -- End-to-end run of `loosely_coupled_k8s_job` against a cluster, and confirmation - of the Kubernetes rows in the readiness checklist. -- Screenshots of UI surfaces and integration-pattern diagrams. -- Platform architecture document update. +- Extend the project-level GitLab pipeline to publish the payload image as well + as the code-location image. diff --git a/README.md b/README.md index 3241130..6e64f87 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ actually run, and how does runtime information get back to the control plane?* | User guide | [documents/user-guide/distributed-execution-guide.md](documents/user-guide/distributed-execution-guide.md) | Complete | | Readiness checklist | [documents/user-guide/readiness-checklist.md](documents/user-guide/readiness-checklist.md) | Complete | | Tightly coupled reference | [src/distributed_execution/tightly_coupled/jobs.py](src/distributed_execution/tightly_coupled/jobs.py) | Runnable | -| Loosely coupled reference | [src/distributed_execution/loosely_coupled/jobs.py](src/distributed_execution/loosely_coupled/jobs.py) | Runnable (subprocess verified) | +| Loosely coupled reference | [src/distributed_execution/loosely_coupled/jobs.py](src/distributed_execution/loosely_coupled/jobs.py) | Runnable and cluster-verified | | External payload | [payload/work.py](payload/work.py) | Runnable | | Example configuration | [yaml/](yaml/) | Complete | @@ -35,7 +35,7 @@ distributed-execution/ │ ├── work.py │ ├── requirements.txt │ └── Dockerfile -├── documents/user-guide/ # AC1-AC4 documentation +├── documents/user-guide/ # Guide, checklist and requirement traceability ├── yaml/ # Working example configuration ├── tests/ ├── Dockerfile @@ -88,10 +88,10 @@ pipeline still has to learn. ## Status -Both execution targets are implemented. The tightly coupled jobs and the loosely -coupled **subprocess** transport are verified by the test suite. The loosely -coupled **Kubernetes** transport is implemented but has not yet been run against -a cluster; see the guide's *Outstanding work* section. +Both execution targets are implemented. Local variants are verified by the test +suite, `tightly_coupled_k8s_job` has run end to end through the platform launcher, +and `loosely_coupled_k8s_job` has run end to end through the standalone cluster +probe. See the guide and readiness checklist for the recorded evidence. ## Licence diff --git a/documents/user-guide/distributed-execution-guide.md b/documents/user-guide/distributed-execution-guide.md index e8f0411..b126b0e 100644 --- a/documents/user-guide/distributed-execution-guide.md +++ b/documents/user-guide/distributed-execution-guide.md @@ -5,14 +5,10 @@ integration pattern for a Dagster workflow on the Simpl orchestration platform. > **Document status.** Both setups are backed by runnable reference -> implementations in this repository. The tightly coupled setup and the loosely -> coupled **subprocess** transport are verified end to end, including runs -> launched from the Dagster UI — see [section 5.6](#56-what-a-verified-run-actually-produced). -> The loosely coupled **Kubernetes** transport was run on sandbox-cat-dat on -> 2026-08-31 and reached `RUN_SUCCESS`; see the readiness checklist for the -> evidence. What remains unproven on a cluster is the *tightly* coupled -> Kubernetes path — `k8s_job_executor` — whose rows stay marked -> **NOT CLUSTER-VERIFIED**. See [Outstanding work](#8-outstanding-work). +> implementations and verified end to end. The loosely coupled Kubernetes path +> reached `RUN_SUCCESS` on sandbox-cat-dat on 2026-08-31; the tightly coupled +> `k8s_job_executor` path reached `RUN_SUCCESS` through the platform launcher on +> 2026-09-01. See the readiness checklist for the recorded evidence. --- @@ -70,7 +66,7 @@ pattern. --- -## 2. AC1 — Choosing an execution target +## 2. Choosing an execution target and cluster setup ### 2.1 Choose tightly coupled when @@ -125,7 +121,7 @@ before it is run for the first time. Section 5.4 explains where each item lives. --- -## 3. AC2 — Decision support +## 3. Decision support and cluster-configuration inputs ### 3.1 Side-by-side comparison @@ -183,9 +179,134 @@ Use these as a fast triage. If several point the same way, that is your answer. succeed while reporting nothing. Two images to build, version and scan instead of one. +### 3.4 Cluster-configuration inputs + +Once the setup is chosen, the container cluster has to be described. The table +below is the full input surface, split into what a run cannot start without and +what only applies under a stated condition. "Mandatory" means the run fails, or +reports nothing, if the input is absent or wrong. + +| Input | Tightly coupled | Loosely coupled | Notes | +|---|---|---|---| +| Cluster / kube context hosting the run worker | Mandatory | Mandatory | The launcher's cluster. Loosely coupled may dispatch to a *second* cluster | +| Namespace for run pods | Mandatory | Mandatory | Supplied by the code location's `DAGSTER_CONTAINER_CONTEXT`; the launcher's `job_namespace` is only a fallback | +| Code location image reference and tag | Mandatory | Mandatory | Never `latest`; check C3 | +| Executor selection | Mandatory | Mandatory | Bounds step granularity; loosely coupled it bounds only dispatch concurrency | +| Service account for run pods | Mandatory | Mandatory | Must have `automountServiceAccountToken: true` | +| Run-pod egress to the metadata database | **Mandatory** | Not applicable | The pod that writes run events | +| Namespace for step pods | Conditional — `k8s_job_executor` only | Not applicable | Defaults to the run pod's namespace | +| RBAC to create and watch Jobs | Conditional — `k8s_job_executor` only | **Mandatory**, in the payload namespace | Same verbs, different namespace | +| Shared RWX volume plus an I/O manager `base_dir` under it | Conditional — `k8s_job_executor` only | Optional | Without it step pods cannot read each other's outputs | +| Per-step resource requests and limits | Optional | Optional | Multiply by step or dispatch concurrency | +| Payload image reference and tag | Not applicable | **Mandatory** | Must share the code location's commit tag | +| Namespace for payload workloads | Not applicable | **Mandatory** | May be in another cluster or trust boundary | +| Message channel and its reader | Not applicable | **Mandatory** | Pod log stream by default; RBAC on `pods/log` *is* the channel | +| Payload wait timeout | Not applicable | Optional | Defaults to 24 h — lower it so scheduling failures surface | +| Vault role and secret paths | Conditional — if the workflow reads secrets | Conditional — launching pod only | Payloads should receive only `extras` and explicit `env` | +| Object-storage endpoint and credentials | Conditional — if the workflow uses S3 | Conditional — if the *payload* uses S3 | | +| Image pull secret | Conditional — non-anonymous registry | Conditional — non-anonymous registry | Applies to the SA that runs the pod, which for dispatched payload Jobs is `default` | + +### 3.5 Minimum input set + +The list below is the reusable baseline: the smallest set of values a team has to +agree on before a workflow can be configured against a cluster. Record it once +per workflow, in the workflow's own repository, and carry it into whatever +configuration format the team already uses. + +**Both setups** + +1. `cluster` — the cluster the run worker lands in. +2. `run_namespace` — namespace for run pods. +3. `code_location_image` — image reference including an immutable tag. +4. `executor` — executor name plus its concurrency bound. +5. `service_account` — the identity run pods assume. + +**Tightly coupled adds** + +6. `metadata_db` — host and port the run and step pods must reach. +7. `step_namespace` — only when `k8s_job_executor` is used. +8. `shared_output_path` — RWX mount path, and the I/O manager `base_dir` pinned + under it, again only for `k8s_job_executor`. + +**Loosely coupled adds** + +6. `payload_image` — image reference sharing the code location's commit tag. +7. `payload_namespace` — where dispatched workloads are created; may be in + another cluster. +8. `message_channel` — the reader in use, and the RBAC verb that carries it. + +This guide deliberately does **not** prescribe a configuration schema or key +names. The names above are descriptive labels for the values, not a required +format; the examples in section 3.6 express the same baseline in two different +shapes, and both are correct. + +### 3.6 Worked input examples + +The reference configuration in [yaml/](../../yaml) shows the baseline expressed +in the formats each setup already uses. Neither is normative. + +**Tightly coupled** — the inputs land in Helm values and in the job's executor +config, because that is where the platform chart and Dagster expect them: + +```yaml +# yaml/tightly-coupled/values-run-launcher.yaml (shape, not a fixed schema) +dagster: + runLauncher: + type: K8sRunLauncher + config: + k8sRunLauncher: + jobNamespace: dagster # run_namespace + failPodOnRunFailure: true + runK8sConfig: + containerConfig: + env: # metadata_db, S3 and Vault reachability + - name: S3_ENDPOINT_URL + value: https://s3.dev.simpl-europe.eu +``` + +```python +# step_namespace, shared_output_path and the executor travel with the job +tightly_coupled_k8s_job = distributed_execution_reference.to_job( + executor_def=k8s_job_executor, + resource_defs={"io_manager": fs_io_manager.configured({"base_dir": SHARED_IO_BASE_DIR})}, + config={"execution": {"config": {"step_k8s_config": STEP_K8S_CONFIG}}}, +) +``` + +**Loosely coupled** — the same baseline, but the cluster-facing half is the +payload image and its namespace. Here they are environment values set on both the +code location and the run launcher, because the dispatching pod is a run pod: + +```yaml +# yaml/loosely-coupled/values-pipes-payload.yaml (shape, not a fixed schema) +dagster: + dagster-user-deployments: + deployments: + - name: distributed-execution + env: + - name: PIPES_PAYLOAD_IMAGE # payload_image, tag-locked + value: /distributed-execution/payload:0.0.0 + - name: PIPES_PAYLOAD_NAMESPACE # payload_namespace + value: dagster +``` + +```python +PAYLOAD_IMAGE = os.environ.get("PIPES_PAYLOAD_IMAGE", ...) +PAYLOAD_NAMESPACE = os.environ.get("PIPES_PAYLOAD_NAMESPACE", "dagster") +``` + +The `message_channel` input has no key in either file. The default reader is the +pod log stream, so it is configured by granting `pods/log` in +[yaml/loosely-coupled/rbac-pipes-dispatch.yaml](../../yaml/loosely-coupled/rbac-pipes-dispatch.yaml). +An input can be mandatory and still not be a configuration field. + +Note what is absent from the loosely coupled example: no metadata database, no +Vault role, no object-storage credentials. Those inputs are not optional there — +they do not exist, which is the operational consequence of the choice. + --- -## 4. AC3 — Readiness checks +## 4. Readiness checks Pre-run validation items, their expected evidence, and the symptoms and corrections for common misconfigurations are maintained separately, in @@ -193,7 +314,7 @@ corrections for common misconfigurations are maintained separately, in --- -## 5. AC4 — Configuring execution-target linkage in code +## 5. Configuring execution-target linkage in code ### 5.1 Configuration constructs @@ -233,12 +354,18 @@ tightly_coupled_local_job = distributed_execution_reference.to_job( ```python tightly_coupled_k8s_job = distributed_execution_reference.to_job( name="tightly_coupled_k8s_job", - executor_def=k8s_job_executor.configured( - { - "image_pull_policy": "IfNotPresent", - "step_k8s_config": STEP_K8S_CONFIG, + executor_def=k8s_job_executor, + resource_defs={ + "io_manager": fs_io_manager.configured({"base_dir": SHARED_IO_BASE_DIR}) + }, + config={ + "execution": { + "config": { + "image_pull_policy": "IfNotPresent", + "step_k8s_config": STEP_K8S_CONFIG, + } } - ), + }, tags={**COMMON_TAGS, "executor": "k8s_job"}, ) ``` @@ -253,6 +380,22 @@ operationally: [yaml/tightly-coupled/rbac-step-executor.yaml](../../yaml/tightly-coupled/rbac-step-executor.yaml). - Per-step resource requests apply per pod, so the aggregate request for a fan-out step is the per-step request multiplied by concurrency. +- **Step outputs must land on storage every step pod can read.** With the default + filesystem I/O manager each pod writes to its own container filesystem, so a + downstream step opens a path that does not exist there. `STEP_K8S_CONFIG` mounts + the RWX `dagster-shared-pvc` at `/dagster/shared`, and the job pins + `fs_io_manager`'s `base_dir` underneath it. +- Pin `base_dir` directly rather than relying on `DAGSTER_HOME`. The default + `base_dir` is `$DAGSTER_HOME/storage`, but the Dagster chart already injects + `DAGSTER_HOME=/tmp/dagster` into the step container; a second entry appended by + `step_k8s_config` does not displace it, so the override is silently ignored and + outputs keep going to `/tmp/dagster/storage`. + +The executor's defaults go through `config=` rather than +`k8s_job_executor.configured(...)`. `.configured()` collapses the executor's +config schema to `Any`, which makes a run-config `execution:` block accepted and +then discarded, and makes `job_image` and `job_namespace` impossible to supply at +launch time. The evidence that the switch took effect is in `summarise_results` output metadata, and the two fields say different things: @@ -485,6 +628,32 @@ on a cluster it is what distinguishes `k8s_job_executor` from holds the namespace the run pod landed in, which is the placement half of the execution target. +### 5.7 Dagster UI reference + +The screenshots below were captured from the local reference service on +2026-09-02. They show the surfaces participants use to inspect job linkage and +run evidence. Cluster placement is evidenced separately by the sandbox run IDs +in the readiness checklist. + +All five reference jobs are registered under one code location: + +![Distributed execution job list](images/distributed-execution-job-list.png) + +The job overview exposes the graph whose executor or dispatch op defines the +integration pattern: + +![Tightly coupled reference graph](images/distributed-execution-job-graph.png) + +The Launchpad shows that `execution_target` and `executor` are job tags persisted +by code configuration, not a separate runtime-only selector: + +![Execution-target tags in Launchpad](images/distributed-execution-launchpad.png) + +A successful local run displays the step events and final status used alongside +output metadata as retained evidence: + +![Successful tightly coupled reference run](images/distributed-execution-successful-run.png) + --- ## 6. Upstream documentation @@ -506,24 +675,59 @@ execution target. ## 8. Outstanding work -Tracked under SIMPL-30451. +Tracked under SIMPL-30451 and SIMPL-30787. -| Item | AC | Status | +| Item | Status | +|---|---| +| Execution-target choices, prerequisites, workflow-level documentation | Complete | +| Comparison, environment-fit indicators, trade-offs | Complete | +| Cluster-configuration inputs, mandatory vs optional, minimum input set | Complete | +| Readiness checklist and evidence mapping | Complete; both Kubernetes paths cluster-verified | +| Misconfiguration symptoms and corrections | Complete; corrected with observed cluster failure modes | +| Configuration constructs and persistence | Complete | +| Before-and-after example, tightly coupled | Complete | +| Before-and-after example, switching to loosely coupled | Complete | +| Runnable tightly coupled reference implementations | Complete | +| Runnable loosely coupled reference implementation | Complete (subprocess and Kubernetes transports both verified) | +| Both images build; payload image passes the isolation check (L11) | Complete | +| Locally runnable jobs launched from the Dagster UI, evidence recorded in section 5.6 | Complete | +| Payload image published to a container registry | Complete on the sandbox Gitea registry; the GitLab registry still pending | +| GitLab pipeline builds both images | **Pending** — the shared `ds.gitlab-ci.yml` template builds one image from the root Dockerfile | +| End-to-end run of `loosely_coupled_k8s_job` on a cluster | Complete — sandbox-cat-dat, 2026-08-31, run `cff9b348…`; see the readiness checklist | +| `distributed-execution` registered as a code location on a platform Dagster | Complete — tightly coupled run `1d8cb167…` launched through the platform webserver | +| Screenshots of UI surfaces | Complete — job list, graph, Launchpad linkage and successful run captured in section 5.7 | +| Platform architecture document update | Complete — root `deployment_diagram.md` includes the service and both runtime-information paths | + +--- + +## 9. Requirement traceability + +The sections above are organised by subject, not by ticket. Two stories share +this material; the mapping below is authoritative for both. + +### SIMPL-30787 — defining the execution container cluster + +| AC | Requirement | Where it is met | |---|---|---| -| Execution-target choices, prerequisites, workflow-level documentation | AC1 | Complete | -| Comparison, environment-fit indicators, trade-offs | AC2 | Complete | -| Readiness checklist and evidence mapping | AC3 | Complete; tightly coupled K8s executor rows not cluster-verified | -| Misconfiguration symptoms and corrections | AC3 | Complete; tightly coupled K8s symptoms not cluster-verified | -| Configuration constructs and persistence | AC4 | Complete | -| Before-and-after example, tightly coupled | AC4 | Complete | -| Before-and-after example, switching to loosely coupled | AC4 | Complete | -| Runnable tightly coupled reference implementations | AC4 | Complete | -| Runnable loosely coupled reference implementation | AC4 | Complete (subprocess and Kubernetes transports both verified) | -| Both images build; payload image passes the isolation check (L11) | Tech details | Complete | -| Locally runnable jobs launched from the Dagster UI, evidence recorded in section 5.6 | Tech details | Complete | -| Payload image published to a container registry | Tech details | Complete on the sandbox Gitea registry; the GitLab registry still pending | -| GitLab pipeline builds both images | Tech details | **Pending** — the shared `ds.gitlab-ci.yml` template builds one image from the root Dockerfile | -| End-to-end run of `loosely_coupled_k8s_job` on a cluster | Tech details | Complete — sandbox-cat-dat, 2026-08-31, run `cff9b348…`; see the readiness checklist | -| `distributed-execution` registered as a code location on a platform Dagster | Tech details | **Pending** — blocked on the 1.13.19 vs 1.12.8 control plane skew | -| Screenshots of UI surfaces | Tech details | **Pending** — needs a deployed platform instance, not a local dev server | -| Platform architecture document update | Tech details | **Pending** | +| AC1 | Prerequisites for both setups | Section 2.3 | +| AC1 | What must be documented before execution | Section 2.4 | +| AC1 | Where cluster-runtime connectivity is required, where decoupled reporting is expected | Sections 1.2 and 2.3; the asymmetry is drawn in the section 1.2 diagram | +| AC2 | Mandatory vs optional inputs per setup | Section 3.4 | +| AC2 | Examples for both setups, no fixed config naming model | Section 3.6, with the closing note in 3.5 | +| AC2 | Reusable minimum input set | Section 3.5 | +| AC3 | Checklist separates tightly from loosely coupled checks | [readiness-checklist.md](readiness-checklist.md) sections 2 and 3; section 1 holds the checks common to both | +| AC3 | Each item mapped to expected evidence | The *Expected evidence* column of every checklist table | +| AC4 | Configuration constructs attaching the pattern to a workflow | Section 5.1 | +| AC4 | Worked before-and-after example across setups | Sections 5.2 and 5.3 | +| AC4 | At least one runnable reference implementation per setup | Section 5.5 — three tightly coupled, two loosely coupled | +| AC4 | Selection persisted in configuration artifacts, not runtime-only | Section 5.4 | +| Tech | Diagrams, screenshots, working config files, upstream links | Sections 1.2, 5.7, [yaml/](../../yaml), section 6 | + +### SIMPL-30451 — execution targets and integration patterns + +| AC | Subject | Where it is met | +|---|---|---| +| AC1 | Choosing an execution target | Section 2 | +| AC2 | Decision support | Sections 3.1 to 3.3 | +| AC3 | Readiness checks | Section 4 and the readiness checklist | +| AC4 | Linkage configured in code | Section 5 | diff --git a/documents/user-guide/images/distributed-execution-job-graph.png b/documents/user-guide/images/distributed-execution-job-graph.png new file mode 100644 index 0000000..79f8890 Binary files /dev/null and b/documents/user-guide/images/distributed-execution-job-graph.png differ diff --git a/documents/user-guide/images/distributed-execution-job-list.png b/documents/user-guide/images/distributed-execution-job-list.png new file mode 100644 index 0000000..57b5371 Binary files /dev/null and b/documents/user-guide/images/distributed-execution-job-list.png differ diff --git a/documents/user-guide/images/distributed-execution-launchpad.png b/documents/user-guide/images/distributed-execution-launchpad.png new file mode 100644 index 0000000..e9a84f0 Binary files /dev/null and b/documents/user-guide/images/distributed-execution-launchpad.png differ diff --git a/documents/user-guide/images/distributed-execution-successful-run.png b/documents/user-guide/images/distributed-execution-successful-run.png new file mode 100644 index 0000000..d693dbe Binary files /dev/null and b/documents/user-guide/images/distributed-execution-successful-run.png differ diff --git a/documents/user-guide/readiness-checklist.md b/documents/user-guide/readiness-checklist.md index 832b7e7..b95d4f1 100644 --- a/documents/user-guide/readiness-checklist.md +++ b/documents/user-guide/readiness-checklist.md @@ -1,18 +1,20 @@ # Distributed Execution: Readiness Checklist Pre-run validation for a workflow's execution target. Companion to the -[user guide](distributed-execution-guide.md); this document covers AC3. +[user guide](distributed-execution-guide.md); this document is the validation +checklist required by SIMPL-30451 AC3 and SIMPL-30787 AC3. Section 1 holds the +checks common to both setups, section 2 the tightly coupled checks and section 3 +the loosely coupled ones; every item states the evidence that proves it. Run these checks **before the first execution** of a workflow, and again after any change to the run launcher, the executor, the target namespace or the code location image. -> **L4–L10 were cleared on sandbox-cat-dat on 2026-08-31.** Checks L1–L3 and L11, -> which exercise the payload contract, the message-parsing path and image-level -> isolation, are verified locally. The **tightly coupled** Kubernetes path has -> been assessed against the same cluster but not executed — C5, C6, T2, T3, T5 -> and T8 are confirmed from the live instance configuration and the deployed code -> locations. See [section 2.1](#21-sandbox-state-2026-08-31). +> **Both Kubernetes paths are cluster-verified.** L4–L10 were cleared on +> sandbox-cat-dat on 2026-08-31. The tightly coupled `k8s_job_executor` path +> reached `RUN_SUCCESS` through the platform launcher on 2026-09-01. Checks +> L1–L3 and L11, which exercise the payload contract, message parsing and image +> isolation, are also verified locally. See [section 2.1](#21-sandbox-state-2026-08-31). --- @@ -31,7 +33,7 @@ location image. | # | Check | How to verify | Expected evidence | |---|---|---|---| -| T1 | Run pod reaches the metadata database | Launch `tightly_coupled_in_process_job` | Run reaches `SUCCESS`; `report_execution_target` output metadata lists `DAGSTER_POSTGRES_HOST`, `DAGSTER_POSTGRES_USER` and `DAGSTER_POSTGRES_DB` under `env_vars_present`, and `env_vars_missing` is empty | +| T1 | Run pod reaches the metadata database | Launch `tightly_coupled_in_process_job` | Run reaches `SUCCESS`; `report_execution_target` metadata lists `DAGSTER_PG_PASSWORD` under `env_vars_present` without exposing its value, and `env_vars_missing` is empty. Successful step startup is the connectivity proof because the step loads run state from Postgres before executing the op | | T2 | Vault injection works | Same run; inspect the run pod | `kubectl -n dagster describe pod ` shows the `vault-env` init container completed; no `vault:` literal remains in the process environment | | T3 | Object storage is reachable | Same run, if the workflow uses S3 | No `EndpointConnectionError` in run logs; `S3_ENDPOINT_URL` present in `env_vars_present` | | T4 | Multiprocess fan-out actually fans out | Launch `tightly_coupled_local_job` | Run succeeds; `summarise_results` metadata shows one entry per unit in `contributing_workers` and a single entry in `contributing_hosts` — separate processes, same machine | @@ -67,7 +69,7 @@ instance rather than the platform's Postgres. `tightly_coupled_in_process_job` also succeeded, reporting a single worker. -### T1's env-var check is wrong for this platform +### T1's env-var check and platform contract The run emitted: @@ -75,19 +77,20 @@ The run emitted: > `DAGSTER_POSTGRES_HOST`, `DAGSTER_POSTGRES_USER`, `DAGSTER_POSTGRES_DB`. > Expected for a loosely coupled target; a misconfiguration for a tightly coupled one. -That warning would fire on a **correctly configured** platform run pod. The chart -does not set those variables: it injects `DAGSTER_PG_PASSWORD` from -`dagster-postgresql-secret` and bakes host, user and database into `dagster.yaml`. -So `TIGHTLY_COUPLED_ENV_VARS` in `preflight.py` names variables Simpl does not -use, and the check reports a misconfiguration that is not one. The reachability -probe above is the part that carries real evidence. +That warning was a false positive on a **correctly configured** platform run +pod. The chart does not set those variables: it injects `DAGSTER_PG_PASSWORD` +from `dagster-postgresql-secret` and stores host, user and database in +`dagster.yaml`. The reference check now tests for `DAGSTER_PG_PASSWORD` and +reports only its name, never its value. Database reachability is evidenced by +successful step startup: a tightly coupled step must load its run state from +the configured Dagster instance before `report_execution_target` can execute. ### T6 and T7, executed 2026-08-31 Running them needed three things the earlier attempt lacked: dagster pinned to 1.12.8 to match the control plane, so writing to the live `dataprovider01_dagster` -could not trigger a migration; `DAGSTER_HOME` moved onto the RWX -`dagster-shared-pvc`, because the default I/O manager writes step outputs under +could not trigger a migration; step outputs moved onto the RWX +`dagster-shared-pvc`, because the default I/O manager writes them under `$DAGSTER_HOME/storage` and each step pod otherwise gets its own empty volume; and `automount_service_account_token: true` in `step_k8s_config`, because `dagster-svc-account` disables it by default. The probe is @@ -125,6 +128,39 @@ A run submitted the normal way, through the webserver to `K8sRunLauncher`, gets T6/T7 end to end, including `summarise_results`, needs the code location registered on the platform Dagster. +### T6 and T7 closed end to end, 2026-09-01 + +With the code location registered on the platform Dagster, +`tightly_coupled_k8s_job` was launched from the webserver and reached +`RUN_SUCCESS` as run `1d8cb167-9fb8-4a34-a036-789eac381e13`. `K8sRunLauncher` +created `dagster-run-1d8cb167-…` in `dataprovider01`, `K8sStepHandler` created one +`dagster-step-…` Job per step, and every step — including `summarise_results` — +completed. No hand-rolled probe, no run-monitoring 403. + +**The last thing in the way was where step outputs are written.** Two earlier +attempts failed on +`FileNotFoundError: /tmp/dagster/storage//generate_work_units/result/unit_0`, +with every step pod using its own container filesystem. Mounting +`dagster-shared-pvc` into the step pods was necessary but not sufficient: setting +`DAGSTER_HOME` through `step_k8s_config` does not take effect, because the Dagster +chart already injects `DAGSTER_HOME=/tmp/dagster` and the appended second entry +does not displace the first. The fix is to bypass `DAGSTER_HOME` and pin the I/O +manager instead — `fs_io_manager.configured({"base_dir": SHARED_IO_BASE_DIR})` on +the job, with `SHARED_IO_BASE_DIR` under the mounted PVC. The successful run logs +confirm it: + +> Handled output "result" using IO manager "io_manager" — +> `/dagster/shared/distributed-execution/storage/1d8cb167-…/generate_work_units/result/unit_0` + +The platform run exposed the false-positive `DAGSTER_POSTGRES_*` warning +described under T1 above. The reference now checks the chart's actual +`DAGSTER_PG_PASSWORD` injection instead. + +> Note: `report_execution_target` reported `namespace: ` on a +> step pod that plainly was in Kubernetes. The check reads the service account +> namespace file, which is absent when the token is not automounted. It is a +> property of the probe, not of the run. + **C6 passes, contrary to what `job_namespace` alone suggests.** The launcher's `job_namespace: dagster` is only a default. Each code location ships a `DAGSTER_CONTAINER_CONTEXT` environment variable — generated by the chart — whose @@ -141,10 +177,8 @@ Both deployed code locations carry it, so both would launch correctly: | `template-code-location` | `dataprovider01` | | `semaphore-ui` | `dataprovider01` | -What is still true is that **no run has ever executed on this cluster** — -`dataprovider01` holds no pods labelled `dagster/run-id` and the daemon log shows -no launch activity. That appears to be because nothing has been launched, not -because anything is broken. +The successful run confirms that the code-location container context places run +and step pods in `dataprovider01` with the required namespace-local dependencies. The residual risk is narrower than a broken namespace, but real: a code location that reaches the launcher *without* a container context namespace inherits @@ -216,6 +250,7 @@ rather than at the omission. | `env_vars_missing` is non-empty in `report_execution_target` metadata | Env vars are set on the code location deployment but not on the run pod | Add them under `runLauncher.config.k8sRunLauncher.runK8sConfig.containerConfig.env` — code location env is **not** inherited by run pods | | A literal `vault:...` string appears as a value at runtime | Vault mutating webhook did not process the pod | Verify the `vault.security.banzaicloud.io/*` annotations are on the **run pod** template, not only the code location pod | | Steps hang in `STARTING` with `k8s_job_executor` | Service account lacks Job create/watch permission | Apply `yaml/tightly-coupled/rbac-step-executor.yaml` and confirm with `kubectl auth can-i` | +| `FileNotFoundError` loading a step input from `/tmp/dagster/storage/...` | Step pods each use their own container filesystem for I/O manager output | Mount an RWX volume into step pods via `step_k8s_config` **and** pin `fs_io_manager`'s `base_dir` under it. Setting `DAGSTER_HOME` in `step_k8s_config` does not work — the chart's `DAGSTER_HOME=/tmp/dagster` is injected first and wins | | `contributing_hosts` shows one host when `k8s_job_executor` is configured | Run tags or Launchpad config overrode the executor, or the image predates the change | Confirm the code location reloaded after the image bump; check the run's *Config* tab for an `execution:` override | | Step pods `OOMKilled` under fan-out | Per-step memory limit applied per pod, aggregate exceeded quota | Raise `step_k8s_config` limits or lower step concurrency; the two multiply | | Postgres refuses connections once fan-out grows | Each step pod is an independent DB client | Reduce step concurrency, raise the Postgres connection limit, or move the fan-out step to a loosely coupled target | @@ -234,8 +269,10 @@ rather than at the omission. | Run cancelled in the UI, payload pod keeps running | Cancellation is not propagated to dispatched workloads automatically | `delete_pod_on_completion` handles the normal path; for cancellation, verify orphaned Jobs and add a cleanup sensor | > **Partly observed.** The pipes rows were exercised on sandbox-cat-dat on -> 2026-08-31. Rows describing the tightly coupled `k8s_job_executor` still follow -> from the dagster-k8s API rather than from observation. +> 2026-08-31. The tightly coupled `k8s_job_executor` rows were closed on +> 2026-09-01 by run `1d8cb167-9fb8-4a34-a036-789eac381e13`; the remaining rows in +> the table above still follow from the dagster-k8s API rather than from +> observation. > > One symptom the cluster run added, absent from the table above: a service > account with `automountServiceAccountToken: false` — which the platform's diff --git a/src/distributed_execution/preflight.py b/src/distributed_execution/preflight.py index 295880c..9920a42 100644 --- a/src/distributed_execution/preflight.py +++ b/src/distributed_execution/preflight.py @@ -12,9 +12,7 @@ from urllib.parse import urlparse # Runtime dependencies a tightly coupled execution pod must be able to resolve. TIGHTLY_COUPLED_ENV_VARS = ( - "DAGSTER_POSTGRES_HOST", - "DAGSTER_POSTGRES_USER", - "DAGSTER_POSTGRES_DB", + "DAGSTER_PG_PASSWORD", ) diff --git a/tests/test_loosely_coupled.py b/tests/test_loosely_coupled.py index 77cba0d..7885706 100644 --- a/tests/test_loosely_coupled.py +++ b/tests/test_loosely_coupled.py @@ -65,9 +65,7 @@ def test_each_unit_is_dispatched_to_its_own_external_worker(): def test_payload_reports_no_orchestration_credentials(monkeypatch): - monkeypatch.delenv("DAGSTER_POSTGRES_HOST", raising=False) - monkeypatch.delenv("DAGSTER_POSTGRES_USER", raising=False) - monkeypatch.delenv("DAGSTER_POSTGRES_DB", raising=False) + monkeypatch.delenv("DAGSTER_PG_PASSWORD", raising=False) result = loosely_coupled_subprocess_job.execute_in_process() mapped = result.output_for_node("dispatch_external_work_subprocess") diff --git a/tests/test_tightly_coupled.py b/tests/test_tightly_coupled.py index 2a4b921..5f80c1d 100644 --- a/tests/test_tightly_coupled.py +++ b/tests/test_tightly_coupled.py @@ -81,6 +81,16 @@ def test_env_var_check_reports_missing_names(): assert result["missing"] == ["DEFINITELY_NOT_SET_12345"] +def test_env_var_check_reports_present_names(monkeypatch): + monkeypatch.setenv("DAGSTER_PG_PASSWORD", "not-exposed-in-evidence") + + result = check_env_vars(("DAGSTER_PG_PASSWORD",)) + + assert result["passed"] is True + assert result["present"] == ["DAGSTER_PG_PASSWORD"] + assert "not-exposed-in-evidence" not in repr(result) + + def test_pod_identity_falls_back_outside_kubernetes(monkeypatch): monkeypatch.delenv("DAGSTER_K8S_PIPELINE_RUN_NAMESPACE", raising=False) diff --git a/yaml/tightly-coupled/values-run-launcher.yaml b/yaml/tightly-coupled/values-run-launcher.yaml index fc230d7..62e7770 100644 --- a/yaml/tightly-coupled/values-run-launcher.yaml +++ b/yaml/tightly-coupled/values-run-launcher.yaml @@ -13,8 +13,8 @@ dagster: config: k8sRunLauncher: # Namespace the run pods land in. Must be a namespace whose NetworkPolicy - # permits egress to Postgres, object storage and Vault - see AC1 - # prerequisites in the user guide. + # permits egress to Postgres, object storage and Vault - see the + # prerequisites in section 2.3 of the user guide. jobNamespace: dagster # Surfaces step failures as pod failures so kubectl and Dagster agree.