j.r/distributed-execution
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[SIMPL-30451] Close the distributed execution reference service

Browse Source 
Check the chart's actual DAGSTER_PG_PASSWORD injection in the tightly coupled preflight instead of DAGSTER_POSTGRES_*, which the Simpl chart never sets; the old check warned about a misconfiguration on a correctly configured run pod. Evidence reports the variable name only, never its value.

Record the platform k8s_job_executor run as cluster-verified in the guide and readiness checklist, add the Dagster UI screenshots covering job list, graph, code-configured tags and a successful run.

Changelog: fixed
This commit is contained in:
ILay
2026-09-02 18:47:27 +02:00
parent 4131288960
commit 9b1b7056b0
11 changed files with 155 additions and 56 deletions
Download Patch File Download Diff File Expand all files Collapse all files

78
documents/user-guide/readiness-checklist.md
View File

@@ -7,12 +7,11 @@ Run these checks **before the first execution** of a workflow, and again after a
change to the run launcher, the executor, the target namespace or the code
location image.
> **L4–L10 were cleared on sandbox-cat-dat on 2026-08-31.** Checks L1–L3 and L11,
> which exercise the payload contract, the message-parsing path and image-level
> isolation, are verified locally. The **tightly coupled** Kubernetes path has
> been assessed against the same cluster but not executed — C5, C6, T2, T3, T5
> and T8 are confirmed from the live instance configuration and the deployed code
> locations. See [section 2.1](#21-sandbox-state-2026-08-31).
> **Both Kubernetes paths are cluster-verified.** L4–L10 were cleared on
> sandbox-cat-dat on 2026-08-31. The tightly coupled `k8s_job_executor` path
> reached `RUN_SUCCESS` through the platform launcher on 2026-09-01. Checks
> L1–L3 and L11, which exercise the payload contract, message parsing and image
> isolation, are also verified locally. See [section 2.1](#21-sandbox-state-2026-08-31).
---
@@ -31,7 +30,7 @@ location image.
| # | Check | How to verify | Expected evidence |
|---|---|---|---|
| T1 | Run pod reaches the metadata database | Launch `tightly_coupled_in_process_job` | Run reaches `SUCCESS`; `report_execution_target` output metadata lists `DAGSTER_POSTGRES_HOST`, `DAGSTER_POSTGRES_USER` and `DAGSTER_POSTGRES_DB` under `env_vars_present`, and `env_vars_missing` is empty |
| T1 | Run pod reaches the metadata database | Launch `tightly_coupled_in_process_job` | Run reaches `SUCCESS`; `report_execution_target` metadata lists `DAGSTER_PG_PASSWORD` under `env_vars_present` without exposing its value, and `env_vars_missing` is empty. Successful step startup is the connectivity proof because the step loads run state from Postgres before executing the op |
| T2 | Vault injection works | Same run; inspect the run pod | `kubectl -n dagster describe pod <run-pod>` shows the `vault-env` init container completed; no `vault:` literal remains in the process environment |
| T3 | Object storage is reachable | Same run, if the workflow uses S3 | No `EndpointConnectionError` in run logs; `S3_ENDPOINT_URL` present in `env_vars_present` |
| T4 | Multiprocess fan-out actually fans out | Launch `tightly_coupled_local_job` | Run succeeds; `summarise_results` metadata shows one entry per unit in `contributing_workers` and a single entry in `contributing_hosts` — separate processes, same machine |
@@ -67,7 +66,7 @@ instance rather than the platform's Postgres.
`tightly_coupled_in_process_job` also succeeded, reporting a single worker.
### T1's env-var check is wrong for this platform
### T1's env-var check and platform contract
The run emitted:
@@ -75,19 +74,20 @@ The run emitted:
> `DAGSTER_POSTGRES_HOST`, `DAGSTER_POSTGRES_USER`, `DAGSTER_POSTGRES_DB`.
> Expected for a loosely coupled target; a misconfiguration for a tightly coupled one.
That warning would fire on a **correctly configured** platform run pod. The chart
does not set those variables: it injects `DAGSTER_PG_PASSWORD` from
`dagster-postgresql-secret` and bakes host, user and database into `dagster.yaml`.
So `TIGHTLY_COUPLED_ENV_VARS` in `preflight.py` names variables Simpl does not
use, and the check reports a misconfiguration that is not one. The reachability
probe above is the part that carries real evidence.
That warning was a false positive on a **correctly configured** platform run
pod. The chart does not set those variables: it injects `DAGSTER_PG_PASSWORD`
from `dagster-postgresql-secret` and stores host, user and database in
`dagster.yaml`. The reference check now tests for `DAGSTER_PG_PASSWORD` and
reports only its name, never its value. Database reachability is evidenced by
successful step startup: a tightly coupled step must load its run state from
the configured Dagster instance before `report_execution_target` can execute.
### T6 and T7, executed 2026-08-31
Running them needed three things the earlier attempt lacked: dagster pinned to
1.12.8 to match the control plane, so writing to the live `dataprovider01_dagster`
could not trigger a migration; `DAGSTER_HOME` moved onto the RWX
`dagster-shared-pvc`, because the default I/O manager writes step outputs under
could not trigger a migration; step outputs moved onto the RWX
`dagster-shared-pvc`, because the default I/O manager writes them under
`$DAGSTER_HOME/storage` and each step pod otherwise gets its own empty volume;
and `automount_service_account_token: true` in `step_k8s_config`, because
`dagster-svc-account` disables it by default. The probe is
@@ -125,6 +125,39 @@ A run submitted the normal way, through the webserver to `K8sRunLauncher`, gets
T6/T7 end to end, including `summarise_results`, needs the code location
registered on the platform Dagster.
### T6 and T7 closed end to end, 2026-09-01
With the code location registered on the platform Dagster,
`tightly_coupled_k8s_job` was launched from the webserver and reached
`RUN_SUCCESS` as run `1d8cb167-9fb8-4a34-a036-789eac381e13`. `K8sRunLauncher`
created `dagster-run-1d8cb167-…` in `dataprovider01`, `K8sStepHandler` created one
`dagster-step-…` Job per step, and every step — including `summarise_results` —
completed. No hand-rolled probe, no run-monitoring 403.
**The last thing in the way was where step outputs are written.** Two earlier
attempts failed on
`FileNotFoundError: /tmp/dagster/storage/<run-id>/generate_work_units/result/unit_0`,
with every step pod using its own container filesystem. Mounting
`dagster-shared-pvc` into the step pods was necessary but not sufficient: setting
`DAGSTER_HOME` through `step_k8s_config` does not take effect, because the Dagster
chart already injects `DAGSTER_HOME=/tmp/dagster` and the appended second entry
does not displace the first. The fix is to bypass `DAGSTER_HOME` and pin the I/O
manager instead — `fs_io_manager.configured({"base_dir": SHARED_IO_BASE_DIR})` on
the job, with `SHARED_IO_BASE_DIR` under the mounted PVC. The successful run logs
confirm it:
> Handled output "result" using IO manager "io_manager" —
> `/dagster/shared/distributed-execution/storage/1d8cb167-…/generate_work_units/result/unit_0`
The platform run exposed the false-positive `DAGSTER_POSTGRES_*` warning
described under T1 above. The reference now checks the chart's actual
`DAGSTER_PG_PASSWORD` injection instead.
> Note: `report_execution_target` reported `namespace: <not-in-kubernetes>` on a
> step pod that plainly was in Kubernetes. The check reads the service account
> namespace file, which is absent when the token is not automounted. It is a
> property of the probe, not of the run.
**C6 passes, contrary to what `job_namespace` alone suggests.** The launcher's
`job_namespace: dagster` is only a default. Each code location ships a
`DAGSTER_CONTAINER_CONTEXT` environment variable — generated by the chart — whose
@@ -141,10 +174,8 @@ Both deployed code locations carry it, so both would launch correctly:
| `template-code-location` | `dataprovider01` |
| `semaphore-ui` | `dataprovider01` |
What is still true is that **no run has ever executed on this cluster** —
`dataprovider01` holds no pods labelled `dagster/run-id` and the daemon log shows
no launch activity. That appears to be because nothing has been launched, not
because anything is broken.
The successful run confirms that the code-location container context places run
and step pods in `dataprovider01` with the required namespace-local dependencies.
The residual risk is narrower than a broken namespace, but real: a code location
that reaches the launcher *without* a container context namespace inherits
@@ -216,6 +247,7 @@ rather than at the omission.
| `env_vars_missing` is non-empty in `report_execution_target` metadata | Env vars are set on the code location deployment but not on the run pod | Add them under `runLauncher.config.k8sRunLauncher.runK8sConfig.containerConfig.env` — code location env is **not** inherited by run pods |
| A literal `vault:...` string appears as a value at runtime | Vault mutating webhook did not process the pod | Verify the `vault.security.banzaicloud.io/*` annotations are on the **run pod** template, not only the code location pod |
| Steps hang in `STARTING` with `k8s_job_executor` | Service account lacks Job create/watch permission | Apply `yaml/tightly-coupled/rbac-step-executor.yaml` and confirm with `kubectl auth can-i` |
| `FileNotFoundError` loading a step input from `/tmp/dagster/storage/...` | Step pods each use their own container filesystem for I/O manager output | Mount an RWX volume into step pods via `step_k8s_config` **and** pin `fs_io_manager`'s `base_dir` under it. Setting `DAGSTER_HOME` in `step_k8s_config` does not work — the chart's `DAGSTER_HOME=/tmp/dagster` is injected first and wins |
| `contributing_hosts` shows one host when `k8s_job_executor` is configured | Run tags or Launchpad config overrode the executor, or the image predates the change | Confirm the code location reloaded after the image bump; check the run's *Config* tab for an `execution:` override |
| Step pods `OOMKilled` under fan-out | Per-step memory limit applied per pod, aggregate exceeded quota | Raise `step_k8s_config` limits or lower step concurrency; the two multiply |
| Postgres refuses connections once fan-out grows | Each step pod is an independent DB client | Reduce step concurrency, raise the Postgres connection limit, or move the fan-out step to a loosely coupled target |
@@ -234,8 +266,10 @@ rather than at the omission.
| Run cancelled in the UI, payload pod keeps running | Cancellation is not propagated to dispatched workloads automatically | `delete_pod_on_completion` handles the normal path; for cancellation, verify orphaned Jobs and add a cleanup sensor |
> **Partly observed.** The pipes rows were exercised on sandbox-cat-dat on
> 2026-08-31. Rows describing the tightly coupled `k8s_job_executor` still follow
> from the dagster-k8s API rather than from observation.
> 2026-08-31. The tightly coupled `k8s_job_executor` rows were closed on
> 2026-09-01 by run `1d8cb167-9fb8-4a34-a036-789eac381e13`; the remaining rows in
> the table above still follow from the dagster-k8s API rather than from
> observation.
>
> One symptom the cluster run added, absent from the table above: a service
> account with `automountServiceAccountToken: false` — which the platform's