j.r/distributed-execution
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[SIMPL-30451] Verify the Kubernetes pipes transport on the sandbox

Browse Source 
Run cff9b348-bfc3-4ac1-ab51-a94892b8e3a0 of loosely_coupled_k8s_job reached RUN_SUCCESS in dataprovider01 on sandbox-cat-dat: four payload Jobs, four distinct payload pod hostnames in contributing_hosts, and the payload log lines in the run log. Checks L4-L10 are now observed rather than derived. Torn down afterwards; the namespace was left as it was found.

Corrects two claims that were wrong. The registry does NOT require a pull secret - a bare GET returns 401, but that is the opening move of the Docker token handshake, and completing it anonymously returns the manifest for both images. And PIPES_PAYLOAD_NAMESPACE was set to a namespace named dagster, which does not exist on that cluster; the release is called dagster but runs in dataprovider01.

Adds the sandbox probe variant. The generic probe cannot run there: sandbox access is Rancher project-scoped, so a new namespace grants its creator nothing and Role creation is denied everywhere. It is not needed either, since dagster-role already carries the exact pipes permissions.

Changelog: added
This commit is contained in:
ILay
2026-08-31 19:41:26 +02:00
parent e686df5710
commit 96caa3f81b
5 changed files with 204 additions and 76 deletions
Download Patch File Download Diff File Expand all files Collapse all files

26
documents/user-guide/distributed-execution-guide.md
View File

@@ -8,8 +8,10 @@ integration pattern for a Dagster workflow on the Simpl orchestration platform.
> implementations in this repository. The tightly coupled setup and the loosely
> coupled **subprocess** transport are verified end to end, including runs
> launched from the Dagster UI — see [section 5.6](#56-what-a-verified-run-actually-produced).
> The loosely coupled **Kubernetes** transport is implemented and reviewed but
> has not yet been run against a cluster; statements specific to it are marked
> The loosely coupled **Kubernetes** transport was run on sandbox-cat-dat on
> 2026-08-31 and reached `RUN_SUCCESS`; see the readiness checklist for the
> evidence. What remains unproven on a cluster is the *tightly* coupled
> Kubernetes path — `k8s_job_executor` — whose rows stay marked
> **NOT CLUSTER-VERIFIED**. See [Outstanding work](#8-outstanding-work).
---
@@ -378,10 +380,12 @@ empty result. If your cluster's logging setup makes that transport unreliable,
switch to an object-storage message reader, which needs a bucket plus
credentials on both sides.
> **NOT CLUSTER-VERIFIED.** The Kubernetes transport above is implemented and
> its RBAC and configuration are recorded, but it has not yet been run against a
> Simpl cluster. The subprocess transport, which exercises the identical payload
> and the identical message-parsing path, is verified by the test suite.
> **CLUSTER-VERIFIED 2026-08-31.** Run `cff9b348…` of `loosely_coupled_k8s_job`
> reached `RUN_SUCCESS` on sandbox-cat-dat. Four payload Jobs were created, one
> per work unit, and `contributing_hosts` held four distinct payload pod names —
> so the work demonstrably ran off the orchestrator. The payload's log lines
> reached the run log, which is the pod log stream serving as the message
> channel exactly as described above.
### 5.4 Where setup selection is persisted
@@ -398,6 +402,7 @@ setting. There is no UI control that switches a workflow between the two pattern
| `yaml/sandbox/values-sandbox-gitea.yaml` | Sandbox registry coordinates, pull secret, and the SHA both images share |
| `yaml/loosely-coupled/rbac-pipes-dispatch.yaml` | Permissions required by the dispatching pod |
| `yaml/loosely-coupled/probe-pipes-k8s.yaml` | Standalone cluster probe for the pipes transport (checklist L4–L9) |
| `yaml/sandbox/probe-pipes-k8s-sandbox.yaml` | The same probe for sandbox-cat-dat, where project-scoped access forbids creating namespaces and RBAC |
| `yaml/values-dagster-distributed-execution.yaml` | Code location image and entry point |
The Launchpad can override *run configuration* — op config, resource config, tags
@@ -507,17 +512,18 @@ Tracked under SIMPL-30451.
|---|---|---|
| Execution-target choices, prerequisites, workflow-level documentation | AC1 | Complete |
| Comparison, environment-fit indicators, trade-offs | AC2 | Complete |
| Readiness checklist and evidence mapping | AC3 | Complete; K8s pipes checks not cluster-verified |
| Misconfiguration symptoms and corrections | AC3 | Complete; K8s pipes symptoms not cluster-verified |
| Readiness checklist and evidence mapping | AC3 | Complete; tightly coupled K8s executor rows not cluster-verified |
| Misconfiguration symptoms and corrections | AC3 | Complete; tightly coupled K8s symptoms not cluster-verified |
| Configuration constructs and persistence | AC4 | Complete |
| Before-and-after example, tightly coupled | AC4 | Complete |
| Before-and-after example, switching to loosely coupled | AC4 | Complete |
| Runnable tightly coupled reference implementations | AC4 | Complete |
| Runnable loosely coupled reference implementation | AC4 | Complete (subprocess verified, K8s not cluster-run) |
| Runnable loosely coupled reference implementation | AC4 | Complete (subprocess and Kubernetes transports both verified) |
| Both images build; payload image passes the isolation check (L11) | Tech details | Complete |
| Locally runnable jobs launched from the Dagster UI, evidence recorded in section 5.6 | Tech details | Complete |
| Payload image published to a container registry | Tech details | Complete on the sandbox Gitea registry; the GitLab registry still pending |
| GitLab pipeline builds both images | Tech details | **Pending** — the shared `ds.gitlab-ci.yml` template builds one image from the root Dockerfile |
| End-to-end run of `loosely_coupled_k8s_job` on a cluster | Tech details | **Pending** |
| End-to-end run of `loosely_coupled_k8s_job` on a cluster | Tech details | Complete — sandbox-cat-dat, 2026-08-31, run `cff9b348…`; see the readiness checklist |
| `distributed-execution` registered as a code location on a platform Dagster | Tech details | **Pending** — blocked on the 1.13.19 vs 1.12.8 control plane skew |
| Screenshots of UI surfaces | Tech details | **Pending** — needs a deployed platform instance, not a local dev server |
| Platform architecture document update | Tech details | **Pending** |

39
documents/user-guide/readiness-checklist.md
View File

@@ -7,11 +7,11 @@ Run these checks **before the first execution** of a workflow, and again after a
change to the run launcher, the executor, the target namespace or the code
location image.
> **NOT CLUSTER-VERIFIED.** Checks L4–L9 and the Kubernetes rows of section 4.3
> are derived from the implemented reference but have not yet been run against a
> Simpl cluster. Checks L1–L3, L10 and L11, which exercise the payload contract,
> the message-parsing path, per-unit dispatch and image-level isolation, are
> verified locally.
> **L4–L10 were cleared on sandbox-cat-dat on 2026-08-31.** Checks L1–L3 and L11,
> which exercise the payload contract, the message-parsing path and image-level
> isolation, are verified locally. What remains unverified on a cluster is the
> **tightly coupled** Kubernetes path — the T rows and the Kubernetes rows of
> section 4.3 — which uses `k8s_job_executor` rather than the pipes client.
---
@@ -46,7 +46,7 @@ location image.
| L1 | Payload contract is intact | `uv run pytest tests/test_loosely_coupled.py` | `test_payload_does_not_import_dagster` passes — the payload imports `dagster_pipes` only |
| L2 | Pipes round trip works | Launch `loosely_coupled_subprocess_job` | Run reaches `SUCCESS`; run logs contain the payload's `External payload started on …` line, proving `pipes.log` crossed the channel |
| L3 | Silence is treated as failure | Same test module | `test_silent_message_path_is_treated_as_failure` passes — an empty message list raises rather than yielding an empty result |
| L4 | Payload image is pullable by the target cluster | `kubectl -n <payload-ns> run pull-probe --image=<payload-image> --restart=Never --command -- true` | Pod reaches `Completed`; no `ImagePullBackOff`. The sandbox Gitea registry rejects anonymous pulls, so the payload namespace needs its own pull secret — see `yaml/sandbox/values-sandbox-gitea.yaml` |
| L4 | Payload image is pullable by the target cluster | `kubectl -n <payload-ns> run pull-probe --image=<payload-image> --restart=Never --command -- true` | Pod reaches `Completed`; no `ImagePullBackOff`. Both Gitea images pull anonymously — a bare registry `GET` returns 401, but that is the start of the Docker token handshake, not a refusal |
| L5 | Dispatcher can create Jobs in the payload namespace | `kubectl -n <payload-ns> auth can-i create jobs --as=system:serviceaccount:dagster:dagster-dev` | Returns `yes` |
| L6 | Dispatcher can read pod logs — **the message channel** | `kubectl -n <payload-ns> auth can-i get pods/log --as=system:serviceaccount:dagster:dagster-dev` | Returns `yes`. A `no` here breaks reporting *without* failing the workload |
| L7 | Payload Job is actually created | Launch `loosely_coupled_k8s_job`, then `kubectl -n <payload-ns> get jobs -l app.kubernetes.io/name=distributed-execution-payload` | One Job per dispatch, labelled `dagster/execution-target=loosely-coupled` |
@@ -63,6 +63,21 @@ throwaway namespace, without deploying a code location or a Dagster control
plane. Prefer it over assembling the cluster checks by hand: the RBAC it grants
is exactly the set L5 and L6 ask about, so a failure localises immediately.
**L4–L10 were cleared on sandbox-cat-dat on 2026-08-31** using the sandbox
variant `yaml/sandbox/probe-pipes-k8s-sandbox.yaml`. Run
`cff9b348-bfc3-4ac1-ab51-a94892b8e3a0` reached `RUN_SUCCESS` in `dataprovider01`:
four payload Jobs, four distinct payload pod hostnames in `contributing_hosts`,
and the payload's `External payload started on …` lines in the dispatcher's log,
which is the pod log stream doing its job as the message channel. No credentials
were needed anywhere.
One trap the run exposed. The platform's `dagster-svc-account` sets
`automountServiceAccountToken: false`, and the Dagster chart overrides it to
`true` on every pod it manages. A hand-written pod that does not is a plausible
future failure: the pipes client selects in-cluster authentication correctly and
then fails on `Service token file does not exist`, which points at Kubernetes
rather than at the omission.
---
## 4. Common misconfiguration symptoms
@@ -101,9 +116,15 @@ is exactly the set L5 and L6 ask about, so a failure localises immediately.
| Payload receives no `units` | `extras` key mismatch between dispatcher and `pipes.get_extra()` | Both sides must use the same key; a typo yields a `KeyError` inside the payload |
| Run cancelled in the UI, payload pod keeps running | Cancellation is not propagated to dispatched workloads automatically | `delete_pod_on_completion` handles the normal path; for cancellation, verify orphaned Jobs and add a cleanup sensor |
> **NOT CLUSTER-VERIFIED.** Rows referencing Kubernetes behaviour follow from the
> implemented reference and the dagster-k8s API, but have not been observed on a
> Simpl cluster. Confirm and amend after the first cluster run.
> **Partly observed.** The pipes rows were exercised on sandbox-cat-dat on
> 2026-08-31. Rows describing the tightly coupled `k8s_job_executor` still follow
> from the dagster-k8s API rather than from observation.
>
> One symptom the cluster run added, absent from the table above: a service
> account with `automountServiceAccountToken: false` — which the platform's
> `dagster-svc-account` uses — makes the pipes client fail with
> `ConfigException: Service token file does not exist`. It reads as a Kubernetes
> fault; the fix is `automountServiceAccountToken: true` on the pod.
---