j.r/distributed-execution
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[SIMPL-30451] Verify the Kubernetes pipes transport on the sandbox

Browse Source 
Run cff9b348-bfc3-4ac1-ab51-a94892b8e3a0 of loosely_coupled_k8s_job reached RUN_SUCCESS in dataprovider01 on sandbox-cat-dat: four payload Jobs, four distinct payload pod hostnames in contributing_hosts, and the payload log lines in the run log. Checks L4-L10 are now observed rather than derived. Torn down afterwards; the namespace was left as it was found.

Corrects two claims that were wrong. The registry does NOT require a pull secret - a bare GET returns 401, but that is the opening move of the Docker token handshake, and completing it anonymously returns the manifest for both images. And PIPES_PAYLOAD_NAMESPACE was set to a namespace named dagster, which does not exist on that cluster; the release is called dagster but runs in dataprovider01.

Adds the sandbox probe variant. The generic probe cannot run there: sandbox access is Rancher project-scoped, so a new namespace grants its creator nothing and Role creation is denied everywhere. It is not needed either, since dagster-role already carries the exact pipes permissions.

Changelog: added
This commit is contained in:
ILay
2026-08-31 19:41:26 +02:00
parent e686df5710
commit 96caa3f81b
5 changed files with 204 additions and 76 deletions
Download Patch File Download Diff File Expand all files Collapse all files

39
documents/user-guide/readiness-checklist.md
View File

@@ -7,11 +7,11 @@ Run these checks **before the first execution** of a workflow, and again after a
change to the run launcher, the executor, the target namespace or the code
location image.
> **NOT CLUSTER-VERIFIED.** Checks L4–L9 and the Kubernetes rows of section 4.3
> are derived from the implemented reference but have not yet been run against a
> Simpl cluster. Checks L1–L3, L10 and L11, which exercise the payload contract,
> the message-parsing path, per-unit dispatch and image-level isolation, are
> verified locally.
> **L4–L10 were cleared on sandbox-cat-dat on 2026-08-31.** Checks L1–L3 and L11,
> which exercise the payload contract, the message-parsing path and image-level
> isolation, are verified locally. What remains unverified on a cluster is the
> **tightly coupled** Kubernetes path — the T rows and the Kubernetes rows of
> section 4.3 — which uses `k8s_job_executor` rather than the pipes client.
---
@@ -46,7 +46,7 @@ location image.
| L1 | Payload contract is intact | `uv run pytest tests/test_loosely_coupled.py` | `test_payload_does_not_import_dagster` passes — the payload imports `dagster_pipes` only |
| L2 | Pipes round trip works | Launch `loosely_coupled_subprocess_job` | Run reaches `SUCCESS`; run logs contain the payload's `External payload started on …` line, proving `pipes.log` crossed the channel |
| L3 | Silence is treated as failure | Same test module | `test_silent_message_path_is_treated_as_failure` passes — an empty message list raises rather than yielding an empty result |
| L4 | Payload image is pullable by the target cluster | `kubectl -n <payload-ns> run pull-probe --image=<payload-image> --restart=Never --command -- true` | Pod reaches `Completed`; no `ImagePullBackOff`. The sandbox Gitea registry rejects anonymous pulls, so the payload namespace needs its own pull secret — see `yaml/sandbox/values-sandbox-gitea.yaml` |
| L4 | Payload image is pullable by the target cluster | `kubectl -n <payload-ns> run pull-probe --image=<payload-image> --restart=Never --command -- true` | Pod reaches `Completed`; no `ImagePullBackOff`. Both Gitea images pull anonymously — a bare registry `GET` returns 401, but that is the start of the Docker token handshake, not a refusal |
| L5 | Dispatcher can create Jobs in the payload namespace | `kubectl -n <payload-ns> auth can-i create jobs --as=system:serviceaccount:dagster:dagster-dev` | Returns `yes` |
| L6 | Dispatcher can read pod logs — **the message channel** | `kubectl -n <payload-ns> auth can-i get pods/log --as=system:serviceaccount:dagster:dagster-dev` | Returns `yes`. A `no` here breaks reporting *without* failing the workload |
| L7 | Payload Job is actually created | Launch `loosely_coupled_k8s_job`, then `kubectl -n <payload-ns> get jobs -l app.kubernetes.io/name=distributed-execution-payload` | One Job per dispatch, labelled `dagster/execution-target=loosely-coupled` |
@@ -63,6 +63,21 @@ throwaway namespace, without deploying a code location or a Dagster control
plane. Prefer it over assembling the cluster checks by hand: the RBAC it grants
is exactly the set L5 and L6 ask about, so a failure localises immediately.
**L4–L10 were cleared on sandbox-cat-dat on 2026-08-31** using the sandbox
variant `yaml/sandbox/probe-pipes-k8s-sandbox.yaml`. Run
`cff9b348-bfc3-4ac1-ab51-a94892b8e3a0` reached `RUN_SUCCESS` in `dataprovider01`:
four payload Jobs, four distinct payload pod hostnames in `contributing_hosts`,
and the payload's `External payload started on …` lines in the dispatcher's log,
which is the pod log stream doing its job as the message channel. No credentials
were needed anywhere.
One trap the run exposed. The platform's `dagster-svc-account` sets
`automountServiceAccountToken: false`, and the Dagster chart overrides it to
`true` on every pod it manages. A hand-written pod that does not is a plausible
future failure: the pipes client selects in-cluster authentication correctly and
then fails on `Service token file does not exist`, which points at Kubernetes
rather than at the omission.
---
## 4. Common misconfiguration symptoms
@@ -101,9 +116,15 @@ is exactly the set L5 and L6 ask about, so a failure localises immediately.
| Payload receives no `units` | `extras` key mismatch between dispatcher and `pipes.get_extra()` | Both sides must use the same key; a typo yields a `KeyError` inside the payload |
| Run cancelled in the UI, payload pod keeps running | Cancellation is not propagated to dispatched workloads automatically | `delete_pod_on_completion` handles the normal path; for cancellation, verify orphaned Jobs and add a cleanup sensor |
> **NOT CLUSTER-VERIFIED.** Rows referencing Kubernetes behaviour follow from the
> implemented reference and the dagster-k8s API, but have not been observed on a
> Simpl cluster. Confirm and amend after the first cluster run.
> **Partly observed.** The pipes rows were exercised on sandbox-cat-dat on
> 2026-08-31. Rows describing the tightly coupled `k8s_job_executor` still follow
> from the dagster-k8s API rather than from observation.
>
> One symptom the cluster run added, absent from the table above: a service
> account with `automountServiceAccountToken: false` — which the platform's
> `dagster-svc-account` uses — makes the pipes client fail with
> `ConfigException: Service token file does not exist`. It reads as a Kubernetes
> fault; the fix is `automountServiceAccountToken: true` on the pod.
---