[SIMPL-30451] Verify the Kubernetes pipes transport on the sandbox

Run cff9b348-bfc3-4ac1-ab51-a94892b8e3a0 of loosely_coupled_k8s_job reached RUN_SUCCESS in dataprovider01 on sandbox-cat-dat: four payload Jobs, four distinct payload pod hostnames in contributing_hosts, and the payload log lines in the run log. Checks L4-L10 are now observed rather than derived. Torn down afterwards; the namespace was left as it was found.

Corrects two claims that were wrong. The registry does NOT require a pull secret - a bare GET returns 401, but that is the opening move of the Docker token handshake, and completing it anonymously returns the manifest for both images. And PIPES_PAYLOAD_NAMESPACE was set to a namespace named dagster, which does not exist on that cluster; the release is called dagster but runs in dataprovider01.

Adds the sandbox probe variant. The generic probe cannot run there: sandbox access is Rancher project-scoped, so a new namespace grants its creator nothing and Role creation is denied everywhere. It is not needed either, since dagster-role already carries the exact pipes permissions.

Changelog: added

documents/user-guide/distributed-execution-guide.md

@@ -8,8 +8,10 @@ integration pattern for a Dagster workflow on the Simpl orchestration platform.
 > implementations in this repository. The tightly coupled setup and the loosely
 > coupled **subprocess** transport are verified end to end, including runs
 > launched from the Dagster UI — see [section 5.6](#56-what-a-verified-run-actually-produced).
-> The loosely coupled **Kubernetes** transport is implemented and reviewed but
+> The loosely coupled **Kubernetes** transport was run on sandbox-cat-dat on
-> has not yet been run against a cluster; statements specific to it are marked
+> 2026-08-31 and reached `RUN_SUCCESS`; see the readiness checklist for the
+> evidence. What remains unproven on a cluster is the *tightly* coupled
+> Kubernetes path — `k8s_job_executor` — whose rows stay marked
 > **NOT CLUSTER-VERIFIED**. See [Outstanding work](#8-outstanding-work).
 
 ---
@@ -378,10 +380,12 @@ empty result. If your cluster's logging setup makes that transport unreliable,
 switch to an object-storage message reader, which needs a bucket plus
 credentials on both sides.
 
-> **NOT CLUSTER-VERIFIED.** The Kubernetes transport above is implemented and
+> **CLUSTER-VERIFIED 2026-08-31.** Run `cff9b348…` of `loosely_coupled_k8s_job`
-> its RBAC and configuration are recorded, but it has not yet been run against a
+> reached `RUN_SUCCESS` on sandbox-cat-dat. Four payload Jobs were created, one
-> Simpl cluster. The subprocess transport, which exercises the identical payload
+> per work unit, and `contributing_hosts` held four distinct payload pod names —
-> and the identical message-parsing path, is verified by the test suite.
+> so the work demonstrably ran off the orchestrator. The payload's log lines
+> reached the run log, which is the pod log stream serving as the message
+> channel exactly as described above.
 
 ### 5.4 Where setup selection is persisted
 
@@ -398,6 +402,7 @@ setting. There is no UI control that switches a workflow between the two pattern
 | `yaml/sandbox/values-sandbox-gitea.yaml` | Sandbox registry coordinates, pull secret, and the SHA both images share |
 | `yaml/loosely-coupled/rbac-pipes-dispatch.yaml` | Permissions required by the dispatching pod |
 | `yaml/loosely-coupled/probe-pipes-k8s.yaml` | Standalone cluster probe for the pipes transport (checklist L4–L9) |
+| `yaml/sandbox/probe-pipes-k8s-sandbox.yaml` | The same probe for sandbox-cat-dat, where project-scoped access forbids creating namespaces and RBAC |
 | `yaml/values-dagster-distributed-execution.yaml` | Code location image and entry point |
 
 The Launchpad can override *run configuration* — op config, resource config, tags
@@ -507,17 +512,18 @@ Tracked under SIMPL-30451.
 |---|---|---|
 | Execution-target choices, prerequisites, workflow-level documentation | AC1 | Complete |
 | Comparison, environment-fit indicators, trade-offs | AC2 | Complete |
-| Readiness checklist and evidence mapping | AC3 | Complete; K8s pipes checks not cluster-verified |
+| Readiness checklist and evidence mapping | AC3 | Complete; tightly coupled K8s executor rows not cluster-verified |
-| Misconfiguration symptoms and corrections | AC3 | Complete; K8s pipes symptoms not cluster-verified |
+| Misconfiguration symptoms and corrections | AC3 | Complete; tightly coupled K8s symptoms not cluster-verified |
 | Configuration constructs and persistence | AC4 | Complete |
 | Before-and-after example, tightly coupled | AC4 | Complete |
 | Before-and-after example, switching to loosely coupled | AC4 | Complete |
 | Runnable tightly coupled reference implementations | AC4 | Complete |
-| Runnable loosely coupled reference implementation | AC4 | Complete (subprocess verified, K8s not cluster-run) |
+| Runnable loosely coupled reference implementation | AC4 | Complete (subprocess and Kubernetes transports both verified) |
 | Both images build; payload image passes the isolation check (L11) | Tech details | Complete |
 | Locally runnable jobs launched from the Dagster UI, evidence recorded in section 5.6 | Tech details | Complete |
 | Payload image published to a container registry | Tech details | Complete on the sandbox Gitea registry; the GitLab registry still pending |
 | GitLab pipeline builds both images | Tech details | **Pending** — the shared `ds.gitlab-ci.yml` template builds one image from the root Dockerfile |
-| End-to-end run of `loosely_coupled_k8s_job` on a cluster | Tech details | **Pending** |
+| End-to-end run of `loosely_coupled_k8s_job` on a cluster | Tech details | Complete — sandbox-cat-dat, 2026-08-31, run `cff9b348…`; see the readiness checklist |
+| `distributed-execution` registered as a code location on a platform Dagster | Tech details | **Pending** — blocked on the 1.13.19 vs 1.12.8 control plane skew |
 | Screenshots of UI surfaces | Tech details | **Pending** — needs a deployed platform instance, not a local dev server |
 | Platform architecture document update | Tech details | **Pending** |

documents/user-guide/readiness-checklist.md

@@ -7,11 +7,11 @@ Run these checks **before the first execution** of a workflow, and again after a
 change to the run launcher, the executor, the target namespace or the code
 location image.
 
-> **NOT CLUSTER-VERIFIED.** Checks L4–L9 and the Kubernetes rows of section 4.3
+> **L4–L10 were cleared on sandbox-cat-dat on 2026-08-31.** Checks L1–L3 and L11,
-> are derived from the implemented reference but have not yet been run against a
+> which exercise the payload contract, the message-parsing path and image-level
-> Simpl cluster. Checks L1–L3, L10 and L11, which exercise the payload contract,
+> isolation, are verified locally. What remains unverified on a cluster is the
-> the message-parsing path, per-unit dispatch and image-level isolation, are
+> **tightly coupled** Kubernetes path — the T rows and the Kubernetes rows of
-> verified locally.
+> section 4.3 — which uses `k8s_job_executor` rather than the pipes client.
 
 ---
 
@@ -46,7 +46,7 @@ location image.
 | L1 | Payload contract is intact | `uv run pytest tests/test_loosely_coupled.py` | `test_payload_does_not_import_dagster` passes — the payload imports `dagster_pipes` only |
 | L2 | Pipes round trip works | Launch `loosely_coupled_subprocess_job` | Run reaches `SUCCESS`; run logs contain the payload's `External payload started on …` line, proving `pipes.log` crossed the channel |
 | L3 | Silence is treated as failure | Same test module | `test_silent_message_path_is_treated_as_failure` passes — an empty message list raises rather than yielding an empty result |
-| L4 | Payload image is pullable by the target cluster | `kubectl -n <payload-ns> run pull-probe --image=<payload-image> --restart=Never --command -- true` | Pod reaches `Completed`; no `ImagePullBackOff`. The sandbox Gitea registry rejects anonymous pulls, so the payload namespace needs its own pull secret — see `yaml/sandbox/values-sandbox-gitea.yaml` |
+| L4 | Payload image is pullable by the target cluster | `kubectl -n <payload-ns> run pull-probe --image=<payload-image> --restart=Never --command -- true` | Pod reaches `Completed`; no `ImagePullBackOff`. Both Gitea images pull anonymously — a bare registry `GET` returns 401, but that is the start of the Docker token handshake, not a refusal |
 | L5 | Dispatcher can create Jobs in the payload namespace | `kubectl -n <payload-ns> auth can-i create jobs --as=system:serviceaccount:dagster:dagster-dev` | Returns `yes` |
 | L6 | Dispatcher can read pod logs — **the message channel** | `kubectl -n <payload-ns> auth can-i get pods/log --as=system:serviceaccount:dagster:dagster-dev` | Returns `yes`. A `no` here breaks reporting *without* failing the workload |
 | L7 | Payload Job is actually created | Launch `loosely_coupled_k8s_job`, then `kubectl -n <payload-ns> get jobs -l app.kubernetes.io/name=distributed-execution-payload` | One Job per dispatch, labelled `dagster/execution-target=loosely-coupled` |
@@ -63,6 +63,21 @@ throwaway namespace, without deploying a code location or a Dagster control
 plane. Prefer it over assembling the cluster checks by hand: the RBAC it grants
 is exactly the set L5 and L6 ask about, so a failure localises immediately.
 
+**L4–L10 were cleared on sandbox-cat-dat on 2026-08-31** using the sandbox
+variant `yaml/sandbox/probe-pipes-k8s-sandbox.yaml`. Run
+`cff9b348-bfc3-4ac1-ab51-a94892b8e3a0` reached `RUN_SUCCESS` in `dataprovider01`:
+four payload Jobs, four distinct payload pod hostnames in `contributing_hosts`,
+and the payload's `External payload started on …` lines in the dispatcher's log,
+which is the pod log stream doing its job as the message channel. No credentials
+were needed anywhere.
+
+One trap the run exposed. The platform's `dagster-svc-account` sets
+`automountServiceAccountToken: false`, and the Dagster chart overrides it to
+`true` on every pod it manages. A hand-written pod that does not is a plausible
+future failure: the pipes client selects in-cluster authentication correctly and
+then fails on `Service token file does not exist`, which points at Kubernetes
+rather than at the omission.
+
 ---
 
 ## 4. Common misconfiguration symptoms
@@ -101,9 +116,15 @@ is exactly the set L5 and L6 ask about, so a failure localises immediately.
 | Payload receives no `units` | `extras` key mismatch between dispatcher and `pipes.get_extra()` | Both sides must use the same key; a typo yields a `KeyError` inside the payload |
 | Run cancelled in the UI, payload pod keeps running | Cancellation is not propagated to dispatched workloads automatically | `delete_pod_on_completion` handles the normal path; for cancellation, verify orphaned Jobs and add a cleanup sensor |
 
-> **NOT CLUSTER-VERIFIED.** Rows referencing Kubernetes behaviour follow from the
+> **Partly observed.** The pipes rows were exercised on sandbox-cat-dat on
-> implemented reference and the dagster-k8s API, but have not been observed on a
+> 2026-08-31. Rows describing the tightly coupled `k8s_job_executor` still follow
-> Simpl cluster. Confirm and amend after the first cluster run.
+> from the dagster-k8s API rather than from observation.
+>
+> One symptom the cluster run added, absent from the table above: a service
+> account with `automountServiceAccountToken: false` — which the platform's
+> `dagster-svc-account` uses — makes the pipes client fail with
+> `ConfigException: Service token file does not exist`. It reads as a Kubernetes
+> fault; the fix is `automountServiceAccountToken: true` on the pod.
 
 ---
 

yaml/loosely-coupled/probe-pipes-k8s.yaml

@@ -12,33 +12,24 @@
 #
 # --- Before applying -------------------------------------------------------
 #
-# 1. Pull secret. The registry rejects anonymous pulls, so both this pod and the
+# Credentials, only if your registry needs them. The Gitea images referenced
-#    payload Jobs it creates need credentials. Use a token with read:package.
+# below do NOT: a bare GET returns 401, but that is the start of the Docker
+# token handshake, and completing it anonymously returns the manifest. A probe
+# pod carrying no credentials pulled and ran on the sandbox on 2026-08-31.
 #
-#      kubectl -n distexec-probe create secret docker-registry gitea-registry \
+# If you point this at a registry that does require authentication, remember the
-#        --docker-server=gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu \
+# payload Jobs need it too. PipesK8sClient sets no serviceAccountName, so they
-#        --docker-username=<gitea-user> \
+# run as `default` and inherit nothing from the pod below; patch that account or
-#        --docker-password=<gitea-token>
+# pass imagePullSecrets through base_pod_spec in dispatch_external_work_k8s. A
-#
+# payload pod that cannot pull leaves the op waiting until pod_wait_timeout,
-# 2. Payload pods inherit the pull secret from the default service account.
+# which reads like a hung workload rather than a missing credential.
-#    PipesK8sClient does not set serviceAccountName on the Jobs it creates, so
-#    they run as `default`, and this service does not yet pass imagePullSecrets
-#    through base_pod_spec:
-#
-#      kubectl -n distexec-probe patch serviceaccount default \
-#        -p '{"imagePullSecrets":[{"name":"gitea-registry"}]}'
-#
-#    Skip this and the payload pod never starts. That surfaces as the op waiting
-#    until pod_wait_timeout, which reads like a hung workload rather than a
-#    missing credential.
-#
-# Order does not matter much: apply this file first and the dispatcher pod sits
-# in ImagePullBackOff until the secret appears, then pulls on the kubelet's next
-# retry. The default service account only has to be patched before the first
-# payload Job is created, which is after the dispatcher starts.
 #
 #      kubectl apply -f yaml/loosely-coupled/probe-pipes-k8s.yaml
 #
+# On sandbox-cat-dat use yaml/sandbox/probe-pipes-k8s-sandbox.yaml instead: that
+# cluster's access is Rancher project-scoped, so this file's namespace and RBAC
+# cannot be created there. The sandbox variant needs neither.
+#
 # `--dry-run=server` reports "namespaces distexec-probe not found" for the four
 # namespaced objects. That is the dry run declining to create the namespace it
 # would need, not a fault in the manifests; `--dry-run=client` passes clean.
@@ -134,6 +125,10 @@ spec:
     spec:
       restartPolicy: Never
       serviceAccountName: pipes-dispatcher
+      # A service account with automountServiceAccountToken: false - which the
+      # platform's dagster-svc-account uses - leaves the pipes client selecting
+      # in-cluster auth and then failing on "Service token file does not exist".
+      automountServiceAccountToken: true
       securityContext:
         runAsNonRoot: true
         runAsUser: 1000

yaml/sandbox/probe-pipes-k8s-sandbox.yaml

@@ -0,0 +1,124 @@
+# Sandbox variant of the pipes cluster probe.
+#
+# Runs `loosely_coupled_k8s_job` as a single Job in dataprovider01, reusing the
+# platform's own dagster-svc-account. Use this instead of
+# yaml/loosely-coupled/probe-pipes-k8s.yaml on sandbox-cat-dat.
+#
+# WHY IT LOOKS DIFFERENT. The generic probe creates its own namespace, service
+# account and RBAC. That cannot run here: sandbox access is Rancher
+# project-scoped, so a freshly created namespace belongs to no project and the
+# creator has no rights inside it, and `create roles` is denied in every
+# namespace. Verified with `kubectl auth can-i`, not assumed.
+#
+# It does not need them anyway. The `dagster-role` bound to dagster-svc-account
+# already grants batch/jobs (full verbs), jobs/status, pods, events, and
+# pods/log - exactly the set the pipes client needs. Readiness checks L5 and L6
+# therefore already pass on this cluster; read the live Role to confirm:
+#
+#   kubectl -n dataprovider01 get role dagster-role -o yaml
+#
+# WHAT THIS COSTS. Unlike the generic probe this is NOT isolated. It borrows a
+# live service account and leaves payload Jobs in a shared namespace until they
+# are deleted. Nothing it does is persistent, but it is visible to anyone else
+# working in dataprovider01.
+#
+# --- Credentials: none needed ----------------------------------------------
+#
+# Both Gitea images are anonymously pullable. A bare GET to the registry returns
+# 401, which looks like a refusal but is just the start of the Docker token
+# handshake; completing it anonymously and re-requesting the manifest returns
+# 200 for both images. Verify with:
+#
+#   TOK=$(curl -sS "https://gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/v2/token?service=container_registry&scope=repository:j.r/distributed-execution:pull" | jq -r .token)
+#   curl -sS -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $TOK" \
+#     https://gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/v2/j.r/distributed-execution/manifests/5122da4691f9
+#
+# This also means the payload Jobs need nothing: they run as the `default`
+# service account, which carries no pull secret, and do not need one.
+#
+# --- Running ---------------------------------------------------------------
+#
+#   kubectl -n dataprovider01 apply -f yaml/sandbox/probe-pipes-k8s-sandbox.yaml
+#   kubectl -n dataprovider01 logs -f job/distexec-pipes-probe
+#   kubectl -n dataprovider01 get jobs -l app.kubernetes.io/name=distributed-execution-payload
+#
+# --- Teardown --------------------------------------------------------------
+#
+#   kubectl -n dataprovider01 delete job distexec-pipes-probe
+#   kubectl -n dataprovider01 delete jobs -l app.kubernetes.io/name=distributed-execution-payload
+#
+# VERIFIED 2026-08-31 on sandbox-cat-dat / dataprovider01. Run
+# cff9b348-bfc3-4ac1-ab51-a94892b8e3a0 reached RUN_SUCCESS: four payload Jobs,
+# four distinct payload pod hostnames in contributing_hosts, messages returned
+# over the pod log stream. Checks L4-L10 pass. Torn down afterwards; the
+# namespace was left as it was found.
+
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: distexec-pipes-probe
+  namespace: dataprovider01
+  labels:
+    app.kubernetes.io/name: distributed-execution
+    app.kubernetes.io/component: probe
+spec:
+  backoffLimit: 0
+  activeDeadlineSeconds: 1800
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: distributed-execution
+        app.kubernetes.io/component: probe
+    spec:
+      restartPolicy: Never
+      # Already holds jobs + pods/log; this probe grants itself nothing.
+      serviceAccountName: dagster-svc-account
+      # dagster-svc-account sets automountServiceAccountToken: false, so every
+      # pod that uses it must opt back in - the chart does this for its own
+      # deployments. Without it the pipes client picks in-cluster auth correctly
+      # and then fails on "Service token file does not exist".
+      automountServiceAccountToken: true
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1000
+        runAsGroup: 1000
+        fsGroup: 1000
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: dispatcher
+          image: gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/j.r/distributed-execution:5122da4691f9
+          imagePullPolicy: IfNotPresent
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop:
+                - ALL
+          command:
+            - dagster
+            - job
+            - execute
+            - -f
+            - src/distributed_execution/repository.py
+            - -j
+            - loosely_coupled_k8s_job
+          env:
+            - name: DAGSTER_HOME
+              value: /dagster-home
+            - name: PIPES_PAYLOAD_IMAGE
+              value: gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/j.r/distributed-execution-payload:5122da4691f9
+            - name: PIPES_PAYLOAD_NAMESPACE
+              value: dataprovider01
+          volumeMounts:
+            - name: dagster-home
+              mountPath: /dagster-home
+          resources:
+            requests:
+              cpu: 200m
+              memory: 512Mi
+            limits:
+              cpu: "1"
+              memory: 1Gi
+      volumes:
+        - name: dagster-home
+          emptyDir: {}

yaml/sandbox/values-sandbox-gitea.yaml

@@ -1,8 +1,9 @@
 # Sandbox deployment values - Gitea registry.
 #
-# NOT CLUSTER-VERIFIED. These values are written against images that exist and a
+# NOT CLUSTER-VERIFIED. The images and the registry behaviour here are confirmed
-# registry whose auth behaviour was checked, but they have not been applied to a
+# - a probe pod ran from them on the sandbox on 2026-08-31 - but these values
-# Simpl cluster. Confirm and amend after the first deployment.
+# have never been applied as a code location. Registering this service against
+# the sandbox Dagster is still blocked on the version skew below.
 #
 # VERSION SKEW, unresolved. These images carry dagster 1.13.19 (requirements.txt
 # pins no upper bound, so the build took whatever was current). The sandbox
@@ -21,39 +22,18 @@
 # the version lock: the code location and the payload it dispatches must come
 # from one commit. Bump both together or not at all.
 #
-# The sandbox Gitea registry requires authentication - an anonymous manifest GET
+# The Gitea images are anonymously pullable, so NO pull secret is needed. A bare
-# returns 401 - so a pull secret is required in every namespace that pulls either
+# GET to the registry returns 401, which reads as a refusal but is only the start
-# image. Create it with a Gitea access token that has read:package scope:
+# of the Docker token handshake; completing it anonymously and re-requesting the
+# manifest returns 200 for both images. This was confirmed on 2026-08-31, and a
+# probe pod carrying no credentials pulled and ran on the sandbox.
 #
-#   kubectl -n <namespace> create secret docker-registry gitea-registry \
+# An earlier revision of this file required a pull secret on both the deployment
-#     --docker-server=gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu \
+# and the run launcher. That was wrong, and wrong in an expensive direction: it
-#     --docker-username=<gitea-user> \
+# sent the reader looking for credentials that do not exist.
-#     --docker-password=<gitea-token>
-#
-# The payload namespace needs it too: PipesK8sClient creates that Job, and a
-# missing pull secret there leaves the op waiting on a pod that never starts,
-# which surfaces as pod_wait_timeout rather than as an image error.
-#
-# PULL SECRET PLACEMENT. Checked against the chart templates in dagster 1.12.8,
-# which is what the sandbox runs. There is no per-deployment imagePullSecrets
-# field - deployment-user.yaml reads the SUBCHART value and the run launcher
-# helper reads the PARENT value - so a pull secret nested under a deployments[]
-# entry is silently discarded, and the pod fails with ImagePullBackOff against a
-# values file that looks correct.
-#
-# Both keys below therefore apply to every code location in the release, not
-# just this one. Extra pull secrets are ignored for registries they do not match,
-# so the existing code locations are unaffected.
 
 dagster:
-  # Run pods, webserver and daemon.
-  imagePullSecrets:
-    - name: gitea-registry
-
   dagster-user-deployments:
-    # Code server pods.
-    imagePullSecrets:
-      - name: gitea-registry
     deployments:
       - name: distributed-execution
         image:
@@ -63,8 +43,10 @@ dagster:
         env:
           - name: PIPES_PAYLOAD_IMAGE
             value: gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/j.r/distributed-execution-payload:5122da4691f9
+          # The sandbox release is named `dagster` but lives in dataprovider01;
+          # there is no namespace called `dagster` on that cluster.
           - name: PIPES_PAYLOAD_NAMESPACE
-            value: dagster
+            value: dataprovider01
 
   runLauncher:
     config:
@@ -75,4 +57,4 @@ dagster:
               - name: PIPES_PAYLOAD_IMAGE
                 value: gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/j.r/distributed-execution-payload:5122da4691f9
               - name: PIPES_PAYLOAD_NAMESPACE
-                value: dagster
+                value: dataprovider01