From 96caa3f81bb555543e0904c739b79877bcfee41a Mon Sep 17 00:00:00 2001 From: ILay Date: Mon, 31 Aug 2026 19:41:26 +0200 Subject: [PATCH] [SIMPL-30451] Verify the Kubernetes pipes transport on the sandbox Run cff9b348-bfc3-4ac1-ab51-a94892b8e3a0 of loosely_coupled_k8s_job reached RUN_SUCCESS in dataprovider01 on sandbox-cat-dat: four payload Jobs, four distinct payload pod hostnames in contributing_hosts, and the payload log lines in the run log. Checks L4-L10 are now observed rather than derived. Torn down afterwards; the namespace was left as it was found. Corrects two claims that were wrong. The registry does NOT require a pull secret - a bare GET returns 401, but that is the opening move of the Docker token handshake, and completing it anonymously returns the manifest for both images. And PIPES_PAYLOAD_NAMESPACE was set to a namespace named dagster, which does not exist on that cluster; the release is called dagster but runs in dataprovider01. Adds the sandbox probe variant. The generic probe cannot run there: sandbox access is Rancher project-scoped, so a new namespace grants its creator nothing and Role creation is denied everywhere. It is not needed either, since dagster-role already carries the exact pipes permissions. Changelog: added --- .../user-guide/distributed-execution-guide.md | 26 ++-- documents/user-guide/readiness-checklist.md | 39 ++++-- yaml/loosely-coupled/probe-pipes-k8s.yaml | 41 +++--- yaml/sandbox/probe-pipes-k8s-sandbox.yaml | 124 ++++++++++++++++++ yaml/sandbox/values-sandbox-gitea.yaml | 50 +++---- 5 files changed, 204 insertions(+), 76 deletions(-) create mode 100644 yaml/sandbox/probe-pipes-k8s-sandbox.yaml diff --git a/documents/user-guide/distributed-execution-guide.md b/documents/user-guide/distributed-execution-guide.md index 8c64d45..e8f0411 100644 --- a/documents/user-guide/distributed-execution-guide.md +++ b/documents/user-guide/distributed-execution-guide.md @@ -8,8 +8,10 @@ integration pattern for a Dagster workflow on the Simpl orchestration platform. > implementations in this repository. The tightly coupled setup and the loosely > coupled **subprocess** transport are verified end to end, including runs > launched from the Dagster UI — see [section 5.6](#56-what-a-verified-run-actually-produced). -> The loosely coupled **Kubernetes** transport is implemented and reviewed but -> has not yet been run against a cluster; statements specific to it are marked +> The loosely coupled **Kubernetes** transport was run on sandbox-cat-dat on +> 2026-08-31 and reached `RUN_SUCCESS`; see the readiness checklist for the +> evidence. What remains unproven on a cluster is the *tightly* coupled +> Kubernetes path — `k8s_job_executor` — whose rows stay marked > **NOT CLUSTER-VERIFIED**. See [Outstanding work](#8-outstanding-work). --- @@ -378,10 +380,12 @@ empty result. If your cluster's logging setup makes that transport unreliable, switch to an object-storage message reader, which needs a bucket plus credentials on both sides. -> **NOT CLUSTER-VERIFIED.** The Kubernetes transport above is implemented and -> its RBAC and configuration are recorded, but it has not yet been run against a -> Simpl cluster. The subprocess transport, which exercises the identical payload -> and the identical message-parsing path, is verified by the test suite. +> **CLUSTER-VERIFIED 2026-08-31.** Run `cff9b348…` of `loosely_coupled_k8s_job` +> reached `RUN_SUCCESS` on sandbox-cat-dat. Four payload Jobs were created, one +> per work unit, and `contributing_hosts` held four distinct payload pod names — +> so the work demonstrably ran off the orchestrator. The payload's log lines +> reached the run log, which is the pod log stream serving as the message +> channel exactly as described above. ### 5.4 Where setup selection is persisted @@ -398,6 +402,7 @@ setting. There is no UI control that switches a workflow between the two pattern | `yaml/sandbox/values-sandbox-gitea.yaml` | Sandbox registry coordinates, pull secret, and the SHA both images share | | `yaml/loosely-coupled/rbac-pipes-dispatch.yaml` | Permissions required by the dispatching pod | | `yaml/loosely-coupled/probe-pipes-k8s.yaml` | Standalone cluster probe for the pipes transport (checklist L4–L9) | +| `yaml/sandbox/probe-pipes-k8s-sandbox.yaml` | The same probe for sandbox-cat-dat, where project-scoped access forbids creating namespaces and RBAC | | `yaml/values-dagster-distributed-execution.yaml` | Code location image and entry point | The Launchpad can override *run configuration* — op config, resource config, tags @@ -507,17 +512,18 @@ Tracked under SIMPL-30451. |---|---|---| | Execution-target choices, prerequisites, workflow-level documentation | AC1 | Complete | | Comparison, environment-fit indicators, trade-offs | AC2 | Complete | -| Readiness checklist and evidence mapping | AC3 | Complete; K8s pipes checks not cluster-verified | -| Misconfiguration symptoms and corrections | AC3 | Complete; K8s pipes symptoms not cluster-verified | +| Readiness checklist and evidence mapping | AC3 | Complete; tightly coupled K8s executor rows not cluster-verified | +| Misconfiguration symptoms and corrections | AC3 | Complete; tightly coupled K8s symptoms not cluster-verified | | Configuration constructs and persistence | AC4 | Complete | | Before-and-after example, tightly coupled | AC4 | Complete | | Before-and-after example, switching to loosely coupled | AC4 | Complete | | Runnable tightly coupled reference implementations | AC4 | Complete | -| Runnable loosely coupled reference implementation | AC4 | Complete (subprocess verified, K8s not cluster-run) | +| Runnable loosely coupled reference implementation | AC4 | Complete (subprocess and Kubernetes transports both verified) | | Both images build; payload image passes the isolation check (L11) | Tech details | Complete | | Locally runnable jobs launched from the Dagster UI, evidence recorded in section 5.6 | Tech details | Complete | | Payload image published to a container registry | Tech details | Complete on the sandbox Gitea registry; the GitLab registry still pending | | GitLab pipeline builds both images | Tech details | **Pending** — the shared `ds.gitlab-ci.yml` template builds one image from the root Dockerfile | -| End-to-end run of `loosely_coupled_k8s_job` on a cluster | Tech details | **Pending** | +| End-to-end run of `loosely_coupled_k8s_job` on a cluster | Tech details | Complete — sandbox-cat-dat, 2026-08-31, run `cff9b348…`; see the readiness checklist | +| `distributed-execution` registered as a code location on a platform Dagster | Tech details | **Pending** — blocked on the 1.13.19 vs 1.12.8 control plane skew | | Screenshots of UI surfaces | Tech details | **Pending** — needs a deployed platform instance, not a local dev server | | Platform architecture document update | Tech details | **Pending** | diff --git a/documents/user-guide/readiness-checklist.md b/documents/user-guide/readiness-checklist.md index 7995271..9d4e928 100644 --- a/documents/user-guide/readiness-checklist.md +++ b/documents/user-guide/readiness-checklist.md @@ -7,11 +7,11 @@ Run these checks **before the first execution** of a workflow, and again after a change to the run launcher, the executor, the target namespace or the code location image. -> **NOT CLUSTER-VERIFIED.** Checks L4–L9 and the Kubernetes rows of section 4.3 -> are derived from the implemented reference but have not yet been run against a -> Simpl cluster. Checks L1–L3, L10 and L11, which exercise the payload contract, -> the message-parsing path, per-unit dispatch and image-level isolation, are -> verified locally. +> **L4–L10 were cleared on sandbox-cat-dat on 2026-08-31.** Checks L1–L3 and L11, +> which exercise the payload contract, the message-parsing path and image-level +> isolation, are verified locally. What remains unverified on a cluster is the +> **tightly coupled** Kubernetes path — the T rows and the Kubernetes rows of +> section 4.3 — which uses `k8s_job_executor` rather than the pipes client. --- @@ -46,7 +46,7 @@ location image. | L1 | Payload contract is intact | `uv run pytest tests/test_loosely_coupled.py` | `test_payload_does_not_import_dagster` passes — the payload imports `dagster_pipes` only | | L2 | Pipes round trip works | Launch `loosely_coupled_subprocess_job` | Run reaches `SUCCESS`; run logs contain the payload's `External payload started on …` line, proving `pipes.log` crossed the channel | | L3 | Silence is treated as failure | Same test module | `test_silent_message_path_is_treated_as_failure` passes — an empty message list raises rather than yielding an empty result | -| L4 | Payload image is pullable by the target cluster | `kubectl -n run pull-probe --image= --restart=Never --command -- true` | Pod reaches `Completed`; no `ImagePullBackOff`. The sandbox Gitea registry rejects anonymous pulls, so the payload namespace needs its own pull secret — see `yaml/sandbox/values-sandbox-gitea.yaml` | +| L4 | Payload image is pullable by the target cluster | `kubectl -n run pull-probe --image= --restart=Never --command -- true` | Pod reaches `Completed`; no `ImagePullBackOff`. Both Gitea images pull anonymously — a bare registry `GET` returns 401, but that is the start of the Docker token handshake, not a refusal | | L5 | Dispatcher can create Jobs in the payload namespace | `kubectl -n auth can-i create jobs --as=system:serviceaccount:dagster:dagster-dev` | Returns `yes` | | L6 | Dispatcher can read pod logs — **the message channel** | `kubectl -n auth can-i get pods/log --as=system:serviceaccount:dagster:dagster-dev` | Returns `yes`. A `no` here breaks reporting *without* failing the workload | | L7 | Payload Job is actually created | Launch `loosely_coupled_k8s_job`, then `kubectl -n get jobs -l app.kubernetes.io/name=distributed-execution-payload` | One Job per dispatch, labelled `dagster/execution-target=loosely-coupled` | @@ -63,6 +63,21 @@ throwaway namespace, without deploying a code location or a Dagster control plane. Prefer it over assembling the cluster checks by hand: the RBAC it grants is exactly the set L5 and L6 ask about, so a failure localises immediately. +**L4–L10 were cleared on sandbox-cat-dat on 2026-08-31** using the sandbox +variant `yaml/sandbox/probe-pipes-k8s-sandbox.yaml`. Run +`cff9b348-bfc3-4ac1-ab51-a94892b8e3a0` reached `RUN_SUCCESS` in `dataprovider01`: +four payload Jobs, four distinct payload pod hostnames in `contributing_hosts`, +and the payload's `External payload started on …` lines in the dispatcher's log, +which is the pod log stream doing its job as the message channel. No credentials +were needed anywhere. + +One trap the run exposed. The platform's `dagster-svc-account` sets +`automountServiceAccountToken: false`, and the Dagster chart overrides it to +`true` on every pod it manages. A hand-written pod that does not is a plausible +future failure: the pipes client selects in-cluster authentication correctly and +then fails on `Service token file does not exist`, which points at Kubernetes +rather than at the omission. + --- ## 4. Common misconfiguration symptoms @@ -101,9 +116,15 @@ is exactly the set L5 and L6 ask about, so a failure localises immediately. | Payload receives no `units` | `extras` key mismatch between dispatcher and `pipes.get_extra()` | Both sides must use the same key; a typo yields a `KeyError` inside the payload | | Run cancelled in the UI, payload pod keeps running | Cancellation is not propagated to dispatched workloads automatically | `delete_pod_on_completion` handles the normal path; for cancellation, verify orphaned Jobs and add a cleanup sensor | -> **NOT CLUSTER-VERIFIED.** Rows referencing Kubernetes behaviour follow from the -> implemented reference and the dagster-k8s API, but have not been observed on a -> Simpl cluster. Confirm and amend after the first cluster run. +> **Partly observed.** The pipes rows were exercised on sandbox-cat-dat on +> 2026-08-31. Rows describing the tightly coupled `k8s_job_executor` still follow +> from the dagster-k8s API rather than from observation. +> +> One symptom the cluster run added, absent from the table above: a service +> account with `automountServiceAccountToken: false` — which the platform's +> `dagster-svc-account` uses — makes the pipes client fail with +> `ConfigException: Service token file does not exist`. It reads as a Kubernetes +> fault; the fix is `automountServiceAccountToken: true` on the pod. --- diff --git a/yaml/loosely-coupled/probe-pipes-k8s.yaml b/yaml/loosely-coupled/probe-pipes-k8s.yaml index 3acf329..cadfb70 100644 --- a/yaml/loosely-coupled/probe-pipes-k8s.yaml +++ b/yaml/loosely-coupled/probe-pipes-k8s.yaml @@ -12,33 +12,24 @@ # # --- Before applying ------------------------------------------------------- # -# 1. Pull secret. The registry rejects anonymous pulls, so both this pod and the -# payload Jobs it creates need credentials. Use a token with read:package. +# Credentials, only if your registry needs them. The Gitea images referenced +# below do NOT: a bare GET returns 401, but that is the start of the Docker +# token handshake, and completing it anonymously returns the manifest. A probe +# pod carrying no credentials pulled and ran on the sandbox on 2026-08-31. # -# kubectl -n distexec-probe create secret docker-registry gitea-registry \ -# --docker-server=gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu \ -# --docker-username= \ -# --docker-password= -# -# 2. Payload pods inherit the pull secret from the default service account. -# PipesK8sClient does not set serviceAccountName on the Jobs it creates, so -# they run as `default`, and this service does not yet pass imagePullSecrets -# through base_pod_spec: -# -# kubectl -n distexec-probe patch serviceaccount default \ -# -p '{"imagePullSecrets":[{"name":"gitea-registry"}]}' -# -# Skip this and the payload pod never starts. That surfaces as the op waiting -# until pod_wait_timeout, which reads like a hung workload rather than a -# missing credential. -# -# Order does not matter much: apply this file first and the dispatcher pod sits -# in ImagePullBackOff until the secret appears, then pulls on the kubelet's next -# retry. The default service account only has to be patched before the first -# payload Job is created, which is after the dispatcher starts. +# If you point this at a registry that does require authentication, remember the +# payload Jobs need it too. PipesK8sClient sets no serviceAccountName, so they +# run as `default` and inherit nothing from the pod below; patch that account or +# pass imagePullSecrets through base_pod_spec in dispatch_external_work_k8s. A +# payload pod that cannot pull leaves the op waiting until pod_wait_timeout, +# which reads like a hung workload rather than a missing credential. # # kubectl apply -f yaml/loosely-coupled/probe-pipes-k8s.yaml # +# On sandbox-cat-dat use yaml/sandbox/probe-pipes-k8s-sandbox.yaml instead: that +# cluster's access is Rancher project-scoped, so this file's namespace and RBAC +# cannot be created there. The sandbox variant needs neither. +# # `--dry-run=server` reports "namespaces distexec-probe not found" for the four # namespaced objects. That is the dry run declining to create the namespace it # would need, not a fault in the manifests; `--dry-run=client` passes clean. @@ -134,6 +125,10 @@ spec: spec: restartPolicy: Never serviceAccountName: pipes-dispatcher + # A service account with automountServiceAccountToken: false - which the + # platform's dagster-svc-account uses - leaves the pipes client selecting + # in-cluster auth and then failing on "Service token file does not exist". + automountServiceAccountToken: true securityContext: runAsNonRoot: true runAsUser: 1000 diff --git a/yaml/sandbox/probe-pipes-k8s-sandbox.yaml b/yaml/sandbox/probe-pipes-k8s-sandbox.yaml new file mode 100644 index 0000000..c5e1feb --- /dev/null +++ b/yaml/sandbox/probe-pipes-k8s-sandbox.yaml @@ -0,0 +1,124 @@ +# Sandbox variant of the pipes cluster probe. +# +# Runs `loosely_coupled_k8s_job` as a single Job in dataprovider01, reusing the +# platform's own dagster-svc-account. Use this instead of +# yaml/loosely-coupled/probe-pipes-k8s.yaml on sandbox-cat-dat. +# +# WHY IT LOOKS DIFFERENT. The generic probe creates its own namespace, service +# account and RBAC. That cannot run here: sandbox access is Rancher +# project-scoped, so a freshly created namespace belongs to no project and the +# creator has no rights inside it, and `create roles` is denied in every +# namespace. Verified with `kubectl auth can-i`, not assumed. +# +# It does not need them anyway. The `dagster-role` bound to dagster-svc-account +# already grants batch/jobs (full verbs), jobs/status, pods, events, and +# pods/log - exactly the set the pipes client needs. Readiness checks L5 and L6 +# therefore already pass on this cluster; read the live Role to confirm: +# +# kubectl -n dataprovider01 get role dagster-role -o yaml +# +# WHAT THIS COSTS. Unlike the generic probe this is NOT isolated. It borrows a +# live service account and leaves payload Jobs in a shared namespace until they +# are deleted. Nothing it does is persistent, but it is visible to anyone else +# working in dataprovider01. +# +# --- Credentials: none needed ---------------------------------------------- +# +# Both Gitea images are anonymously pullable. A bare GET to the registry returns +# 401, which looks like a refusal but is just the start of the Docker token +# handshake; completing it anonymously and re-requesting the manifest returns +# 200 for both images. Verify with: +# +# TOK=$(curl -sS "https://gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/v2/token?service=container_registry&scope=repository:j.r/distributed-execution:pull" | jq -r .token) +# curl -sS -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $TOK" \ +# https://gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/v2/j.r/distributed-execution/manifests/5122da4691f9 +# +# This also means the payload Jobs need nothing: they run as the `default` +# service account, which carries no pull secret, and do not need one. +# +# --- Running --------------------------------------------------------------- +# +# kubectl -n dataprovider01 apply -f yaml/sandbox/probe-pipes-k8s-sandbox.yaml +# kubectl -n dataprovider01 logs -f job/distexec-pipes-probe +# kubectl -n dataprovider01 get jobs -l app.kubernetes.io/name=distributed-execution-payload +# +# --- Teardown -------------------------------------------------------------- +# +# kubectl -n dataprovider01 delete job distexec-pipes-probe +# kubectl -n dataprovider01 delete jobs -l app.kubernetes.io/name=distributed-execution-payload +# +# VERIFIED 2026-08-31 on sandbox-cat-dat / dataprovider01. Run +# cff9b348-bfc3-4ac1-ab51-a94892b8e3a0 reached RUN_SUCCESS: four payload Jobs, +# four distinct payload pod hostnames in contributing_hosts, messages returned +# over the pod log stream. Checks L4-L10 pass. Torn down afterwards; the +# namespace was left as it was found. + +apiVersion: batch/v1 +kind: Job +metadata: + name: distexec-pipes-probe + namespace: dataprovider01 + labels: + app.kubernetes.io/name: distributed-execution + app.kubernetes.io/component: probe +spec: + backoffLimit: 0 + activeDeadlineSeconds: 1800 + template: + metadata: + labels: + app.kubernetes.io/name: distributed-execution + app.kubernetes.io/component: probe + spec: + restartPolicy: Never + # Already holds jobs + pods/log; this probe grants itself nothing. + serviceAccountName: dagster-svc-account + # dagster-svc-account sets automountServiceAccountToken: false, so every + # pod that uses it must opt back in - the chart does this for its own + # deployments. Without it the pipes client picks in-cluster auth correctly + # and then fails on "Service token file does not exist". + automountServiceAccountToken: true + securityContext: + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + fsGroup: 1000 + seccompProfile: + type: RuntimeDefault + containers: + - name: dispatcher + image: gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/j.r/distributed-execution:5122da4691f9 + imagePullPolicy: IfNotPresent + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + command: + - dagster + - job + - execute + - -f + - src/distributed_execution/repository.py + - -j + - loosely_coupled_k8s_job + env: + - name: DAGSTER_HOME + value: /dagster-home + - name: PIPES_PAYLOAD_IMAGE + value: gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/j.r/distributed-execution-payload:5122da4691f9 + - name: PIPES_PAYLOAD_NAMESPACE + value: dataprovider01 + volumeMounts: + - name: dagster-home + mountPath: /dagster-home + resources: + requests: + cpu: 200m + memory: 512Mi + limits: + cpu: "1" + memory: 1Gi + volumes: + - name: dagster-home + emptyDir: {} diff --git a/yaml/sandbox/values-sandbox-gitea.yaml b/yaml/sandbox/values-sandbox-gitea.yaml index 1d7be11..8ac1b30 100644 --- a/yaml/sandbox/values-sandbox-gitea.yaml +++ b/yaml/sandbox/values-sandbox-gitea.yaml @@ -1,8 +1,9 @@ # Sandbox deployment values - Gitea registry. # -# NOT CLUSTER-VERIFIED. These values are written against images that exist and a -# registry whose auth behaviour was checked, but they have not been applied to a -# Simpl cluster. Confirm and amend after the first deployment. +# NOT CLUSTER-VERIFIED. The images and the registry behaviour here are confirmed +# - a probe pod ran from them on the sandbox on 2026-08-31 - but these values +# have never been applied as a code location. Registering this service against +# the sandbox Dagster is still blocked on the version skew below. # # VERSION SKEW, unresolved. These images carry dagster 1.13.19 (requirements.txt # pins no upper bound, so the build took whatever was current). The sandbox @@ -21,39 +22,18 @@ # the version lock: the code location and the payload it dispatches must come # from one commit. Bump both together or not at all. # -# The sandbox Gitea registry requires authentication - an anonymous manifest GET -# returns 401 - so a pull secret is required in every namespace that pulls either -# image. Create it with a Gitea access token that has read:package scope: +# The Gitea images are anonymously pullable, so NO pull secret is needed. A bare +# GET to the registry returns 401, which reads as a refusal but is only the start +# of the Docker token handshake; completing it anonymously and re-requesting the +# manifest returns 200 for both images. This was confirmed on 2026-08-31, and a +# probe pod carrying no credentials pulled and ran on the sandbox. # -# kubectl -n create secret docker-registry gitea-registry \ -# --docker-server=gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu \ -# --docker-username= \ -# --docker-password= -# -# The payload namespace needs it too: PipesK8sClient creates that Job, and a -# missing pull secret there leaves the op waiting on a pod that never starts, -# which surfaces as pod_wait_timeout rather than as an image error. -# -# PULL SECRET PLACEMENT. Checked against the chart templates in dagster 1.12.8, -# which is what the sandbox runs. There is no per-deployment imagePullSecrets -# field - deployment-user.yaml reads the SUBCHART value and the run launcher -# helper reads the PARENT value - so a pull secret nested under a deployments[] -# entry is silently discarded, and the pod fails with ImagePullBackOff against a -# values file that looks correct. -# -# Both keys below therefore apply to every code location in the release, not -# just this one. Extra pull secrets are ignored for registries they do not match, -# so the existing code locations are unaffected. +# An earlier revision of this file required a pull secret on both the deployment +# and the run launcher. That was wrong, and wrong in an expensive direction: it +# sent the reader looking for credentials that do not exist. dagster: - # Run pods, webserver and daemon. - imagePullSecrets: - - name: gitea-registry - dagster-user-deployments: - # Code server pods. - imagePullSecrets: - - name: gitea-registry deployments: - name: distributed-execution image: @@ -63,8 +43,10 @@ dagster: env: - name: PIPES_PAYLOAD_IMAGE value: gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/j.r/distributed-execution-payload:5122da4691f9 + # The sandbox release is named `dagster` but lives in dataprovider01; + # there is no namespace called `dagster` on that cluster. - name: PIPES_PAYLOAD_NAMESPACE - value: dagster + value: dataprovider01 runLauncher: config: @@ -75,4 +57,4 @@ dagster: - name: PIPES_PAYLOAD_IMAGE value: gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/j.r/distributed-execution-payload:5122da4691f9 - name: PIPES_PAYLOAD_NAMESPACE - value: dagster + value: dataprovider01