[SIMPL-30451] Verify the Kubernetes pipes transport on the sandbox

Run cff9b348-bfc3-4ac1-ab51-a94892b8e3a0 of loosely_coupled_k8s_job reached RUN_SUCCESS in dataprovider01 on sandbox-cat-dat: four payload Jobs, four distinct payload pod hostnames in contributing_hosts, and the payload log lines in the run log. Checks L4-L10 are now observed rather than derived. Torn down afterwards; the namespace was left as it was found.

Corrects two claims that were wrong. The registry does NOT require a pull secret - a bare GET returns 401, but that is the opening move of the Docker token handshake, and completing it anonymously returns the manifest for both images. And PIPES_PAYLOAD_NAMESPACE was set to a namespace named dagster, which does not exist on that cluster; the release is called dagster but runs in dataprovider01.

Adds the sandbox probe variant. The generic probe cannot run there: sandbox access is Rancher project-scoped, so a new namespace grants its creator nothing and Role creation is denied everywhere. It is not needed either, since dagster-role already carries the exact pipes permissions.

Changelog: added

yaml/loosely-coupled/probe-pipes-k8s.yaml

@@ -12,33 +12,24 @@
 #
 # --- Before applying -------------------------------------------------------
 #
-# 1. Pull secret. The registry rejects anonymous pulls, so both this pod and the
-#    payload Jobs it creates need credentials. Use a token with read:package.
+# Credentials, only if your registry needs them. The Gitea images referenced
+# below do NOT: a bare GET returns 401, but that is the start of the Docker
+# token handshake, and completing it anonymously returns the manifest. A probe
+# pod carrying no credentials pulled and ran on the sandbox on 2026-08-31.
 #
-#      kubectl -n distexec-probe create secret docker-registry gitea-registry \
-#        --docker-server=gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu \
-#        --docker-username=<gitea-user> \
-#        --docker-password=<gitea-token>
-#
-# 2. Payload pods inherit the pull secret from the default service account.
-#    PipesK8sClient does not set serviceAccountName on the Jobs it creates, so
-#    they run as `default`, and this service does not yet pass imagePullSecrets
-#    through base_pod_spec:
-#
-#      kubectl -n distexec-probe patch serviceaccount default \
-#        -p '{"imagePullSecrets":[{"name":"gitea-registry"}]}'
-#
-#    Skip this and the payload pod never starts. That surfaces as the op waiting
-#    until pod_wait_timeout, which reads like a hung workload rather than a
-#    missing credential.
-#
-# Order does not matter much: apply this file first and the dispatcher pod sits
-# in ImagePullBackOff until the secret appears, then pulls on the kubelet's next
-# retry. The default service account only has to be patched before the first
-# payload Job is created, which is after the dispatcher starts.
+# If you point this at a registry that does require authentication, remember the
+# payload Jobs need it too. PipesK8sClient sets no serviceAccountName, so they
+# run as `default` and inherit nothing from the pod below; patch that account or
+# pass imagePullSecrets through base_pod_spec in dispatch_external_work_k8s. A
+# payload pod that cannot pull leaves the op waiting until pod_wait_timeout,
+# which reads like a hung workload rather than a missing credential.
 #
 #      kubectl apply -f yaml/loosely-coupled/probe-pipes-k8s.yaml
 #
+# On sandbox-cat-dat use yaml/sandbox/probe-pipes-k8s-sandbox.yaml instead: that
+# cluster's access is Rancher project-scoped, so this file's namespace and RBAC
+# cannot be created there. The sandbox variant needs neither.
+#
 # `--dry-run=server` reports "namespaces distexec-probe not found" for the four
 # namespaced objects. That is the dry run declining to create the namespace it
 # would need, not a fault in the manifests; `--dry-run=client` passes clean.
@@ -134,6 +125,10 @@ spec:
     spec:
       restartPolicy: Never
       serviceAccountName: pipes-dispatcher
+      # A service account with automountServiceAccountToken: false - which the
+      # platform's dagster-svc-account uses - leaves the pipes client selecting
+      # in-cluster auth and then failing on "Service token file does not exist".
+      automountServiceAccountToken: true
       securityContext:
         runAsNonRoot: true
         runAsUser: 1000