3 Commits

Author SHA1 Message Date
ILay
96caa3f81b [SIMPL-30451] Verify the Kubernetes pipes transport on the sandbox
Run cff9b348-bfc3-4ac1-ab51-a94892b8e3a0 of loosely_coupled_k8s_job reached RUN_SUCCESS in dataprovider01 on sandbox-cat-dat: four payload Jobs, four distinct payload pod hostnames in contributing_hosts, and the payload log lines in the run log. Checks L4-L10 are now observed rather than derived. Torn down afterwards; the namespace was left as it was found.

Corrects two claims that were wrong. The registry does NOT require a pull secret - a bare GET returns 401, but that is the opening move of the Docker token handshake, and completing it anonymously returns the manifest for both images. And PIPES_PAYLOAD_NAMESPACE was set to a namespace named dagster, which does not exist on that cluster; the release is called dagster but runs in dataprovider01.

Adds the sandbox probe variant. The generic probe cannot run there: sandbox access is Rancher project-scoped, so a new namespace grants its creator nothing and Role creation is denied everywhere. It is not needed either, since dagster-role already carries the exact pipes permissions.

Changelog: added
2026-08-31 19:41:26 +02:00
ILay
e686df5710 [SIMPL-30451] Harden the probe pod against restricted Pod Security
The int cluster did not enforce Pod Security when this was written - no namespace labels, no Kyverno or Gatekeeper. The sandbox was NOT checked; there is no kubeconfig context for it. A cluster that does enforce it rejects the pod outright rather than degrading, which is slow to diagnose from an unrelated namespace.

Records the matching gap it cannot fix: PipesK8sClient builds the payload pod specs, and this service passes no securityContext through base_pod_spec, so a restricted namespace would admit the dispatcher and reject every payload it creates.

Changelog: changed
2026-08-31 18:41:03 +02:00
ILay
515eef7736 [SIMPL-30451] Add a standalone cluster probe for the pipes transport
Runs loosely_coupled_k8s_job from one throwaway pod with only the RBAC the pipes client needs, clearing readiness checks L4-L9 without deploying a code location, a webserver or a database. Because the pod is its own control plane it also sidesteps the 1.13.19 vs 1.12.8 skew that blocks registering this service against the sandbox Dagster.

Verified in the container beforehand: the CLI form reaches RUN_SUCCESS, the multiprocess executor fans out to four dynamic steps, and PipesK8sClient selects in-cluster credentials from KUBERNETES_SERVICE_HOST, so no code change is needed to run it in a pod.

Changelog: added
2026-08-31 17:46:37 +02:00