j.r/distributed-execution
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[SIMPL-30451] Add a standalone cluster probe for the pipes transport

Browse Source 
Runs loosely_coupled_k8s_job from one throwaway pod with only the RBAC the pipes client needs, clearing readiness checks L4-L9 without deploying a code location, a webserver or a database. Because the pod is its own control plane it also sidesteps the 1.13.19 vs 1.12.8 skew that blocks registering this service against the sandbox Dagster.

Verified in the container beforehand: the CLI form reaches RUN_SUCCESS, the multiprocess executor fans out to four dynamic steps, and PipesK8sClient selects in-cluster credentials from KUBERNETES_SERVICE_HOST, so no code change is needed to run it in a pod.

Changelog: added
This commit is contained in:
ILay
2026-08-31 17:46:37 +02:00
parent b47c530d46
commit 515eef7736
3 changed files with 176 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
documents/user-guide/distributed-execution-guide.md
View File

@@ -397,6 +397,7 @@ setting. There is no UI control that switches a workflow between the two pattern
| `yaml/loosely-coupled/values-pipes-payload.yaml` | Payload image reference and target namespace |
| `yaml/sandbox/values-sandbox-gitea.yaml` | Sandbox registry coordinates, pull secret, and the SHA both images share |
| `yaml/loosely-coupled/rbac-pipes-dispatch.yaml` | Permissions required by the dispatching pod |
| `yaml/loosely-coupled/probe-pipes-k8s.yaml` | Standalone cluster probe for the pipes transport (checklist L4–L9) |
| `yaml/values-dagster-distributed-execution.yaml` | Code location image and entry point |
The Launchpad can override *run configuration* — op config, resource config, tags

5
documents/user-guide/readiness-checklist.md
View File

@@ -58,6 +58,11 @@ location image.
Checks L4–L9 require a cluster. L1–L3, L10 and L11 run on a laptop and should
gate every change to the payload or the dispatching op.
`yaml/loosely-coupled/probe-pipes-k8s.yaml` clears L4–L9 in a single run from a
throwaway namespace, without deploying a code location or a Dagster control
plane. Prefer it over assembling the cluster checks by hand: the RBAC it grants
is exactly the set L5 and L6 ask about, so a failure localises immediately.
---
## 4. Common misconfiguration symptoms