diff --git a/documents/user-guide/distributed-execution-guide.md b/documents/user-guide/distributed-execution-guide.md index ab2b94f..8c64d45 100644 --- a/documents/user-guide/distributed-execution-guide.md +++ b/documents/user-guide/distributed-execution-guide.md @@ -397,6 +397,7 @@ setting. There is no UI control that switches a workflow between the two pattern | `yaml/loosely-coupled/values-pipes-payload.yaml` | Payload image reference and target namespace | | `yaml/sandbox/values-sandbox-gitea.yaml` | Sandbox registry coordinates, pull secret, and the SHA both images share | | `yaml/loosely-coupled/rbac-pipes-dispatch.yaml` | Permissions required by the dispatching pod | +| `yaml/loosely-coupled/probe-pipes-k8s.yaml` | Standalone cluster probe for the pipes transport (checklist L4–L9) | | `yaml/values-dagster-distributed-execution.yaml` | Code location image and entry point | The Launchpad can override *run configuration* — op config, resource config, tags diff --git a/documents/user-guide/readiness-checklist.md b/documents/user-guide/readiness-checklist.md index f187b43..7995271 100644 --- a/documents/user-guide/readiness-checklist.md +++ b/documents/user-guide/readiness-checklist.md @@ -58,6 +58,11 @@ location image. Checks L4–L9 require a cluster. L1–L3, L10 and L11 run on a laptop and should gate every change to the payload or the dispatching op. +`yaml/loosely-coupled/probe-pipes-k8s.yaml` clears L4–L9 in a single run from a +throwaway namespace, without deploying a code location or a Dagster control +plane. Prefer it over assembling the cluster checks by hand: the RBAC it grants +is exactly the set L5 and L6 ask about, so a failure localises immediately. + --- ## 4. Common misconfiguration symptoms diff --git a/yaml/loosely-coupled/probe-pipes-k8s.yaml b/yaml/loosely-coupled/probe-pipes-k8s.yaml new file mode 100644 index 0000000..d35cb9c --- /dev/null +++ b/yaml/loosely-coupled/probe-pipes-k8s.yaml @@ -0,0 +1,170 @@ +# Cluster probe for the loosely coupled Kubernetes transport. +# +# Runs `loosely_coupled_k8s_job` from a single throwaway pod. The point is to +# prove the part of this service that no test can reach: that the dispatching +# process creates payload Jobs and that messages come back over the pod log +# stream. It clears readiness checks L4-L9 in one run. +# +# Deliberately NOT a Dagster deployment. No webserver, no daemon, no database, +# no code location registration. That sidesteps the control-plane version skew - +# the pod is its own control plane, so the 1.13.19 images here have nothing to be +# compatible with. Everything lives in one namespace you delete afterwards. +# +# --- Before applying ------------------------------------------------------- +# +# 1. Pull secret. The registry rejects anonymous pulls, so both this pod and the +# payload Jobs it creates need credentials. Use a token with read:package. +# +# kubectl -n distexec-probe create secret docker-registry gitea-registry \ +# --docker-server=gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu \ +# --docker-username= \ +# --docker-password= +# +# 2. Payload pods inherit the pull secret from the default service account. +# PipesK8sClient does not set serviceAccountName on the Jobs it creates, so +# they run as `default`, and this service does not yet pass imagePullSecrets +# through base_pod_spec: +# +# kubectl -n distexec-probe patch serviceaccount default \ +# -p '{"imagePullSecrets":[{"name":"gitea-registry"}]}' +# +# Skip this and the payload pod never starts. That surfaces as the op waiting +# until pod_wait_timeout, which reads like a hung workload rather than a +# missing credential. +# +# Order does not matter much: apply this file first and the dispatcher pod sits +# in ImagePullBackOff until the secret appears, then pulls on the kubelet's next +# retry. The default service account only has to be patched before the first +# payload Job is created, which is after the dispatcher starts. +# +# kubectl apply -f yaml/loosely-coupled/probe-pipes-k8s.yaml +# +# `--dry-run=server` reports "namespaces distexec-probe not found" for the four +# namespaced objects. That is the dry run declining to create the namespace it +# would need, not a fault in the manifests; `--dry-run=client` passes clean. +# +# --- Reading the result ---------------------------------------------------- +# +# kubectl -n distexec-probe logs -f job/pipes-probe +# kubectl -n distexec-probe get jobs -l app.kubernetes.io/name=distributed-execution-payload +# +# Success is RUN_SUCCESS plus four payload Jobs, one per work unit. The +# summarise_results output carries contributing_workers; on a cluster those +# should be four distinct pod hostnames, which is the claim the guide makes and +# a laptop cannot demonstrate. +# +# A run that reaches RUN_FAILURE with "no pipes messages received" means the +# payload ran but its log stream never came back - check the pods/log rule +# below before suspecting the payload. +# +# --- Teardown -------------------------------------------------------------- +# +# kubectl delete namespace distexec-probe +# +# NOT CLUSTER-VERIFIED. Written against the published images and the RBAC the +# pipes client is documented to need, but not yet applied to a cluster. + +apiVersion: v1 +kind: Namespace +metadata: + name: distexec-probe + labels: + app.kubernetes.io/name: distributed-execution + app.kubernetes.io/component: probe +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: pipes-dispatcher + namespace: distexec-probe +imagePullSecrets: + - name: gitea-registry +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: pipes-dispatcher + namespace: distexec-probe +rules: + - apiGroups: ["batch"] + resources: ["jobs", "jobs/status"] + verbs: ["create", "get", "list", "watch", "delete"] + # pods/log is the message channel for PipesK8sPodLogsMessageReader, not just + # an observability convenience. Without it the payload runs to completion and + # reports nothing, and the op fails on an empty message list. + - apiGroups: [""] + resources: ["pods", "pods/log", "pods/status"] + verbs: ["get", "list", "watch"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: pipes-dispatcher + namespace: distexec-probe +subjects: + - kind: ServiceAccount + name: pipes-dispatcher + namespace: distexec-probe +roleRef: + kind: Role + name: pipes-dispatcher + apiGroup: rbac.authorization.k8s.io +--- +apiVersion: batch/v1 +kind: Job +metadata: + name: pipes-probe + namespace: distexec-probe +spec: + # One shot. A retry would obscure which attempt produced which payload Jobs. + backoffLimit: 0 + activeDeadlineSeconds: 1800 + template: + metadata: + labels: + app.kubernetes.io/name: distributed-execution + app.kubernetes.io/component: probe + spec: + restartPolicy: Never + serviceAccountName: pipes-dispatcher + securityContext: + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + # Makes the emptyDir writable by UID 1000, which DAGSTER_HOME needs. + fsGroup: 1000 + containers: + - name: dispatcher + image: gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/j.r/distributed-execution:5122da4691f9 + imagePullPolicy: IfNotPresent + command: + - dagster + - job + - execute + - -f + - src/distributed_execution/repository.py + - -j + - loosely_coupled_k8s_job + env: + - name: DAGSTER_HOME + value: /dagster-home + # Must be the same commit as the image above. + - name: PIPES_PAYLOAD_IMAGE + value: gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/j.r/distributed-execution-payload:5122da4691f9 + - name: PIPES_PAYLOAD_NAMESPACE + value: distexec-probe + volumeMounts: + - name: dagster-home + mountPath: /dagster-home + resources: + requests: + cpu: 200m + memory: 512Mi + limits: + cpu: "1" + memory: 1Gi + volumes: + # The job runs on multiprocess_executor, so the instance has to be a real + # sqlite instance on disk rather than an ephemeral one. + - name: dagster-home + emptyDir: {}