From 18cc46d132184e2f48dda1995cf35c460c82f6f5 Mon Sep 17 00:00:00 2001 From: ILay Date: Mon, 31 Aug 2026 19:57:38 +0200 Subject: [PATCH] [SIMPL-30451] Assess the tightly coupled path against the sandbox T2, T3, T5 and T8 are confirmed from the live dagster-instance ConfigMap and from dagster-role. None of the T checks could be executed: the tightly coupled path needs this code location registered on the platform Dagster, and unlike the pipes transport it cannot run from a standalone pod, because every step pod connects to the metadata database itself and so they must share instance storage rather than a local SQLite file. C6 fails on that cluster and explains why no run has ever executed there. The launcher sends run pods to namespace dagster while its instance_config_map, postgres_password_secret and dagster-shared-pvc all live in dataprovider01; those references do not cross namespaces. dataprovider01 holds no pods labelled dagster/run-id and the daemon log shows no launch activity. C6 now tells the reader to check the launcher's namespace-local dependencies, not just the namespace. Changelog: changed --- documents/user-guide/readiness-checklist.md | 48 ++++++++++++++++++--- 1 file changed, 43 insertions(+), 5 deletions(-) diff --git a/documents/user-guide/readiness-checklist.md b/documents/user-guide/readiness-checklist.md index 9d4e928..d158d46 100644 --- a/documents/user-guide/readiness-checklist.md +++ b/documents/user-guide/readiness-checklist.md @@ -9,9 +9,10 @@ location image. > **L4–L10 were cleared on sandbox-cat-dat on 2026-08-31.** Checks L1–L3 and L11, > which exercise the payload contract, the message-parsing path and image-level -> isolation, are verified locally. What remains unverified on a cluster is the -> **tightly coupled** Kubernetes path — the T rows and the Kubernetes rows of -> section 4.3 — which uses `k8s_job_executor` rather than the pipes client. +> isolation, are verified locally. The **tightly coupled** Kubernetes path has +> been assessed against the same cluster but not executed — T2, T3, T5 and T8 +> are confirmed from the live instance configuration and C6 fails there. See +> [section 2.1](#21-sandbox-state-2026-08-31). --- @@ -23,8 +24,8 @@ location image. | C2 | Jobs are registered | Dagster UI → **Jobs** | The jobs listed in the guide's section 5.5 appear under the code location | | C3 | Image tag matches the intended release | `kubectl -n dagster get deploy -l dagster/code-location=distributed-execution -o jsonpath='{.items[*].spec.template.spec.containers[*].image}'` | Tag equals the version in `pipeline.variables.sh`; never `latest` | | C4 | Image architecture matches the nodes | `docker manifest inspect ` | Includes `linux/amd64`; a manifest with only `linux/arm64` produces `no match for platform` at pull time | -| C5 | Run launcher type is as intended | `kubectl -n dagster get cm dagster-instance -o yaml` | `run_launcher` block shows `K8sRunLauncher` and the expected `job_namespace` | -| C6 | Target namespace exists and is schedulable | `kubectl get ns ` and `kubectl -n get resourcequota` | Namespace is `Active`; remaining quota exceeds the job's aggregate requests | +| C5 | Run launcher type is as intended | `kubectl -n get cm dagster-instance -o yaml` | `run_launcher` block shows `K8sRunLauncher` and the expected `job_namespace` | +| C6 | Target namespace exists and is schedulable | `kubectl get ns ` and `kubectl -n get resourcequota` | Namespace is `Active`; remaining quota exceeds the job's aggregate requests. Also confirm the launcher's `instance_config_map`, `postgres_password_secret` and any PVC volumes exist **in that same namespace** — these references do not cross namespaces, and on sandbox-cat-dat they do not match `job_namespace` (see section 2.1) | ## 2. Tightly coupled checks @@ -39,6 +40,43 @@ location image. | T7 | Step pod egress is permitted | Same run | Steps do not hang in `STARTING`; run logs contain no connection timeouts to port 5432 | | T8 | Failure surfaces as a pod failure | Force a step failure in a scratch namespace | `failPodOnRunFailure: true` is set, and the step pod reports `Failed` rather than `Completed` | +### 2.1 Sandbox state, 2026-08-31 + +The T checks were assessed against sandbox-cat-dat by reading the live +`dagster-instance` ConfigMap in `dataprovider01`. None of them could be *executed*: +the tightly coupled Kubernetes path needs this code location registered on the +platform Dagster, which is blocked on the control plane skew, and unlike the +pipes transport it cannot be exercised from a standalone pod — every step pod +connects to the metadata database on its own, so they must share instance +storage rather than a local SQLite file. + +What the configuration shows: + +| # | Finding | +|---|---| +| C5 | `run_launcher` is `K8sRunLauncher` with `job_namespace: dagster` | +| T2 | Vault injection is configured — `pod_template_spec_metadata` carries the banzaicloud annotations with role `sandbox-cat-dat-role` | +| T3 | `S3_ENDPOINT_URL` is `https://s3.sandbox-cat-dat.simpl-europe.eu`, with access keys injected from Vault | +| T5 | Passes. `dagster-role`, bound to `dagster-svc-account`, grants `batch/jobs` with create, delete, get, list, patch, update and watch | +| T8 | `fail_pod_on_run_failure: true` is set | + +**C6 fails, and it is the reason nothing has ever run here.** The launcher sends +run pods to namespace `dagster`, but every namespace-local dependency it names — +`instance_config_map: dagster-instance`, `postgres_password_secret: +dagster-postgresql-secret`, and the `dagster-shared-pvc` volume — exists in +`dataprovider01`. A ConfigMap, Secret or PVC reference resolves only within the +pod's own namespace, so a run pod placed in `dagster` would reference three +objects that are not there. Consistent with that, `dataprovider01` holds no pods +labelled `dagster/run-id` and the daemon log shows no launch activity. + +One caveat on the evidence: project-scoped access means `kubectl get ns dagster` +returns `Forbidden` rather than `NotFound`, so whether that namespace exists is +unconfirmed. It does not change the conclusion — the dependencies are in the +wrong namespace either way. + +This is a platform configuration defect, not a defect in this service. It is +exactly what C6 exists to catch. + ## 3. Loosely coupled checks | # | Check | How to verify | Expected evidence |