diff --git a/documents/user-guide/readiness-checklist.md b/documents/user-guide/readiness-checklist.md index d158d46..cf8f5b7 100644 --- a/documents/user-guide/readiness-checklist.md +++ b/documents/user-guide/readiness-checklist.md @@ -10,9 +10,9 @@ location image. > **L4–L10 were cleared on sandbox-cat-dat on 2026-08-31.** Checks L1–L3 and L11, > which exercise the payload contract, the message-parsing path and image-level > isolation, are verified locally. The **tightly coupled** Kubernetes path has -> been assessed against the same cluster but not executed — T2, T3, T5 and T8 -> are confirmed from the live instance configuration and C6 fails there. See -> [section 2.1](#21-sandbox-state-2026-08-31). +> been assessed against the same cluster but not executed — C5, C6, T2, T3, T5 +> and T8 are confirmed from the live instance configuration and the deployed code +> locations. See [section 2.1](#21-sandbox-state-2026-08-31). --- @@ -24,8 +24,8 @@ location image. | C2 | Jobs are registered | Dagster UI → **Jobs** | The jobs listed in the guide's section 5.5 appear under the code location | | C3 | Image tag matches the intended release | `kubectl -n dagster get deploy -l dagster/code-location=distributed-execution -o jsonpath='{.items[*].spec.template.spec.containers[*].image}'` | Tag equals the version in `pipeline.variables.sh`; never `latest` | | C4 | Image architecture matches the nodes | `docker manifest inspect ` | Includes `linux/amd64`; a manifest with only `linux/arm64` produces `no match for platform` at pull time | -| C5 | Run launcher type is as intended | `kubectl -n get cm dagster-instance -o yaml` | `run_launcher` block shows `K8sRunLauncher` and the expected `job_namespace` | -| C6 | Target namespace exists and is schedulable | `kubectl get ns ` and `kubectl -n get resourcequota` | Namespace is `Active`; remaining quota exceeds the job's aggregate requests. Also confirm the launcher's `instance_config_map`, `postgres_password_secret` and any PVC volumes exist **in that same namespace** — these references do not cross namespaces, and on sandbox-cat-dat they do not match `job_namespace` (see section 2.1) | +| C5 | Run launcher type is as intended | `kubectl -n get cm dagster-instance -o yaml` | `run_launcher` block shows `K8sRunLauncher`. Note that `job_namespace` here is only a default — a code location's `DAGSTER_CONTAINER_CONTEXT` overrides it, so read that too before concluding where run pods land | +| C6 | Target namespace exists and is schedulable | `kubectl get ns ` and `kubectl -n get resourcequota` | Namespace is `Active`; remaining quota exceeds the job's aggregate requests. Also confirm the launcher's `instance_config_map`, `postgres_password_secret` and any PVC volumes exist **in the namespace run pods actually use** — these references do not cross namespaces (see section 2.1) | ## 2. Tightly coupled checks @@ -54,28 +54,42 @@ What the configuration shows: | # | Finding | |---|---| -| C5 | `run_launcher` is `K8sRunLauncher` with `job_namespace: dagster` | +| C5 | `run_launcher` is `K8sRunLauncher`; `job_namespace: dagster` is the default, overridden per code location to `dataprovider01` | | T2 | Vault injection is configured — `pod_template_spec_metadata` carries the banzaicloud annotations with role `sandbox-cat-dat-role` | | T3 | `S3_ENDPOINT_URL` is `https://s3.sandbox-cat-dat.simpl-europe.eu`, with access keys injected from Vault | | T5 | Passes. `dagster-role`, bound to `dagster-svc-account`, grants `batch/jobs` with create, delete, get, list, patch, update and watch | | T8 | `fail_pod_on_run_failure: true` is set | -**C6 fails, and it is the reason nothing has ever run here.** The launcher sends -run pods to namespace `dagster`, but every namespace-local dependency it names — -`instance_config_map: dagster-instance`, `postgres_password_secret: -dagster-postgresql-secret`, and the `dagster-shared-pvc` volume — exists in -`dataprovider01`. A ConfigMap, Secret or PVC reference resolves only within the -pod's own namespace, so a run pod placed in `dagster` would reference three -objects that are not there. Consistent with that, `dataprovider01` holds no pods -labelled `dagster/run-id` and the daemon log shows no launch activity. +**C6 passes, contrary to what `job_namespace` alone suggests.** The launcher's +`job_namespace: dagster` is only a default. Each code location ships a +`DAGSTER_CONTAINER_CONTEXT` environment variable — generated by the chart — whose +`k8s.namespace` is `dataprovider01`, and the code location's context is merged +*over* the launcher's, so it wins (`dagster_k8s/container_context.py`: `namespace += other.namespace if other.namespace else self.namespace`). Run pods therefore +land in `dataprovider01`, where `dagster-instance`, `dagster-postgresql-secret` +and `dagster-shared-pvc` all exist. -One caveat on the evidence: project-scoped access means `kubectl get ns dagster` -returns `Forbidden` rather than `NotFound`, so whether that namespace exists is -unconfirmed. It does not change the conclusion — the dependencies are in the -wrong namespace either way. +Both deployed code locations carry it, so both would launch correctly: -This is a platform configuration defect, not a defect in this service. It is -exactly what C6 exists to catch. +| Code location | `DAGSTER_CONTAINER_CONTEXT` `k8s.namespace` | +|---|---| +| `template-code-location` | `dataprovider01` | +| `semaphore-ui` | `dataprovider01` | + +What is still true is that **no run has ever executed on this cluster** — +`dataprovider01` holds no pods labelled `dagster/run-id` and the daemon log shows +no launch activity. That appears to be because nothing has been launched, not +because anything is broken. + +The residual risk is narrower than a broken namespace, but real: a code location +that reaches the launcher *without* a container context namespace inherits +`dagster` and would fail on three missing objects. The chart always sets it, so +this bites only hand-rolled pods — the reason the C6 row asks you to check the +launcher's namespace-local dependencies rather than just the namespace. + +> An earlier revision of this section claimed C6 failed and that this explained +> the absence of runs. That was wrong: it read `job_namespace` without accounting +> for the container-context override. ## 3. Loosely coupled checks