[SIMPL-30451] Correct the C6 finding: the container context overrides job_namespace

The previous revision claimed C6 failed on the sandbox because job_namespace is dagster while the launcher's ConfigMap, Secret and PVC live in dataprovider01. That read job_namespace in isolation. Each code location ships a DAGSTER_CONTAINER_CONTEXT whose k8s.namespace is dataprovider01, and the code location context is merged over the launcher's, so it wins - dagster_k8s/container_context.py resolves namespace as 'other.namespace if other.namespace else self.namespace'. Both deployed code locations carry it, so run pods land where their dependencies are.

Still true: no run has ever executed on that cluster. That now looks like nothing having been launched rather than anything being broken. The residual risk is a code location reaching the launcher without a container context namespace, which the chart always sets, so it bites only hand-rolled pods.

C5 now warns that job_namespace is a default rather than the effective namespace.

Changelog: fixed
This commit is contained in:
ILay
2026-08-31 20:09:50 +02:00
parent 18cc46d132
commit 2bbecb1bdd

View File

@@ -10,9 +10,9 @@ location image.
> **L4L10 were cleared on sandbox-cat-dat on 2026-08-31.** Checks L1L3 and L11, > **L4L10 were cleared on sandbox-cat-dat on 2026-08-31.** Checks L1L3 and L11,
> which exercise the payload contract, the message-parsing path and image-level > which exercise the payload contract, the message-parsing path and image-level
> isolation, are verified locally. The **tightly coupled** Kubernetes path has > isolation, are verified locally. The **tightly coupled** Kubernetes path has
> been assessed against the same cluster but not executed — T2, T3, T5 and T8 > been assessed against the same cluster but not executed — C5, C6, T2, T3, T5
> are confirmed from the live instance configuration and C6 fails there. See > and T8 are confirmed from the live instance configuration and the deployed code
> [section 2.1](#21-sandbox-state-2026-08-31). > locations. See [section 2.1](#21-sandbox-state-2026-08-31).
--- ---
@@ -24,8 +24,8 @@ location image.
| C2 | Jobs are registered | Dagster UI → **Jobs** | The jobs listed in the guide's section 5.5 appear under the code location | | C2 | Jobs are registered | Dagster UI → **Jobs** | The jobs listed in the guide's section 5.5 appear under the code location |
| C3 | Image tag matches the intended release | `kubectl -n dagster get deploy -l dagster/code-location=distributed-execution -o jsonpath='{.items[*].spec.template.spec.containers[*].image}'` | Tag equals the version in `pipeline.variables.sh`; never `latest` | | C3 | Image tag matches the intended release | `kubectl -n dagster get deploy -l dagster/code-location=distributed-execution -o jsonpath='{.items[*].spec.template.spec.containers[*].image}'` | Tag equals the version in `pipeline.variables.sh`; never `latest` |
| C4 | Image architecture matches the nodes | `docker manifest inspect <image>` | Includes `linux/amd64`; a manifest with only `linux/arm64` produces `no match for platform` at pull time | | C4 | Image architecture matches the nodes | `docker manifest inspect <image>` | Includes `linux/amd64`; a manifest with only `linux/arm64` produces `no match for platform` at pull time |
| C5 | Run launcher type is as intended | `kubectl -n <namespace> get cm dagster-instance -o yaml` | `run_launcher` block shows `K8sRunLauncher` and the expected `job_namespace` | | C5 | Run launcher type is as intended | `kubectl -n <namespace> get cm dagster-instance -o yaml` | `run_launcher` block shows `K8sRunLauncher`. Note that `job_namespace` here is only a default — a code location's `DAGSTER_CONTAINER_CONTEXT` overrides it, so read that too before concluding where run pods land |
| C6 | Target namespace exists and is schedulable | `kubectl get ns <namespace>` and `kubectl -n <namespace> get resourcequota` | Namespace is `Active`; remaining quota exceeds the job's aggregate requests. Also confirm the launcher's `instance_config_map`, `postgres_password_secret` and any PVC volumes exist **in that same namespace** — these references do not cross namespaces, and on sandbox-cat-dat they do not match `job_namespace` (see section 2.1) | | C6 | Target namespace exists and is schedulable | `kubectl get ns <namespace>` and `kubectl -n <namespace> get resourcequota` | Namespace is `Active`; remaining quota exceeds the job's aggregate requests. Also confirm the launcher's `instance_config_map`, `postgres_password_secret` and any PVC volumes exist **in the namespace run pods actually use** — these references do not cross namespaces (see section 2.1) |
## 2. Tightly coupled checks ## 2. Tightly coupled checks
@@ -54,28 +54,42 @@ What the configuration shows:
| # | Finding | | # | Finding |
|---|---| |---|---|
| C5 | `run_launcher` is `K8sRunLauncher` with `job_namespace: dagster` | | C5 | `run_launcher` is `K8sRunLauncher`; `job_namespace: dagster` is the default, overridden per code location to `dataprovider01` |
| T2 | Vault injection is configured — `pod_template_spec_metadata` carries the banzaicloud annotations with role `sandbox-cat-dat-role` | | T2 | Vault injection is configured — `pod_template_spec_metadata` carries the banzaicloud annotations with role `sandbox-cat-dat-role` |
| T3 | `S3_ENDPOINT_URL` is `https://s3.sandbox-cat-dat.simpl-europe.eu`, with access keys injected from Vault | | T3 | `S3_ENDPOINT_URL` is `https://s3.sandbox-cat-dat.simpl-europe.eu`, with access keys injected from Vault |
| T5 | Passes. `dagster-role`, bound to `dagster-svc-account`, grants `batch/jobs` with create, delete, get, list, patch, update and watch | | T5 | Passes. `dagster-role`, bound to `dagster-svc-account`, grants `batch/jobs` with create, delete, get, list, patch, update and watch |
| T8 | `fail_pod_on_run_failure: true` is set | | T8 | `fail_pod_on_run_failure: true` is set |
**C6 fails, and it is the reason nothing has ever run here.** The launcher sends **C6 passes, contrary to what `job_namespace` alone suggests.** The launcher's
run pods to namespace `dagster`, but every namespace-local dependency it names — `job_namespace: dagster` is only a default. Each code location ships a
`instance_config_map: dagster-instance`, `postgres_password_secret: `DAGSTER_CONTAINER_CONTEXT` environment variable — generated by the chart — whose
dagster-postgresql-secret`, and the `dagster-shared-pvc` volume — exists in `k8s.namespace` is `dataprovider01`, and the code location's context is merged
`dataprovider01`. A ConfigMap, Secret or PVC reference resolves only within the *over* the launcher's, so it wins (`dagster_k8s/container_context.py`: `namespace
pod's own namespace, so a run pod placed in `dagster` would reference three = other.namespace if other.namespace else self.namespace`). Run pods therefore
objects that are not there. Consistent with that, `dataprovider01` holds no pods land in `dataprovider01`, where `dagster-instance`, `dagster-postgresql-secret`
labelled `dagster/run-id` and the daemon log shows no launch activity. and `dagster-shared-pvc` all exist.
One caveat on the evidence: project-scoped access means `kubectl get ns dagster` Both deployed code locations carry it, so both would launch correctly:
returns `Forbidden` rather than `NotFound`, so whether that namespace exists is
unconfirmed. It does not change the conclusion — the dependencies are in the
wrong namespace either way.
This is a platform configuration defect, not a defect in this service. It is | Code location | `DAGSTER_CONTAINER_CONTEXT` `k8s.namespace` |
exactly what C6 exists to catch. |---|---|
| `template-code-location` | `dataprovider01` |
| `semaphore-ui` | `dataprovider01` |
What is still true is that **no run has ever executed on this cluster**
`dataprovider01` holds no pods labelled `dagster/run-id` and the daemon log shows
no launch activity. That appears to be because nothing has been launched, not
because anything is broken.
The residual risk is narrower than a broken namespace, but real: a code location
that reaches the launcher *without* a container context namespace inherits
`dagster` and would fail on three missing objects. The chart always sets it, so
this bites only hand-rolled pods — the reason the C6 row asks you to check the
launcher's namespace-local dependencies rather than just the namespace.
> An earlier revision of this section claimed C6 failed and that this explained
> the absence of runs. That was wrong: it read `job_namespace` without accounting
> for the container-context override.
## 3. Loosely coupled checks ## 3. Loosely coupled checks