[SIMPL-30451] Correct the C6 finding: the container context overrides job_namespace
The previous revision claimed C6 failed on the sandbox because job_namespace is dagster while the launcher's ConfigMap, Secret and PVC live in dataprovider01. That read job_namespace in isolation. Each code location ships a DAGSTER_CONTAINER_CONTEXT whose k8s.namespace is dataprovider01, and the code location context is merged over the launcher's, so it wins - dagster_k8s/container_context.py resolves namespace as 'other.namespace if other.namespace else self.namespace'. Both deployed code locations carry it, so run pods land where their dependencies are. Still true: no run has ever executed on that cluster. That now looks like nothing having been launched rather than anything being broken. The residual risk is a code location reaching the launcher without a container context namespace, which the chart always sets, so it bites only hand-rolled pods. C5 now warns that job_namespace is a default rather than the effective namespace. Changelog: fixed
This commit is contained in:
@@ -10,9 +10,9 @@ location image.
|
|||||||
> **L4–L10 were cleared on sandbox-cat-dat on 2026-08-31.** Checks L1–L3 and L11,
|
> **L4–L10 were cleared on sandbox-cat-dat on 2026-08-31.** Checks L1–L3 and L11,
|
||||||
> which exercise the payload contract, the message-parsing path and image-level
|
> which exercise the payload contract, the message-parsing path and image-level
|
||||||
> isolation, are verified locally. The **tightly coupled** Kubernetes path has
|
> isolation, are verified locally. The **tightly coupled** Kubernetes path has
|
||||||
> been assessed against the same cluster but not executed — T2, T3, T5 and T8
|
> been assessed against the same cluster but not executed — C5, C6, T2, T3, T5
|
||||||
> are confirmed from the live instance configuration and C6 fails there. See
|
> and T8 are confirmed from the live instance configuration and the deployed code
|
||||||
> [section 2.1](#21-sandbox-state-2026-08-31).
|
> locations. See [section 2.1](#21-sandbox-state-2026-08-31).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -24,8 +24,8 @@ location image.
|
|||||||
| C2 | Jobs are registered | Dagster UI → **Jobs** | The jobs listed in the guide's section 5.5 appear under the code location |
|
| C2 | Jobs are registered | Dagster UI → **Jobs** | The jobs listed in the guide's section 5.5 appear under the code location |
|
||||||
| C3 | Image tag matches the intended release | `kubectl -n dagster get deploy -l dagster/code-location=distributed-execution -o jsonpath='{.items[*].spec.template.spec.containers[*].image}'` | Tag equals the version in `pipeline.variables.sh`; never `latest` |
|
| C3 | Image tag matches the intended release | `kubectl -n dagster get deploy -l dagster/code-location=distributed-execution -o jsonpath='{.items[*].spec.template.spec.containers[*].image}'` | Tag equals the version in `pipeline.variables.sh`; never `latest` |
|
||||||
| C4 | Image architecture matches the nodes | `docker manifest inspect <image>` | Includes `linux/amd64`; a manifest with only `linux/arm64` produces `no match for platform` at pull time |
|
| C4 | Image architecture matches the nodes | `docker manifest inspect <image>` | Includes `linux/amd64`; a manifest with only `linux/arm64` produces `no match for platform` at pull time |
|
||||||
| C5 | Run launcher type is as intended | `kubectl -n <namespace> get cm dagster-instance -o yaml` | `run_launcher` block shows `K8sRunLauncher` and the expected `job_namespace` |
|
| C5 | Run launcher type is as intended | `kubectl -n <namespace> get cm dagster-instance -o yaml` | `run_launcher` block shows `K8sRunLauncher`. Note that `job_namespace` here is only a default — a code location's `DAGSTER_CONTAINER_CONTEXT` overrides it, so read that too before concluding where run pods land |
|
||||||
| C6 | Target namespace exists and is schedulable | `kubectl get ns <namespace>` and `kubectl -n <namespace> get resourcequota` | Namespace is `Active`; remaining quota exceeds the job's aggregate requests. Also confirm the launcher's `instance_config_map`, `postgres_password_secret` and any PVC volumes exist **in that same namespace** — these references do not cross namespaces, and on sandbox-cat-dat they do not match `job_namespace` (see section 2.1) |
|
| C6 | Target namespace exists and is schedulable | `kubectl get ns <namespace>` and `kubectl -n <namespace> get resourcequota` | Namespace is `Active`; remaining quota exceeds the job's aggregate requests. Also confirm the launcher's `instance_config_map`, `postgres_password_secret` and any PVC volumes exist **in the namespace run pods actually use** — these references do not cross namespaces (see section 2.1) |
|
||||||
|
|
||||||
## 2. Tightly coupled checks
|
## 2. Tightly coupled checks
|
||||||
|
|
||||||
@@ -54,28 +54,42 @@ What the configuration shows:
|
|||||||
|
|
||||||
| # | Finding |
|
| # | Finding |
|
||||||
|---|---|
|
|---|---|
|
||||||
| C5 | `run_launcher` is `K8sRunLauncher` with `job_namespace: dagster` |
|
| C5 | `run_launcher` is `K8sRunLauncher`; `job_namespace: dagster` is the default, overridden per code location to `dataprovider01` |
|
||||||
| T2 | Vault injection is configured — `pod_template_spec_metadata` carries the banzaicloud annotations with role `sandbox-cat-dat-role` |
|
| T2 | Vault injection is configured — `pod_template_spec_metadata` carries the banzaicloud annotations with role `sandbox-cat-dat-role` |
|
||||||
| T3 | `S3_ENDPOINT_URL` is `https://s3.sandbox-cat-dat.simpl-europe.eu`, with access keys injected from Vault |
|
| T3 | `S3_ENDPOINT_URL` is `https://s3.sandbox-cat-dat.simpl-europe.eu`, with access keys injected from Vault |
|
||||||
| T5 | Passes. `dagster-role`, bound to `dagster-svc-account`, grants `batch/jobs` with create, delete, get, list, patch, update and watch |
|
| T5 | Passes. `dagster-role`, bound to `dagster-svc-account`, grants `batch/jobs` with create, delete, get, list, patch, update and watch |
|
||||||
| T8 | `fail_pod_on_run_failure: true` is set |
|
| T8 | `fail_pod_on_run_failure: true` is set |
|
||||||
|
|
||||||
**C6 fails, and it is the reason nothing has ever run here.** The launcher sends
|
**C6 passes, contrary to what `job_namespace` alone suggests.** The launcher's
|
||||||
run pods to namespace `dagster`, but every namespace-local dependency it names —
|
`job_namespace: dagster` is only a default. Each code location ships a
|
||||||
`instance_config_map: dagster-instance`, `postgres_password_secret:
|
`DAGSTER_CONTAINER_CONTEXT` environment variable — generated by the chart — whose
|
||||||
dagster-postgresql-secret`, and the `dagster-shared-pvc` volume — exists in
|
`k8s.namespace` is `dataprovider01`, and the code location's context is merged
|
||||||
`dataprovider01`. A ConfigMap, Secret or PVC reference resolves only within the
|
*over* the launcher's, so it wins (`dagster_k8s/container_context.py`: `namespace
|
||||||
pod's own namespace, so a run pod placed in `dagster` would reference three
|
= other.namespace if other.namespace else self.namespace`). Run pods therefore
|
||||||
objects that are not there. Consistent with that, `dataprovider01` holds no pods
|
land in `dataprovider01`, where `dagster-instance`, `dagster-postgresql-secret`
|
||||||
labelled `dagster/run-id` and the daemon log shows no launch activity.
|
and `dagster-shared-pvc` all exist.
|
||||||
|
|
||||||
One caveat on the evidence: project-scoped access means `kubectl get ns dagster`
|
Both deployed code locations carry it, so both would launch correctly:
|
||||||
returns `Forbidden` rather than `NotFound`, so whether that namespace exists is
|
|
||||||
unconfirmed. It does not change the conclusion — the dependencies are in the
|
|
||||||
wrong namespace either way.
|
|
||||||
|
|
||||||
This is a platform configuration defect, not a defect in this service. It is
|
| Code location | `DAGSTER_CONTAINER_CONTEXT` `k8s.namespace` |
|
||||||
exactly what C6 exists to catch.
|
|---|---|
|
||||||
|
| `template-code-location` | `dataprovider01` |
|
||||||
|
| `semaphore-ui` | `dataprovider01` |
|
||||||
|
|
||||||
|
What is still true is that **no run has ever executed on this cluster** —
|
||||||
|
`dataprovider01` holds no pods labelled `dagster/run-id` and the daemon log shows
|
||||||
|
no launch activity. That appears to be because nothing has been launched, not
|
||||||
|
because anything is broken.
|
||||||
|
|
||||||
|
The residual risk is narrower than a broken namespace, but real: a code location
|
||||||
|
that reaches the launcher *without* a container context namespace inherits
|
||||||
|
`dagster` and would fail on three missing objects. The chart always sets it, so
|
||||||
|
this bites only hand-rolled pods — the reason the C6 row asks you to check the
|
||||||
|
launcher's namespace-local dependencies rather than just the namespace.
|
||||||
|
|
||||||
|
> An earlier revision of this section claimed C6 failed and that this explained
|
||||||
|
> the absence of runs. That was wrong: it read `job_namespace` without accounting
|
||||||
|
> for the container-context override.
|
||||||
|
|
||||||
## 3. Loosely coupled checks
|
## 3. Loosely coupled checks
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user