j.r/distributed-execution
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[SIMPL-30451] Run T1 and T4 on the sandbox

Browse Source 
Executed from a pod in dataprovider01 with a local SQLite instance, deliberately not the platform's Postgres. T1 reachability passes: check_tcp_reachable against pg-cluster.common01.svc.cluster.local:5432 returned True. T4 passes: tightly_coupled_local_job reached RUN_SUCCESS with one entry in contributing_hosts and four distinct PIDs in contributing_workers, which is what the row predicts. Torn down afterwards.

The run exposed a defect in this service. report_execution_target warns that DAGSTER_POSTGRES_HOST, DAGSTER_POSTGRES_USER and DAGSTER_POSTGRES_DB are missing and calls that a tightly coupled misconfiguration, but the Simpl chart never sets them - it injects DAGSTER_PG_PASSWORD and bakes host, user and database into dagster.yaml. TIGHTLY_COUPLED_ENV_VARS therefore reports a misconfiguration that is not one, and would do so on a correctly configured platform run pod.

It also made the T6 obstacle concrete rather than theoretical: step outputs were written to /dagster-home/storage via PickledObjectFilesystemIOManager, which under k8s_job_executor lands on a volume no other step pod can read.

Changelog: added
This commit is contained in:
ILay
2026-08-31 20:17:45 +02:00
parent 2bbecb1bdd
commit 554c92ec92
2 changed files with 145 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

52
documents/user-guide/readiness-checklist.md
View File

@@ -43,12 +43,12 @@ location image.
### 2.1 Sandbox state, 2026-08-31
The T checks were assessed against sandbox-cat-dat by reading the live
`dagster-instance` ConfigMap in `dataprovider01`. None of them could be *executed*:
the tightly coupled Kubernetes path needs this code location registered on the
platform Dagster, which is blocked on the control plane skew, and unlike the
pipes transport it cannot be exercised from a standalone pod — every step pod
connects to the metadata database on its own, so they must share instance
storage rather than a local SQLite file.
`dagster-instance` ConfigMap in `dataprovider01` and by running T1 and T4 from a
pod there. T6 and T7 could not be executed: the tightly coupled Kubernetes path
needs this code location registered on the platform Dagster, which is blocked on
the control plane skew, and unlike the pipes transport it cannot be exercised
from a self-contained pod — every step pod connects to the metadata database on
its own, so they must share instance storage rather than a local SQLite file.
What the configuration shows:
@@ -60,6 +60,46 @@ What the configuration shows:
| T5 | Passes. `dagster-role`, bound to `dagster-svc-account`, grants `batch/jobs` with create, delete, get, list, patch, update and watch |
| T8 | `fail_pod_on_run_failure: true` is set |
**T1 and T4 were executed on 2026-08-31** from a pod in `dataprovider01`
(`yaml/sandbox/probe-tightly-coupled-sandbox.yaml`), using a local SQLite
instance rather than the platform's Postgres.
| # | Result |
|---|---|
| T1 | Reachability **passes** — `check_tcp_reachable('pg-cluster.common01.svc.cluster.local', 5432)` returned `passed: True`. The env-var half of the check does not hold; see below |
| T4 | **Passes.** `tightly_coupled_local_job` reached `RUN_SUCCESS` with `contributing_hosts: ['distexec-tc-probe-hbdfl']` — one entry — and four distinct PIDs in `contributing_workers`. Separate processes, same machine, exactly as the row predicts |
`tightly_coupled_in_process_job` also succeeded, reporting a single worker.
### T1's env-var check is wrong for this platform
The run emitted:
> Orchestration runtime env vars not visible to this process:
> `DAGSTER_POSTGRES_HOST`, `DAGSTER_POSTGRES_USER`, `DAGSTER_POSTGRES_DB`.
> Expected for a loosely coupled target; a misconfiguration for a tightly coupled one.
That warning would fire on a **correctly configured** platform run pod. The chart
does not set those variables: it injects `DAGSTER_PG_PASSWORD` from
`dagster-postgresql-secret` and bakes host, user and database into `dagster.yaml`.
So `TIGHTLY_COUPLED_ENV_VARS` in `preflight.py` names variables Simpl does not
use, and the check reports a misconfiguration that is not one. The reachability
probe above is the part that carries real evidence.
### Why T6 and T7 remain unexecuted
The run made the obstacle concrete. Step outputs went to
`/dagster-home/storage/<run-id>/…` via `PickledObjectFilesystemIOManager`. Under
`k8s_job_executor` each step is its own pod with its own volume, so a downstream
step cannot read an upstream output. Running it needs both a shared I/O manager —
S3, or the RWX `dagster-shared-pvc` the launcher already mounts at
`/dagster/shared` — and shared instance storage, which means Postgres.
The only Postgres available is the platform's live `dataprovider01_dagster`, and
these images carry dagster 1.13.19 against a 1.12.8 control plane. Pointing a
newer Dagster at an older schema is not a safe thing to try on a shared database.
Pin dagster to the control plane version and rebuild first.
**C6 passes, contrary to what `job_namespace` alone suggests.** The launcher's
`job_namespace: dagster` is only a default. Each code location ships a
`DAGSTER_CONTAINER_CONTEXT` environment variable — generated by the chart — whose