j.r/distributed-execution
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[SIMPL-30451] Execute T6 and T7 on the sandbox

Browse Source 
Adds the k8s_job_executor probe and records the result. Six step Jobs were created, one per step, and four steps executed and succeeded against the platform's live Postgres. The run then failed before summarise_results: run monitoring polls for a dagster-run-<id> Job that a hand-rolled probe never creates, and with no launcher-supplied container context it looked in the launcher's default namespace, where dagster-svc-account is denied jobs/status. Two step pods had already started and logged 'Skipping step execution' before exiting 0, which is why all six Jobs read Complete while only four steps ran.

Changelog: added
This commit is contained in:
ILay
2026-08-31 22:10:12 +02:00
parent 3f8175a17f
commit 64cace2952
2 changed files with 203 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files

59
documents/user-guide/readiness-checklist.md
View File

@@ -43,12 +43,8 @@ location image.
### 2.1 Sandbox state, 2026-08-31
The T checks were assessed against sandbox-cat-dat by reading the live
`dagster-instance` ConfigMap in `dataprovider01` and by running T1 and T4 from a
pod there. T6 and T7 could not be executed: the tightly coupled Kubernetes path
needs this code location registered on the platform Dagster, which is blocked on
the control plane skew, and unlike the pipes transport it cannot be exercised
from a self-contained pod — every step pod connects to the metadata database on
its own, so they must share instance storage rather than a local SQLite file.
`dagster-instance` ConfigMap in `dataprovider01` and by running T1, T4, T6 and T7
from pods there.
What the configuration shows:
@@ -86,19 +82,48 @@ So `TIGHTLY_COUPLED_ENV_VARS` in `preflight.py` names variables Simpl does not
use, and the check reports a misconfiguration that is not one. The reachability
probe above is the part that carries real evidence.
### Why T6 and T7 remain unexecuted
### T6 and T7, executed 2026-08-31
The run made the obstacle concrete. Step outputs went to
`/dagster-home/storage/<run-id>/…` via `PickledObjectFilesystemIOManager`. Under
`k8s_job_executor` each step is its own pod with its own volume, so a downstream
step cannot read an upstream output. Running it needs both a shared I/O manager —
S3, or the RWX `dagster-shared-pvc` the launcher already mounts at
`/dagster/shared` — and shared instance storage, which means Postgres.
Running them needed three things the earlier attempt lacked: dagster pinned to
1.12.8 to match the control plane, so writing to the live `dataprovider01_dagster`
could not trigger a migration; `DAGSTER_HOME` moved onto the RWX
`dagster-shared-pvc`, because the default I/O manager writes step outputs under
`$DAGSTER_HOME/storage` and each step pod otherwise gets its own empty volume;
and `automount_service_account_token: true` in `step_k8s_config`, because
`dagster-svc-account` disables it by default. The probe is
`yaml/sandbox/probe-k8s-executor-sandbox.yaml`; the run is
`d05c736a-8ec7-4e59-8cb0-9cb238db40dc`.
The only Postgres available is the platform's live `dataprovider01_dagster`, and
these images carry dagster 1.13.19 against a 1.12.8 control plane. Pointing a
newer Dagster at an older schema is not a safe thing to try on a shared database.
Pin dagster to the control plane version and rebuild first.
| # | Result |
|---|---|
| T6 | **Passes.** Six Jobs labelled `dagster/job=tightly_coupled_k8s_job` were created, one per step, each in its own pod |
| T7 | **Passes.** No step hung in `STARTING` and no step reported a connection timeout to port 5432. Every step pod opened the platform Postgres itself — the outputs and inputs below were all recorded through it |
Four steps executed and succeeded: `generate_work_units` (four dynamic outputs),
`report_execution_target`, and `process_work_unit[unit_0]` and `[unit_1]`, each
loading its input from the shared volume that a different pod wrote.
**The run then failed, and `summarise_results` never started.** The cause is a
property of the probe, not of the code or the platform:
> `Detected run worker status UNKNOWN: DagsterK8sUnrecoverableAPIError … jobs.batch`
> `"dagster-run-d05c736a-…" is forbidden: User "system:serviceaccount:dataprovider01:`
> `dagster-svc-account" cannot get resource "jobs/status" … in the namespace "dagster"`
Run monitoring polls for a run worker Job named `dagster-run-<run-id>`. The probe
starts the run with `dagster job execute` from a hand-rolled Job, so no such
object exists; and with no launcher-supplied container context the monitor looked
in the launcher's default namespace, `dagster`, where `dagster-svc-account` holds
no rights. The 403 turned *absent* into *unknown*, and the monitor failed the run
at its first poll. `process_work_unit[unit_2]` and `[unit_3]` had their pods
started already; both logged `Skipping step execution … since the run is in status
DagsterRunStatus.FAILURE` and exited 0 — which is why all six Jobs read `Complete`
while only four steps ran.
A run submitted the normal way, through the webserver to `K8sRunLauncher`, gets a
`dagster-run-<run-id>` Job in `dataprovider01` and is not exposed to this. Closing
T6/T7 end to end, including `summarise_results`, needs the code location
registered on the platform Dagster.
**C6 passes, contrary to what `job_namespace` alone suggests.** The launcher's
`job_namespace: dagster` is only a default. Each code location ships a