[SIMPL-30451] Execute T6 and T7 on the sandbox

Adds the k8s_job_executor probe and records the result. Six step Jobs were created, one per step, and four steps executed and succeeded against the platform's live Postgres. The run then failed before summarise_results: run monitoring polls for a dagster-run-<id> Job that a hand-rolled probe never creates, and with no launcher-supplied container context it looked in the launcher's default namespace, where dagster-svc-account is denied jobs/status. Two step pods had already started and logged 'Skipping step execution' before exiting 0, which is why all six Jobs read Complete while only four steps ran.

Changelog: added

documents/user-guide/readiness-checklist.md

@@ -43,12 +43,8 @@ location image.
 ### 2.1 Sandbox state, 2026-08-31
 
 The T checks were assessed against sandbox-cat-dat by reading the live
-`dagster-instance` ConfigMap in `dataprovider01` and by running T1 and T4 from a
+`dagster-instance` ConfigMap in `dataprovider01` and by running T1, T4, T6 and T7
-pod there. T6 and T7 could not be executed: the tightly coupled Kubernetes path
+from pods there.
-needs this code location registered on the platform Dagster, which is blocked on
-the control plane skew, and unlike the pipes transport it cannot be exercised
-from a self-contained pod — every step pod connects to the metadata database on
-its own, so they must share instance storage rather than a local SQLite file.
 
 What the configuration shows:
 
@@ -86,19 +82,48 @@ So `TIGHTLY_COUPLED_ENV_VARS` in `preflight.py` names variables Simpl does not
 use, and the check reports a misconfiguration that is not one. The reachability
 probe above is the part that carries real evidence.
 
-### Why T6 and T7 remain unexecuted
+### T6 and T7, executed 2026-08-31
 
-The run made the obstacle concrete. Step outputs went to
+Running them needed three things the earlier attempt lacked: dagster pinned to
-`/dagster-home/storage/<run-id>/…` via `PickledObjectFilesystemIOManager`. Under
+1.12.8 to match the control plane, so writing to the live `dataprovider01_dagster`
-`k8s_job_executor` each step is its own pod with its own volume, so a downstream
+could not trigger a migration; `DAGSTER_HOME` moved onto the RWX
-step cannot read an upstream output. Running it needs both a shared I/O manager —
+`dagster-shared-pvc`, because the default I/O manager writes step outputs under
-S3, or the RWX `dagster-shared-pvc` the launcher already mounts at
+`$DAGSTER_HOME/storage` and each step pod otherwise gets its own empty volume;
-`/dagster/shared` — and shared instance storage, which means Postgres.
+and `automount_service_account_token: true` in `step_k8s_config`, because
+`dagster-svc-account` disables it by default. The probe is
+`yaml/sandbox/probe-k8s-executor-sandbox.yaml`; the run is
+`d05c736a-8ec7-4e59-8cb0-9cb238db40dc`.
 
-The only Postgres available is the platform's live `dataprovider01_dagster`, and
+| # | Result |
-these images carry dagster 1.13.19 against a 1.12.8 control plane. Pointing a
+|---|---|
-newer Dagster at an older schema is not a safe thing to try on a shared database.
+| T6 | **Passes.** Six Jobs labelled `dagster/job=tightly_coupled_k8s_job` were created, one per step, each in its own pod |
-Pin dagster to the control plane version and rebuild first.
+| T7 | **Passes.** No step hung in `STARTING` and no step reported a connection timeout to port 5432. Every step pod opened the platform Postgres itself — the outputs and inputs below were all recorded through it |
+
+Four steps executed and succeeded: `generate_work_units` (four dynamic outputs),
+`report_execution_target`, and `process_work_unit[unit_0]` and `[unit_1]`, each
+loading its input from the shared volume that a different pod wrote.
+
+**The run then failed, and `summarise_results` never started.** The cause is a
+property of the probe, not of the code or the platform:
+
+> `Detected run worker status UNKNOWN: DagsterK8sUnrecoverableAPIError … jobs.batch`
+> `"dagster-run-d05c736a-…" is forbidden: User "system:serviceaccount:dataprovider01:`
+> `dagster-svc-account" cannot get resource "jobs/status" … in the namespace "dagster"`
+
+Run monitoring polls for a run worker Job named `dagster-run-<run-id>`. The probe
+starts the run with `dagster job execute` from a hand-rolled Job, so no such
+object exists; and with no launcher-supplied container context the monitor looked
+in the launcher's default namespace, `dagster`, where `dagster-svc-account` holds
+no rights. The 403 turned *absent* into *unknown*, and the monitor failed the run
+at its first poll. `process_work_unit[unit_2]` and `[unit_3]` had their pods
+started already; both logged `Skipping step execution … since the run is in status
+DagsterRunStatus.FAILURE` and exited 0 — which is why all six Jobs read `Complete`
+while only four steps ran.
+
+A run submitted the normal way, through the webserver to `K8sRunLauncher`, gets a
+`dagster-run-<run-id>` Job in `dataprovider01` and is not exposed to this. Closing
+T6/T7 end to end, including `summarise_results`, needs the code location
+registered on the platform Dagster.
 
 **C6 passes, contrary to what `job_namespace` alone suggests.** The launcher's
 `job_namespace: dagster` is only a default. Each code location ships a

yaml/sandbox/probe-k8s-executor-sandbox.yaml

@@ -0,0 +1,161 @@
+# T6/T7 probe - tightly coupled via k8s_job_executor, one pod per step.
+#
+# This is the invasive one. Unlike the pipes probe it writes to the platform's
+# live metadata database, because that is what tightly coupled means: every step
+# pod opens the database itself, so they must share instance storage. Safe only
+# because the images are pinned to dagster 1.12.8, exactly matching the control
+# plane, so no schema migration can be triggered.
+#
+# Two things the loosely coupled probe did not need:
+#
+#   1. Shared instance storage. DAGSTER_HOME is copied from the platform's
+#      `dagster-instance` ConfigMap into a writable volume, so run and event
+#      storage are the same Postgres the platform uses. Step pods get the same
+#      config via the run launcher's `instance_config_map`.
+#
+#   2. A shared I/O manager. The default writes step outputs to pod-local disk,
+#      which under this executor means a downstream step cannot read an upstream
+#      output. base_dir is redirected to the RWX `dagster-shared-pvc` the
+#      launcher already declares.
+#
+# The run config lives in its own ConfigMap rather than a heredoc inside the
+# container command. A heredoc nested in a YAML block scalar silently produced an
+# empty file on the first attempt, and an empty run config is valid YAML, so the
+# run started and only failed later with "No image included in either executor
+# config or the job" - a confusing distance from the actual cause. The command
+# echoes the config before using it so that failure mode cannot recur silently.
+#
+# --- Running ---------------------------------------------------------------
+#
+#   kubectl -n dataprovider01 apply -f yaml/sandbox/probe-k8s-executor-sandbox.yaml
+#   kubectl -n dataprovider01 logs -f job/distexec-k8sexec-probe
+#   kubectl -n dataprovider01 get jobs -l dagster/job=tightly_coupled_k8s_job   # T6
+#
+# --- Teardown --------------------------------------------------------------
+#
+#   kubectl -n dataprovider01 delete job distexec-k8sexec-probe
+#   kubectl -n dataprovider01 delete jobs -l dagster/job=tightly_coupled_k8s_job
+#   kubectl -n dataprovider01 delete cm distexec-probe-run-config
+#
+# The run itself stays in the platform's Dagster database and will be visible in
+# its UI. That is not reversible with kubectl.
+
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: distexec-probe-run-config
+  namespace: dataprovider01
+  labels:
+    app.kubernetes.io/name: distributed-execution
+    app.kubernetes.io/component: probe
+data:
+  run-config.yaml: |
+    execution:
+      config:
+        job_namespace: dataprovider01
+        job_image: gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/j.r/distributed-execution:3f8175a17f0a
+        service_account_name: dagster-svc-account
+        image_pull_policy: IfNotPresent
+        max_concurrent: 2
+        # Step pods inherit DAGSTER_HOME from the launching process otherwise,
+        # which points at a volume only that pod has.
+        env_vars:
+          - DAGSTER_HOME=/dagster/shared/distexec-probe/home
+        volumes:
+          - name: dagster-shared-storage
+            persistent_volume_claim:
+              claim_name: dagster-shared-pvc
+        volume_mounts:
+          - name: dagster-shared-storage
+            mount_path: /dagster/shared
+        step_k8s_config:
+          pod_spec_config:
+            automount_service_account_token: true
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: distexec-k8sexec-probe
+  namespace: dataprovider01
+  labels:
+    app.kubernetes.io/name: distributed-execution
+    app.kubernetes.io/component: probe
+spec:
+  backoffLimit: 0
+  activeDeadlineSeconds: 1800
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: distributed-execution
+        app.kubernetes.io/component: probe
+    spec:
+      restartPolicy: Never
+      serviceAccountName: dagster-svc-account
+      automountServiceAccountToken: true
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1000
+        runAsGroup: 1000
+        fsGroup: 1000
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: runner
+          image: gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/j.r/distributed-execution:3f8175a17f0a
+          imagePullPolicy: IfNotPresent
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop:
+                - ALL
+          env:
+            # On the RWX volume, not a pod-local one: the default I/O manager writes
+            # step outputs under $DAGSTER_HOME/storage, and every step pod has to
+            # read what the previous one wrote.
+            - name: DAGSTER_HOME
+              value: /dagster/shared/distexec-probe/home
+            # How the chart tells a code location which image its steps run as.
+            - name: DAGSTER_CURRENT_IMAGE
+              value: gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/j.r/distributed-execution:3f8175a17f0a
+            - name: DAGSTER_PG_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: dagster-postgresql-secret
+                  key: postgresql-password
+          command:
+            - sh
+            - -c
+            - |
+              set -e
+              mkdir -p "$DAGSTER_HOME"
+              cp /instance-cm/dagster.yaml "$DAGSTER_HOME/dagster.yaml"
+              echo "===== run config in use ====="
+              cat /run-config/run-config.yaml
+              echo "===== T6/T7: tightly_coupled_k8s_job ====="
+              dagster job execute -f src/distributed_execution/repository.py -j tightly_coupled_k8s_job -c /run-config/run-config.yaml
+          volumeMounts:
+            - name: instance-cm
+              mountPath: /instance-cm
+              readOnly: true
+            - name: run-config
+              mountPath: /run-config
+              readOnly: true
+            - name: dagster-shared-storage
+              mountPath: /dagster/shared
+          resources:
+            requests:
+              cpu: 200m
+              memory: 512Mi
+            limits:
+              cpu: "1"
+              memory: 1Gi
+      volumes:
+        - name: instance-cm
+          configMap:
+            name: dagster-instance
+        - name: run-config
+          configMap:
+            name: distexec-probe-run-config
+        - name: dagster-shared-storage
+          persistentVolumeClaim:
+            claimName: dagster-shared-pvc