documents/user-guide/readiness-checklist.md

@@ -43,12 +43,8 @@ location image.
 ### 2.1 Sandbox state, 2026-08-31
 
 The T checks were assessed against sandbox-cat-dat by reading the live
-`dagster-instance` ConfigMap in `dataprovider01` and by running T1 and T4 from a
+`dagster-instance` ConfigMap in `dataprovider01` and by running T1, T4, T6 and T7
-pod there. T6 and T7 could not be executed: the tightly coupled Kubernetes path
+from pods there.
-needs this code location registered on the platform Dagster, which is blocked on
-the control plane skew, and unlike the pipes transport it cannot be exercised
-from a self-contained pod — every step pod connects to the metadata database on
-its own, so they must share instance storage rather than a local SQLite file.
 
 What the configuration shows:
 
@@ -86,19 +82,48 @@ So `TIGHTLY_COUPLED_ENV_VARS` in `preflight.py` names variables Simpl does not
 use, and the check reports a misconfiguration that is not one. The reachability
 probe above is the part that carries real evidence.
 
-### Why T6 and T7 remain unexecuted
+### T6 and T7, executed 2026-08-31
 
-The run made the obstacle concrete. Step outputs went to
+Running them needed three things the earlier attempt lacked: dagster pinned to
-`/dagster-home/storage/<run-id>/…` via `PickledObjectFilesystemIOManager`. Under
+1.12.8 to match the control plane, so writing to the live `dataprovider01_dagster`
-`k8s_job_executor` each step is its own pod with its own volume, so a downstream
+could not trigger a migration; `DAGSTER_HOME` moved onto the RWX
-step cannot read an upstream output. Running it needs both a shared I/O manager —
+`dagster-shared-pvc`, because the default I/O manager writes step outputs under
-S3, or the RWX `dagster-shared-pvc` the launcher already mounts at
+`$DAGSTER_HOME/storage` and each step pod otherwise gets its own empty volume;
-`/dagster/shared` — and shared instance storage, which means Postgres.
+and `automount_service_account_token: true` in `step_k8s_config`, because
+`dagster-svc-account` disables it by default. The probe is
+`yaml/sandbox/probe-k8s-executor-sandbox.yaml`; the run is
+`d05c736a-8ec7-4e59-8cb0-9cb238db40dc`.
 
-The only Postgres available is the platform's live `dataprovider01_dagster`, and
+| # | Result |
-these images carry dagster 1.13.19 against a 1.12.8 control plane. Pointing a
+|---|---|
-newer Dagster at an older schema is not a safe thing to try on a shared database.
+| T6 | **Passes.** Six Jobs labelled `dagster/job=tightly_coupled_k8s_job` were created, one per step, each in its own pod |
-Pin dagster to the control plane version and rebuild first.
+| T7 | **Passes.** No step hung in `STARTING` and no step reported a connection timeout to port 5432. Every step pod opened the platform Postgres itself — the outputs and inputs below were all recorded through it |
+
+Four steps executed and succeeded: `generate_work_units` (four dynamic outputs),
+`report_execution_target`, and `process_work_unit[unit_0]` and `[unit_1]`, each
+loading its input from the shared volume that a different pod wrote.
+
+**The run then failed, and `summarise_results` never started.** The cause is a
+property of the probe, not of the code or the platform:
+
+> `Detected run worker status UNKNOWN: DagsterK8sUnrecoverableAPIError … jobs.batch`
+> `"dagster-run-d05c736a-…" is forbidden: User "system:serviceaccount:dataprovider01:`
+> `dagster-svc-account" cannot get resource "jobs/status" … in the namespace "dagster"`
+
+Run monitoring polls for a run worker Job named `dagster-run-<run-id>`. The probe
+starts the run with `dagster job execute` from a hand-rolled Job, so no such
+object exists; and with no launcher-supplied container context the monitor looked
+in the launcher's default namespace, `dagster`, where `dagster-svc-account` holds
+no rights. The 403 turned *absent* into *unknown*, and the monitor failed the run
+at its first poll. `process_work_unit[unit_2]` and `[unit_3]` had their pods
+started already; both logged `Skipping step execution … since the run is in status
+DagsterRunStatus.FAILURE` and exited 0 — which is why all six Jobs read `Complete`
+while only four steps ran.
+
+A run submitted the normal way, through the webserver to `K8sRunLauncher`, gets a
+`dagster-run-<run-id>` Job in `dataprovider01` and is not exposed to this. Closing
+T6/T7 end to end, including `summarise_results`, needs the code location
+registered on the platform Dagster.
 
 **C6 passes, contrary to what `job_namespace` alone suggests.** The launcher's
 `job_namespace: dagster` is only a default. Each code location ships a

yaml/sandbox/probe-k8s-executor-sandbox.yaml

@@ -0,0 +1,161 @@
+# T6/T7 probe - tightly coupled via k8s_job_executor, one pod per step.
+#
+# This is the invasive one. Unlike the pipes probe it writes to the platform's
+# live metadata database, because that is what tightly coupled means: every step
+# pod opens the database itself, so they must share instance storage. Safe only
+# because the images are pinned to dagster 1.12.8, exactly matching the control
+# plane, so no schema migration can be triggered.
+#
+# Two things the loosely coupled probe did not need:
+#
+#   1. Shared instance storage. DAGSTER_HOME is copied from the platform's
+#      `dagster-instance` ConfigMap into a writable volume, so run and event
+#      storage are the same Postgres the platform uses. Step pods get the same
+#      config via the run launcher's `instance_config_map`.
+#
+#   2. A shared I/O manager. The default writes step outputs to pod-local disk,
+#      which under this executor means a downstream step cannot read an upstream
+#      output. base_dir is redirected to the RWX `dagster-shared-pvc` the
+#      launcher already declares.
+#
+# The run config lives in its own ConfigMap rather than a heredoc inside the
+# container command. A heredoc nested in a YAML block scalar silently produced an
+# empty file on the first attempt, and an empty run config is valid YAML, so the
+# run started and only failed later with "No image included in either executor
+# config or the job" - a confusing distance from the actual cause. The command
+# echoes the config before using it so that failure mode cannot recur silently.
+#
+# --- Running ---------------------------------------------------------------
+#
+#   kubectl -n dataprovider01 apply -f yaml/sandbox/probe-k8s-executor-sandbox.yaml
+#   kubectl -n dataprovider01 logs -f job/distexec-k8sexec-probe
+#   kubectl -n dataprovider01 get jobs -l dagster/job=tightly_coupled_k8s_job   # T6
+#
+# --- Teardown --------------------------------------------------------------
+#
+#   kubectl -n dataprovider01 delete job distexec-k8sexec-probe
+#   kubectl -n dataprovider01 delete jobs -l dagster/job=tightly_coupled_k8s_job
+#   kubectl -n dataprovider01 delete cm distexec-probe-run-config
+#
+# The run itself stays in the platform's Dagster database and will be visible in
+# its UI. That is not reversible with kubectl.
+
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: distexec-probe-run-config
+  namespace: dataprovider01
+  labels:
+    app.kubernetes.io/name: distributed-execution
+    app.kubernetes.io/component: probe
+data:
+  run-config.yaml: |
+    execution:
+      config:
+        job_namespace: dataprovider01
+        job_image: gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/j.r/distributed-execution:3f8175a17f0a
+        service_account_name: dagster-svc-account
+        image_pull_policy: IfNotPresent
+        max_concurrent: 2
+        # Step pods inherit DAGSTER_HOME from the launching process otherwise,
+        # which points at a volume only that pod has.
+        env_vars:
+          - DAGSTER_HOME=/dagster/shared/distexec-probe/home
+        volumes:
+          - name: dagster-shared-storage
+            persistent_volume_claim:
+              claim_name: dagster-shared-pvc
+        volume_mounts:
+          - name: dagster-shared-storage
+            mount_path: /dagster/shared
+        step_k8s_config:
+          pod_spec_config:
+            automount_service_account_token: true
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: distexec-k8sexec-probe
+  namespace: dataprovider01
+  labels:
+    app.kubernetes.io/name: distributed-execution
+    app.kubernetes.io/component: probe
+spec:
+  backoffLimit: 0
+  activeDeadlineSeconds: 1800
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: distributed-execution
+        app.kubernetes.io/component: probe
+    spec:
+      restartPolicy: Never
+      serviceAccountName: dagster-svc-account
+      automountServiceAccountToken: true
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1000
+        runAsGroup: 1000
+        fsGroup: 1000
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: runner
+          image: gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/j.r/distributed-execution:3f8175a17f0a
+          imagePullPolicy: IfNotPresent
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop:
+                - ALL
+          env:
+            # On the RWX volume, not a pod-local one: the default I/O manager writes
+            # step outputs under $DAGSTER_HOME/storage, and every step pod has to
+            # read what the previous one wrote.
+            - name: DAGSTER_HOME
+              value: /dagster/shared/distexec-probe/home
+            # How the chart tells a code location which image its steps run as.
+            - name: DAGSTER_CURRENT_IMAGE
+              value: gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/j.r/distributed-execution:3f8175a17f0a
+            - name: DAGSTER_PG_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: dagster-postgresql-secret
+                  key: postgresql-password
+          command:
+            - sh
+            - -c
+            - |
+              set -e
+              mkdir -p "$DAGSTER_HOME"
+              cp /instance-cm/dagster.yaml "$DAGSTER_HOME/dagster.yaml"
+              echo "===== run config in use ====="
+              cat /run-config/run-config.yaml
+              echo "===== T6/T7: tightly_coupled_k8s_job ====="
+              dagster job execute -f src/distributed_execution/repository.py -j tightly_coupled_k8s_job -c /run-config/run-config.yaml
+          volumeMounts:
+            - name: instance-cm
+              mountPath: /instance-cm
+              readOnly: true
+            - name: run-config
+              mountPath: /run-config
+              readOnly: true
+            - name: dagster-shared-storage
+              mountPath: /dagster/shared
+          resources:
+            requests:
+              cpu: 200m
+              memory: 512Mi
+            limits:
+              cpu: "1"
+              memory: 1Gi
+      volumes:
+        - name: instance-cm
+          configMap:
+            name: dagster-instance
+        - name: run-config
+          configMap:
+            name: distexec-probe-run-config
+        - name: dagster-shared-storage
+          persistentVolumeClaim:
+            claimName: dagster-shared-pvc