diff --git a/documents/user-guide/readiness-checklist.md b/documents/user-guide/readiness-checklist.md index 672666d..832b7e7 100644 --- a/documents/user-guide/readiness-checklist.md +++ b/documents/user-guide/readiness-checklist.md @@ -43,12 +43,8 @@ location image. ### 2.1 Sandbox state, 2026-08-31 The T checks were assessed against sandbox-cat-dat by reading the live -`dagster-instance` ConfigMap in `dataprovider01` and by running T1 and T4 from a -pod there. T6 and T7 could not be executed: the tightly coupled Kubernetes path -needs this code location registered on the platform Dagster, which is blocked on -the control plane skew, and unlike the pipes transport it cannot be exercised -from a self-contained pod — every step pod connects to the metadata database on -its own, so they must share instance storage rather than a local SQLite file. +`dagster-instance` ConfigMap in `dataprovider01` and by running T1, T4, T6 and T7 +from pods there. What the configuration shows: @@ -86,19 +82,48 @@ So `TIGHTLY_COUPLED_ENV_VARS` in `preflight.py` names variables Simpl does not use, and the check reports a misconfiguration that is not one. The reachability probe above is the part that carries real evidence. -### Why T6 and T7 remain unexecuted +### T6 and T7, executed 2026-08-31 -The run made the obstacle concrete. Step outputs went to -`/dagster-home/storage//…` via `PickledObjectFilesystemIOManager`. Under -`k8s_job_executor` each step is its own pod with its own volume, so a downstream -step cannot read an upstream output. Running it needs both a shared I/O manager — -S3, or the RWX `dagster-shared-pvc` the launcher already mounts at -`/dagster/shared` — and shared instance storage, which means Postgres. +Running them needed three things the earlier attempt lacked: dagster pinned to +1.12.8 to match the control plane, so writing to the live `dataprovider01_dagster` +could not trigger a migration; `DAGSTER_HOME` moved onto the RWX +`dagster-shared-pvc`, because the default I/O manager writes step outputs under +`$DAGSTER_HOME/storage` and each step pod otherwise gets its own empty volume; +and `automount_service_account_token: true` in `step_k8s_config`, because +`dagster-svc-account` disables it by default. The probe is +`yaml/sandbox/probe-k8s-executor-sandbox.yaml`; the run is +`d05c736a-8ec7-4e59-8cb0-9cb238db40dc`. -The only Postgres available is the platform's live `dataprovider01_dagster`, and -these images carry dagster 1.13.19 against a 1.12.8 control plane. Pointing a -newer Dagster at an older schema is not a safe thing to try on a shared database. -Pin dagster to the control plane version and rebuild first. +| # | Result | +|---|---| +| T6 | **Passes.** Six Jobs labelled `dagster/job=tightly_coupled_k8s_job` were created, one per step, each in its own pod | +| T7 | **Passes.** No step hung in `STARTING` and no step reported a connection timeout to port 5432. Every step pod opened the platform Postgres itself — the outputs and inputs below were all recorded through it | + +Four steps executed and succeeded: `generate_work_units` (four dynamic outputs), +`report_execution_target`, and `process_work_unit[unit_0]` and `[unit_1]`, each +loading its input from the shared volume that a different pod wrote. + +**The run then failed, and `summarise_results` never started.** The cause is a +property of the probe, not of the code or the platform: + +> `Detected run worker status UNKNOWN: DagsterK8sUnrecoverableAPIError … jobs.batch` +> `"dagster-run-d05c736a-…" is forbidden: User "system:serviceaccount:dataprovider01:` +> `dagster-svc-account" cannot get resource "jobs/status" … in the namespace "dagster"` + +Run monitoring polls for a run worker Job named `dagster-run-`. The probe +starts the run with `dagster job execute` from a hand-rolled Job, so no such +object exists; and with no launcher-supplied container context the monitor looked +in the launcher's default namespace, `dagster`, where `dagster-svc-account` holds +no rights. The 403 turned *absent* into *unknown*, and the monitor failed the run +at its first poll. `process_work_unit[unit_2]` and `[unit_3]` had their pods +started already; both logged `Skipping step execution … since the run is in status +DagsterRunStatus.FAILURE` and exited 0 — which is why all six Jobs read `Complete` +while only four steps ran. + +A run submitted the normal way, through the webserver to `K8sRunLauncher`, gets a +`dagster-run-` Job in `dataprovider01` and is not exposed to this. Closing +T6/T7 end to end, including `summarise_results`, needs the code location +registered on the platform Dagster. **C6 passes, contrary to what `job_namespace` alone suggests.** The launcher's `job_namespace: dagster` is only a default. Each code location ships a diff --git a/yaml/sandbox/probe-k8s-executor-sandbox.yaml b/yaml/sandbox/probe-k8s-executor-sandbox.yaml new file mode 100644 index 0000000..7ab8448 --- /dev/null +++ b/yaml/sandbox/probe-k8s-executor-sandbox.yaml @@ -0,0 +1,161 @@ +# T6/T7 probe - tightly coupled via k8s_job_executor, one pod per step. +# +# This is the invasive one. Unlike the pipes probe it writes to the platform's +# live metadata database, because that is what tightly coupled means: every step +# pod opens the database itself, so they must share instance storage. Safe only +# because the images are pinned to dagster 1.12.8, exactly matching the control +# plane, so no schema migration can be triggered. +# +# Two things the loosely coupled probe did not need: +# +# 1. Shared instance storage. DAGSTER_HOME is copied from the platform's +# `dagster-instance` ConfigMap into a writable volume, so run and event +# storage are the same Postgres the platform uses. Step pods get the same +# config via the run launcher's `instance_config_map`. +# +# 2. A shared I/O manager. The default writes step outputs to pod-local disk, +# which under this executor means a downstream step cannot read an upstream +# output. base_dir is redirected to the RWX `dagster-shared-pvc` the +# launcher already declares. +# +# The run config lives in its own ConfigMap rather than a heredoc inside the +# container command. A heredoc nested in a YAML block scalar silently produced an +# empty file on the first attempt, and an empty run config is valid YAML, so the +# run started and only failed later with "No image included in either executor +# config or the job" - a confusing distance from the actual cause. The command +# echoes the config before using it so that failure mode cannot recur silently. +# +# --- Running --------------------------------------------------------------- +# +# kubectl -n dataprovider01 apply -f yaml/sandbox/probe-k8s-executor-sandbox.yaml +# kubectl -n dataprovider01 logs -f job/distexec-k8sexec-probe +# kubectl -n dataprovider01 get jobs -l dagster/job=tightly_coupled_k8s_job # T6 +# +# --- Teardown -------------------------------------------------------------- +# +# kubectl -n dataprovider01 delete job distexec-k8sexec-probe +# kubectl -n dataprovider01 delete jobs -l dagster/job=tightly_coupled_k8s_job +# kubectl -n dataprovider01 delete cm distexec-probe-run-config +# +# The run itself stays in the platform's Dagster database and will be visible in +# its UI. That is not reversible with kubectl. + +apiVersion: v1 +kind: ConfigMap +metadata: + name: distexec-probe-run-config + namespace: dataprovider01 + labels: + app.kubernetes.io/name: distributed-execution + app.kubernetes.io/component: probe +data: + run-config.yaml: | + execution: + config: + job_namespace: dataprovider01 + job_image: gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/j.r/distributed-execution:3f8175a17f0a + service_account_name: dagster-svc-account + image_pull_policy: IfNotPresent + max_concurrent: 2 + # Step pods inherit DAGSTER_HOME from the launching process otherwise, + # which points at a volume only that pod has. + env_vars: + - DAGSTER_HOME=/dagster/shared/distexec-probe/home + volumes: + - name: dagster-shared-storage + persistent_volume_claim: + claim_name: dagster-shared-pvc + volume_mounts: + - name: dagster-shared-storage + mount_path: /dagster/shared + step_k8s_config: + pod_spec_config: + automount_service_account_token: true +--- +apiVersion: batch/v1 +kind: Job +metadata: + name: distexec-k8sexec-probe + namespace: dataprovider01 + labels: + app.kubernetes.io/name: distributed-execution + app.kubernetes.io/component: probe +spec: + backoffLimit: 0 + activeDeadlineSeconds: 1800 + template: + metadata: + labels: + app.kubernetes.io/name: distributed-execution + app.kubernetes.io/component: probe + spec: + restartPolicy: Never + serviceAccountName: dagster-svc-account + automountServiceAccountToken: true + securityContext: + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + fsGroup: 1000 + seccompProfile: + type: RuntimeDefault + containers: + - name: runner + image: gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/j.r/distributed-execution:3f8175a17f0a + imagePullPolicy: IfNotPresent + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + env: + # On the RWX volume, not a pod-local one: the default I/O manager writes + # step outputs under $DAGSTER_HOME/storage, and every step pod has to + # read what the previous one wrote. + - name: DAGSTER_HOME + value: /dagster/shared/distexec-probe/home + # How the chart tells a code location which image its steps run as. + - name: DAGSTER_CURRENT_IMAGE + value: gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/j.r/distributed-execution:3f8175a17f0a + - name: DAGSTER_PG_PASSWORD + valueFrom: + secretKeyRef: + name: dagster-postgresql-secret + key: postgresql-password + command: + - sh + - -c + - | + set -e + mkdir -p "$DAGSTER_HOME" + cp /instance-cm/dagster.yaml "$DAGSTER_HOME/dagster.yaml" + echo "===== run config in use =====" + cat /run-config/run-config.yaml + echo "===== T6/T7: tightly_coupled_k8s_job =====" + dagster job execute -f src/distributed_execution/repository.py -j tightly_coupled_k8s_job -c /run-config/run-config.yaml + volumeMounts: + - name: instance-cm + mountPath: /instance-cm + readOnly: true + - name: run-config + mountPath: /run-config + readOnly: true + - name: dagster-shared-storage + mountPath: /dagster/shared + resources: + requests: + cpu: 200m + memory: 512Mi + limits: + cpu: "1" + memory: 1Gi + volumes: + - name: instance-cm + configMap: + name: dagster-instance + - name: run-config + configMap: + name: distexec-probe-run-config + - name: dagster-shared-storage + persistentVolumeClaim: + claimName: dagster-shared-pvc