From 7692891fffa53880b44ee6b3d2de0ee3b9714677 Mon Sep 17 00:00:00 2001 From: ILay Date: Wed, 26 Aug 2026 19:28:19 +0200 Subject: [PATCH] [SIMPL-30451] Record real run evidence for both locally runnable jobs Both jobs were launched from the Dagster UI, so the numbers in the new section 5.6 come from the daemon and run launcher path rather than from execute_in_process. The tightly coupled run reports one contributing worker across seven steps; the loosely coupled subprocess run reports four, which is the difference the two patterns are supposed to make visible. Hostnames are redacted, process IDs are not. Also ignores a local .dagster_home, which the README's dagster dev instructions will create for anyone who sets DAGSTER_HOME alongside the repo. Changelog: added --- .gitignore | 1 + .../user-guide/distributed-execution-guide.md | 53 +++++++++++++++++-- 2 files changed, 50 insertions(+), 4 deletions(-) diff --git a/.gitignore b/.gitignore index d287118..2c6029a 100644 --- a/.gitignore +++ b/.gitignore @@ -5,3 +5,4 @@ .coverage .tmp_dagster_home_*/ +.dagster_home/ diff --git a/documents/user-guide/distributed-execution-guide.md b/documents/user-guide/distributed-execution-guide.md index 09ffefe..89d264b 100644 --- a/documents/user-guide/distributed-execution-guide.md +++ b/documents/user-guide/distributed-execution-guide.md @@ -6,9 +6,10 @@ integration pattern for a Dagster workflow on the Simpl orchestration platform. > **Document status.** Both setups are backed by runnable reference > implementations in this repository. The tightly coupled setup and the loosely -> coupled **subprocess** transport are verified end to end. The loosely coupled -> **Kubernetes** transport is implemented and reviewed but has not yet been run -> against a cluster; statements specific to it are marked +> coupled **subprocess** transport are verified end to end, including runs +> launched from the Dagster UI — see [section 5.6](#56-what-a-verified-run-actually-produced). +> The loosely coupled **Kubernetes** transport is implemented and reviewed but +> has not yet been run against a cluster; statements specific to it are marked > **NOT CLUSTER-VERIFIED**. See [Outstanding work](#8-outstanding-work). --- @@ -434,6 +435,49 @@ uv run dagster dev -f src/distributed_execution/repository.py environment variables as output metadata on every run, which is the evidence the readiness checklist refers to. +### 5.6 What a verified run actually produced + +The two locally runnable jobs were launched from the Dagster UI, so the evidence +below comes from the full daemon → run launcher → executor path rather than from +a unit test. Hostnames are redacted; process IDs are real. + +`tightly_coupled_in_process_job`: + +```text +succeeded steps: 7 + generate_work_units, report_execution_target, + process_work_unit[unit_0..unit_3], summarise_results + +units = 4 +contributing_hosts = ['dev-host'] +contributing_workers = ['dev-host#25264'] +``` + +`loosely_coupled_subprocess_job`: + +```text +succeeded steps: 7 + generate_work_units, report_execution_target, + dispatch_external_work_subprocess[unit_0..unit_3], summarise_results + +units = 4 +contributing_hosts = ['dev-host'] +contributing_workers = ['dev-host#13532', 'dev-host#27116', + 'dev-host#32984', 'dev-host#31224'] +``` + +Read the two blocks together. The step lists are the same shape, which is the +point of section 5.3 — the graphs differ by one node. `contributing_workers` is +what separates them: one entry means the work happened inside the Dagster +process; four means four external processes did it and reported back over the +pipes channel. `contributing_hosts` stays at one because a laptop is one machine; +on a cluster it is what distinguishes `k8s_job_executor` from +`multiprocess_executor`. + +`launcher_namespace` reads `` in both. In a cluster run it +holds the namespace the run pod landed in, which is the placement half of the +execution target. + --- ## 6. Upstream documentation @@ -469,7 +513,8 @@ Tracked under SIMPL-30451. | Runnable tightly coupled reference implementations | AC4 | Complete | | Runnable loosely coupled reference implementation | AC4 | Complete (subprocess verified, K8s not cluster-run) | | Both images build; payload image passes the isolation check (L11) | Tech details | Complete | +| Locally runnable jobs launched from the Dagster UI, evidence recorded in section 5.6 | Tech details | Complete | | Payload image published to the container registry | Tech details | **Pending** — needs the CI build job | | End-to-end run of `loosely_coupled_k8s_job` on a cluster | Tech details | **Pending** | -| Screenshots of UI surfaces | Tech details | **Pending** | +| Screenshots of UI surfaces | Tech details | **Pending** — needs a deployed platform instance, not a local dev server | | Platform architecture document update | Tech details | **Pending** |