diff --git a/.gitignore b/.gitignore index d287118..2c6029a 100644 --- a/.gitignore +++ b/.gitignore @@ -5,3 +5,4 @@ .coverage .tmp_dagster_home_*/ +.dagster_home/ diff --git a/documents/user-guide/distributed-execution-guide.md b/documents/user-guide/distributed-execution-guide.md index 09ffefe..89d264b 100644 --- a/documents/user-guide/distributed-execution-guide.md +++ b/documents/user-guide/distributed-execution-guide.md @@ -6,9 +6,10 @@ integration pattern for a Dagster workflow on the Simpl orchestration platform. > **Document status.** Both setups are backed by runnable reference > implementations in this repository. The tightly coupled setup and the loosely -> coupled **subprocess** transport are verified end to end. The loosely coupled -> **Kubernetes** transport is implemented and reviewed but has not yet been run -> against a cluster; statements specific to it are marked +> coupled **subprocess** transport are verified end to end, including runs +> launched from the Dagster UI — see [section 5.6](#56-what-a-verified-run-actually-produced). +> The loosely coupled **Kubernetes** transport is implemented and reviewed but +> has not yet been run against a cluster; statements specific to it are marked > **NOT CLUSTER-VERIFIED**. See [Outstanding work](#8-outstanding-work). --- @@ -434,6 +435,49 @@ uv run dagster dev -f src/distributed_execution/repository.py environment variables as output metadata on every run, which is the evidence the readiness checklist refers to. +### 5.6 What a verified run actually produced + +The two locally runnable jobs were launched from the Dagster UI, so the evidence +below comes from the full daemon → run launcher → executor path rather than from +a unit test. Hostnames are redacted; process IDs are real. + +`tightly_coupled_in_process_job`: + +```text +succeeded steps: 7 + generate_work_units, report_execution_target, + process_work_unit[unit_0..unit_3], summarise_results + +units = 4 +contributing_hosts = ['dev-host'] +contributing_workers = ['dev-host#25264'] +``` + +`loosely_coupled_subprocess_job`: + +```text +succeeded steps: 7 + generate_work_units, report_execution_target, + dispatch_external_work_subprocess[unit_0..unit_3], summarise_results + +units = 4 +contributing_hosts = ['dev-host'] +contributing_workers = ['dev-host#13532', 'dev-host#27116', + 'dev-host#32984', 'dev-host#31224'] +``` + +Read the two blocks together. The step lists are the same shape, which is the +point of section 5.3 — the graphs differ by one node. `contributing_workers` is +what separates them: one entry means the work happened inside the Dagster +process; four means four external processes did it and reported back over the +pipes channel. `contributing_hosts` stays at one because a laptop is one machine; +on a cluster it is what distinguishes `k8s_job_executor` from +`multiprocess_executor`. + +`launcher_namespace` reads `` in both. In a cluster run it +holds the namespace the run pod landed in, which is the placement half of the +execution target. + --- ## 6. Upstream documentation @@ -469,7 +513,8 @@ Tracked under SIMPL-30451. | Runnable tightly coupled reference implementations | AC4 | Complete | | Runnable loosely coupled reference implementation | AC4 | Complete (subprocess verified, K8s not cluster-run) | | Both images build; payload image passes the isolation check (L11) | Tech details | Complete | +| Locally runnable jobs launched from the Dagster UI, evidence recorded in section 5.6 | Tech details | Complete | | Payload image published to the container registry | Tech details | **Pending** — needs the CI build job | | End-to-end run of `loosely_coupled_k8s_job` on a cluster | Tech details | **Pending** | -| Screenshots of UI surfaces | Tech details | **Pending** | +| Screenshots of UI surfaces | Tech details | **Pending** — needs a deployed platform instance, not a local dev server | | Platform architecture document update | Tech details | **Pending** |