[SIMPL-30451] Fan out work units so the execution target is observable

The reference graph processed every work unit inside a single op, so exactly one worker was ever reported regardless of executor. That made the guide's claim that k8s_job_executor yields several distinct contributing_hosts false, and left the reference implementations unable to demonstrate the executor choice at all. generate_work_units is now a DynamicOut and both graphs map over it, so one step is created per unit and the loosely coupled pattern dispatches one external workload per unit.

Evidence is split into contributing_workers (host and pid, differs per process) and contributing_hosts (differs only across machines), because the previous single field could not distinguish multiprocess fan-out from no fan-out. Tests now assert the mapped step keys rather than a host count, since execute_in_process ignores executor_def and cannot prove executor behaviour on its own. Adds a Windows note: multiprocess_executor did not complete during authoring and left orphaned processes.

Changelog: fixed
This commit is contained in:
ILay
2026-08-26 18:50:01 +02:00
parent 43dbb81a95
commit ccc2e94c2a
11 changed files with 148 additions and 65 deletions

View File

@@ -53,9 +53,13 @@ uv run dagster dev -f src/distributed_execution/repository.py
```
The Dagster UI is then available at <http://localhost:3000>. Two jobs run end to
end on a laptop with no cluster: `tightly_coupled_local_job` and
`loosely_coupled_subprocess_job`. The Kubernetes variants of each are documented
in the user guide.
end on a laptop with no cluster: `tightly_coupled_in_process_job` and
`loosely_coupled_subprocess_job`. The Kubernetes variants of each require a
cluster and are documented in the user guide.
> `tightly_coupled_local_job` uses `multiprocess_executor`. It did not complete on
> a Windows development machine during authoring — see the Windows note in the
> user guide's section 5.5.
### Running tests