[SIMPL-30451] Fan out work units so the execution target is observable
The reference graph processed every work unit inside a single op, so exactly one worker was ever reported regardless of executor. That made the guide's claim that k8s_job_executor yields several distinct contributing_hosts false, and left the reference implementations unable to demonstrate the executor choice at all. generate_work_units is now a DynamicOut and both graphs map over it, so one step is created per unit and the loosely coupled pattern dispatches one external workload per unit. Evidence is split into contributing_workers (host and pid, differs per process) and contributing_hosts (differs only across machines), because the previous single field could not distinguish multiprocess fan-out from no fan-out. Tests now assert the mapped step keys rather than a host count, since execute_in_process ignores executor_def and cannot prove executor behaviour on its own. Adds a Windows note: multiprocess_executor did not complete during authoring and left orphaned processes. Changelog: fixed
This commit is contained in:
10
README.md
10
README.md
@@ -53,9 +53,13 @@ uv run dagster dev -f src/distributed_execution/repository.py
|
||||
```
|
||||
|
||||
The Dagster UI is then available at <http://localhost:3000>. Two jobs run end to
|
||||
end on a laptop with no cluster: `tightly_coupled_local_job` and
|
||||
`loosely_coupled_subprocess_job`. The Kubernetes variants of each are documented
|
||||
in the user guide.
|
||||
end on a laptop with no cluster: `tightly_coupled_in_process_job` and
|
||||
`loosely_coupled_subprocess_job`. The Kubernetes variants of each require a
|
||||
cluster and are documented in the user guide.
|
||||
|
||||
> `tightly_coupled_local_job` uses `multiprocess_executor`. It did not complete on
|
||||
> a Windows development machine during authoring — see the Windows note in the
|
||||
> user guide's section 5.5.
|
||||
|
||||
### Running tests
|
||||
|
||||
|
||||
Reference in New Issue
Block a user