[SIMPL-30451] Fan out work units so the execution target is observable

The reference graph processed every work unit inside a single op, so exactly one worker was ever reported regardless of executor. That made the guide's claim that k8s_job_executor yields several distinct contributing_hosts false, and left the reference implementations unable to demonstrate the executor choice at all. generate_work_units is now a DynamicOut and both graphs map over it, so one step is created per unit and the loosely coupled pattern dispatches one external workload per unit.

Evidence is split into contributing_workers (host and pid, differs per process) and contributing_hosts (differs only across machines), because the previous single field could not distinguish multiprocess fan-out from no fan-out. Tests now assert the mapped step keys rather than a host count, since execute_in_process ignores executor_def and cannot prove executor behaviour on its own. Adds a Windows note: multiprocess_executor did not complete during authoring and left orphaned processes.

Changelog: fixed
This commit is contained in:
ILay
2026-08-26 18:50:01 +02:00
parent 43dbb81a95
commit ccc2e94c2a
11 changed files with 148 additions and 65 deletions

View File

@@ -17,7 +17,7 @@ from dagster_k8s import k8s_job_executor
from distributed_execution.ops import (
generate_work_units,
process_work_units,
process_work_unit,
report_execution_target,
summarise_results,
)
@@ -47,7 +47,7 @@ def distributed_execution_reference():
"""Shared topology, so the three jobs differ only by execution target."""
target_report = report_execution_target()
units = generate_work_units()
results = process_work_units(units)
results = units.map(process_work_unit).collect()
return summarise_results(results, target_report)