[SIMPL-30451] Fan out work units so the execution target is observable
The reference graph processed every work unit inside a single op, so exactly one worker was ever reported regardless of executor. That made the guide's claim that k8s_job_executor yields several distinct contributing_hosts false, and left the reference implementations unable to demonstrate the executor choice at all. generate_work_units is now a DynamicOut and both graphs map over it, so one step is created per unit and the loosely coupled pattern dispatches one external workload per unit. Evidence is split into contributing_workers (host and pid, differs per process) and contributing_hosts (differs only across machines), because the previous single field could not distinguish multiprocess fan-out from no fan-out. Tests now assert the mapped step keys rather than a host count, since execute_in_process ignores executor_def and cannot prove executor behaviour on its own. Adds a Windows note: multiprocess_executor did not complete during authoring and left orphaned processes. Changelog: fixed
This commit is contained in:
@@ -17,7 +17,7 @@ from dagster_k8s import k8s_job_executor
|
||||
|
||||
from distributed_execution.ops import (
|
||||
generate_work_units,
|
||||
process_work_units,
|
||||
process_work_unit,
|
||||
report_execution_target,
|
||||
summarise_results,
|
||||
)
|
||||
@@ -47,7 +47,7 @@ def distributed_execution_reference():
|
||||
"""Shared topology, so the three jobs differ only by execution target."""
|
||||
target_report = report_execution_target()
|
||||
units = generate_work_units()
|
||||
results = process_work_units(units)
|
||||
results = units.map(process_work_unit).collect()
|
||||
return summarise_results(results, target_report)
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user