[SIMPL-30451] Fan out work units so the execution target is observable

The reference graph processed every work unit inside a single op, so exactly one worker was ever reported regardless of executor. That made the guide's claim that k8s_job_executor yields several distinct contributing_hosts false, and left the reference implementations unable to demonstrate the executor choice at all. generate_work_units is now a DynamicOut and both graphs map over it, so one step is created per unit and the loosely coupled pattern dispatches one external workload per unit.

Evidence is split into contributing_workers (host and pid, differs per process) and contributing_hosts (differs only across machines), because the previous single field could not distinguish multiprocess fan-out from no fan-out. Tests now assert the mapped step keys rather than a host count, since execute_in_process ignores executor_def and cannot prove executor behaviour on its own. Adds a Windows note: multiprocess_executor did not complete during authoring and left orphaned processes.

Changelog: fixed
This commit is contained in:
ILay
2026-08-26 18:50:01 +02:00
parent 43dbb81a95
commit ccc2e94c2a
11 changed files with 148 additions and 65 deletions

View File

@@ -27,10 +27,13 @@ def main() -> None:
with open_dagster_pipes() as pipes:
units = pipes.get_extra("units")
host = socket.gethostname()
worker = f"{host}#{os.getpid()}"
pipes.log.info(f"External payload started on {host} with {len(units)} work units")
pipes.log.info(f"External payload started on {worker} with {len(units)} work units")
results = [{"unit": unit, "squared": unit * unit, "host": host} for unit in units]
results = [
{"unit": unit, "squared": unit * unit, "host": host, "worker": worker} for unit in units
]
leaked = [name for name in ORCHESTRATION_ENV_VARS if os.environ.get(name)]
if leaked:
@@ -44,11 +47,12 @@ def main() -> None:
{
"results": results,
"host": host,
"worker": worker,
"orchestration_env_visible": leaked,
}
)
pipes.log.info(f"External payload finished on {host}")
pipes.log.info(f"External payload finished on {worker}")
if __name__ == "__main__":