[SIMPL-30451] Fan out work units so the execution target is observable
The reference graph processed every work unit inside a single op, so exactly one worker was ever reported regardless of executor. That made the guide's claim that k8s_job_executor yields several distinct contributing_hosts false, and left the reference implementations unable to demonstrate the executor choice at all. generate_work_units is now a DynamicOut and both graphs map over it, so one step is created per unit and the loosely coupled pattern dispatches one external workload per unit. Evidence is split into contributing_workers (host and pid, differs per process) and contributing_hosts (differs only across machines), because the previous single field could not distinguish multiprocess fan-out from no fan-out. Tests now assert the mapped step keys rather than a host count, since execute_in_process ignores executor_def and cannot prove executor behaviour on its own. Adds a Windows note: multiprocess_executor did not complete during authoring and left orphaned processes. Changelog: fixed
This commit is contained in:
@@ -27,10 +27,13 @@ def main() -> None:
|
||||
with open_dagster_pipes() as pipes:
|
||||
units = pipes.get_extra("units")
|
||||
host = socket.gethostname()
|
||||
worker = f"{host}#{os.getpid()}"
|
||||
|
||||
pipes.log.info(f"External payload started on {host} with {len(units)} work units")
|
||||
pipes.log.info(f"External payload started on {worker} with {len(units)} work units")
|
||||
|
||||
results = [{"unit": unit, "squared": unit * unit, "host": host} for unit in units]
|
||||
results = [
|
||||
{"unit": unit, "squared": unit * unit, "host": host, "worker": worker} for unit in units
|
||||
]
|
||||
|
||||
leaked = [name for name in ORCHESTRATION_ENV_VARS if os.environ.get(name)]
|
||||
if leaked:
|
||||
@@ -44,11 +47,12 @@ def main() -> None:
|
||||
{
|
||||
"results": results,
|
||||
"host": host,
|
||||
"worker": worker,
|
||||
"orchestration_env_visible": leaked,
|
||||
}
|
||||
)
|
||||
|
||||
pipes.log.info(f"External payload finished on {host}")
|
||||
pipes.log.info(f"External payload finished on {worker}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
Reference in New Issue
Block a user