[SIMPL-30451] Fan out work units so the execution target is observable

The reference graph processed every work unit inside a single op, so exactly one worker was ever reported regardless of executor. That made the guide's claim that k8s_job_executor yields several distinct contributing_hosts false, and left the reference implementations unable to demonstrate the executor choice at all. generate_work_units is now a DynamicOut and both graphs map over it, so one step is created per unit and the loosely coupled pattern dispatches one external workload per unit.

Evidence is split into contributing_workers (host and pid, differs per process) and contributing_hosts (differs only across machines), because the previous single field could not distinguish multiprocess fan-out from no fan-out. Tests now assert the mapped step keys rather than a host count, since execute_in_process ignores executor_def and cannot prove executor behaviour on its own. Adds a Windows note: multiprocess_executor did not complete during authoring and left orphaned processes.

Changelog: fixed
This commit is contained in:
ILay
2026-08-26 18:50:01 +02:00
parent 43dbb81a95
commit ccc2e94c2a
11 changed files with 148 additions and 65 deletions

View File

@@ -52,9 +52,12 @@ def check_endpoint_reachable(url: str, timeout: float = 3.0) -> dict:
def describe_pod_identity() -> dict:
"""Where this process is actually running - the primary execution-target evidence."""
hostname = socket.gethostname()
return {
"hostname": socket.gethostname(),
"hostname": hostname,
"pid": os.getpid(),
# Distinguishes processes on one host, so multiprocess fan-out is visible locally.
"worker": f"{hostname}#{os.getpid()}",
"namespace": os.environ.get("DAGSTER_K8S_PIPELINE_RUN_NAMESPACE", "<not-in-kubernetes>"),
"run_id": os.environ.get("DAGSTER_RUN_ID", "<unset>"),
"image": os.environ.get("DAGSTER_K8S_PIPELINE_RUN_IMAGE", "<unset>"),