[SIMPL-30451] Fan out work units so the execution target is observable
The reference graph processed every work unit inside a single op, so exactly one worker was ever reported regardless of executor. That made the guide's claim that k8s_job_executor yields several distinct contributing_hosts false, and left the reference implementations unable to demonstrate the executor choice at all. generate_work_units is now a DynamicOut and both graphs map over it, so one step is created per unit and the loosely coupled pattern dispatches one external workload per unit. Evidence is split into contributing_workers (host and pid, differs per process) and contributing_hosts (differs only across machines), because the previous single field could not distinguish multiprocess fan-out from no fan-out. Tests now assert the mapped step keys rather than a host count, since execute_in_process ignores executor_def and cannot prove executor behaviour on its own. Adds a Windows note: multiprocess_executor did not complete during authoring and left orphaned processes. Changelog: fixed
This commit is contained in:
@@ -52,9 +52,12 @@ def check_endpoint_reachable(url: str, timeout: float = 3.0) -> dict:
|
||||
|
||||
def describe_pod_identity() -> dict:
|
||||
"""Where this process is actually running - the primary execution-target evidence."""
|
||||
hostname = socket.gethostname()
|
||||
return {
|
||||
"hostname": socket.gethostname(),
|
||||
"hostname": hostname,
|
||||
"pid": os.getpid(),
|
||||
# Distinguishes processes on one host, so multiprocess fan-out is visible locally.
|
||||
"worker": f"{hostname}#{os.getpid()}",
|
||||
"namespace": os.environ.get("DAGSTER_K8S_PIPELINE_RUN_NAMESPACE", "<not-in-kubernetes>"),
|
||||
"run_id": os.environ.get("DAGSTER_RUN_ID", "<unset>"),
|
||||
"image": os.environ.get("DAGSTER_K8S_PIPELINE_RUN_IMAGE", "<unset>"),
|
||||
|
||||
Reference in New Issue
Block a user