[SIMPL-30451] Fan out work units so the execution target is observable

The reference graph processed every work unit inside a single op, so exactly one worker was ever reported regardless of executor. That made the guide's claim that k8s_job_executor yields several distinct contributing_hosts false, and left the reference implementations unable to demonstrate the executor choice at all. generate_work_units is now a DynamicOut and both graphs map over it, so one step is created per unit and the loosely coupled pattern dispatches one external workload per unit.

Evidence is split into contributing_workers (host and pid, differs per process) and contributing_hosts (differs only across machines), because the previous single field could not distinguish multiprocess fan-out from no fan-out. Tests now assert the mapped step keys rather than a host count, since execute_in_process ignores executor_def and cannot prove executor behaviour on its own. Adds a Windows note: multiprocess_executor did not complete during authoring and left orphaned processes.

Changelog: fixed
This commit is contained in:
ILay
2026-08-26 18:50:01 +02:00
parent 43dbb81a95
commit ccc2e94c2a
11 changed files with 148 additions and 65 deletions

View File

@@ -250,8 +250,20 @@ operationally:
[yaml/tightly-coupled/rbac-step-executor.yaml](../../yaml/tightly-coupled/rbac-step-executor.yaml).
- Per-step resource requests apply per pod, so the aggregate request for a
fan-out step is the per-step request multiplied by concurrency.
- `summarise_results` now reports several distinct `contributing_hosts` instead of
one. That output metadata is the evidence that the switch actually took effect.
The evidence that the switch took effect is in `summarise_results` output
metadata, and the two fields say different things:
| Executor | `contributing_workers` | `contributing_hosts` |
|---|---|---|
| `in_process_executor` | 1 | 1 |
| `multiprocess_executor` | one per unit | 1 — same machine |
| `k8s_job_executor` | one per unit | one per unit — separate pods |
This only works because `generate_work_units` is a `DynamicOut` and the graph
does `units.map(process_work_unit).collect()`. A single op looping over all units
internally would report one worker under *every* executor, because one step
cannot span processes or pods.
### 5.3 Worked example — switching to loosely coupled
@@ -266,7 +278,7 @@ written, so the diff below is the real one.
def distributed_execution_reference():
target_report = report_execution_target()
units = generate_work_units()
results = process_work_units(units) # runs in-process
results = units.map(process_work_unit).collect() # runs in Dagster
return summarise_results(results, target_report)
```
@@ -277,27 +289,27 @@ def distributed_execution_reference():
def loosely_coupled_k8s_reference():
target_report = report_execution_target()
units = generate_work_units()
results = dispatch_external_work_k8s(units) # dispatches, then listens
results = units.map(dispatch_external_work_k8s).collect() # dispatches, then listens
return summarise_results(results, target_report)
```
The dispatching op replaces direct computation with a pipes client call:
```python
@op(out=Out(list))
@op(out=Out(dict))
def dispatch_external_work_k8s(
context: OpExecutionContext,
units: list,
unit: int,
pipes_k8s_client: PipesK8sClient,
) -> list:
) -> dict:
completed = pipes_k8s_client.run(
context=context,
image=PAYLOAD_IMAGE,
command=["python", "/app/work.py"],
namespace=PAYLOAD_NAMESPACE,
extras={"units": units},
extras={"units": [unit]},
)
return _results_from_pipes(context, completed)
return _result_from_pipes(context, completed)
```
and the job supplies the client as a resource instead of an executor:
@@ -343,6 +355,11 @@ else from the Dagster ecosystem — a test asserts this, because the moment the
payload imports `dagster` the isolation argument for choosing this pattern
collapses.
Because the graph maps over the dynamic output, one external workload is
dispatched **per unit**. A test asserts that the four units come back from four
distinct external workers, which is the loosely coupled equivalent of the
`contributing_workers` evidence in section 5.2.
#### Message channel choice
`PipesK8sClient` defaults to `PipesK8sPodLogsMessageReader`, which is what the
@@ -383,7 +400,7 @@ Those take effect only when a new image is built and the code location reloads.
| Job | Executor / transport | Runs locally | Demonstrates |
|---|---|---|---|
| `tightly_coupled_in_process_job` | `in_process_executor` | Yes | Baseline; steps inside the run worker |
| `tightly_coupled_local_job` | `multiprocess_executor` | Yes | Subprocess fan-out bounded by the run pod |
| `tightly_coupled_local_job` | `multiprocess_executor` | Linux/macOS | Subprocess fan-out bounded by the run pod |
| `tightly_coupled_k8s_job` | `k8s_job_executor` | No — needs a cluster | One Kubernetes Job per step |
| `loosely_coupled_subprocess_job` | `PipesSubprocessClient` | Yes | The pipes contract end to end, no cluster |
| `loosely_coupled_k8s_job` | `PipesK8sClient` | No — needs a cluster | External Job dispatch, messages over pod logs |
@@ -392,6 +409,14 @@ The last two dispatch the same [payload/work.py](../../payload/work.py). The
subprocess variant exists so the pipes contract can be exercised, tested and
demonstrated without any infrastructure.
> **Windows note.** `tightly_coupled_local_job` did not complete on a Windows
> development machine during authoring: the `multiprocess_executor` spawned step
> subprocesses that never terminated, and required manual cleanup. This was not
> reproduced on Linux and is not expected to affect cluster deployments, where
> run workers are Linux pods. For a laptop demonstration on Windows, prefer
> `tightly_coupled_in_process_job` or `loosely_coupled_subprocess_job`, both of
> which are covered by the test suite.
Run the local variants with:
```bash