[SIMPL-30451] Fan out work units so the execution target is observable
The reference graph processed every work unit inside a single op, so exactly one worker was ever reported regardless of executor. That made the guide's claim that k8s_job_executor yields several distinct contributing_hosts false, and left the reference implementations unable to demonstrate the executor choice at all. generate_work_units is now a DynamicOut and both graphs map over it, so one step is created per unit and the loosely coupled pattern dispatches one external workload per unit. Evidence is split into contributing_workers (host and pid, differs per process) and contributing_hosts (differs only across machines), because the previous single field could not distinguish multiprocess fan-out from no fan-out. Tests now assert the mapped step keys rather than a host count, since execute_in_process ignores executor_def and cannot prove executor behaviour on its own. Adds a Windows note: multiprocess_executor did not complete during authoring and left orphaned processes. Changelog: fixed
This commit is contained in:
@@ -250,8 +250,20 @@ operationally:
|
||||
[yaml/tightly-coupled/rbac-step-executor.yaml](../../yaml/tightly-coupled/rbac-step-executor.yaml).
|
||||
- Per-step resource requests apply per pod, so the aggregate request for a
|
||||
fan-out step is the per-step request multiplied by concurrency.
|
||||
- `summarise_results` now reports several distinct `contributing_hosts` instead of
|
||||
one. That output metadata is the evidence that the switch actually took effect.
|
||||
|
||||
The evidence that the switch took effect is in `summarise_results` output
|
||||
metadata, and the two fields say different things:
|
||||
|
||||
| Executor | `contributing_workers` | `contributing_hosts` |
|
||||
|---|---|---|
|
||||
| `in_process_executor` | 1 | 1 |
|
||||
| `multiprocess_executor` | one per unit | 1 — same machine |
|
||||
| `k8s_job_executor` | one per unit | one per unit — separate pods |
|
||||
|
||||
This only works because `generate_work_units` is a `DynamicOut` and the graph
|
||||
does `units.map(process_work_unit).collect()`. A single op looping over all units
|
||||
internally would report one worker under *every* executor, because one step
|
||||
cannot span processes or pods.
|
||||
|
||||
### 5.3 Worked example — switching to loosely coupled
|
||||
|
||||
@@ -266,7 +278,7 @@ written, so the diff below is the real one.
|
||||
def distributed_execution_reference():
|
||||
target_report = report_execution_target()
|
||||
units = generate_work_units()
|
||||
results = process_work_units(units) # runs in-process
|
||||
results = units.map(process_work_unit).collect() # runs in Dagster
|
||||
return summarise_results(results, target_report)
|
||||
```
|
||||
|
||||
@@ -277,27 +289,27 @@ def distributed_execution_reference():
|
||||
def loosely_coupled_k8s_reference():
|
||||
target_report = report_execution_target()
|
||||
units = generate_work_units()
|
||||
results = dispatch_external_work_k8s(units) # dispatches, then listens
|
||||
results = units.map(dispatch_external_work_k8s).collect() # dispatches, then listens
|
||||
return summarise_results(results, target_report)
|
||||
```
|
||||
|
||||
The dispatching op replaces direct computation with a pipes client call:
|
||||
|
||||
```python
|
||||
@op(out=Out(list))
|
||||
@op(out=Out(dict))
|
||||
def dispatch_external_work_k8s(
|
||||
context: OpExecutionContext,
|
||||
units: list,
|
||||
unit: int,
|
||||
pipes_k8s_client: PipesK8sClient,
|
||||
) -> list:
|
||||
) -> dict:
|
||||
completed = pipes_k8s_client.run(
|
||||
context=context,
|
||||
image=PAYLOAD_IMAGE,
|
||||
command=["python", "/app/work.py"],
|
||||
namespace=PAYLOAD_NAMESPACE,
|
||||
extras={"units": units},
|
||||
extras={"units": [unit]},
|
||||
)
|
||||
return _results_from_pipes(context, completed)
|
||||
return _result_from_pipes(context, completed)
|
||||
```
|
||||
|
||||
and the job supplies the client as a resource instead of an executor:
|
||||
@@ -343,6 +355,11 @@ else from the Dagster ecosystem — a test asserts this, because the moment the
|
||||
payload imports `dagster` the isolation argument for choosing this pattern
|
||||
collapses.
|
||||
|
||||
Because the graph maps over the dynamic output, one external workload is
|
||||
dispatched **per unit**. A test asserts that the four units come back from four
|
||||
distinct external workers, which is the loosely coupled equivalent of the
|
||||
`contributing_workers` evidence in section 5.2.
|
||||
|
||||
#### Message channel choice
|
||||
|
||||
`PipesK8sClient` defaults to `PipesK8sPodLogsMessageReader`, which is what the
|
||||
@@ -383,7 +400,7 @@ Those take effect only when a new image is built and the code location reloads.
|
||||
| Job | Executor / transport | Runs locally | Demonstrates |
|
||||
|---|---|---|---|
|
||||
| `tightly_coupled_in_process_job` | `in_process_executor` | Yes | Baseline; steps inside the run worker |
|
||||
| `tightly_coupled_local_job` | `multiprocess_executor` | Yes | Subprocess fan-out bounded by the run pod |
|
||||
| `tightly_coupled_local_job` | `multiprocess_executor` | Linux/macOS | Subprocess fan-out bounded by the run pod |
|
||||
| `tightly_coupled_k8s_job` | `k8s_job_executor` | No — needs a cluster | One Kubernetes Job per step |
|
||||
| `loosely_coupled_subprocess_job` | `PipesSubprocessClient` | Yes | The pipes contract end to end, no cluster |
|
||||
| `loosely_coupled_k8s_job` | `PipesK8sClient` | No — needs a cluster | External Job dispatch, messages over pod logs |
|
||||
@@ -392,6 +409,14 @@ The last two dispatch the same [payload/work.py](../../payload/work.py). The
|
||||
subprocess variant exists so the pipes contract can be exercised, tested and
|
||||
demonstrated without any infrastructure.
|
||||
|
||||
> **Windows note.** `tightly_coupled_local_job` did not complete on a Windows
|
||||
> development machine during authoring: the `multiprocess_executor` spawned step
|
||||
> subprocesses that never terminated, and required manual cleanup. This was not
|
||||
> reproduced on Linux and is not expected to affect cluster deployments, where
|
||||
> run workers are Linux pods. For a laptop demonstration on Windows, prefer
|
||||
> `tightly_coupled_in_process_job` or `loosely_coupled_subprocess_job`, both of
|
||||
> which are covered by the test suite.
|
||||
|
||||
Run the local variants with:
|
||||
|
||||
```bash
|
||||
|
||||
Reference in New Issue
Block a user