j.r/distributed-execution
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[SIMPL-30451] Fan out work units so the execution target is observable

Browse Source 
The reference graph processed every work unit inside a single op, so exactly one worker was ever reported regardless of executor. That made the guide's claim that k8s_job_executor yields several distinct contributing_hosts false, and left the reference implementations unable to demonstrate the executor choice at all. generate_work_units is now a DynamicOut and both graphs map over it, so one step is created per unit and the loosely coupled pattern dispatches one external workload per unit.

Evidence is split into contributing_workers (host and pid, differs per process) and contributing_hosts (differs only across machines), because the previous single field could not distinguish multiprocess fan-out from no fan-out. Tests now assert the mapped step keys rather than a host count, since execute_in_process ignores executor_def and cannot prove executor behaviour on its own. Adds a Windows note: multiprocess_executor did not complete during authoring and left orphaned processes.

Changelog: fixed
This commit is contained in:
ILay
2026-08-26 18:50:01 +02:00
parent 43dbb81a95
commit ccc2e94c2a
11 changed files with 148 additions and 65 deletions
Download Patch File Download Diff File Expand all files Collapse all files

45
documents/user-guide/distributed-execution-guide.md
View File

@@ -250,8 +250,20 @@ operationally:
[yaml/tightly-coupled/rbac-step-executor.yaml](../../yaml/tightly-coupled/rbac-step-executor.yaml).
- Per-step resource requests apply per pod, so the aggregate request for a
fan-out step is the per-step request multiplied by concurrency.
- `summarise_results` now reports several distinct `contributing_hosts` instead of
one. That output metadata is the evidence that the switch actually took effect.
The evidence that the switch took effect is in `summarise_results` output
metadata, and the two fields say different things:
| Executor | `contributing_workers` | `contributing_hosts` |
|---|---|---|
| `in_process_executor` | 1 | 1 |
| `multiprocess_executor` | one per unit | 1 — same machine |
| `k8s_job_executor` | one per unit | one per unit — separate pods |
This only works because `generate_work_units` is a `DynamicOut` and the graph
does `units.map(process_work_unit).collect()`. A single op looping over all units
internally would report one worker under *every* executor, because one step
cannot span processes or pods.
### 5.3 Worked example — switching to loosely coupled
@@ -266,7 +278,7 @@ written, so the diff below is the real one.
def distributed_execution_reference():
target_report = report_execution_target()
units = generate_work_units()
results = process_work_units(units) # runs in-process
results = units.map(process_work_unit).collect() # runs in Dagster
return summarise_results(results, target_report)
```
@@ -277,27 +289,27 @@ def distributed_execution_reference():
def loosely_coupled_k8s_reference():
target_report = report_execution_target()
units = generate_work_units()
results = dispatch_external_work_k8s(units) # dispatches, then listens
results = units.map(dispatch_external_work_k8s).collect() # dispatches, then listens
return summarise_results(results, target_report)
```
The dispatching op replaces direct computation with a pipes client call:
```python
@op(out=Out(list))
@op(out=Out(dict))
def dispatch_external_work_k8s(
context: OpExecutionContext,
units: list,
unit: int,
pipes_k8s_client: PipesK8sClient,
) -> list:
) -> dict:
completed = pipes_k8s_client.run(
context=context,
image=PAYLOAD_IMAGE,
command=["python", "/app/work.py"],
namespace=PAYLOAD_NAMESPACE,
extras={"units": units},
extras={"units": [unit]},
)
return _results_from_pipes(context, completed)
return _result_from_pipes(context, completed)
```
and the job supplies the client as a resource instead of an executor:
@@ -343,6 +355,11 @@ else from the Dagster ecosystem — a test asserts this, because the moment the
payload imports `dagster` the isolation argument for choosing this pattern
collapses.
Because the graph maps over the dynamic output, one external workload is
dispatched **per unit**. A test asserts that the four units come back from four
distinct external workers, which is the loosely coupled equivalent of the
`contributing_workers` evidence in section 5.2.
#### Message channel choice
`PipesK8sClient` defaults to `PipesK8sPodLogsMessageReader`, which is what the
@@ -383,7 +400,7 @@ Those take effect only when a new image is built and the code location reloads.
| Job | Executor / transport | Runs locally | Demonstrates |
|---|---|---|---|
| `tightly_coupled_in_process_job` | `in_process_executor` | Yes | Baseline; steps inside the run worker |
| `tightly_coupled_local_job` | `multiprocess_executor` | Yes | Subprocess fan-out bounded by the run pod |
| `tightly_coupled_local_job` | `multiprocess_executor` | Linux/macOS | Subprocess fan-out bounded by the run pod |
| `tightly_coupled_k8s_job` | `k8s_job_executor` | No — needs a cluster | One Kubernetes Job per step |
| `loosely_coupled_subprocess_job` | `PipesSubprocessClient` | Yes | The pipes contract end to end, no cluster |
| `loosely_coupled_k8s_job` | `PipesK8sClient` | No — needs a cluster | External Job dispatch, messages over pod logs |
@@ -392,6 +409,14 @@ The last two dispatch the same [payload/work.py](../../payload/work.py). The
subprocess variant exists so the pipes contract can be exercised, tested and
demonstrated without any infrastructure.
> **Windows note.** `tightly_coupled_local_job` did not complete on a Windows
> development machine during authoring: the `multiprocess_executor` spawned step
> subprocesses that never terminated, and required manual cleanup. This was not
> reproduced on Linux and is not expected to affect cluster deployments, where
> run workers are Linux pods. For a laptop demonstration on Windows, prefer
> `tightly_coupled_in_process_job` or `loosely_coupled_subprocess_job`, both of
> which are covered by the test suite.
Run the local variants with:
```bash

15
documents/user-guide/readiness-checklist.md
View File

@@ -9,8 +9,8 @@ location image.
> **NOT CLUSTER-VERIFIED.** Checks L4–L9 and the Kubernetes rows of section 4.3
> are derived from the implemented reference but have not yet been run against a
> Simpl cluster. Checks L1–L3, which exercise the payload contract and the
> message-parsing path, are covered by the test suite.
> Simpl cluster. Checks L1–L3 and L10, which exercise the payload contract, the
> message-parsing path and per-unit dispatch, are covered by the test suite.
---
@@ -32,9 +32,9 @@ location image.
| T1 | Run pod reaches the metadata database | Launch `tightly_coupled_in_process_job` | Run reaches `SUCCESS`; `report_execution_target` output metadata lists `DAGSTER_POSTGRES_HOST`, `DAGSTER_POSTGRES_USER` and `DAGSTER_POSTGRES_DB` under `env_vars_present`, and `env_vars_missing` is empty |
| T2 | Vault injection works | Same run; inspect the run pod | `kubectl -n dagster describe pod <run-pod>` shows the `vault-env` init container completed; no `vault:` literal remains in the process environment |
| T3 | Object storage is reachable | Same run, if the workflow uses S3 | No `EndpointConnectionError` in run logs; `S3_ENDPOINT_URL` present in `env_vars_present` |
| T4 | Multiprocess fan-out is bounded correctly | Launch `tightly_coupled_local_job` | Run succeeds; `summarise_results` metadata shows exactly one entry in `contributing_hosts`, confirming steps stayed on the run worker |
| T4 | Multiprocess fan-out actually fans out | Launch `tightly_coupled_local_job` | Run succeeds; `summarise_results` metadata shows one entry per unit in `contributing_workers` and a single entry in `contributing_hosts` — separate processes, same machine |
| T5 | RBAC permits step Jobs | `kubectl -n dagster auth can-i create jobs --as=system:serviceaccount:dagster:dagster-dev` | Returns `yes`; required only for `k8s_job_executor` |
| T6 | Step pods are actually created | Launch `tightly_coupled_k8s_job`, then `kubectl -n dagster get jobs -l dagster/run-id=<run-id>` | One Job per step; `summarise_results` metadata shows **several** distinct `contributing_hosts` |
| T6 | Step pods are actually created | Launch `tightly_coupled_k8s_job`, then `kubectl -n dagster get jobs -l dagster/run-id=<run-id>` | One Job per mapped unit; `contributing_hosts` now shows one entry **per unit**, not one |
| T7 | Step pod egress is permitted | Same run | Steps do not hang in `STARTING`; run logs contain no connection timeouts to port 5432 |
| T8 | Failure surfaces as a pod failure | Force a step failure in a scratch namespace | `failPodOnRunFailure: true` is set, and the step pod reports `Failed` rather than `Completed` |
@@ -50,10 +50,11 @@ location image.
| L6 | Dispatcher can read pod logs — **the message channel** | `kubectl -n <payload-ns> auth can-i get pods/log --as=system:serviceaccount:dagster:dagster-dev` | Returns `yes`. A `no` here breaks reporting *without* failing the workload |
| L7 | Payload Job is actually created | Launch `loosely_coupled_k8s_job`, then `kubectl -n <payload-ns> get jobs -l app.kubernetes.io/name=distributed-execution-payload` | One Job per dispatch, labelled `dagster/execution-target=loosely-coupled` |
| L8 | Payload has **no** orchestration connectivity | Same run; inspect run logs | No `Payload could see orchestration runtime credentials` warning. This is a positive check — absence of errors is not sufficient |
| L9 | Work ran off-platform | Same run | `summarise_results` metadata shows `contributing_hosts` containing the payload pod name, not the run worker's hostname |
| L9 | Work ran off-platform | Same run | `summarise_results` metadata shows `contributing_hosts` containing the payload pod names, not the run worker's hostname |
| L10 | One workload dispatched per unit | Same run, or `uv run pytest -k dispatched_to_its_own` locally | `contributing_workers` has one entry per unit; the local test asserts four distinct external workers |
Checks L4–L9 require a cluster. L1–L3 run on a laptop and should gate every
change to the payload or the dispatching op.
Checks L4–L9 require a cluster. L1–L3 and L10 run on a laptop and should gate
every change to the payload or the dispatching op.
---