j.r/distributed-execution
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[SIMPL-30451] Correct stale references in the guide and verify both images build

Browse Source 
The rename of process_work_units and _results_from_pipes during the fan-out refactor left three references behind, in the guide, the checklist and the loosely coupled module docstring. The guide also showed loosely_coupled_k8s_job without its executor_def and claimed the pipes client replaced the executor, which stopped being true once the graph fanned out.

Both images were built and smoke tested: the code location image loads its definitions, and the payload image ships dagster_pipes without dagster. That check is now recorded as L11 in the readiness checklist, so the isolation argument is verified at image level and not only in the source.

Changelog: fixed
This commit is contained in:
ILay
2026-08-26 19:21:41 +02:00
parent ccc2e94c2a
commit 1ce332fa1b
4 changed files with 25 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files

13
documents/user-guide/distributed-execution-guide.md
View File

@@ -312,16 +312,22 @@ def dispatch_external_work_k8s(
return _result_from_pipes(context, completed)
```
and the job supplies the client as a resource instead of an executor:
and the job supplies the client as a resource:
```python
loosely_coupled_k8s_job = loosely_coupled_k8s_reference.to_job(
name="loosely_coupled_k8s_job",
executor_def=multiprocess_executor.configured({"max_concurrent": 2}),
resource_defs={"pipes_k8s_client": PipesK8sClient()},
tags={**COMMON_TAGS, "transport": "k8s_pod_logs"},
)
```
The executor is still present, but its meaning has changed. It no longer decides
where the work runs — the payload image and `namespace` do that. It only bounds
how many dispatching ops wait on the message channel at once. The two decisions
that were fused in the tightly coupled setup are now independent.
#### What changes operationally
| | Tightly coupled | Loosely coupled |
@@ -366,7 +372,7 @@ distinct external workers, which is the loosely coupled equivalent of the
reference uses. It needs no extra infrastructure, but it couples message
delivery to the pod log stream — a log shipper that intercepts or truncates
stdout will break reporting while the workload still exits 0. That is why
`_results_from_pipes` raises on an empty message list rather than returning an
`_result_from_pipes` raises on an empty message list rather than returning an
empty result. If your cluster's logging setup makes that transport unreliable,
switch to an object-storage message reader, which needs a bucket plus
credentials on both sides.
@@ -462,7 +468,8 @@ Tracked under SIMPL-30451.
| Before-and-after example, switching to loosely coupled | AC4 | Complete |
| Runnable tightly coupled reference implementations | AC4 | Complete |
| Runnable loosely coupled reference implementation | AC4 | Complete (subprocess verified, K8s not cluster-run) |
| Payload image published to the registry | Tech details | **Pending** |
| Both images build; payload image passes the isolation check (L11) | Tech details | Complete |
| Payload image published to the container registry | Tech details | **Pending** — needs the CI build job |
| End-to-end run of `loosely_coupled_k8s_job` on a cluster | Tech details | **Pending** |
| Screenshots of UI surfaces | Tech details | **Pending** |
| Platform architecture document update | Tech details | **Pending** |