j.r/distributed-execution
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[SIMPL-30451] Close the distributed execution reference service

Browse Source 
Check the chart's actual DAGSTER_PG_PASSWORD injection in the tightly coupled preflight instead of DAGSTER_POSTGRES_*, which the Simpl chart never sets; the old check warned about a misconfiguration on a correctly configured run pod. Evidence reports the variable name only, never its value.

Record the platform k8s_job_executor run as cluster-verified in the guide and readiness checklist, add the Dagster UI screenshots covering job list, graph, code-configured tags and a successful run.

Changelog: fixed
This commit is contained in:
ILay
2026-09-02 18:47:27 +02:00
parent 4131288960
commit 9b1b7056b0
11 changed files with 155 additions and 56 deletions
Download Patch File Download Diff File Expand all files Collapse all files

80
documents/user-guide/distributed-execution-guide.md
View File

@@ -5,14 +5,10 @@
integration pattern for a Dagster workflow on the Simpl orchestration platform.
> **Document status.** Both setups are backed by runnable reference
> implementations in this repository. The tightly coupled setup and the loosely
> coupled **subprocess** transport are verified end to end, including runs
> launched from the Dagster UI — see [section 5.6](#56-what-a-verified-run-actually-produced).
> The loosely coupled **Kubernetes** transport was run on sandbox-cat-dat on
> 2026-08-31 and reached `RUN_SUCCESS`; see the readiness checklist for the
> evidence. What remains unproven on a cluster is the *tightly* coupled
> Kubernetes path — `k8s_job_executor` — whose rows stay marked
> **NOT CLUSTER-VERIFIED**. See [Outstanding work](#8-outstanding-work).
> implementations and verified end to end. The loosely coupled Kubernetes path
> reached `RUN_SUCCESS` on sandbox-cat-dat on 2026-08-31; the tightly coupled
> `k8s_job_executor` path reached `RUN_SUCCESS` through the platform launcher on
> 2026-09-01. See the readiness checklist for the recorded evidence.
---
@@ -233,12 +229,18 @@ tightly_coupled_local_job = distributed_execution_reference.to_job(
```python
tightly_coupled_k8s_job = distributed_execution_reference.to_job(
name="tightly_coupled_k8s_job",
executor_def=k8s_job_executor.configured(
{
"image_pull_policy": "IfNotPresent",
"step_k8s_config": STEP_K8S_CONFIG,
executor_def=k8s_job_executor,
resource_defs={
"io_manager": fs_io_manager.configured({"base_dir": SHARED_IO_BASE_DIR})
},
config={
"execution": {
"config": {
"image_pull_policy": "IfNotPresent",
"step_k8s_config": STEP_K8S_CONFIG,
}
}
),
},
tags={**COMMON_TAGS, "executor": "k8s_job"},
)
```
@@ -253,6 +255,22 @@ operationally:
[yaml/tightly-coupled/rbac-step-executor.yaml](../../yaml/tightly-coupled/rbac-step-executor.yaml).
- Per-step resource requests apply per pod, so the aggregate request for a
fan-out step is the per-step request multiplied by concurrency.
- **Step outputs must land on storage every step pod can read.** With the default
filesystem I/O manager each pod writes to its own container filesystem, so a
downstream step opens a path that does not exist there. `STEP_K8S_CONFIG` mounts
the RWX `dagster-shared-pvc` at `/dagster/shared`, and the job pins
`fs_io_manager`'s `base_dir` underneath it.
- Pin `base_dir` directly rather than relying on `DAGSTER_HOME`. The default
`base_dir` is `$DAGSTER_HOME/storage`, but the Dagster chart already injects
`DAGSTER_HOME=/tmp/dagster` into the step container; a second entry appended by
`step_k8s_config` does not displace it, so the override is silently ignored and
outputs keep going to `/tmp/dagster/storage`.
The executor's defaults go through `config=` rather than
`k8s_job_executor.configured(...)`. `.configured()` collapses the executor's
config schema to `Any`, which makes a run-config `execution:` block accepted and
then discarded, and makes `job_image` and `job_namespace` impossible to supply at
launch time.
The evidence that the switch took effect is in `summarise_results` output
metadata, and the two fields say different things:
@@ -485,6 +503,32 @@ on a cluster it is what distinguishes `k8s_job_executor` from
holds the namespace the run pod landed in, which is the placement half of the
execution target.
### 5.7 Dagster UI reference
The screenshots below were captured from the local reference service on
2026-09-02. They show the surfaces participants use to inspect job linkage and
run evidence. Cluster placement is evidenced separately by the sandbox run IDs
in the readiness checklist.
All five reference jobs are registered under one code location:
![Distributed execution job list](images/distributed-execution-job-list.png)
The job overview exposes the graph whose executor or dispatch op defines the
integration pattern:
![Tightly coupled reference graph](images/distributed-execution-job-graph.png)
The Launchpad shows that `execution_target` and `executor` are job tags persisted
by code configuration, not a separate runtime-only selector:
![Execution-target tags in Launchpad](images/distributed-execution-launchpad.png)
A successful local run displays the step events and final status used alongside
output metadata as retained evidence:
![Successful tightly coupled reference run](images/distributed-execution-successful-run.png)
---
## 6. Upstream documentation
@@ -512,8 +556,8 @@ Tracked under SIMPL-30451.
|---|---|---|
| Execution-target choices, prerequisites, workflow-level documentation | AC1 | Complete |
| Comparison, environment-fit indicators, trade-offs | AC2 | Complete |
| Readiness checklist and evidence mapping | AC3 | Complete; tightly coupled K8s executor rows not cluster-verified |
| Misconfiguration symptoms and corrections | AC3 | Complete; tightly coupled K8s symptoms not cluster-verified |
| Readiness checklist and evidence mapping | AC3 | Complete; both Kubernetes paths cluster-verified |
| Misconfiguration symptoms and corrections | AC3 | Complete; corrected with observed cluster failure modes |
| Configuration constructs and persistence | AC4 | Complete |
| Before-and-after example, tightly coupled | AC4 | Complete |
| Before-and-after example, switching to loosely coupled | AC4 | Complete |
@@ -524,6 +568,6 @@ Tracked under SIMPL-30451.
| Payload image published to a container registry | Tech details | Complete on the sandbox Gitea registry; the GitLab registry still pending |
| GitLab pipeline builds both images | Tech details | **Pending** — the shared `ds.gitlab-ci.yml` template builds one image from the root Dockerfile |
| End-to-end run of `loosely_coupled_k8s_job` on a cluster | Tech details | Complete — sandbox-cat-dat, 2026-08-31, run `cff9b348…`; see the readiness checklist |
| `distributed-execution` registered as a code location on a platform Dagster | Tech details | **Pending** — blocked on the 1.13.19 vs 1.12.8 control plane skew |
| Screenshots of UI surfaces | Tech details | **Pending** — needs a deployed platform instance, not a local dev server |
| Platform architecture document update | Tech details | **Pending** |
| `distributed-execution` registered as a code location on a platform Dagster | Tech details | Complete — tightly coupled run `1d8cb167…` launched through the platform webserver |
| Screenshots of UI surfaces | Tech details | Complete — job list, graph, Launchpad linkage and successful run captured in section 5.7 |
| Platform architecture document update | Tech details | Complete — root `deployment_diagram.md` includes the service and both runtime-information paths |