j.r/distributed-execution
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[SIMPL-30451] Distributed execution reference service and container-cluster configuration guide #1

Merged
j.r merged 23 commits from feature/SIMPL-30451-distributed-execution into develop 2026-09-02 17:38:23 +00:00
Conversation 0 Commits 23 Files Changed 28 +193 -27
5 changed files with 193 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Showing only changes of commit aa49420ca6 - Show all commits

3
CHANGELOG.md
View File

@@ -4,6 +4,9 @@
- Repository skeleton for the `distributed-execution` supporting data service. - Repository skeleton for the `distributed-execution` supporting data service.
- User guide covering execution-target selection, decision support and readiness checks. - User guide covering execution-target selection, decision support and readiness checks.
- Cluster-configuration input reference: mandatory versus optional inputs per setup,
a reusable minimum input set and worked examples in both setups' own formats.
- Requirement traceability mapping the guide's sections onto SIMPL-30451 and SIMPL-30787.
- Runnable tightly coupled reference implementations (`in_process`, `multiprocess`, `k8s_job_executor`). - Runnable tightly coupled reference implementations (`in_process`, `multiprocess`, `k8s_job_executor`).
- Runnable loosely coupled reference implementations via dagster-pipes - Runnable loosely coupled reference implementations via dagster-pipes
(`PipesSubprocessClient`, `PipesK8sClient`) with a standalone external payload image. (`PipesSubprocessClient`, `PipesK8sClient`) with a standalone external payload image.

2
README.md
View File

@@ -35,7 +35,7 @@ distributed-execution/
│ ├── work.py │ ├── work.py
│ ├── requirements.txt │ ├── requirements.txt
│ └── Dockerfile │ └── Dockerfile
├── documents/user-guide/ # AC1-AC4 documentation ├── documents/user-guide/ # Guide, checklist and requirement traceability
├── yaml/ # Working example configuration ├── yaml/ # Working example configuration
├── tests/ ├── tests/
├── Dockerfile ├── Dockerfile

206
documents/user-guide/distributed-execution-guide.md
View File

@@ -66,7 +66,7 @@ pattern.
--- ---
## 2. AC1 — Choosing an execution target ## 2. Choosing an execution target and cluster setup
### 2.1 Choose tightly coupled when ### 2.1 Choose tightly coupled when
@@ -121,7 +121,7 @@ before it is run for the first time. Section 5.4 explains where each item lives.
--- ---
## 3. AC2 — Decision support ## 3. Decision support and cluster-configuration inputs
### 3.1 Side-by-side comparison ### 3.1 Side-by-side comparison
@@ -179,9 +179,134 @@ Use these as a fast triage. If several point the same way, that is your answer.
succeed while reporting nothing. Two images to build, version and scan instead succeed while reporting nothing. Two images to build, version and scan instead
of one. of one.
### 3.4 Cluster-configuration inputs
Once the setup is chosen, the container cluster has to be described. The table
below is the full input surface, split into what a run cannot start without and
what only applies under a stated condition. "Mandatory" means the run fails, or
reports nothing, if the input is absent or wrong.
| Input | Tightly coupled | Loosely coupled | Notes |
|---|---|---|---|
| Cluster / kube context hosting the run worker | Mandatory | Mandatory | The launcher's cluster. Loosely coupled may dispatch to a *second* cluster |
| Namespace for run pods | Mandatory | Mandatory | Supplied by the code location's `DAGSTER_CONTAINER_CONTEXT`; the launcher's `job_namespace` is only a fallback |
| Code location image reference and tag | Mandatory | Mandatory | Never `latest`; check C3 |
| Executor selection | Mandatory | Mandatory | Bounds step granularity; loosely coupled it bounds only dispatch concurrency |
| Service account for run pods | Mandatory | Mandatory | Must have `automountServiceAccountToken: true` |
| Run-pod egress to the metadata database | **Mandatory** | Not applicable | The pod that writes run events |
| Namespace for step pods | Conditional — `k8s_job_executor` only | Not applicable | Defaults to the run pod's namespace |
| RBAC to create and watch Jobs | Conditional — `k8s_job_executor` only | **Mandatory**, in the payload namespace | Same verbs, different namespace |
| Shared RWX volume plus an I/O manager `base_dir` under it | Conditional — `k8s_job_executor` only | Optional | Without it step pods cannot read each other's outputs |
| Per-step resource requests and limits | Optional | Optional | Multiply by step or dispatch concurrency |
| Payload image reference and tag | Not applicable | **Mandatory** | Must share the code location's commit tag |
| Namespace for payload workloads | Not applicable | **Mandatory** | May be in another cluster or trust boundary |
| Message channel and its reader | Not applicable | **Mandatory** | Pod log stream by default; RBAC on `pods/log` *is* the channel |
| Payload wait timeout | Not applicable | Optional | Defaults to 24 h — lower it so scheduling failures surface |
| Vault role and secret paths | Conditional — if the workflow reads secrets | Conditional — launching pod only | Payloads should receive only `extras` and explicit `env` |
| Object-storage endpoint and credentials | Conditional — if the workflow uses S3 | Conditional — if the *payload* uses S3 | |
| Image pull secret | Conditional — non-anonymous registry | Conditional — non-anonymous registry | Applies to the SA that runs the pod, which for dispatched payload Jobs is `default` |
### 3.5 Minimum input set
The list below is the reusable baseline: the smallest set of values a team has to
agree on before a workflow can be configured against a cluster. Record it once
per workflow, in the workflow's own repository, and carry it into whatever
configuration format the team already uses.
**Both setups**
1. `cluster` — the cluster the run worker lands in.
2. `run_namespace` — namespace for run pods.
3. `code_location_image` — image reference including an immutable tag.
4. `executor` — executor name plus its concurrency bound.
5. `service_account` — the identity run pods assume.
**Tightly coupled adds**
6. `metadata_db` — host and port the run and step pods must reach.
7. `step_namespace` — only when `k8s_job_executor` is used.
8. `shared_output_path` — RWX mount path, and the I/O manager `base_dir` pinned
under it, again only for `k8s_job_executor`.
**Loosely coupled adds**
6. `payload_image` — image reference sharing the code location's commit tag.
7. `payload_namespace` — where dispatched workloads are created; may be in
another cluster.
8. `message_channel` — the reader in use, and the RBAC verb that carries it.
This guide deliberately does **not** prescribe a configuration schema or key
names. The names above are descriptive labels for the values, not a required
format; the examples in section 3.6 express the same baseline in two different
shapes, and both are correct.
### 3.6 Worked input examples
The reference configuration in [yaml/](../../yaml) shows the baseline expressed
in the formats each setup already uses. Neither is normative.
**Tightly coupled** — the inputs land in Helm values and in the job's executor
config, because that is where the platform chart and Dagster expect them:
```yaml
# yaml/tightly-coupled/values-run-launcher.yaml (shape, not a fixed schema)
dagster:
runLauncher:
type: K8sRunLauncher
config:
k8sRunLauncher:
jobNamespace: dagster # run_namespace
failPodOnRunFailure: true
runK8sConfig:
containerConfig:
env: # metadata_db, S3 and Vault reachability
- name: S3_ENDPOINT_URL
value: https://s3.dev.simpl-europe.eu
```
```python
# step_namespace, shared_output_path and the executor travel with the job
tightly_coupled_k8s_job = distributed_execution_reference.to_job(
executor_def=k8s_job_executor,
resource_defs={"io_manager": fs_io_manager.configured({"base_dir": SHARED_IO_BASE_DIR})},
config={"execution": {"config": {"step_k8s_config": STEP_K8S_CONFIG}}},
)
```
**Loosely coupled** — the same baseline, but the cluster-facing half is the
payload image and its namespace. Here they are environment values set on both the
code location and the run launcher, because the dispatching pod is a run pod:
```yaml
# yaml/loosely-coupled/values-pipes-payload.yaml (shape, not a fixed schema)
dagster:
dagster-user-deployments:
deployments:
- name: distributed-execution
env:
- name: PIPES_PAYLOAD_IMAGE # payload_image, tag-locked
value: <registry>/distributed-execution/payload:0.0.0
- name: PIPES_PAYLOAD_NAMESPACE # payload_namespace
value: dagster
```
```python
PAYLOAD_IMAGE = os.environ.get("PIPES_PAYLOAD_IMAGE", ...)
PAYLOAD_NAMESPACE = os.environ.get("PIPES_PAYLOAD_NAMESPACE", "dagster")
```
The `message_channel` input has no key in either file. The default reader is the
pod log stream, so it is configured by granting `pods/log` in
[yaml/loosely-coupled/rbac-pipes-dispatch.yaml](../../yaml/loosely-coupled/rbac-pipes-dispatch.yaml).
An input can be mandatory and still not be a configuration field.
Note what is absent from the loosely coupled example: no metadata database, no
Vault role, no object-storage credentials. Those inputs are not optional there —
they do not exist, which is the operational consequence of the choice.
--- ---
## 4. AC3 — Readiness checks ## 4. Readiness checks
Pre-run validation items, their expected evidence, and the symptoms and Pre-run validation items, their expected evidence, and the symptoms and
corrections for common misconfigurations are maintained separately, in corrections for common misconfigurations are maintained separately, in
@@ -189,7 +314,7 @@ corrections for common misconfigurations are maintained separately, in
--- ---
## 5. AC4 — Configuring execution-target linkage in code ## 5. Configuring execution-target linkage in code
### 5.1 Configuration constructs ### 5.1 Configuration constructs
@@ -550,24 +675,59 @@ output metadata as retained evidence:
## 8. Outstanding work ## 8. Outstanding work
Tracked under SIMPL-30451. Tracked under SIMPL-30451 and SIMPL-30787.
| Item | AC | Status | | Item | Status |
|---|---|
| Execution-target choices, prerequisites, workflow-level documentation | Complete |
| Comparison, environment-fit indicators, trade-offs | Complete |
| Cluster-configuration inputs, mandatory vs optional, minimum input set | Complete |
| Readiness checklist and evidence mapping | Complete; both Kubernetes paths cluster-verified |
| Misconfiguration symptoms and corrections | Complete; corrected with observed cluster failure modes |
| Configuration constructs and persistence | Complete |
| Before-and-after example, tightly coupled | Complete |
| Before-and-after example, switching to loosely coupled | Complete |
| Runnable tightly coupled reference implementations | Complete |
| Runnable loosely coupled reference implementation | Complete (subprocess and Kubernetes transports both verified) |
| Both images build; payload image passes the isolation check (L11) | Complete |
| Locally runnable jobs launched from the Dagster UI, evidence recorded in section 5.6 | Complete |
| Payload image published to a container registry | Complete on the sandbox Gitea registry; the GitLab registry still pending |
| GitLab pipeline builds both images | **Pending** — the shared `ds.gitlab-ci.yml` template builds one image from the root Dockerfile |
| End-to-end run of `loosely_coupled_k8s_job` on a cluster | Complete — sandbox-cat-dat, 2026-08-31, run `cff9b348…`; see the readiness checklist |
| `distributed-execution` registered as a code location on a platform Dagster | Complete — tightly coupled run `1d8cb167…` launched through the platform webserver |
| Screenshots of UI surfaces | Complete — job list, graph, Launchpad linkage and successful run captured in section 5.7 |
| Platform architecture document update | Complete — root `deployment_diagram.md` includes the service and both runtime-information paths |
---
## 9. Requirement traceability
The sections above are organised by subject, not by ticket. Two stories share
this material; the mapping below is authoritative for both.
### SIMPL-30787 — defining the execution container cluster
| AC | Requirement | Where it is met |
|---|---|---| |---|---|---|
| Execution-target choices, prerequisites, workflow-level documentation | AC1 | Complete | | AC1 | Prerequisites for both setups | Section 2.3 |
| Comparison, environment-fit indicators, trade-offs | AC2 | Complete | | AC1 | What must be documented before execution | Section 2.4 |
| Readiness checklist and evidence mapping | AC3 | Complete; both Kubernetes paths cluster-verified | | AC1 | Where cluster-runtime connectivity is required, where decoupled reporting is expected | Sections 1.2 and 2.3; the asymmetry is drawn in the section 1.2 diagram |
| Misconfiguration symptoms and corrections | AC3 | Complete; corrected with observed cluster failure modes | | AC2 | Mandatory vs optional inputs per setup | Section 3.4 |
| Configuration constructs and persistence | AC4 | Complete | | AC2 | Examples for both setups, no fixed config naming model | Section 3.6, with the closing note in 3.5 |
| Before-and-after example, tightly coupled | AC4 | Complete | | AC2 | Reusable minimum input set | Section 3.5 |
| Before-and-after example, switching to loosely coupled | AC4 | Complete | | AC3 | Checklist separates tightly from loosely coupled checks | [readiness-checklist.md](readiness-checklist.md) sections 2 and 3; section 1 holds the checks common to both |
| Runnable tightly coupled reference implementations | AC4 | Complete | | AC3 | Each item mapped to expected evidence | The *Expected evidence* column of every checklist table |
| Runnable loosely coupled reference implementation | AC4 | Complete (subprocess and Kubernetes transports both verified) | | AC4 | Configuration constructs attaching the pattern to a workflow | Section 5.1 |
| Both images build; payload image passes the isolation check (L11) | Tech details | Complete | | AC4 | Worked before-and-after example across setups | Sections 5.2 and 5.3 |
| Locally runnable jobs launched from the Dagster UI, evidence recorded in section 5.6 | Tech details | Complete | | AC4 | At least one runnable reference implementation per setup | Section 5.5 — three tightly coupled, two loosely coupled |
| Payload image published to a container registry | Tech details | Complete on the sandbox Gitea registry; the GitLab registry still pending | | AC4 | Selection persisted in configuration artifacts, not runtime-only | Section 5.4 |
| GitLab pipeline builds both images | Tech details | **Pending** — the shared `ds.gitlab-ci.yml` template builds one image from the root Dockerfile | | Tech | Diagrams, screenshots, working config files, upstream links | Sections 1.2, 5.7, [yaml/](../../yaml), section 6 |
| End-to-end run of `loosely_coupled_k8s_job` on a cluster | Tech details | Complete — sandbox-cat-dat, 2026-08-31, run `cff9b348…`; see the readiness checklist |
| `distributed-execution` registered as a code location on a platform Dagster | Tech details | Complete — tightly coupled run `1d8cb167…` launched through the platform webserver | ### SIMPL-30451 — execution targets and integration patterns
| Screenshots of UI surfaces | Tech details | Complete — job list, graph, Launchpad linkage and successful run captured in section 5.7 |
| Platform architecture document update | Tech details | Complete — root `deployment_diagram.md` includes the service and both runtime-information paths | | AC | Subject | Where it is met |
|---|---|---|
| AC1 | Choosing an execution target | Section 2 |
| AC2 | Decision support | Sections 3.1 to 3.3 |
| AC3 | Readiness checks | Section 4 and the readiness checklist |
| AC4 | Linkage configured in code | Section 5 |

5
documents/user-guide/readiness-checklist.md
View File

@@ -1,7 +1,10 @@
# Distributed Execution: Readiness Checklist # Distributed Execution: Readiness Checklist
Pre-run validation for a workflow's execution target. Companion to the Pre-run validation for a workflow's execution target. Companion to the
[user guide](distributed-execution-guide.md); this document covers AC3. [user guide](distributed-execution-guide.md); this document is the validation
checklist required by SIMPL-30451 AC3 and SIMPL-30787 AC3. Section 1 holds the
checks common to both setups, section 2 the tightly coupled checks and section 3
the loosely coupled ones; every item states the evidence that proves it.
Run these checks **before the first execution** of a workflow, and again after any Run these checks **before the first execution** of a workflow, and again after any
change to the run launcher, the executor, the target namespace or the code change to the run launcher, the executor, the target namespace or the code

4
yaml/tightly-coupled/values-run-launcher.yaml
View File

@@ -13,8 +13,8 @@ dagster:
config: config:
k8sRunLauncher: k8sRunLauncher:
# Namespace the run pods land in. Must be a namespace whose NetworkPolicy # Namespace the run pods land in. Must be a namespace whose NetworkPolicy
# permits egress to Postgres, object storage and Vault - see AC1 # permits egress to Postgres, object storage and Vault - see the
# prerequisites in the user guide. # prerequisites in section 2.3 of the user guide.
jobNamespace: dagster jobNamespace: dagster
# Surfaces step failures as pod failures so kubectl and Dagster agree. # Surfaces step failures as pod failures so kubectl and Dagster agree.