j.r/distributed-execution
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[SIMPL-30787] Document container-cluster configuration inputs

Browse Source 
Add the cluster-configuration input reference the guide was missing: which inputs are mandatory, conditional or optional in each setup, a reusable minimum input set, and worked examples expressed in the format each setup already uses. No configuration schema or key naming is prescribed.

Retitle the subject sections so they no longer carry one story's AC numbering, and add a traceability section mapping SIMPL-30787 and SIMPL-30451 acceptance criteria onto them.

Changelog: added
This commit is contained in:
ILay
2026-09-02 18:48:01 +02:00
parent 9b1b7056b0
commit aa49420ca6
5 changed files with 193 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files

206
documents/user-guide/distributed-execution-guide.md
View File

@@ -66,7 +66,7 @@ pattern.
---
## 2. AC1 — Choosing an execution target
## 2. Choosing an execution target and cluster setup
### 2.1 Choose tightly coupled when
@@ -121,7 +121,7 @@ before it is run for the first time. Section 5.4 explains where each item lives.
---
## 3. AC2 — Decision support
## 3. Decision support and cluster-configuration inputs
### 3.1 Side-by-side comparison
@@ -179,9 +179,134 @@ Use these as a fast triage. If several point the same way, that is your answer.
succeed while reporting nothing. Two images to build, version and scan instead
of one.
### 3.4 Cluster-configuration inputs
Once the setup is chosen, the container cluster has to be described. The table
below is the full input surface, split into what a run cannot start without and
what only applies under a stated condition. "Mandatory" means the run fails, or
reports nothing, if the input is absent or wrong.
| Input | Tightly coupled | Loosely coupled | Notes |
|---|---|---|---|
| Cluster / kube context hosting the run worker | Mandatory | Mandatory | The launcher's cluster. Loosely coupled may dispatch to a *second* cluster |
| Namespace for run pods | Mandatory | Mandatory | Supplied by the code location's `DAGSTER_CONTAINER_CONTEXT`; the launcher's `job_namespace` is only a fallback |
| Code location image reference and tag | Mandatory | Mandatory | Never `latest`; check C3 |
| Executor selection | Mandatory | Mandatory | Bounds step granularity; loosely coupled it bounds only dispatch concurrency |
| Service account for run pods | Mandatory | Mandatory | Must have `automountServiceAccountToken: true` |
| Run-pod egress to the metadata database | **Mandatory** | Not applicable | The pod that writes run events |
| Namespace for step pods | Conditional — `k8s_job_executor` only | Not applicable | Defaults to the run pod's namespace |
| RBAC to create and watch Jobs | Conditional — `k8s_job_executor` only | **Mandatory**, in the payload namespace | Same verbs, different namespace |
| Shared RWX volume plus an I/O manager `base_dir` under it | Conditional — `k8s_job_executor` only | Optional | Without it step pods cannot read each other's outputs |
| Per-step resource requests and limits | Optional | Optional | Multiply by step or dispatch concurrency |
| Payload image reference and tag | Not applicable | **Mandatory** | Must share the code location's commit tag |
| Namespace for payload workloads | Not applicable | **Mandatory** | May be in another cluster or trust boundary |
| Message channel and its reader | Not applicable | **Mandatory** | Pod log stream by default; RBAC on `pods/log` *is* the channel |
| Payload wait timeout | Not applicable | Optional | Defaults to 24 h — lower it so scheduling failures surface |
| Vault role and secret paths | Conditional — if the workflow reads secrets | Conditional — launching pod only | Payloads should receive only `extras` and explicit `env` |
| Object-storage endpoint and credentials | Conditional — if the workflow uses S3 | Conditional — if the *payload* uses S3 | |
| Image pull secret | Conditional — non-anonymous registry | Conditional — non-anonymous registry | Applies to the SA that runs the pod, which for dispatched payload Jobs is `default` |
### 3.5 Minimum input set
The list below is the reusable baseline: the smallest set of values a team has to
agree on before a workflow can be configured against a cluster. Record it once
per workflow, in the workflow's own repository, and carry it into whatever
configuration format the team already uses.
**Both setups**
1. `cluster` — the cluster the run worker lands in.
2. `run_namespace` — namespace for run pods.
3. `code_location_image` — image reference including an immutable tag.
4. `executor` — executor name plus its concurrency bound.
5. `service_account` — the identity run pods assume.
**Tightly coupled adds**
6. `metadata_db` — host and port the run and step pods must reach.
7. `step_namespace` — only when `k8s_job_executor` is used.
8. `shared_output_path` — RWX mount path, and the I/O manager `base_dir` pinned
under it, again only for `k8s_job_executor`.
**Loosely coupled adds**
6. `payload_image` — image reference sharing the code location's commit tag.
7. `payload_namespace` — where dispatched workloads are created; may be in
another cluster.
8. `message_channel` — the reader in use, and the RBAC verb that carries it.
This guide deliberately does **not** prescribe a configuration schema or key
names. The names above are descriptive labels for the values, not a required
format; the examples in section 3.6 express the same baseline in two different
shapes, and both are correct.
### 3.6 Worked input examples
The reference configuration in [yaml/](../../yaml) shows the baseline expressed
in the formats each setup already uses. Neither is normative.
**Tightly coupled** — the inputs land in Helm values and in the job's executor
config, because that is where the platform chart and Dagster expect them:
```yaml
# yaml/tightly-coupled/values-run-launcher.yaml (shape, not a fixed schema)
dagster:
runLauncher:
type: K8sRunLauncher
config:
k8sRunLauncher:
jobNamespace: dagster # run_namespace
failPodOnRunFailure: true
runK8sConfig:
containerConfig:
env: # metadata_db, S3 and Vault reachability
- name: S3_ENDPOINT_URL
value: https://s3.dev.simpl-europe.eu
```
```python
# step_namespace, shared_output_path and the executor travel with the job
tightly_coupled_k8s_job = distributed_execution_reference.to_job(
executor_def=k8s_job_executor,
resource_defs={"io_manager": fs_io_manager.configured({"base_dir": SHARED_IO_BASE_DIR})},
config={"execution": {"config": {"step_k8s_config": STEP_K8S_CONFIG}}},
)
```
**Loosely coupled** — the same baseline, but the cluster-facing half is the
payload image and its namespace. Here they are environment values set on both the
code location and the run launcher, because the dispatching pod is a run pod:
```yaml
# yaml/loosely-coupled/values-pipes-payload.yaml (shape, not a fixed schema)
dagster:
dagster-user-deployments:
deployments:
- name: distributed-execution
env:
- name: PIPES_PAYLOAD_IMAGE # payload_image, tag-locked
value: <registry>/distributed-execution/payload:0.0.0
- name: PIPES_PAYLOAD_NAMESPACE # payload_namespace
value: dagster
```
```python
PAYLOAD_IMAGE = os.environ.get("PIPES_PAYLOAD_IMAGE", ...)
PAYLOAD_NAMESPACE = os.environ.get("PIPES_PAYLOAD_NAMESPACE", "dagster")
```
The `message_channel` input has no key in either file. The default reader is the
pod log stream, so it is configured by granting `pods/log` in
[yaml/loosely-coupled/rbac-pipes-dispatch.yaml](../../yaml/loosely-coupled/rbac-pipes-dispatch.yaml).
An input can be mandatory and still not be a configuration field.
Note what is absent from the loosely coupled example: no metadata database, no
Vault role, no object-storage credentials. Those inputs are not optional there —
they do not exist, which is the operational consequence of the choice.
---
## 4. AC3 — Readiness checks
## 4. Readiness checks
Pre-run validation items, their expected evidence, and the symptoms and
corrections for common misconfigurations are maintained separately, in
@@ -189,7 +314,7 @@ corrections for common misconfigurations are maintained separately, in
---
## 5. AC4 — Configuring execution-target linkage in code
## 5. Configuring execution-target linkage in code
### 5.1 Configuration constructs
@@ -550,24 +675,59 @@ output metadata as retained evidence:
## 8. Outstanding work
Tracked under SIMPL-30451.
Tracked under SIMPL-30451 and SIMPL-30787.
| Item | AC | Status |
| Item | Status |
|---|---|
| Execution-target choices, prerequisites, workflow-level documentation | Complete |
| Comparison, environment-fit indicators, trade-offs | Complete |
| Cluster-configuration inputs, mandatory vs optional, minimum input set | Complete |
| Readiness checklist and evidence mapping | Complete; both Kubernetes paths cluster-verified |
| Misconfiguration symptoms and corrections | Complete; corrected with observed cluster failure modes |
| Configuration constructs and persistence | Complete |
| Before-and-after example, tightly coupled | Complete |
| Before-and-after example, switching to loosely coupled | Complete |
| Runnable tightly coupled reference implementations | Complete |
| Runnable loosely coupled reference implementation | Complete (subprocess and Kubernetes transports both verified) |
| Both images build; payload image passes the isolation check (L11) | Complete |
| Locally runnable jobs launched from the Dagster UI, evidence recorded in section 5.6 | Complete |
| Payload image published to a container registry | Complete on the sandbox Gitea registry; the GitLab registry still pending |
| GitLab pipeline builds both images | **Pending** — the shared `ds.gitlab-ci.yml` template builds one image from the root Dockerfile |
| End-to-end run of `loosely_coupled_k8s_job` on a cluster | Complete — sandbox-cat-dat, 2026-08-31, run `cff9b348…`; see the readiness checklist |
| `distributed-execution` registered as a code location on a platform Dagster | Complete — tightly coupled run `1d8cb167…` launched through the platform webserver |
| Screenshots of UI surfaces | Complete — job list, graph, Launchpad linkage and successful run captured in section 5.7 |
| Platform architecture document update | Complete — root `deployment_diagram.md` includes the service and both runtime-information paths |
---
## 9. Requirement traceability
The sections above are organised by subject, not by ticket. Two stories share
this material; the mapping below is authoritative for both.
### SIMPL-30787 — defining the execution container cluster
| AC | Requirement | Where it is met |
|---|---|---|
| Execution-target choices, prerequisites, workflow-level documentation | AC1 | Complete |
| Comparison, environment-fit indicators, trade-offs | AC2 | Complete |
| Readiness checklist and evidence mapping | AC3 | Complete; both Kubernetes paths cluster-verified |
| Misconfiguration symptoms and corrections | AC3 | Complete; corrected with observed cluster failure modes |
| Configuration constructs and persistence | AC4 | Complete |
| Before-and-after example, tightly coupled | AC4 | Complete |
| Before-and-after example, switching to loosely coupled | AC4 | Complete |
| Runnable tightly coupled reference implementations | AC4 | Complete |
| Runnable loosely coupled reference implementation | AC4 | Complete (subprocess and Kubernetes transports both verified) |
| Both images build; payload image passes the isolation check (L11) | Tech details | Complete |
| Locally runnable jobs launched from the Dagster UI, evidence recorded in section 5.6 | Tech details | Complete |
| Payload image published to a container registry | Tech details | Complete on the sandbox Gitea registry; the GitLab registry still pending |
| GitLab pipeline builds both images | Tech details | **Pending** — the shared `ds.gitlab-ci.yml` template builds one image from the root Dockerfile |
| End-to-end run of `loosely_coupled_k8s_job` on a cluster | Tech details | Complete — sandbox-cat-dat, 2026-08-31, run `cff9b348…`; see the readiness checklist |
| `distributed-execution` registered as a code location on a platform Dagster | Tech details | Complete — tightly coupled run `1d8cb167…` launched through the platform webserver |
| Screenshots of UI surfaces | Tech details | Complete — job list, graph, Launchpad linkage and successful run captured in section 5.7 |
| Platform architecture document update | Tech details | Complete — root `deployment_diagram.md` includes the service and both runtime-information paths |
| AC1 | Prerequisites for both setups | Section 2.3 |
| AC1 | What must be documented before execution | Section 2.4 |
| AC1 | Where cluster-runtime connectivity is required, where decoupled reporting is expected | Sections 1.2 and 2.3; the asymmetry is drawn in the section 1.2 diagram |
| AC2 | Mandatory vs optional inputs per setup | Section 3.4 |
| AC2 | Examples for both setups, no fixed config naming model | Section 3.6, with the closing note in 3.5 |
| AC2 | Reusable minimum input set | Section 3.5 |
| AC3 | Checklist separates tightly from loosely coupled checks | [readiness-checklist.md](readiness-checklist.md) sections 2 and 3; section 1 holds the checks common to both |
| AC3 | Each item mapped to expected evidence | The *Expected evidence* column of every checklist table |
| AC4 | Configuration constructs attaching the pattern to a workflow | Section 5.1 |
| AC4 | Worked before-and-after example across setups | Sections 5.2 and 5.3 |
| AC4 | At least one runnable reference implementation per setup | Section 5.5 — three tightly coupled, two loosely coupled |
| AC4 | Selection persisted in configuration artifacts, not runtime-only | Section 5.4 |
| Tech | Diagrams, screenshots, working config files, upstream links | Sections 1.2, 5.7, [yaml/](../../yaml), section 6 |
### SIMPL-30451 — execution targets and integration patterns
| AC | Subject | Where it is met |
|---|---|---|
| AC1 | Choosing an execution target | Section 2 |
| AC2 | Decision support | Sections 3.1 to 3.3 |
| AC3 | Readiness checks | Section 4 and the readiness checklist |
| AC4 | Linkage configured in code | Section 5 |

5
documents/user-guide/readiness-checklist.md
View File

@@ -1,7 +1,10 @@
# Distributed Execution: Readiness Checklist
Pre-run validation for a workflow's execution target. Companion to the
[user guide](distributed-execution-guide.md); this document covers AC3.
[user guide](distributed-execution-guide.md); this document is the validation
checklist required by SIMPL-30451 AC3 and SIMPL-30787 AC3. Section 1 holds the
checks common to both setups, section 2 the tightly coupled checks and section 3
the loosely coupled ones; every item states the evidence that proves it.
Run these checks **before the first execution** of a workflow, and again after any
change to the run launcher, the executor, the target namespace or the code