j.r/distributed-execution
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Deploy to sandbox: develop --> main #2

Merged
j.r merged 6 commits from develop into main 2026-09-02 17:43:37 +00:00
Conversation 0 Commits 6 Files Changed 12 +193 -27
5 changed files with 193 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Showing only changes of commit aa49420ca6 - Show all commits

3
CHANGELOG.md
View File

@@ -4,6 +4,9 @@
- Repository skeleton for the `distributed-execution` supporting data service.
- User guide covering execution-target selection, decision support and readiness checks.
- Cluster-configuration input reference: mandatory versus optional inputs per setup,
a reusable minimum input set and worked examples in both setups' own formats.
- Requirement traceability mapping the guide's sections onto SIMPL-30451 and SIMPL-30787.
- Runnable tightly coupled reference implementations (`in_process`, `multiprocess`, `k8s_job_executor`).
- Runnable loosely coupled reference implementations via dagster-pipes
(`PipesSubprocessClient`, `PipesK8sClient`) with a standalone external payload image.

2
README.md
View File

@@ -35,7 +35,7 @@ distributed-execution/
│ ├── work.py
│ ├── requirements.txt
│ └── Dockerfile
├── documents/user-guide/ # AC1-AC4 documentation
├── documents/user-guide/ # Guide, checklist and requirement traceability
├── yaml/ # Working example configuration
├── tests/
├── Dockerfile

206
documents/user-guide/distributed-execution-guide.md
View File

@@ -66,7 +66,7 @@ pattern.
---
## 2. AC1 — Choosing an execution target
## 2. Choosing an execution target and cluster setup
### 2.1 Choose tightly coupled when
@@ -121,7 +121,7 @@ before it is run for the first time. Section 5.4 explains where each item lives.
---
## 3. AC2 — Decision support
## 3. Decision support and cluster-configuration inputs
### 3.1 Side-by-side comparison
@@ -179,9 +179,134 @@ Use these as a fast triage. If several point the same way, that is your answer.
succeed while reporting nothing. Two images to build, version and scan instead
of one.
### 3.4 Cluster-configuration inputs
Once the setup is chosen, the container cluster has to be described. The table
below is the full input surface, split into what a run cannot start without and
what only applies under a stated condition. "Mandatory" means the run fails, or
reports nothing, if the input is absent or wrong.
| Input | Tightly coupled | Loosely coupled | Notes |
|---|---|---|---|
| Cluster / kube context hosting the run worker | Mandatory | Mandatory | The launcher's cluster. Loosely coupled may dispatch to a *second* cluster |
| Namespace for run pods | Mandatory | Mandatory | Supplied by the code location's `DAGSTER_CONTAINER_CONTEXT`; the launcher's `job_namespace` is only a fallback |
| Code location image reference and tag | Mandatory | Mandatory | Never `latest`; check C3 |
| Executor selection | Mandatory | Mandatory | Bounds step granularity; loosely coupled it bounds only dispatch concurrency |
| Service account for run pods | Mandatory | Mandatory | Must have `automountServiceAccountToken: true` |
| Run-pod egress to the metadata database | **Mandatory** | Not applicable | The pod that writes run events |
| Namespace for step pods | Conditional — `k8s_job_executor` only | Not applicable | Defaults to the run pod's namespace |
| RBAC to create and watch Jobs | Conditional — `k8s_job_executor` only | **Mandatory**, in the payload namespace | Same verbs, different namespace |
| Shared RWX volume plus an I/O manager `base_dir` under it | Conditional — `k8s_job_executor` only | Optional | Without it step pods cannot read each other's outputs |
| Per-step resource requests and limits | Optional | Optional | Multiply by step or dispatch concurrency |
| Payload image reference and tag | Not applicable | **Mandatory** | Must share the code location's commit tag |
| Namespace for payload workloads | Not applicable | **Mandatory** | May be in another cluster or trust boundary |
| Message channel and its reader | Not applicable | **Mandatory** | Pod log stream by default; RBAC on `pods/log` *is* the channel |
| Payload wait timeout | Not applicable | Optional | Defaults to 24 h — lower it so scheduling failures surface |
| Vault role and secret paths | Conditional — if the workflow reads secrets | Conditional — launching pod only | Payloads should receive only `extras` and explicit `env` |
| Object-storage endpoint and credentials | Conditional — if the workflow uses S3 | Conditional — if the *payload* uses S3 | |
| Image pull secret | Conditional — non-anonymous registry | Conditional — non-anonymous registry | Applies to the SA that runs the pod, which for dispatched payload Jobs is `default` |
### 3.5 Minimum input set
The list below is the reusable baseline: the smallest set of values a team has to
agree on before a workflow can be configured against a cluster. Record it once
per workflow, in the workflow's own repository, and carry it into whatever
configuration format the team already uses.
**Both setups**
1. `cluster` — the cluster the run worker lands in.
2. `run_namespace` — namespace for run pods.
3. `code_location_image` — image reference including an immutable tag.
4. `executor` — executor name plus its concurrency bound.
5. `service_account` — the identity run pods assume.
**Tightly coupled adds**
6. `metadata_db` — host and port the run and step pods must reach.
7. `step_namespace` — only when `k8s_job_executor` is used.
8. `shared_output_path` — RWX mount path, and the I/O manager `base_dir` pinned
under it, again only for `k8s_job_executor`.
**Loosely coupled adds**
6. `payload_image` — image reference sharing the code location's commit tag.
7. `payload_namespace` — where dispatched workloads are created; may be in
another cluster.
8. `message_channel` — the reader in use, and the RBAC verb that carries it.
This guide deliberately does **not** prescribe a configuration schema or key
names. The names above are descriptive labels for the values, not a required
format; the examples in section 3.6 express the same baseline in two different
shapes, and both are correct.
### 3.6 Worked input examples
The reference configuration in [yaml/](../../yaml) shows the baseline expressed
in the formats each setup already uses. Neither is normative.
**Tightly coupled** — the inputs land in Helm values and in the job's executor
config, because that is where the platform chart and Dagster expect them:
```yaml
# yaml/tightly-coupled/values-run-launcher.yaml (shape, not a fixed schema)
dagster:
runLauncher:
type: K8sRunLauncher
config:
k8sRunLauncher:
jobNamespace: dagster # run_namespace
failPodOnRunFailure: true
runK8sConfig:
containerConfig:
env: # metadata_db, S3 and Vault reachability
- name: S3_ENDPOINT_URL
value: https://s3.dev.simpl-europe.eu
```
```python
# step_namespace, shared_output_path and the executor travel with the job
tightly_coupled_k8s_job = distributed_execution_reference.to_job(
executor_def=k8s_job_executor,
resource_defs={"io_manager": fs_io_manager.configured({"base_dir": SHARED_IO_BASE_DIR})},
config={"execution": {"config": {"step_k8s_config": STEP_K8S_CONFIG}}},
)
```
**Loosely coupled** — the same baseline, but the cluster-facing half is the
payload image and its namespace. Here they are environment values set on both the
code location and the run launcher, because the dispatching pod is a run pod:
```yaml
# yaml/loosely-coupled/values-pipes-payload.yaml (shape, not a fixed schema)
dagster:
dagster-user-deployments:
deployments:
- name: distributed-execution
env:
- name: PIPES_PAYLOAD_IMAGE # payload_image, tag-locked
value: <registry>/distributed-execution/payload:0.0.0
- name: PIPES_PAYLOAD_NAMESPACE # payload_namespace
value: dagster
```
```python
PAYLOAD_IMAGE = os.environ.get("PIPES_PAYLOAD_IMAGE", ...)
PAYLOAD_NAMESPACE = os.environ.get("PIPES_PAYLOAD_NAMESPACE", "dagster")
```
The `message_channel` input has no key in either file. The default reader is the
pod log stream, so it is configured by granting `pods/log` in
[yaml/loosely-coupled/rbac-pipes-dispatch.yaml](../../yaml/loosely-coupled/rbac-pipes-dispatch.yaml).
An input can be mandatory and still not be a configuration field.
Note what is absent from the loosely coupled example: no metadata database, no
Vault role, no object-storage credentials. Those inputs are not optional there —
they do not exist, which is the operational consequence of the choice.
---
## 4. AC3 — Readiness checks
## 4. Readiness checks
Pre-run validation items, their expected evidence, and the symptoms and
corrections for common misconfigurations are maintained separately, in
@@ -189,7 +314,7 @@ corrections for common misconfigurations are maintained separately, in
---
## 5. AC4 — Configuring execution-target linkage in code
## 5. Configuring execution-target linkage in code
### 5.1 Configuration constructs
@@ -550,24 +675,59 @@ output metadata as retained evidence:
## 8. Outstanding work
Tracked under SIMPL-30451.
Tracked under SIMPL-30451 and SIMPL-30787.
| Item | AC | Status |
| Item | Status |
|---|---|
| Execution-target choices, prerequisites, workflow-level documentation | Complete |
| Comparison, environment-fit indicators, trade-offs | Complete |
| Cluster-configuration inputs, mandatory vs optional, minimum input set | Complete |
| Readiness checklist and evidence mapping | Complete; both Kubernetes paths cluster-verified |
| Misconfiguration symptoms and corrections | Complete; corrected with observed cluster failure modes |
| Configuration constructs and persistence | Complete |
| Before-and-after example, tightly coupled | Complete |
| Before-and-after example, switching to loosely coupled | Complete |
| Runnable tightly coupled reference implementations | Complete |
| Runnable loosely coupled reference implementation | Complete (subprocess and Kubernetes transports both verified) |
| Both images build; payload image passes the isolation check (L11) | Complete |
| Locally runnable jobs launched from the Dagster UI, evidence recorded in section 5.6 | Complete |
| Payload image published to a container registry | Complete on the sandbox Gitea registry; the GitLab registry still pending |
| GitLab pipeline builds both images | **Pending** — the shared `ds.gitlab-ci.yml` template builds one image from the root Dockerfile |
| End-to-end run of `loosely_coupled_k8s_job` on a cluster | Complete — sandbox-cat-dat, 2026-08-31, run `cff9b348…`; see the readiness checklist |
| `distributed-execution` registered as a code location on a platform Dagster | Complete — tightly coupled run `1d8cb167…` launched through the platform webserver |
| Screenshots of UI surfaces | Complete — job list, graph, Launchpad linkage and successful run captured in section 5.7 |
| Platform architecture document update | Complete — root `deployment_diagram.md` includes the service and both runtime-information paths |
---
## 9. Requirement traceability
The sections above are organised by subject, not by ticket. Two stories share
this material; the mapping below is authoritative for both.
### SIMPL-30787 — defining the execution container cluster
| AC | Requirement | Where it is met |
|---|---|---|
| Execution-target choices, prerequisites, workflow-level documentation | AC1 | Complete |
| Comparison, environment-fit indicators, trade-offs | AC2 | Complete |
| Readiness checklist and evidence mapping | AC3 | Complete; both Kubernetes paths cluster-verified |
| Misconfiguration symptoms and corrections | AC3 | Complete; corrected with observed cluster failure modes |
| Configuration constructs and persistence | AC4 | Complete |
| Before-and-after example, tightly coupled | AC4 | Complete |
| Before-and-after example, switching to loosely coupled | AC4 | Complete |
| Runnable tightly coupled reference implementations | AC4 | Complete |
| Runnable loosely coupled reference implementation | AC4 | Complete (subprocess and Kubernetes transports both verified) |
| Both images build; payload image passes the isolation check (L11) | Tech details | Complete |
| Locally runnable jobs launched from the Dagster UI, evidence recorded in section 5.6 | Tech details | Complete |
| Payload image published to a container registry | Tech details | Complete on the sandbox Gitea registry; the GitLab registry still pending |
| GitLab pipeline builds both images | Tech details | **Pending** — the shared `ds.gitlab-ci.yml` template builds one image from the root Dockerfile |
| End-to-end run of `loosely_coupled_k8s_job` on a cluster | Tech details | Complete — sandbox-cat-dat, 2026-08-31, run `cff9b348…`; see the readiness checklist |
| `distributed-execution` registered as a code location on a platform Dagster | Tech details | Complete — tightly coupled run `1d8cb167…` launched through the platform webserver |
| Screenshots of UI surfaces | Tech details | Complete — job list, graph, Launchpad linkage and successful run captured in section 5.7 |
| Platform architecture document update | Tech details | Complete — root `deployment_diagram.md` includes the service and both runtime-information paths |
| AC1 | Prerequisites for both setups | Section 2.3 |
| AC1 | What must be documented before execution | Section 2.4 |
| AC1 | Where cluster-runtime connectivity is required, where decoupled reporting is expected | Sections 1.2 and 2.3; the asymmetry is drawn in the section 1.2 diagram |
| AC2 | Mandatory vs optional inputs per setup | Section 3.4 |
| AC2 | Examples for both setups, no fixed config naming model | Section 3.6, with the closing note in 3.5 |
| AC2 | Reusable minimum input set | Section 3.5 |
| AC3 | Checklist separates tightly from loosely coupled checks | [readiness-checklist.md](readiness-checklist.md) sections 2 and 3; section 1 holds the checks common to both |
| AC3 | Each item mapped to expected evidence | The *Expected evidence* column of every checklist table |
| AC4 | Configuration constructs attaching the pattern to a workflow | Section 5.1 |
| AC4 | Worked before-and-after example across setups | Sections 5.2 and 5.3 |
| AC4 | At least one runnable reference implementation per setup | Section 5.5 — three tightly coupled, two loosely coupled |
| AC4 | Selection persisted in configuration artifacts, not runtime-only | Section 5.4 |
| Tech | Diagrams, screenshots, working config files, upstream links | Sections 1.2, 5.7, [yaml/](../../yaml), section 6 |
### SIMPL-30451 — execution targets and integration patterns
| AC | Subject | Where it is met |
|---|---|---|
| AC1 | Choosing an execution target | Section 2 |
| AC2 | Decision support | Sections 3.1 to 3.3 |
| AC3 | Readiness checks | Section 4 and the readiness checklist |
| AC4 | Linkage configured in code | Section 5 |

5
documents/user-guide/readiness-checklist.md
View File

@@ -1,7 +1,10 @@
# Distributed Execution: Readiness Checklist
Pre-run validation for a workflow's execution target. Companion to the
[user guide](distributed-execution-guide.md); this document covers AC3.
[user guide](distributed-execution-guide.md); this document is the validation
checklist required by SIMPL-30451 AC3 and SIMPL-30787 AC3. Section 1 holds the
checks common to both setups, section 2 the tightly coupled checks and section 3
the loosely coupled ones; every item states the evidence that proves it.
Run these checks **before the first execution** of a workflow, and again after any
change to the run launcher, the executor, the target namespace or the code

4
yaml/tightly-coupled/values-run-launcher.yaml
View File

@@ -13,8 +13,8 @@ dagster:
config:
k8sRunLauncher:
# Namespace the run pods land in. Must be a namespace whose NetworkPolicy
# permits egress to Postgres, object storage and Vault - see AC1
# prerequisites in the user guide.
# permits egress to Postgres, object storage and Vault - see the
# prerequisites in section 2.3 of the user guide.
jobNamespace: dagster
# Surfaces step failures as pod failures so kubectl and Dagster agree.