diff --git a/CHANGELOG.md b/CHANGELOG.md index 7d53f02..a34a2fd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,9 @@ - Repository skeleton for the `distributed-execution` supporting data service. - User guide covering execution-target selection, decision support and readiness checks. +- Cluster-configuration input reference: mandatory versus optional inputs per setup, + a reusable minimum input set and worked examples in both setups' own formats. +- Requirement traceability mapping the guide's sections onto SIMPL-30451 and SIMPL-30787. - Runnable tightly coupled reference implementations (`in_process`, `multiprocess`, `k8s_job_executor`). - Runnable loosely coupled reference implementations via dagster-pipes (`PipesSubprocessClient`, `PipesK8sClient`) with a standalone external payload image. diff --git a/README.md b/README.md index 75a92e0..6e64f87 100644 --- a/README.md +++ b/README.md @@ -35,7 +35,7 @@ distributed-execution/ │ ├── work.py │ ├── requirements.txt │ └── Dockerfile -├── documents/user-guide/ # AC1-AC4 documentation +├── documents/user-guide/ # Guide, checklist and requirement traceability ├── yaml/ # Working example configuration ├── tests/ ├── Dockerfile diff --git a/documents/user-guide/distributed-execution-guide.md b/documents/user-guide/distributed-execution-guide.md index 18a881c..b126b0e 100644 --- a/documents/user-guide/distributed-execution-guide.md +++ b/documents/user-guide/distributed-execution-guide.md @@ -66,7 +66,7 @@ pattern. --- -## 2. AC1 — Choosing an execution target +## 2. Choosing an execution target and cluster setup ### 2.1 Choose tightly coupled when @@ -121,7 +121,7 @@ before it is run for the first time. Section 5.4 explains where each item lives. --- -## 3. AC2 — Decision support +## 3. Decision support and cluster-configuration inputs ### 3.1 Side-by-side comparison @@ -179,9 +179,134 @@ Use these as a fast triage. If several point the same way, that is your answer. succeed while reporting nothing. Two images to build, version and scan instead of one. +### 3.4 Cluster-configuration inputs + +Once the setup is chosen, the container cluster has to be described. The table +below is the full input surface, split into what a run cannot start without and +what only applies under a stated condition. "Mandatory" means the run fails, or +reports nothing, if the input is absent or wrong. + +| Input | Tightly coupled | Loosely coupled | Notes | +|---|---|---|---| +| Cluster / kube context hosting the run worker | Mandatory | Mandatory | The launcher's cluster. Loosely coupled may dispatch to a *second* cluster | +| Namespace for run pods | Mandatory | Mandatory | Supplied by the code location's `DAGSTER_CONTAINER_CONTEXT`; the launcher's `job_namespace` is only a fallback | +| Code location image reference and tag | Mandatory | Mandatory | Never `latest`; check C3 | +| Executor selection | Mandatory | Mandatory | Bounds step granularity; loosely coupled it bounds only dispatch concurrency | +| Service account for run pods | Mandatory | Mandatory | Must have `automountServiceAccountToken: true` | +| Run-pod egress to the metadata database | **Mandatory** | Not applicable | The pod that writes run events | +| Namespace for step pods | Conditional — `k8s_job_executor` only | Not applicable | Defaults to the run pod's namespace | +| RBAC to create and watch Jobs | Conditional — `k8s_job_executor` only | **Mandatory**, in the payload namespace | Same verbs, different namespace | +| Shared RWX volume plus an I/O manager `base_dir` under it | Conditional — `k8s_job_executor` only | Optional | Without it step pods cannot read each other's outputs | +| Per-step resource requests and limits | Optional | Optional | Multiply by step or dispatch concurrency | +| Payload image reference and tag | Not applicable | **Mandatory** | Must share the code location's commit tag | +| Namespace for payload workloads | Not applicable | **Mandatory** | May be in another cluster or trust boundary | +| Message channel and its reader | Not applicable | **Mandatory** | Pod log stream by default; RBAC on `pods/log` *is* the channel | +| Payload wait timeout | Not applicable | Optional | Defaults to 24 h — lower it so scheduling failures surface | +| Vault role and secret paths | Conditional — if the workflow reads secrets | Conditional — launching pod only | Payloads should receive only `extras` and explicit `env` | +| Object-storage endpoint and credentials | Conditional — if the workflow uses S3 | Conditional — if the *payload* uses S3 | | +| Image pull secret | Conditional — non-anonymous registry | Conditional — non-anonymous registry | Applies to the SA that runs the pod, which for dispatched payload Jobs is `default` | + +### 3.5 Minimum input set + +The list below is the reusable baseline: the smallest set of values a team has to +agree on before a workflow can be configured against a cluster. Record it once +per workflow, in the workflow's own repository, and carry it into whatever +configuration format the team already uses. + +**Both setups** + +1. `cluster` — the cluster the run worker lands in. +2. `run_namespace` — namespace for run pods. +3. `code_location_image` — image reference including an immutable tag. +4. `executor` — executor name plus its concurrency bound. +5. `service_account` — the identity run pods assume. + +**Tightly coupled adds** + +6. `metadata_db` — host and port the run and step pods must reach. +7. `step_namespace` — only when `k8s_job_executor` is used. +8. `shared_output_path` — RWX mount path, and the I/O manager `base_dir` pinned + under it, again only for `k8s_job_executor`. + +**Loosely coupled adds** + +6. `payload_image` — image reference sharing the code location's commit tag. +7. `payload_namespace` — where dispatched workloads are created; may be in + another cluster. +8. `message_channel` — the reader in use, and the RBAC verb that carries it. + +This guide deliberately does **not** prescribe a configuration schema or key +names. The names above are descriptive labels for the values, not a required +format; the examples in section 3.6 express the same baseline in two different +shapes, and both are correct. + +### 3.6 Worked input examples + +The reference configuration in [yaml/](../../yaml) shows the baseline expressed +in the formats each setup already uses. Neither is normative. + +**Tightly coupled** — the inputs land in Helm values and in the job's executor +config, because that is where the platform chart and Dagster expect them: + +```yaml +# yaml/tightly-coupled/values-run-launcher.yaml (shape, not a fixed schema) +dagster: + runLauncher: + type: K8sRunLauncher + config: + k8sRunLauncher: + jobNamespace: dagster # run_namespace + failPodOnRunFailure: true + runK8sConfig: + containerConfig: + env: # metadata_db, S3 and Vault reachability + - name: S3_ENDPOINT_URL + value: https://s3.dev.simpl-europe.eu +``` + +```python +# step_namespace, shared_output_path and the executor travel with the job +tightly_coupled_k8s_job = distributed_execution_reference.to_job( + executor_def=k8s_job_executor, + resource_defs={"io_manager": fs_io_manager.configured({"base_dir": SHARED_IO_BASE_DIR})}, + config={"execution": {"config": {"step_k8s_config": STEP_K8S_CONFIG}}}, +) +``` + +**Loosely coupled** — the same baseline, but the cluster-facing half is the +payload image and its namespace. Here they are environment values set on both the +code location and the run launcher, because the dispatching pod is a run pod: + +```yaml +# yaml/loosely-coupled/values-pipes-payload.yaml (shape, not a fixed schema) +dagster: + dagster-user-deployments: + deployments: + - name: distributed-execution + env: + - name: PIPES_PAYLOAD_IMAGE # payload_image, tag-locked + value: /distributed-execution/payload:0.0.0 + - name: PIPES_PAYLOAD_NAMESPACE # payload_namespace + value: dagster +``` + +```python +PAYLOAD_IMAGE = os.environ.get("PIPES_PAYLOAD_IMAGE", ...) +PAYLOAD_NAMESPACE = os.environ.get("PIPES_PAYLOAD_NAMESPACE", "dagster") +``` + +The `message_channel` input has no key in either file. The default reader is the +pod log stream, so it is configured by granting `pods/log` in +[yaml/loosely-coupled/rbac-pipes-dispatch.yaml](../../yaml/loosely-coupled/rbac-pipes-dispatch.yaml). +An input can be mandatory and still not be a configuration field. + +Note what is absent from the loosely coupled example: no metadata database, no +Vault role, no object-storage credentials. Those inputs are not optional there — +they do not exist, which is the operational consequence of the choice. + --- -## 4. AC3 — Readiness checks +## 4. Readiness checks Pre-run validation items, their expected evidence, and the symptoms and corrections for common misconfigurations are maintained separately, in @@ -189,7 +314,7 @@ corrections for common misconfigurations are maintained separately, in --- -## 5. AC4 — Configuring execution-target linkage in code +## 5. Configuring execution-target linkage in code ### 5.1 Configuration constructs @@ -550,24 +675,59 @@ output metadata as retained evidence: ## 8. Outstanding work -Tracked under SIMPL-30451. +Tracked under SIMPL-30451 and SIMPL-30787. -| Item | AC | Status | +| Item | Status | +|---|---| +| Execution-target choices, prerequisites, workflow-level documentation | Complete | +| Comparison, environment-fit indicators, trade-offs | Complete | +| Cluster-configuration inputs, mandatory vs optional, minimum input set | Complete | +| Readiness checklist and evidence mapping | Complete; both Kubernetes paths cluster-verified | +| Misconfiguration symptoms and corrections | Complete; corrected with observed cluster failure modes | +| Configuration constructs and persistence | Complete | +| Before-and-after example, tightly coupled | Complete | +| Before-and-after example, switching to loosely coupled | Complete | +| Runnable tightly coupled reference implementations | Complete | +| Runnable loosely coupled reference implementation | Complete (subprocess and Kubernetes transports both verified) | +| Both images build; payload image passes the isolation check (L11) | Complete | +| Locally runnable jobs launched from the Dagster UI, evidence recorded in section 5.6 | Complete | +| Payload image published to a container registry | Complete on the sandbox Gitea registry; the GitLab registry still pending | +| GitLab pipeline builds both images | **Pending** — the shared `ds.gitlab-ci.yml` template builds one image from the root Dockerfile | +| End-to-end run of `loosely_coupled_k8s_job` on a cluster | Complete — sandbox-cat-dat, 2026-08-31, run `cff9b348…`; see the readiness checklist | +| `distributed-execution` registered as a code location on a platform Dagster | Complete — tightly coupled run `1d8cb167…` launched through the platform webserver | +| Screenshots of UI surfaces | Complete — job list, graph, Launchpad linkage and successful run captured in section 5.7 | +| Platform architecture document update | Complete — root `deployment_diagram.md` includes the service and both runtime-information paths | + +--- + +## 9. Requirement traceability + +The sections above are organised by subject, not by ticket. Two stories share +this material; the mapping below is authoritative for both. + +### SIMPL-30787 — defining the execution container cluster + +| AC | Requirement | Where it is met | |---|---|---| -| Execution-target choices, prerequisites, workflow-level documentation | AC1 | Complete | -| Comparison, environment-fit indicators, trade-offs | AC2 | Complete | -| Readiness checklist and evidence mapping | AC3 | Complete; both Kubernetes paths cluster-verified | -| Misconfiguration symptoms and corrections | AC3 | Complete; corrected with observed cluster failure modes | -| Configuration constructs and persistence | AC4 | Complete | -| Before-and-after example, tightly coupled | AC4 | Complete | -| Before-and-after example, switching to loosely coupled | AC4 | Complete | -| Runnable tightly coupled reference implementations | AC4 | Complete | -| Runnable loosely coupled reference implementation | AC4 | Complete (subprocess and Kubernetes transports both verified) | -| Both images build; payload image passes the isolation check (L11) | Tech details | Complete | -| Locally runnable jobs launched from the Dagster UI, evidence recorded in section 5.6 | Tech details | Complete | -| Payload image published to a container registry | Tech details | Complete on the sandbox Gitea registry; the GitLab registry still pending | -| GitLab pipeline builds both images | Tech details | **Pending** — the shared `ds.gitlab-ci.yml` template builds one image from the root Dockerfile | -| End-to-end run of `loosely_coupled_k8s_job` on a cluster | Tech details | Complete — sandbox-cat-dat, 2026-08-31, run `cff9b348…`; see the readiness checklist | -| `distributed-execution` registered as a code location on a platform Dagster | Tech details | Complete — tightly coupled run `1d8cb167…` launched through the platform webserver | -| Screenshots of UI surfaces | Tech details | Complete — job list, graph, Launchpad linkage and successful run captured in section 5.7 | -| Platform architecture document update | Tech details | Complete — root `deployment_diagram.md` includes the service and both runtime-information paths | +| AC1 | Prerequisites for both setups | Section 2.3 | +| AC1 | What must be documented before execution | Section 2.4 | +| AC1 | Where cluster-runtime connectivity is required, where decoupled reporting is expected | Sections 1.2 and 2.3; the asymmetry is drawn in the section 1.2 diagram | +| AC2 | Mandatory vs optional inputs per setup | Section 3.4 | +| AC2 | Examples for both setups, no fixed config naming model | Section 3.6, with the closing note in 3.5 | +| AC2 | Reusable minimum input set | Section 3.5 | +| AC3 | Checklist separates tightly from loosely coupled checks | [readiness-checklist.md](readiness-checklist.md) sections 2 and 3; section 1 holds the checks common to both | +| AC3 | Each item mapped to expected evidence | The *Expected evidence* column of every checklist table | +| AC4 | Configuration constructs attaching the pattern to a workflow | Section 5.1 | +| AC4 | Worked before-and-after example across setups | Sections 5.2 and 5.3 | +| AC4 | At least one runnable reference implementation per setup | Section 5.5 — three tightly coupled, two loosely coupled | +| AC4 | Selection persisted in configuration artifacts, not runtime-only | Section 5.4 | +| Tech | Diagrams, screenshots, working config files, upstream links | Sections 1.2, 5.7, [yaml/](../../yaml), section 6 | + +### SIMPL-30451 — execution targets and integration patterns + +| AC | Subject | Where it is met | +|---|---|---| +| AC1 | Choosing an execution target | Section 2 | +| AC2 | Decision support | Sections 3.1 to 3.3 | +| AC3 | Readiness checks | Section 4 and the readiness checklist | +| AC4 | Linkage configured in code | Section 5 | diff --git a/documents/user-guide/readiness-checklist.md b/documents/user-guide/readiness-checklist.md index 9c721a8..b95d4f1 100644 --- a/documents/user-guide/readiness-checklist.md +++ b/documents/user-guide/readiness-checklist.md @@ -1,7 +1,10 @@ # Distributed Execution: Readiness Checklist Pre-run validation for a workflow's execution target. Companion to the -[user guide](distributed-execution-guide.md); this document covers AC3. +[user guide](distributed-execution-guide.md); this document is the validation +checklist required by SIMPL-30451 AC3 and SIMPL-30787 AC3. Section 1 holds the +checks common to both setups, section 2 the tightly coupled checks and section 3 +the loosely coupled ones; every item states the evidence that proves it. Run these checks **before the first execution** of a workflow, and again after any change to the run launcher, the executor, the target namespace or the code diff --git a/yaml/tightly-coupled/values-run-launcher.yaml b/yaml/tightly-coupled/values-run-launcher.yaml index fc230d7..62e7770 100644 --- a/yaml/tightly-coupled/values-run-launcher.yaml +++ b/yaml/tightly-coupled/values-run-launcher.yaml @@ -13,8 +13,8 @@ dagster: config: k8sRunLauncher: # Namespace the run pods land in. Must be a namespace whose NetworkPolicy - # permits egress to Postgres, object storage and Vault - see AC1 - # prerequisites in the user guide. + # permits egress to Postgres, object storage and Vault - see the + # prerequisites in section 2.3 of the user guide. jobNamespace: dagster # Surfaces step failures as pod failures so kubectl and Dagster agree.