[SIMPL-30451] Add distributed-execution service with guide and reference implementations

Canonical location for documentation, example workflows and reference service implementations covering distributed execution patterns. Covers AC1-AC4: execution-target selection, decision support, readiness checks and code-level linkage. Tightly coupled jobs and the loosely coupled subprocess transport are verified by the test suite; the Kubernetes pipes transport is implemented but not yet cluster-run and is marked as such in the guide.

Changelog: added

.gitignore

@@ -0,0 +1,5 @@
+*.egg-info/
+**/__pycache__/
+.venv/
+.pytest_cache/
+.coverage

.gitlab/CODEOWNERS

@@ -0,0 +1,3 @@
+# All subgroup members as Code Owners for all files
+ 
+* @simpl/simpl-open/data/supporting-data-services

CHANGELOG.md

@@ -0,0 +1,18 @@
+## 0.1.0 (unreleased)
+
+### added
+
+- Repository skeleton for the `distributed-execution` supporting data service.
+- User guide covering execution-target selection, decision support and readiness checks.
+- Runnable tightly coupled reference implementations (`in_process`, `multiprocess`, `k8s_job_executor`).
+- Runnable loosely coupled reference implementations via dagster-pipes
+  (`PipesSubprocessClient`, `PipesK8sClient`) with a standalone external payload image.
+- Working example configuration and RBAC for both execution targets.
+
+### pending
+
+- Publish the payload image to the container registry.
+- End-to-end run of `loosely_coupled_k8s_job` against a cluster, and confirmation
+  of the Kubernetes rows in the readiness checklist.
+- Screenshots of UI surfaces and integration-pattern diagrams.
+- Platform architecture document update.

Dockerfile

@@ -0,0 +1,43 @@
+FROM python:3.12-slim-bookworm
+
+COPY --from=ghcr.io/astral-sh/uv:0.10.8 /uv /uvx /bin/
+
+RUN python -m pip install --no-cache-dir --upgrade "pip==26.1.2"
+
+WORKDIR /app
+
+# Explicit UID 1000 matches the runtime UID expected by the Dagster chart
+RUN addgroup --gid 1000 appgroup && \
+    adduser --uid 1000 --gid 1000 --disabled-password --gecos "" appuser
+
+RUN apt-get update && apt-get upgrade -y \
+    && apt-get install -y --no-install-recommends \
+    git=1:2.39.5-0+deb12u3 \
+    curl=7.88.1-10+deb12u15 \
+    && apt-get clean \
+    && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
+
+RUN chown -R appuser:appgroup /app /opt
+
+COPY pyproject.toml .
+COPY uv.lock .
+COPY src/ ./src/
+COPY payload/ ./payload/
+COPY workspace.yaml .
+
+ENV UV_COMPILE_BYTECODE=1
+ENV UV_LINK_MODE=copy
+
+RUN --mount=type=cache,target=/root/.cache/uv \
+    uv sync --locked --no-dev
+
+ENV PATH="/app/.venv/bin:${PATH}"
+ENV PYTHONPATH="/app/src"
+
+USER appuser
+
+RUN dagster --version
+
+EXPOSE 4000
+
+CMD ["dagster", "code-server", "start", "-h", "0.0.0.0", "-p", "4000", "-f", "src/distributed_execution/repository.py"]

LICENSE

@@ -0,0 +1,3 @@
+# EUROPEAN UNION PUBLIC LICENCE v. 1.2
+ 
+Refer to [licence description](https://eupl.eu/1.2/en/)

README.md

@@ -0,0 +1,85 @@
+# Distributed Execution
+
+Canonical location for documentation, example workflows and reference service
+implementations covering **distributed execution patterns** in the Simpl
+orchestration platform.
+
+The service answers one question for every workflow: *where does the compute
+actually run, and how does runtime information get back to the control plane?*
+
+## Contents
+
+| Area | Location | Status |
+|---|---|---|
+| User guide | [documents/user-guide/distributed-execution-guide.md](documents/user-guide/distributed-execution-guide.md) | Complete |
+| Readiness checklist | [documents/user-guide/readiness-checklist.md](documents/user-guide/readiness-checklist.md) | Complete |
+| Tightly coupled reference | [src/distributed_execution/tightly_coupled/jobs.py](src/distributed_execution/tightly_coupled/jobs.py) | Runnable |
+| Loosely coupled reference | [src/distributed_execution/loosely_coupled/jobs.py](src/distributed_execution/loosely_coupled/jobs.py) | Runnable (subprocess verified) |
+| External payload | [payload/work.py](payload/work.py) | Runnable |
+| Example configuration | [yaml/](yaml/) | Complete |
+
+## Project structure
+
+```text
+distributed-execution/
+├── src/
+│   └── distributed_execution/
+│       ├── repository.py           # Dagster definitions (entry point)
+│       ├── ops.py                  # Shared ops used by both patterns
+│       ├── preflight.py            # Readiness checks backing the checklist
+│       ├── tightly_coupled/
+│       │   └── jobs.py             # in_process / multiprocess / k8s_job_executor
+│       └── loosely_coupled/
+│           └── jobs.py             # PipesSubprocessClient / PipesK8sClient
+├── payload/                        # External workload: dagster-pipes ONLY
+│   ├── work.py
+│   ├── requirements.txt
+│   └── Dockerfile
+├── documents/user-guide/           # AC1-AC4 documentation
+├── yaml/                           # Working example configuration
+├── tests/
+├── Dockerfile
+├── pyproject.toml
+└── workspace.yaml
+```
+
+## Getting started
+
+Prerequisites: Python 3.12+ and `uv`.
+
+```bash
+uv sync --dev
+uv run dagster dev -f src/distributed_execution/repository.py
+```
+
+The Dagster UI is then available at <http://localhost:3000>. Two jobs run end to
+end on a laptop with no cluster: `tightly_coupled_local_job` and
+`loosely_coupled_subprocess_job`. The Kubernetes variants of each are documented
+in the user guide.
+
+### Running tests
+
+```bash
+uv run pytest
+```
+
+### Building the images
+
+Two images, deliberately: the code location and the external payload are
+versioned and scanned independently.
+
+```bash
+docker build -t distributed-execution:0.1.0 .
+docker build -f payload/Dockerfile -t distributed-execution-payload:0.1.0 payload/
+```
+
+## Status
+
+Both execution targets are implemented. The tightly coupled jobs and the loosely
+coupled **subprocess** transport are verified by the test suite. The loosely
+coupled **Kubernetes** transport is implemented but has not yet been run against
+a cluster; see the guide's *Outstanding work* section.
+
+## Licence
+
+European Union Public Licence v1.2 — see [LICENSE](LICENSE).

documents/user-guide/distributed-execution-guide.md

@@ -0,0 +1,443 @@
+# Distributed Execution: Execution Targets and Integration Patterns
+
+**Audience:** Participants with permission to configure workflows.
+**Scope:** How to select, document and configure the execution target and
+integration pattern for a Dagster workflow on the Simpl orchestration platform.
+
+> **Document status.** Both setups are backed by runnable reference
+> implementations in this repository. The tightly coupled setup and the loosely
+> coupled **subprocess** transport are verified end to end. The loosely coupled
+> **Kubernetes** transport is implemented and reviewed but has not yet been run
+> against a cluster; statements specific to it are marked
+> **NOT CLUSTER-VERIFIED**. See [Outstanding work](#8-outstanding-work).
+
+---
+
+## 1. Concepts
+
+### 1.1 What an execution target is
+
+The **execution target** is where the compute for a workflow actually runs. It is
+not a single Dagster setting — it is the combination of three decisions.
+
+| # | Decision | Where it is configured | Scope |
+|---|---|---|---|
+| 1 | **Placement** — which cluster and namespace the run lands in | Run launcher, in the Dagster instance / Helm values | Per deployment |
+| 2 | **Granularity** — one process for the whole run, or one per step | `executor_def`, in workflow code | Per job |
+| 3 | **Integration pattern** — whether the compute participates in Dagster's internals or reports back over a message channel | Job topology in workflow code | Per job |
+
+Decisions 1 and 2 are always present. Decision 3 is what this guide calls the
+*integration pattern*, and it is the one with the largest operational
+consequences, because it determines what the execution pod must be able to reach.
+
+### 1.2 The two integration patterns
+
+**Tightly coupled.** The process doing the work *is* a Dagster process. It imports
+the code location, opens a connection to the metadata database and writes run
+events directly. Runtime information reaches the control plane because the worker
+*is* part of the control plane's data path.
+
+**Loosely coupled.** Dagster only dispatches. It creates a workload on an external
+cluster running an arbitrary image, then acts as a listener. The external process
+writes structured messages over the dagster-pipes protocol; Dagster reads them and
+materialises them as logs and metadata. The execution pod never connects to the
+metadata database.
+
+```mermaid
+flowchart LR
+    subgraph TC["Tightly coupled"]
+        direction TB
+        D1[Dagster daemon] -->|K8sRunLauncher| RW1[Run worker pod]
+        RW1 -->|k8s_job_executor| SP1[Step pods]
+        RW1 -.->|events| DB1[(Metadata DB)]
+        SP1 -.->|events| DB1
+    end
+    subgraph LC["Loosely coupled"]
+        direction TB
+        D2[Dagster daemon] -->|K8sRunLauncher| RW2[Run worker pod]
+        RW2 -->|PipesK8sClient| EXT[External workload<br/>any image]
+        EXT -.->|pipes messages| RW2
+        RW2 -.->|events| DB2[(Metadata DB)]
+    end
+```
+
+Note the asymmetry in the diagram: in the loosely coupled pattern only the run
+worker touches the metadata database. That single arrow is the whole point of the
+pattern.
+
+---
+
+## 2. AC1 — Choosing an execution target
+
+### 2.1 Choose tightly coupled when
+
+- The workflow logic is written in Python and lives in a Dagster code location.
+- Execution pods can be granted network egress to the metadata database, object
+  storage and Vault.
+- You want per-step retries, step-level concurrency limits and asset
+  materialisation handled natively by Dagster.
+- The compute belongs to the same trust boundary as the orchestration platform.
+
+### 2.2 Choose loosely coupled when
+
+- The workload is not Python, or cannot take a dependency on the `dagster`
+  package (different runtime, vendor image, GPU base image).
+- The target cluster is administratively separate and **must not** be granted
+  connectivity to the orchestration metadata database.
+- The compute belongs to a different trust boundary, and you want the blast
+  radius of a compromised execution pod limited to the message channel.
+- The workload already exists as a container and you want to orchestrate it
+  without rewriting it.
+
+### 2.3 Prerequisites
+
+| Prerequisite | Tightly coupled | Loosely coupled |
+|---|---|---|
+| Run launcher configured | Required — `K8sRunLauncher` | Required — `K8sRunLauncher` |
+| Code location image contains workflow code | Required | Required for the *launching* job only |
+| Execution pod → metadata database (Postgres 5432) | **Required** | Not required |
+| Execution pod → object storage | Required if the workflow reads/writes objects | Required only if the payload itself does |
+| Execution pod → Vault | Required for secret injection | Only for the launching pod |
+| Kubernetes RBAC to create Jobs | Required for `k8s_job_executor` only | **Required** in the target namespace |
+| Message channel provisioned | Not applicable | **Required** — pod log stream or object-storage reader |
+| Control-plane reporting | Automatic — worker writes events directly | Via pipes messages; **fails silently if the channel breaks** |
+
+### 2.4 What to document before the first execution
+
+Record the following at workflow level, in the repository holding the workflow,
+before it is run for the first time. Section 5.4 explains where each item lives.
+
+1. **Chosen integration pattern** — tightly or loosely coupled, with a one-line
+   rationale referencing section 2.1 or 2.2.
+2. **Executor and its configuration** — including `max_concurrent` where relevant.
+3. **Target namespace** for run pods and, for `k8s_job_executor`, step pods.
+4. **Runtime dependencies** the execution pods must reach, as host/port pairs.
+   This is the input to the NetworkPolicy review.
+5. **Secrets consumed**, by Vault path — not by value.
+6. **Expected evidence of a healthy run** — which of the checks in the
+   [readiness checklist](readiness-checklist.md) apply, and what a passing run
+   looks like in the UI.
+7. **For loosely coupled only:** the payload image reference and the message
+   channel type.
+
+---
+
+## 3. AC2 — Decision support
+
+### 3.1 Side-by-side comparison
+
+| Dimension | Tightly coupled | Loosely coupled |
+|---|---|---|
+| What runs the work | A Dagster process importing your code location | Any container image |
+| Language constraint | Python, matching the code location's environment | None |
+| Dependency on `dagster` package | Full package in the execution image | Only `dagster-pipes` in the payload image |
+| Metadata DB connectivity from execution pod | **Required** | **Not required** |
+| Runtime information flow | Direct writes to run storage | Structured messages → run worker → run storage |
+| Log capture | Native, per step | Via the pipes message stream |
+| Step-level retries | Native | Manual, at the dispatching op |
+| Asset materialisation | Native, from any step | Reported by the payload, materialised by the launching op |
+| Failure granularity | Per step | Per dispatched workload |
+| Secret distribution | To every execution pod | To the launching pod, plus whatever the payload needs |
+| Blast radius of a compromised execution pod | Orchestration runtime credentials | Message channel only |
+| Cross-cluster execution | Not supported | Supported |
+
+### 3.2 Environment-fit indicators
+
+Use these as a fast triage. If several point the same way, that is your answer.
+
+| Observation about your environment | Points to |
+|---|---|
+| Workflow is pure Python and shares a repository with existing code locations | Tightly coupled |
+| Steps are short and numerous, and you want native retries | Tightly coupled |
+| NetworkPolicy review for the execution namespace is already approved | Tightly coupled |
+| Compute must run in a cluster you do not administer | Loosely coupled |
+| Security review objects to distributing DB credentials to execution pods | Loosely coupled |
+| Payload is an existing vendor or third-party image | Loosely coupled |
+| Workload needs GPUs or a base image incompatible with the code location image | Loosely coupled |
+| Team owning the payload is different from the team owning the workflow | Loosely coupled |
+
+### 3.3 Trade-offs
+
+**Tightly coupled**
+
+- *Gains:* the richest observability with no extra work — every step's logs,
+  metadata, retries and asset materialisations are native. Simplest mental model.
+- *Costs:* the widest connectivity surface. Every execution pod is a credential
+  holder and a database client. Scaling step pods scales database connections,
+  which is the usual first bottleneck. Execution image and code location image are
+  coupled, so a dependency conflict in one workflow can affect others sharing the
+  location.
+
+**Loosely coupled**
+
+- *Gains:* a narrow connectivity surface and genuine dependency isolation. The
+  payload team can ship on its own cadence. Cross-cluster and cross-trust-boundary
+  execution become possible.
+- *Costs:* observability is only as good as the payload's instrumentation — an
+  uninstrumented payload appears in Dagster as one opaque step. Retry and
+  cancellation semantics must be implemented at the dispatching op. The message
+  channel becomes a new failure mode, and when it breaks the run can appear to
+  succeed while reporting nothing. Two images to build, version and scan instead
+  of one.
+
+---
+
+## 4. AC3 — Readiness checks
+
+Pre-run validation items, their expected evidence, and the symptoms and
+corrections for common misconfigurations are maintained separately, in
+[readiness-checklist.md](readiness-checklist.md).
+
+---
+
+## 5. AC4 — Configuring execution-target linkage in code
+
+### 5.1 Configuration constructs
+
+Five constructs bind a workflow to an execution target. Three live in code and
+travel with the workflow; two live in deployment configuration.
+
+| Construct | Lives in | Binds |
+|---|---|---|
+| `executor_def` on `@job` / `to_job()` | Workflow code | Step granularity and placement |
+| `tags` on `@job` | Workflow code | Run-level metadata; `dagster-k8s/config` overrides pod shape |
+| Job topology — direct ops vs. a dispatching op using a pipes client | Workflow code | Integration pattern |
+| `runLauncher` | Helm values | Run worker placement and reachable dependencies |
+| `dagster-user-deployments` entry | Helm values | Which image the code location runs |
+
+The first three are the answer to "which configuration constructs bind the
+execution target to a workflow" — and they are all versioned alongside the
+workflow, which is why setup selection is a code review concern and not a runtime
+toggle.
+
+### 5.2 Worked example — switching within the tightly coupled setup
+
+The reference jobs share one graph and differ only by `executor_def`. This is the
+smallest possible demonstration that granularity is a code-level decision.
+
+**Before — steps run as subprocesses inside the run worker:**
+
+```python
+tightly_coupled_local_job = distributed_execution_reference.to_job(
+    name="tightly_coupled_local_job",
+    executor_def=multiprocess_executor.configured({"max_concurrent": 2}),
+    tags={**COMMON_TAGS, "executor": "multiprocess"},
+)
+```
+
+**After — one Kubernetes Job per step:**
+
+```python
+tightly_coupled_k8s_job = distributed_execution_reference.to_job(
+    name="tightly_coupled_k8s_job",
+    executor_def=k8s_job_executor.configured(
+        {
+            "image_pull_policy": "IfNotPresent",
+            "step_k8s_config": STEP_K8S_CONFIG,
+        }
+    ),
+    tags={**COMMON_TAGS, "executor": "k8s_job"},
+)
+```
+
+The graph, the ops and the run launcher are all unchanged. What changes
+operationally:
+
+- Step pods are now created in the launcher's `jobNamespace`, so that namespace's
+  NetworkPolicy must permit egress to Postgres, object storage and Vault — not
+  just the run worker's namespace.
+- The service account needs RBAC to create and watch Jobs. See
+  [yaml/tightly-coupled/rbac-step-executor.yaml](../../yaml/tightly-coupled/rbac-step-executor.yaml).
+- Per-step resource requests apply per pod, so the aggregate request for a
+  fan-out step is the per-step request multiplied by concurrency.
+- `summarise_results` now reports several distinct `contributing_hosts` instead of
+  one. That output metadata is the evidence that the switch actually took effect.
+
+### 5.3 Worked example — switching to loosely coupled
+
+The graphs for the two patterns differ by **exactly one node**. This is not a
+simplification for the guide; it is how the reference implementations are
+written, so the diff below is the real one.
+
+**Before — tightly coupled. The work happens in a Dagster process:**
+
+```python
+@graph
+def distributed_execution_reference():
+    target_report = report_execution_target()
+    units = generate_work_units()
+    results = process_work_units(units)          # runs in-process
+    return summarise_results(results, target_report)
+```
+
+**After — loosely coupled. The work is dispatched to an external image:**
+
+```python
+@graph
+def loosely_coupled_k8s_reference():
+    target_report = report_execution_target()
+    units = generate_work_units()
+    results = dispatch_external_work_k8s(units)  # dispatches, then listens
+    return summarise_results(results, target_report)
+```
+
+The dispatching op replaces direct computation with a pipes client call:
+
+```python
+@op(out=Out(list))
+def dispatch_external_work_k8s(
+    context: OpExecutionContext,
+    units: list,
+    pipes_k8s_client: PipesK8sClient,
+) -> list:
+    completed = pipes_k8s_client.run(
+        context=context,
+        image=PAYLOAD_IMAGE,
+        command=["python", "/app/work.py"],
+        namespace=PAYLOAD_NAMESPACE,
+        extras={"units": units},
+    )
+    return _results_from_pipes(context, completed)
+```
+
+and the job supplies the client as a resource instead of an executor:
+
+```python
+loosely_coupled_k8s_job = loosely_coupled_k8s_reference.to_job(
+    name="loosely_coupled_k8s_job",
+    resource_defs={"pipes_k8s_client": PipesK8sClient()},
+    tags={**COMMON_TAGS, "transport": "k8s_pod_logs"},
+)
+```
+
+#### What changes operationally
+
+| | Tightly coupled | Loosely coupled |
+|---|---|---|
+| Images to build | 1 (code location) | 2 (code location + payload) |
+| Payload dependency surface | Full `dagster` package | `dagster-pipes` only |
+| Execution pod → Postgres | Egress rule **required** | Egress rule can be **removed** |
+| Execution pod → Vault / S3 | Credentials required | None needed by the payload |
+| RBAC | Only for `k8s_job_executor` | Required on the **dispatching** pod |
+| New failure mode | — | Broken message path |
+
+The NetworkPolicy rules you can *remove* are the point of the exercise. See
+[yaml/loosely-coupled/rbac-pipes-dispatch.yaml](../../yaml/loosely-coupled/rbac-pipes-dispatch.yaml)
+for what replaces them — note that `pods/log` is the message channel, so
+removing that verb silently breaks reporting without failing the workload.
+
+#### The payload contract
+
+[payload/work.py](../../payload/work.py) is the whole contract:
+
+```python
+with open_dagster_pipes() as pipes:
+    units = pipes.get_extra("units")
+    pipes.log.info(f"External payload started on {host}")
+    pipes.report_custom_message({"results": results, "host": host, ...})
+```
+
+`pipes.log` entries appear in the Dagster run logs; `report_custom_message` is
+read back by the dispatching op. The payload imports `dagster_pipes` and nothing
+else from the Dagster ecosystem — a test asserts this, because the moment the
+payload imports `dagster` the isolation argument for choosing this pattern
+collapses.
+
+#### Message channel choice
+
+`PipesK8sClient` defaults to `PipesK8sPodLogsMessageReader`, which is what the
+reference uses. It needs no extra infrastructure, but it couples message
+delivery to the pod log stream — a log shipper that intercepts or truncates
+stdout will break reporting while the workload still exits 0. That is why
+`_results_from_pipes` raises on an empty message list rather than returning an
+empty result. If your cluster's logging setup makes that transport unreliable,
+switch to an object-storage message reader, which needs a bucket plus
+credentials on both sides.
+
+> **NOT CLUSTER-VERIFIED.** The Kubernetes transport above is implemented and
+> its RBAC and configuration are recorded, but it has not yet been run against a
+> Simpl cluster. The subprocess transport, which exercises the identical payload
+> and the identical message-parsing path, is verified by the test suite.
+
+### 5.4 Where setup selection is persisted
+
+Setup selection is persisted in configuration artifacts, never as a runtime-only
+setting. There is no UI control that switches a workflow between the two patterns.
+
+| Artifact | Holds |
+|---|---|
+| `src/<package>/**/jobs.py` | `executor_def`, pipes client resource, tags, job topology |
+| `payload/work.py` + `payload/Dockerfile` | The external payload contract and its image |
+| `yaml/tightly-coupled/values-run-launcher.yaml` | Run launcher type, namespace, pod env |
+| `yaml/tightly-coupled/rbac-step-executor.yaml` | Permissions required by `k8s_job_executor` |
+| `yaml/loosely-coupled/values-pipes-payload.yaml` | Payload image reference and target namespace |
+| `yaml/loosely-coupled/rbac-pipes-dispatch.yaml` | Permissions required by the dispatching pod |
+| `yaml/values-dagster-distributed-execution.yaml` | Code location image and entry point |
+
+The Launchpad can override *run configuration* — op config, resource config, tags
+— for a single run. It cannot change the executor or the integration pattern.
+Those take effect only when a new image is built and the code location reloads.
+
+### 5.5 Reference implementations in this repository
+
+| Job | Executor / transport | Runs locally | Demonstrates |
+|---|---|---|---|
+| `tightly_coupled_in_process_job` | `in_process_executor` | Yes | Baseline; steps inside the run worker |
+| `tightly_coupled_local_job` | `multiprocess_executor` | Yes | Subprocess fan-out bounded by the run pod |
+| `tightly_coupled_k8s_job` | `k8s_job_executor` | No — needs a cluster | One Kubernetes Job per step |
+| `loosely_coupled_subprocess_job` | `PipesSubprocessClient` | Yes | The pipes contract end to end, no cluster |
+| `loosely_coupled_k8s_job` | `PipesK8sClient` | No — needs a cluster | External Job dispatch, messages over pod logs |
+
+The last two dispatch the same [payload/work.py](../../payload/work.py). The
+subprocess variant exists so the pipes contract can be exercised, tested and
+demonstrated without any infrastructure.
+
+Run the local variants with:
+
+```bash
+uv sync --dev
+uv run dagster dev -f src/distributed_execution/repository.py
+```
+
+`report_execution_target` emits pod identity and the visible orchestration
+environment variables as output metadata on every run, which is the evidence the
+readiness checklist refers to.
+
+---
+
+## 6. Upstream documentation
+
+- Dagster — [run launchers](https://docs.dagster.io/deployment/execution/run-launchers)
+- Dagster — [executors](https://docs.dagster.io/deployment/execution/executors)
+- Dagster — [customizing your Kubernetes deployment](https://docs.dagster.io/deployment/oss/deployment-options/kubernetes/customizing-your-deployment)
+- Dagster — [Dagster Pipes](https://docs.dagster.io/guides/build/external-pipelines)
+- Dagster — [Dagster Pipes with Kubernetes](https://docs.dagster.io/guides/build/external-pipelines/kubernetes-pipeline)
+- Kubernetes — [Jobs](https://kubernetes.io/docs/concepts/workloads/controllers/job/)
+- Kubernetes — [network policies](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
+- Kubernetes — [RBAC authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/)
+
+## 7. Related Simpl documentation
+
+- [dagster/documents/user-manual/Orchestration.md](../../../dagster/documents/user-manual/Orchestration.md) — workflow development lifecycle
+- [dagster/documents/installation-guide/Installation Guide.md](../../../dagster/documents/installation-guide/Installation%20Guide.md) — full run launcher parameter reference
+- [dagster/documents/user-manual/Observability.md](../../../dagster/documents/user-manual/Observability.md) — logging and monitoring
+
+## 8. Outstanding work
+
+Tracked under SIMPL-30451.
+
+| Item | AC | Status |
+|---|---|---|
+| Execution-target choices, prerequisites, workflow-level documentation | AC1 | Complete |
+| Comparison, environment-fit indicators, trade-offs | AC2 | Complete |
+| Readiness checklist and evidence mapping | AC3 | Complete; K8s pipes checks not cluster-verified |
+| Misconfiguration symptoms and corrections | AC3 | Complete; K8s pipes symptoms not cluster-verified |
+| Configuration constructs and persistence | AC4 | Complete |
+| Before-and-after example, tightly coupled | AC4 | Complete |
+| Before-and-after example, switching to loosely coupled | AC4 | Complete |
+| Runnable tightly coupled reference implementations | AC4 | Complete |
+| Runnable loosely coupled reference implementation | AC4 | Complete (subprocess verified, K8s not cluster-run) |
+| Payload image published to the registry | Tech details | **Pending** |
+| End-to-end run of `loosely_coupled_k8s_job` on a cluster | Tech details | **Pending** |
+| Screenshots of UI surfaces | Tech details | **Pending** |
+| Platform architecture document update | Tech details | **Pending** |

documents/user-guide/readiness-checklist.md

@@ -0,0 +1,118 @@
+# Distributed Execution: Readiness Checklist
+
+Pre-run validation for a workflow's execution target. Companion to the
+[user guide](distributed-execution-guide.md); this document covers AC3.
+
+Run these checks **before the first execution** of a workflow, and again after any
+change to the run launcher, the executor, the target namespace or the code
+location image.
+
+> **NOT CLUSTER-VERIFIED.** Checks L4–L9 and the Kubernetes rows of section 4.3
+> are derived from the implemented reference but have not yet been run against a
+> Simpl cluster. Checks L1–L3, which exercise the payload contract and the
+> message-parsing path, are covered by the test suite.
+
+---
+
+## 1. Common checks — both setups
+
+| # | Check | How to verify | Expected evidence |
+|---|---|---|---|
+| C1 | Code location loads | Dagster UI → **Deployment** → **Code locations** | Location `distributed-execution` shows status *Loaded*, with a recent load timestamp and no error banner |
+| C2 | Jobs are registered | Dagster UI → **Jobs** | The jobs listed in the guide's section 5.5 appear under the code location |
+| C3 | Image tag matches the intended release | `kubectl -n dagster get deploy -l dagster/code-location=distributed-execution -o jsonpath='{.items[*].spec.template.spec.containers[*].image}'` | Tag equals the version in `pipeline.variables.sh`; never `latest` |
+| C4 | Image architecture matches the nodes | `docker manifest inspect <image>` | Includes `linux/amd64`; a manifest with only `linux/arm64` produces `no match for platform` at pull time |
+| C5 | Run launcher type is as intended | `kubectl -n dagster get cm dagster-instance -o yaml` | `run_launcher` block shows `K8sRunLauncher` and the expected `job_namespace` |
+| C6 | Target namespace exists and is schedulable | `kubectl get ns <namespace>` and `kubectl -n <namespace> get resourcequota` | Namespace is `Active`; remaining quota exceeds the job's aggregate requests |
+
+## 2. Tightly coupled checks
+
+| # | Check | How to verify | Expected evidence |
+|---|---|---|---|
+| T1 | Run pod reaches the metadata database | Launch `tightly_coupled_in_process_job` | Run reaches `SUCCESS`; `report_execution_target` output metadata lists `DAGSTER_POSTGRES_HOST`, `DAGSTER_POSTGRES_USER` and `DAGSTER_POSTGRES_DB` under `env_vars_present`, and `env_vars_missing` is empty |
+| T2 | Vault injection works | Same run; inspect the run pod | `kubectl -n dagster describe pod <run-pod>` shows the `vault-env` init container completed; no `vault:` literal remains in the process environment |
+| T3 | Object storage is reachable | Same run, if the workflow uses S3 | No `EndpointConnectionError` in run logs; `S3_ENDPOINT_URL` present in `env_vars_present` |
+| T4 | Multiprocess fan-out is bounded correctly | Launch `tightly_coupled_local_job` | Run succeeds; `summarise_results` metadata shows exactly one entry in `contributing_hosts`, confirming steps stayed on the run worker |
+| T5 | RBAC permits step Jobs | `kubectl -n dagster auth can-i create jobs --as=system:serviceaccount:dagster:dagster-dev` | Returns `yes`; required only for `k8s_job_executor` |
+| T6 | Step pods are actually created | Launch `tightly_coupled_k8s_job`, then `kubectl -n dagster get jobs -l dagster/run-id=<run-id>` | One Job per step; `summarise_results` metadata shows **several** distinct `contributing_hosts` |
+| T7 | Step pod egress is permitted | Same run | Steps do not hang in `STARTING`; run logs contain no connection timeouts to port 5432 |
+| T8 | Failure surfaces as a pod failure | Force a step failure in a scratch namespace | `failPodOnRunFailure: true` is set, and the step pod reports `Failed` rather than `Completed` |
+
+## 3. Loosely coupled checks
+
+| # | Check | How to verify | Expected evidence |
+|---|---|---|---|
+| L1 | Payload contract is intact | `uv run pytest tests/test_loosely_coupled.py` | `test_payload_does_not_import_dagster` passes — the payload imports `dagster_pipes` only |
+| L2 | Pipes round trip works | Launch `loosely_coupled_subprocess_job` | Run reaches `SUCCESS`; run logs contain the payload's `External payload started on …` line, proving `pipes.log` crossed the channel |
+| L3 | Silence is treated as failure | Same test module | `test_silent_message_path_is_treated_as_failure` passes — an empty message list raises rather than yielding an empty result |
+| L4 | Payload image is pullable by the target cluster | `kubectl -n <payload-ns> run pull-probe --image=<payload-image> --restart=Never --command -- true` | Pod reaches `Completed`; no `ImagePullBackOff` |
+| L5 | Dispatcher can create Jobs in the payload namespace | `kubectl -n <payload-ns> auth can-i create jobs --as=system:serviceaccount:dagster:dagster-dev` | Returns `yes` |
+| L6 | Dispatcher can read pod logs — **the message channel** | `kubectl -n <payload-ns> auth can-i get pods/log --as=system:serviceaccount:dagster:dagster-dev` | Returns `yes`. A `no` here breaks reporting *without* failing the workload |
+| L7 | Payload Job is actually created | Launch `loosely_coupled_k8s_job`, then `kubectl -n <payload-ns> get jobs -l app.kubernetes.io/name=distributed-execution-payload` | One Job per dispatch, labelled `dagster/execution-target=loosely-coupled` |
+| L8 | Payload has **no** orchestration connectivity | Same run; inspect run logs | No `Payload could see orchestration runtime credentials` warning. This is a positive check — absence of errors is not sufficient |
+| L9 | Work ran off-platform | Same run | `summarise_results` metadata shows `contributing_hosts` containing the payload pod name, not the run worker's hostname |
+
+Checks L4–L9 require a cluster. L1–L3 run on a laptop and should gate every
+change to the payload or the dispatching op.
+
+---
+
+## 4. Common misconfiguration symptoms
+
+### 4.1 Both setups
+
+| Symptom | Likely cause | Correction |
+|---|---|---|
+| Code location stuck in *Loading*, then errors | Entry point path in `codeServerArgs` does not match the image layout | Confirm `--python-file` matches `workspace.yaml`; both must be `src/distributed_execution/repository.py` |
+| `ImagePullBackOff` with `no match for platform` | Image published for a single non-matching architecture | Rebuild multi-arch with `docker buildx`, and pin a version tag rather than `latest` |
+| Run stays in `QUEUED` indefinitely | Run coordinator concurrency limit reached, or no schedulable node | Check `max_concurrent_runs` and tag concurrency limits; check node capacity and resource quota |
+| Run fails immediately with a serialisation error | Code location image and Dagster control-plane versions diverge | Align the `dagster` version in `pyproject.toml` with the chart's version and rebuild |
+
+### 4.2 Tightly coupled
+
+| Symptom | Likely cause | Correction |
+|---|---|---|
+| Run pod starts, then fails with a connection timeout to port 5432 | Execution namespace NetworkPolicy does not permit egress to Postgres | Add an egress rule for the metadata database, or move run pods to an already-approved namespace via `jobNamespace` |
+| `env_vars_missing` is non-empty in `report_execution_target` metadata | Env vars are set on the code location deployment but not on the run pod | Add them under `runLauncher.config.k8sRunLauncher.runK8sConfig.containerConfig.env` — code location env is **not** inherited by run pods |
+| A literal `vault:...` string appears as a value at runtime | Vault mutating webhook did not process the pod | Verify the `vault.security.banzaicloud.io/*` annotations are on the **run pod** template, not only the code location pod |
+| Steps hang in `STARTING` with `k8s_job_executor` | Service account lacks Job create/watch permission | Apply `yaml/tightly-coupled/rbac-step-executor.yaml` and confirm with `kubectl auth can-i` |
+| `contributing_hosts` shows one host when `k8s_job_executor` is configured | Run tags or Launchpad config overrode the executor, or the image predates the change | Confirm the code location reloaded after the image bump; check the run's *Config* tab for an `execution:` override |
+| Step pods `OOMKilled` under fan-out | Per-step memory limit applied per pod, aggregate exceeded quota | Raise `step_k8s_config` limits or lower step concurrency; the two multiply |
+| Postgres refuses connections once fan-out grows | Each step pod is an independent DB client | Reduce step concurrency, raise the Postgres connection limit, or move the fan-out step to a loosely coupled target |
+
+### 4.3 Loosely coupled
+
+| Symptom | Likely cause | Correction |
+|---|---|---|
+| Op fails with `No pipes messages received from the external payload` | The message path is broken, not the workload | Work through L6 then L4. The payload very likely ran and succeeded; only its reporting was lost |
+| Payload pod `Completed`, but Dagster shows no payload log lines | A log shipper is intercepting or truncating stdout | Exclude the payload namespace from the shipper, or switch to an object-storage message reader |
+| Op hangs until `pod_wait_timeout` (default 24 h) | Payload Job never scheduled — quota, node selector or image pull | Check `kubectl -n <payload-ns> describe job <name>`; lower `pod_wait_timeout` so the failure surfaces quickly |
+| `403 Forbidden` creating the Job | Dispatcher service account lacks Job create permission | Apply `yaml/loosely-coupled/rbac-pipes-dispatch.yaml` in the **payload** namespace |
+| Warning: `Payload could see orchestration runtime credentials` | Payload pod inherited run-pod env or a Vault annotation | Remove the inherited env; the payload should receive only what `extras` and explicit `env` pass it |
+| Payload exits non-zero but the run reports success | Exit status not being checked, or messages read before failure | Confirm the dispatching op returns through `_results_from_pipes`; do not swallow `PipesClientCompletedInvocation` errors |
+| Payload receives no `units` | `extras` key mismatch between dispatcher and `pipes.get_extra()` | Both sides must use the same key; a typo yields a `KeyError` inside the payload |
+| Run cancelled in the UI, payload pod keeps running | Cancellation is not propagated to dispatched workloads automatically | `delete_pod_on_completion` handles the normal path; for cancellation, verify orphaned Jobs and add a cleanup sensor |
+
+> **NOT CLUSTER-VERIFIED.** Rows referencing Kubernetes behaviour follow from the
+> implemented reference and the dagster-k8s API, but have not been observed on a
+> Simpl cluster. Confirm and amend after the first cluster run.
+
+---
+
+## 5. Evidence retention
+
+For each workflow's first execution, attach to the workflow's repository or
+change record:
+
+1. The run ID and its final status.
+2. The `report_execution_target` output metadata block (pod identity, namespace,
+   env var presence).
+3. The `summarise_results` metadata block (`contributing_hosts`), which proves
+   which execution target was actually used.
+4. For `k8s_job_executor`, the output of `kubectl get jobs -l dagster/run-id=<run-id>`.
+5. For the loosely coupled target, the payload image digest and the run log line
+   emitted by `pipes.log` — together they prove which payload version ran and
+   that the message channel was open.
+
+Items 2 and 3 together are sufficient to demonstrate that the configured
+execution target is the one that ran — which is the point of the checklist.

payload/Dockerfile

@@ -0,0 +1,30 @@
+# External payload image for the loosely coupled execution target.
+#
+# Built and versioned independently of the code location image. It contains no
+# workflow code and no `dagster` package - only the pipes protocol client.
+#
+#   docker build -f payload/Dockerfile -t distributed-execution-payload:0.1.0 payload/
+
+FROM python:3.12-slim-bookworm
+
+RUN python -m pip install --no-cache-dir --upgrade "pip==26.1.2"
+
+WORKDIR /app
+
+RUN addgroup --gid 1000 appgroup && \
+    adduser --uid 1000 --gid 1000 --disabled-password --gecos "" appuser
+
+RUN apt-get update && apt-get upgrade -y \
+    && apt-get clean \
+    && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
+
+COPY requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+
+COPY work.py .
+
+RUN chown -R appuser:appgroup /app
+
+USER appuser
+
+CMD ["python", "/app/work.py"]

payload/requirements.txt

@@ -0,0 +1,3 @@
+# The payload's entire dependency surface. Note the absence of `dagster`:
+# dependency isolation is the point of the loosely coupled pattern.
+dagster-pipes>=1.13.0

payload/work.py

@@ -0,0 +1,55 @@
+"""External payload for the loosely coupled execution target.
+
+This script is deliberately NOT a Dagster code location. It depends only on
+``dagster-pipes``, never on ``dagster``, and it never opens a connection to the
+orchestration metadata database. Everything it reports reaches the control plane
+through the pipes message channel.
+
+Run by ``distributed_execution.loosely_coupled.jobs`` via either
+``PipesSubprocessClient`` (local) or ``PipesK8sClient`` (cluster).
+"""
+
+import os
+import socket
+
+from dagster_pipes import open_dagster_pipes
+
+# Presence of any of these would mean the payload was granted orchestration
+# runtime credentials it has no business holding.
+ORCHESTRATION_ENV_VARS = (
+    "DAGSTER_POSTGRES_HOST",
+    "DAGSTER_POSTGRES_USER",
+    "DAGSTER_POSTGRES_DB",
+)
+
+
+def main() -> None:
+    with open_dagster_pipes() as pipes:
+        units = pipes.get_extra("units")
+        host = socket.gethostname()
+
+        pipes.log.info(f"External payload started on {host} with {len(units)} work units")
+
+        results = [{"unit": unit, "squared": unit * unit, "host": host} for unit in units]
+
+        leaked = [name for name in ORCHESTRATION_ENV_VARS if os.environ.get(name)]
+        if leaked:
+            pipes.log.warning(
+                "Orchestration runtime credentials are visible to this payload: "
+                f"{', '.join(leaked)}. A loosely coupled target should not have them."
+            )
+
+        # The only channel back to the control plane.
+        pipes.report_custom_message(
+            {
+                "results": results,
+                "host": host,
+                "orchestration_env_visible": leaked,
+            }
+        )
+
+        pipes.log.info(f"External payload finished on {host}")
+
+
+if __name__ == "__main__":
+    main()

pipeline.variables.sh

@@ -0,0 +1 @@
+PROJECT_VERSION_NUMBER="0.1.0"

pyproject.toml

@@ -0,0 +1,40 @@
+[project]
+name = "distributed_execution"
+version = "0.1.0"
+description = "Reference implementations and user guide for Dagster distributed execution targets and integration patterns"
+requires-python = ">=3.12"
+dependencies = [
+    "dagster",
+    "dagster-k8s>=0.27.16",
+    "dagster-postgres>=0.27.16",
+    "dagster-webserver>=1.11.16",
+    "pip>=26.1.2",
+    "pydantic>=2.0",
+]
+
+[dependency-groups]
+dev = [
+    "pytest>=8.4.2",
+    "pytest-cov>=7.0.0",
+]
+
+[build-system]
+requires = ["setuptools", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools]
+package-dir = {"" = "src"}
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+distributed_execution = ["*.py"]
+
+[tool.black]
+line-length = 120
+target-version = ['py312']
+
+[tool.pytest.ini_options]
+pythonpath = ["src"]
+testpaths = ["tests"]

requirements.txt

@@ -0,0 +1,6 @@
+dagster
+dagster-k8s>=0.27.16
+dagster-postgres>=0.27.16
+dagster-webserver>=1.11.16
+pip>=26.1.2
+pydantic>=2.0

src/distributed_execution/__init__.py

@@ -0,0 +1,3 @@
+"""Reference implementations for Dagster distributed execution targets."""
+
+__version__ = "0.1.0"

src/distributed_execution/loosely_coupled/__init__.py

@@ -0,0 +1 @@
+"""Loosely coupled (dagster-pipes) execution-target reference implementations."""

src/distributed_execution/loosely_coupled/jobs.py

@@ -0,0 +1,154 @@
+"""Loosely coupled reference implementations.
+
+Dagster does not run the work here. It dispatches an external payload and then
+acts as a listener: the payload writes structured messages over the
+dagster-pipes protocol, which Dagster materialises as logs and metadata. The
+payload never connects to the metadata database.
+
+Two transports are provided. Both dispatch the *same* payload script and produce
+the same output shape as the tightly coupled ``process_work_units`` op, so the
+graphs differ by exactly one node - see the guide's section 5.3.
+"""
+
+import os
+import sys
+from pathlib import Path
+
+from dagster import Failure, OpExecutionContext, Out, PipesSubprocessClient, graph, op
+from dagster_k8s import PipesK8sClient
+
+from distributed_execution.ops import (
+    generate_work_units,
+    report_execution_target,
+    summarise_results,
+)
+
+# Payload lives outside src/ so it can be built into its own image.
+_REPO_ROOT = Path(__file__).resolve().parents[3]
+PAYLOAD_SCRIPT = os.environ.get("PIPES_PAYLOAD_SCRIPT", str(_REPO_ROOT / "payload" / "work.py"))
+
+PAYLOAD_IMAGE = os.environ.get(
+    "PIPES_PAYLOAD_IMAGE",
+    "code.europa.eu:4567/simpl/simpl-open/data/supporting-data-services/distributed-execution/payload:0.1.0",
+)
+PAYLOAD_NAMESPACE = os.environ.get("PIPES_PAYLOAD_NAMESPACE", "dagster")
+
+COMMON_TAGS = {
+    "execution_target": "loosely_coupled",
+    "business_operation": "DISTRIBUTED_EXECUTION_REFERENCE",
+}
+
+
+def _results_from_pipes(context: OpExecutionContext, completed) -> list:
+    """Read the payload's results off the message channel.
+
+    A broken message path is the defining failure mode of this pattern: the
+    external workload can exit 0 while reporting nothing, so silence is treated
+    as a failure rather than an empty result.
+    """
+    messages = completed.get_custom_messages()
+    if not messages:
+        raise Failure(
+            description=(
+                "No pipes messages received from the external payload. The workload may have "
+                "run successfully while its messages were lost. Verify the message path: for "
+                "the Kubernetes transport, confirm the pod log stream is readable and not "
+                "being truncated or intercepted by a log shipper."
+            )
+        )
+
+    payload = messages[-1]
+    leaked = payload.get("orchestration_env_visible") or []
+    if leaked:
+        context.log.warning(
+            "Payload could see orchestration runtime credentials: %s. "
+            "This defeats the isolation the loosely coupled target is chosen for.",
+            ", ".join(leaked),
+        )
+
+    context.log.info("Received %s results from external host %s", len(payload["results"]), payload["host"])
+    return payload["results"]
+
+
+@op(
+    description="Dispatches the external payload as a local subprocess and listens on the pipes channel.",
+    out=Out(list),
+)
+def dispatch_external_work_subprocess(
+    context: OpExecutionContext,
+    units: list,
+    pipes_subprocess_client: PipesSubprocessClient,
+) -> list:
+    completed = pipes_subprocess_client.run(
+        context=context,
+        command=[sys.executable, PAYLOAD_SCRIPT],
+        extras={"units": units},
+    )
+    return _results_from_pipes(context, completed)
+
+
+@op(
+    description="Dispatches the external payload as a Kubernetes Job and listens on the pod log stream.",
+    out=Out(list),
+)
+def dispatch_external_work_k8s(
+    context: OpExecutionContext,
+    units: list,
+    pipes_k8s_client: PipesK8sClient,
+) -> list:
+    completed = pipes_k8s_client.run(
+        context=context,
+        image=PAYLOAD_IMAGE,
+        command=["python", "/app/work.py"],
+        namespace=PAYLOAD_NAMESPACE,
+        extras={"units": units},
+        base_pod_meta={
+            "labels": {
+                "app.kubernetes.io/name": "distributed-execution-payload",
+                "dagster/execution-target": "loosely-coupled",
+            }
+        },
+    )
+    return _results_from_pipes(context, completed)
+
+
+@graph
+def loosely_coupled_subprocess_reference():
+    target_report = report_execution_target()
+    units = generate_work_units()
+    results = dispatch_external_work_subprocess(units)
+    return summarise_results(results, target_report)
+
+
+@graph
+def loosely_coupled_k8s_reference():
+    target_report = report_execution_target()
+    units = generate_work_units()
+    results = dispatch_external_work_k8s(units)
+    return summarise_results(results, target_report)
+
+
+loosely_coupled_subprocess_job = loosely_coupled_subprocess_reference.to_job(
+    name="loosely_coupled_subprocess_job",
+    description=(
+        "Loosely coupled via subprocess transport. Demonstrates the pipes contract end to end "
+        "on a laptop, with no cluster and no metadata database access from the payload."
+    ),
+    resource_defs={"pipes_subprocess_client": PipesSubprocessClient()},
+    tags={**COMMON_TAGS, "transport": "subprocess"},
+)
+
+loosely_coupled_k8s_job = loosely_coupled_k8s_reference.to_job(
+    name="loosely_coupled_k8s_job",
+    description=(
+        "Loosely coupled via Kubernetes Job dispatch. Messages return over the pod log stream. "
+        "Requires a cluster and RBAC to create Jobs in the target namespace."
+    ),
+    resource_defs={"pipes_k8s_client": PipesK8sClient()},
+    tags={**COMMON_TAGS, "transport": "k8s_pod_logs"},
+)
+
+LOOSELY_COUPLED_JOBS = [
+    loosely_coupled_subprocess_job,
+    loosely_coupled_k8s_job,
+]

src/distributed_execution/ops.py

@@ -0,0 +1,83 @@
+"""Ops shared by every execution-target reference implementation.
+
+The ops deliberately do trivial work. What they demonstrate is *where* the work
+happens and *how* that fact reaches the control plane.
+"""
+
+# No `from __future__ import annotations`: it stringifies the `context` hint and
+# Dagster's op context validation then rejects it.
+from dagster import Config, MetadataValue, OpExecutionContext, Out, op
+
+from distributed_execution.preflight import (
+    TIGHTLY_COUPLED_ENV_VARS,
+    check_env_vars,
+    describe_pod_identity,
+)
+
+
+class WorkUnitsConfig(Config):
+    count: int = 4
+
+
+@op(
+    description="Emits execution-target evidence: pod identity plus reachability of orchestration dependencies.",
+    out=Out(dict),
+)
+def report_execution_target(context: OpExecutionContext) -> dict:
+    identity = describe_pod_identity()
+    env_check = check_env_vars(TIGHTLY_COUPLED_ENV_VARS)
+
+    context.log.info("Executing on %s (pid %s), namespace %s", identity["hostname"], identity["pid"], identity["namespace"])
+    if env_check["missing"]:
+        context.log.warning(
+            "Orchestration runtime env vars not visible to this process: %s. "
+            "Expected for a loosely coupled target; a misconfiguration for a tightly coupled one.",
+            ", ".join(env_check["missing"]),
+        )
+
+    context.add_output_metadata(
+        {
+            "hostname": MetadataValue.text(identity["hostname"]),
+            "namespace": MetadataValue.text(identity["namespace"]),
+            "run_id": MetadataValue.text(identity["run_id"]),
+            "image": MetadataValue.text(identity["image"]),
+            "env_vars_present": MetadataValue.json(env_check["present"]),
+            "env_vars_missing": MetadataValue.json(env_check["missing"]),
+        }
+    )
+    return {"identity": identity, "env_check": env_check}
+
+
+@op(description="Produces the work units that later steps fan out over.", out=Out(list))
+def generate_work_units(context: OpExecutionContext, config: WorkUnitsConfig) -> list[int]:
+    units = list(range(config.count))
+    context.log.info("Generated %s work units", len(units))
+    return units
+
+
+@op(description="Performs one unit of work per element; runs once per step-execution slot.", out=Out(list))
+def process_work_units(context: OpExecutionContext, units: list[int]) -> list[dict]:
+    identity = describe_pod_identity()
+    results = [{"unit": unit, "squared": unit * unit, "host": identity["hostname"]} for unit in units]
+    context.log.info("Processed %s units on %s", len(results), identity["hostname"])
+    return results
+
+
+@op(description="Aggregates results and attaches the distinct hosts that contributed.")
+def summarise_results(context: OpExecutionContext, results: list[dict], target_report: dict) -> dict:
+    hosts = sorted({row["host"] for row in results})
+    summary = {
+        "units": len(results),
+        "total": sum(row["squared"] for row in results),
+        "contributing_hosts": hosts,
+        "launcher_namespace": target_report["identity"]["namespace"],
+    }
+    context.add_output_metadata(
+        {
+            "units": MetadataValue.int(summary["units"]),
+            "contributing_hosts": MetadataValue.json(hosts),
+            "launcher_namespace": MetadataValue.text(summary["launcher_namespace"]),
+        }
+    )
+    context.log.info("Summary: %s", summary)
+    return summary

src/distributed_execution/preflight.py

@@ -0,0 +1,61 @@
+"""Readiness checks backing the pre-run validation checklist.
+
+Each function returns a plain dict so the same result can be logged as Dagster
+metadata (evidence for the checklist) or asserted in a test.
+"""
+
+from __future__ import annotations
+
+import os
+import socket
+from urllib.parse import urlparse
+
+# Runtime dependencies a tightly coupled execution pod must be able to resolve.
+TIGHTLY_COUPLED_ENV_VARS = (
+    "DAGSTER_POSTGRES_HOST",
+    "DAGSTER_POSTGRES_USER",
+    "DAGSTER_POSTGRES_DB",
+)
+
+
+def check_env_vars(names: tuple[str, ...] | list[str]) -> dict:
+    """Report which of ``names`` are present, without echoing their values."""
+    present = [name for name in names if os.environ.get(name)]
+    missing = [name for name in names if not os.environ.get(name)]
+    return {
+        "check": "env_vars",
+        "passed": not missing,
+        "present": present,
+        "missing": missing,
+    }
+
+
+def check_tcp_reachable(host: str, port: int, timeout: float = 3.0) -> dict:
+    """Open a TCP connection to prove the execution pod can reach a dependency."""
+    result = {"check": "tcp_reachable", "target": f"{host}:{port}", "passed": False, "error": None}
+    try:
+        with socket.create_connection((host, port), timeout=timeout):
+            result["passed"] = True
+    except OSError as exc:
+        result["error"] = str(exc)
+    return result
+
+
+def check_endpoint_reachable(url: str, timeout: float = 3.0) -> dict:
+    """TCP-level reachability for an endpoint expressed as a URL."""
+    parsed = urlparse(url)
+    if not parsed.hostname:
+        return {"check": "tcp_reachable", "target": url, "passed": False, "error": "no hostname in URL"}
+    port = parsed.port or (443 if parsed.scheme == "https" else 80)
+    return check_tcp_reachable(parsed.hostname, port, timeout=timeout)
+
+
+def describe_pod_identity() -> dict:
+    """Where this process is actually running - the primary execution-target evidence."""
+    return {
+        "hostname": socket.gethostname(),
+        "pid": os.getpid(),
+        "namespace": os.environ.get("DAGSTER_K8S_PIPELINE_RUN_NAMESPACE", "<not-in-kubernetes>"),
+        "run_id": os.environ.get("DAGSTER_RUN_ID", "<unset>"),
+        "image": os.environ.get("DAGSTER_K8S_PIPELINE_RUN_IMAGE", "<unset>"),
+    }

src/distributed_execution/repository.py

@@ -0,0 +1,10 @@
+"""Dagster definitions for the distributed-execution code location."""
+
+from dagster import Definitions
+
+from distributed_execution.loosely_coupled.jobs import LOOSELY_COUPLED_JOBS
+from distributed_execution.tightly_coupled.jobs import TIGHTLY_COUPLED_JOBS
+
+defs = Definitions(
+    jobs=[*TIGHTLY_COUPLED_JOBS, *LOOSELY_COUPLED_JOBS],
+)

src/distributed_execution/tightly_coupled/__init__.py

@@ -0,0 +1 @@
+"""Tightly coupled execution-target reference implementations."""

src/distributed_execution/tightly_coupled/jobs.py

@@ -0,0 +1,97 @@
+"""Tightly coupled reference implementations.
+
+Tightly coupled means the process doing the work *is* a Dagster process: it
+imports the code location, connects to the metadata database and writes run
+events directly. Every execution pod therefore needs network reachability to the
+orchestration runtime dependencies.
+
+Three jobs are provided, differing only in their ``executor_def``. That single
+construct is the code-level half of the execution-target binding; the other half
+is the instance-level run launcher (see ``yaml/tightly-coupled/``).
+"""
+
+from __future__ import annotations
+
+from dagster import graph, in_process_executor, multiprocess_executor
+from dagster_k8s import k8s_job_executor
+
+from distributed_execution.ops import (
+    generate_work_units,
+    process_work_units,
+    report_execution_target,
+    summarise_results,
+)
+
+# Applied to run pods by the K8sRunLauncher; surfaces in the Dagster UI run tags.
+COMMON_TAGS = {
+    "execution_target": "tightly_coupled",
+    "business_operation": "DISTRIBUTED_EXECUTION_REFERENCE",
+}
+
+# Per-step pod shape. Only honoured by k8s_job_executor.
+STEP_K8S_CONFIG = {
+    "container_config": {
+        "resources": {
+            "requests": {"cpu": "100m", "memory": "128Mi"},
+            "limits": {"cpu": "500m", "memory": "512Mi"},
+        },
+    },
+    "pod_spec_config": {
+        "restart_policy": "Never",
+    },
+}
+
+
+@graph
+def distributed_execution_reference():
+    """Shared topology, so the three jobs differ only by execution target."""
+    target_report = report_execution_target()
+    units = generate_work_units()
+    results = process_work_units(units)
+    return summarise_results(results, target_report)
+
+
+# 1. Single process. Steps run inside the run worker itself - no fan-out at all.
+tightly_coupled_in_process_job = distributed_execution_reference.to_job(
+    name="tightly_coupled_in_process_job",
+    description=(
+        "Tightly coupled, single-process. Steps execute inside the run worker. "
+        "Runs unchanged on a laptop and in Kubernetes."
+    ),
+    executor_def=in_process_executor,
+    tags={**COMMON_TAGS, "executor": "in_process"},
+)
+
+# 2. Subprocesses on the run worker. Fan-out bounded by that one pod's resources.
+tightly_coupled_local_job = distributed_execution_reference.to_job(
+    name="tightly_coupled_local_job",
+    description=(
+        "Tightly coupled, multiprocess. Steps execute as subprocesses of the run worker; "
+        "concurrency is bounded by the run pod's CPU and memory limits."
+    ),
+    executor_def=multiprocess_executor.configured({"max_concurrent": 2}),
+    tags={**COMMON_TAGS, "executor": "multiprocess"},
+)
+
+# 3. One Kubernetes Job per step. Requires a cluster; each step pod connects to
+#    the metadata database on its own, which is what makes this tightly coupled.
+tightly_coupled_k8s_job = distributed_execution_reference.to_job(
+    name="tightly_coupled_k8s_job",
+    description=(
+        "Tightly coupled, one Kubernetes Job per step. Each step pod must reach the metadata "
+        "database, object storage and Vault. Requires a cluster - not runnable locally."
+    ),
+    executor_def=k8s_job_executor.configured(
+        {
+            "image_pull_policy": "IfNotPresent",
+            "step_k8s_config": STEP_K8S_CONFIG,
+        }
+    ),
+    tags={**COMMON_TAGS, "executor": "k8s_job"},
+)
+
+TIGHTLY_COUPLED_JOBS = [
+    tightly_coupled_in_process_job,
+    tightly_coupled_local_job,
+    tightly_coupled_k8s_job,
+]

tests/__init__.py

@@ -0,0 +1 @@
+"""Tests for the distributed-execution reference implementations."""

tests/test_loosely_coupled.py

@@ -0,0 +1,86 @@
+"""Verifies the loosely coupled pipes reference end to end via the subprocess transport."""
+
+import pytest
+from dagster import DagsterInvariantViolationError, Failure
+
+from distributed_execution.loosely_coupled.jobs import (
+    LOOSELY_COUPLED_JOBS,
+    PAYLOAD_SCRIPT,
+    loosely_coupled_k8s_job,
+    loosely_coupled_subprocess_job,
+)
+from distributed_execution.repository import defs
+
+
+def test_payload_script_exists():
+    from pathlib import Path
+
+    assert Path(PAYLOAD_SCRIPT).is_file()
+
+
+def test_payload_does_not_import_dagster():
+    from pathlib import Path
+
+    source = Path(PAYLOAD_SCRIPT).read_text(encoding="utf-8")
+    assert "from dagster_pipes import" in source
+    assert "import dagster\n" not in source
+    assert "from dagster import" not in source
+
+
+def test_both_transports_are_registered():
+    names = {job.name for job in defs.jobs}
+    assert {"loosely_coupled_subprocess_job", "loosely_coupled_k8s_job"} <= names
+    assert len(LOOSELY_COUPLED_JOBS) == 2
+
+
+@pytest.mark.parametrize(
+    ("job", "transport"),
+    [
+        (loosely_coupled_subprocess_job, "subprocess"),
+        (loosely_coupled_k8s_job, "k8s_pod_logs"),
+    ],
+)
+def test_jobs_declare_their_execution_target(job, transport):
+    assert job.tags["execution_target"] == "loosely_coupled"
+    assert job.tags["transport"] == transport
+
+
+def test_subprocess_job_runs_end_to_end():
+    result = loosely_coupled_subprocess_job.execute_in_process()
+
+    assert result.success
+    summary = result.output_for_node("summarise_results")
+    # Same shape as the tightly coupled reference: only the middle node changed.
+    assert summary["units"] == 4
+    assert summary["total"] == 14
+
+
+def test_payload_reports_no_orchestration_credentials(monkeypatch):
+    monkeypatch.delenv("DAGSTER_POSTGRES_HOST", raising=False)
+    monkeypatch.delenv("DAGSTER_POSTGRES_USER", raising=False)
+    monkeypatch.delenv("DAGSTER_POSTGRES_DB", raising=False)
+
+    result = loosely_coupled_subprocess_job.execute_in_process()
+    results = result.output_for_node("dispatch_external_work_subprocess")
+
+    assert len(results) == 4
+    assert all(row["host"] for row in results)
+
+
+def test_silent_message_path_is_treated_as_failure():
+    from distributed_execution.loosely_coupled.jobs import _results_from_pipes
+
+    class _Silent:
+        def get_custom_messages(self):
+            return []
+
+    class _Ctx:
+        class log:  # noqa: N801
+            @staticmethod
+            def warning(*_args, **_kwargs): ...
+
+            @staticmethod
+            def info(*_args, **_kwargs): ...
+
+    with pytest.raises(Failure, match="No pipes messages received"):
+        _results_from_pipes(_Ctx(), _Silent())

tests/test_tightly_coupled.py

@@ -0,0 +1,59 @@
+"""Verifies that execution-target linkage is what the guide claims it is."""
+
+import pytest
+
+from distributed_execution.preflight import check_env_vars, describe_pod_identity
+from distributed_execution.repository import defs
+from distributed_execution.tightly_coupled.jobs import (
+    tightly_coupled_in_process_job,
+    tightly_coupled_k8s_job,
+    tightly_coupled_local_job,
+)
+
+EXPECTED_JOBS = {
+    "tightly_coupled_in_process_job",
+    "tightly_coupled_local_job",
+    "tightly_coupled_k8s_job",
+}
+
+
+def test_repository_registers_tightly_coupled_jobs():
+    names = {job.name for job in defs.jobs}
+    assert EXPECTED_JOBS <= names
+
+
+@pytest.mark.parametrize(
+    ("job", "expected_executor"),
+    [
+        (tightly_coupled_in_process_job, "in_process"),
+        (tightly_coupled_local_job, "multiprocess"),
+        (tightly_coupled_k8s_job, "k8s_job"),
+    ],
+)
+def test_each_job_declares_its_execution_target(job, expected_executor):
+    assert job.tags["execution_target"] == "tightly_coupled"
+    assert job.tags["executor"] == expected_executor
+
+
+def test_in_process_job_runs_end_to_end():
+    result = tightly_coupled_in_process_job.execute_in_process()
+
+    assert result.success
+    summary = result.output_for_node("summarise_results")
+    assert summary["units"] == 4
+    assert summary["total"] == 14  # 0 + 1 + 4 + 9
+    # In-process execution never fans out beyond the run worker.
+    assert len(summary["contributing_hosts"]) == 1
+
+
+def test_env_var_check_reports_missing_names():
+    result = check_env_vars(("DEFINITELY_NOT_SET_12345",))
+
+    assert result["passed"] is False
+    assert result["missing"] == ["DEFINITELY_NOT_SET_12345"]
+
+
+def test_pod_identity_falls_back_outside_kubernetes(monkeypatch):
+    monkeypatch.delenv("DAGSTER_K8S_PIPELINE_RUN_NAMESPACE", raising=False)
+
+    assert describe_pod_identity()["namespace"] == "<not-in-kubernetes>"

workspace.yaml

@@ -0,0 +1,3 @@
+load_from:
+  - python_file:
+      relative_path: src/distributed_execution/repository.py

yaml/loosely-coupled/rbac-pipes-dispatch.yaml

@@ -0,0 +1,41 @@
+# Kubernetes RBAC for the loosely coupled execution target.
+#
+# Granted to the service account running the Dagster RUN pods - the pod that
+# calls PipesK8sClient.run() is the one that creates and watches the payload Job.
+#
+# Note what is NOT here: the payload itself needs no RBAC, no database
+# credentials and no Vault role. Its only channel back to the control plane is
+# the pod log stream, which the dispatching pod reads.
+#
+# If the payload runs in a different namespace from the dispatcher, apply this
+# Role in the PAYLOAD namespace and keep the RoleBinding subject pointing at the
+# dispatcher's service account.
+
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: distributed-execution-pipes-dispatcher
+  namespace: dagster
+rules:
+  - apiGroups: ["batch"]
+    resources: ["jobs", "jobs/status"]
+    verbs: ["create", "get", "list", "watch", "delete"]
+  # pods/log is the message channel for PipesK8sPodLogsMessageReader.
+  # Without it the payload runs but reports nothing.
+  - apiGroups: [""]
+    resources: ["pods", "pods/log", "pods/status"]
+    verbs: ["get", "list", "watch"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: distributed-execution-pipes-dispatcher
+  namespace: dagster
+subjects:
+  - kind: ServiceAccount
+    name: dagster-dev
+    namespace: dagster
+roleRef:
+  kind: Role
+  name: distributed-execution-pipes-dispatcher
+  apiGroup: rbac.authorization.k8s.io

yaml/loosely-coupled/values-pipes-payload.yaml

@@ -0,0 +1,32 @@
+# Loosely coupled execution target - code location overrides.
+#
+# Merge into the dagster-user-deployments entry for this code location. These
+# values point the pipes client at the payload image and target namespace.
+#
+# Contrast with yaml/tightly-coupled/values-run-launcher.yaml: there, every
+# execution pod needed S3 and Vault credentials. Here the payload needs none -
+# only the dispatching pod does.
+
+dagster:
+  dagster-user-deployments:
+    deployments:
+      - name: distributed-execution
+        envSecrets: []
+        env:
+          - name: PIPES_PAYLOAD_IMAGE
+            value: code.europa.eu:4567/simpl/simpl-open/data/supporting-data-services/distributed-execution/payload:0.1.0
+          - name: PIPES_PAYLOAD_NAMESPACE
+            value: dagster
+
+  # The dispatching RUN pod creates the payload Job, so the image reference and
+  # namespace must also be visible to run pods, not just the code server.
+  runLauncher:
+    config:
+      k8sRunLauncher:
+        runK8sConfig:
+          containerConfig:
+            env:
+              - name: PIPES_PAYLOAD_IMAGE
+                value: code.europa.eu:4567/simpl/simpl-open/data/supporting-data-services/distributed-execution/payload:0.1.0
+              - name: PIPES_PAYLOAD_NAMESPACE
+                value: dagster

yaml/tightly-coupled/rbac-step-executor.yaml

@@ -0,0 +1,34 @@
+# Kubernetes RBAC required by the k8s_job_executor.
+#
+# The in_process and multiprocess executors need NONE of this - the run worker
+# does all the work itself. Only tightly_coupled_k8s_job, which creates one
+# Kubernetes Job per step, needs permission to manage Jobs and read their pods.
+#
+# Bind to the service account used by the Dagster run pods (dagster.serviceAccount.name).
+
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: distributed-execution-step-runner
+  namespace: dagster
+rules:
+  - apiGroups: ["batch"]
+    resources: ["jobs", "jobs/status"]
+    verbs: ["create", "get", "list", "watch", "delete"]
+  - apiGroups: [""]
+    resources: ["pods", "pods/log", "pods/status"]
+    verbs: ["get", "list", "watch"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: distributed-execution-step-runner
+  namespace: dagster
+subjects:
+  - kind: ServiceAccount
+    name: dagster-dev
+    namespace: dagster
+roleRef:
+  kind: Role
+  name: distributed-execution-step-runner
+  apiGroup: rbac.authorization.k8s.io

yaml/tightly-coupled/values-run-launcher.yaml

@@ -0,0 +1,54 @@
+# Tightly coupled execution target - instance-level run launcher.
+#
+# Merge into the Dagster chart values. This is the half of the execution-target
+# binding that lives OUTSIDE workflow code: it decides where the run worker for
+# an entire run is placed, and what that pod can reach.
+#
+# Companion to src/distributed_execution/tightly_coupled/jobs.py, which decides
+# how the steps INSIDE that run worker execute.
+
+dagster:
+  runLauncher:
+    type: K8sRunLauncher
+    config:
+      k8sRunLauncher:
+        # Namespace the run pods land in. Must be a namespace whose NetworkPolicy
+        # permits egress to Postgres, object storage and Vault - see AC1
+        # prerequisites in the user guide.
+        jobNamespace: dagster
+
+        # Surfaces step failures as pod failures so kubectl and Dagster agree.
+        failPodOnRunFailure: true
+
+        runK8sConfig:
+          podTemplateSpecMetadata:
+            annotations:
+              vault.security.banzaicloud.io/vault-inject: "true"
+              vault.security.banzaicloud.io/vault-addr: https://secrets.common01.dev.simpl-europe.eu
+              vault.security.banzaicloud.io/vault-role: dev-role
+              vault.security.banzaicloud.io/vault-skip-verify: "true"
+              vault.security.banzaicloud.io/vault-path: kubernetes
+
+          containerConfig:
+            resources:
+              requests:
+                cpu: 100m
+                memory: 128Mi
+              limits:
+                cpu: 500m
+                memory: 512Mi
+
+            # Everything below is what "tightly coupled" costs you: each run pod
+            # needs credentials for, and network reachability to, the full
+            # orchestration runtime. A loosely coupled target needs none of it.
+            env:
+              - name: DAGSTER_TELEMETRY_ENABLED
+                value: "false"
+              - name: TOKEN
+                value: "vault:dev/data/dagster/dagster-workflow-vault-secret#VAULT_ACCESS_TOKEN"
+              - name: S3_ENDPOINT_URL
+                value: "https://s3.dev.simpl-europe.eu"
+              - name: S3_ACCESS_KEY
+                value: "vault:dev/data/dev-orchestration-platform#S3_ACCESS_KEY"
+              - name: S3_SECRET_KEY
+                value: "vault:dev/data/dev-orchestration-platform#S3_SECRET_KEY"

yaml/values-dagster-distributed-execution.yaml

@@ -0,0 +1,54 @@
+dagster:
+  # ===========================================================================
+  # CODE LOCATION - distributed-execution
+  # ===========================================================================
+  # Deploys this repository as a Dagster code location so the reference jobs
+  # appear in the UI alongside production workflows.
+  # MANAGED BY GITLAB CI/CD - DO NOT CHANGE MANUALLY
+  dagster-user-deployments:
+    deployments:
+      - name: distributed-execution
+        location_name: distributed-execution
+        enabled: true
+        enableSubchart: true
+
+        codeServerArgs:
+          - "--host"
+          - "0.0.0.0"
+          - "--port"
+          - "4000"
+          - "--python-file"
+          - "src/distributed_execution/repository.py"
+
+        envFrom:
+          - secretRef:
+              name: distributed-execution-dagster-db-secret
+          - configMapRef:
+              name: distributed-execution-dagster-db-config
+
+        image:
+          pullPolicy: IfNotPresent
+          repository: code.europa.eu:4567/simpl/simpl-open/data/supporting-data-services/distributed-execution
+          tag: 0.0.0 # DO NOT CHANGE - replaced automatically by the pipeline before releasing
+
+        port: 4000
+
+        resources:
+          requests:
+            cpu: 100m
+            memory: 256Mi
+          limits:
+            cpu: 500m
+            memory: 512Mi
+
+        livenessProbe:
+          enabled: true
+          periodSeconds: 30
+          timeoutSeconds: 5
+          failureThreshold: 3
+
+        readinessProbe:
+          enabled: true
+          periodSeconds: 10
+          timeoutSeconds: 3
+          failureThreshold: 3