distributed-execution/documents/user-guide/distributed-execution-guide.md

# Distributed Execution: Execution Targets and Integration Patterns

**Audience:** Participants with permission to configure workflows.
**Scope:** How to select, document and configure the execution target and
integration pattern for a Dagster workflow on the Simpl orchestration platform.

> **Document status.** Both setups are backed by runnable reference
> implementations and verified end to end. The loosely coupled Kubernetes path
> reached `RUN_SUCCESS` on sandbox-cat-dat on 2026-08-31; the tightly coupled
> `k8s_job_executor` path reached `RUN_SUCCESS` through the platform launcher on
> 2026-09-01. See the readiness checklist for the recorded evidence.

---

## 1. Concepts

### 1.1 What an execution target is

The **execution target** is where the compute for a workflow actually runs. It is
not a single Dagster setting — it is the combination of three decisions.

| # | Decision | Where it is configured | Scope |
|---|---|---|---|
| 1 | **Placement** — which cluster and namespace the run lands in | Run launcher, in the Dagster instance / Helm values | Per deployment |
| 2 | **Granularity** — one process for the whole run, or one per step | `executor_def`, in workflow code | Per job |
| 3 | **Integration pattern** — whether the compute participates in Dagster's internals or reports back over a message channel | Job topology in workflow code | Per job |

Decisions 1 and 2 are always present. Decision 3 is what this guide calls the
*integration pattern*, and it is the one with the largest operational
consequences, because it determines what the execution pod must be able to reach.

### 1.2 The two integration patterns

**Tightly coupled.** The process doing the work *is* a Dagster process. It imports
the code location, opens a connection to the metadata database and writes run
events directly. Runtime information reaches the control plane because the worker
*is* part of the control plane's data path.

**Loosely coupled.** Dagster only dispatches. It creates a workload on an external
cluster running an arbitrary image, then acts as a listener. The external process
writes structured messages over the dagster-pipes protocol; Dagster reads them and
materialises them as logs and metadata. The execution pod never connects to the
metadata database.

```mermaid
flowchart LR
    subgraph TC["Tightly coupled"]
        direction TB
        D1[Dagster daemon] -->|K8sRunLauncher| RW1[Run worker pod]
        RW1 -->|k8s_job_executor| SP1[Step pods]
        RW1 -.->|events| DB1[(Metadata DB)]
        SP1 -.->|events| DB1
    end
    subgraph LC["Loosely coupled"]
        direction TB
        D2[Dagster daemon] -->|K8sRunLauncher| RW2[Run worker pod]
        RW2 -->|PipesK8sClient| EXT[External workload<br/>any image]
        EXT -.->|pipes messages| RW2
        RW2 -.->|events| DB2[(Metadata DB)]
    end
```

Note the asymmetry in the diagram: in the loosely coupled pattern only the run
worker touches the metadata database. That single arrow is the whole point of the
pattern.

---

## 2. Choosing an execution target and cluster setup

### 2.1 Choose tightly coupled when

- The workflow logic is written in Python and lives in a Dagster code location.
- Execution pods can be granted network egress to the metadata database, object
  storage and Vault.
- You want per-step retries, step-level concurrency limits and asset
  materialisation handled natively by Dagster.
- The compute belongs to the same trust boundary as the orchestration platform.

### 2.2 Choose loosely coupled when

- The workload is not Python, or cannot take a dependency on the `dagster`
  package (different runtime, vendor image, GPU base image).
- The target cluster is administratively separate and **must not** be granted
  connectivity to the orchestration metadata database.
- The compute belongs to a different trust boundary, and you want the blast
  radius of a compromised execution pod limited to the message channel.
- The workload already exists as a container and you want to orchestrate it
  without rewriting it.

### 2.3 Prerequisites

| Prerequisite | Tightly coupled | Loosely coupled |
|---|---|---|
| Run launcher configured | Required — `K8sRunLauncher` | Required — `K8sRunLauncher` |
| Code location image contains workflow code | Required | Required for the *launching* job only |
| Execution pod → metadata database (Postgres 5432) | **Required** | Not required |
| Execution pod → object storage | Required if the workflow reads/writes objects | Required only if the payload itself does |
| Execution pod → Vault | Required for secret injection | Only for the launching pod |
| Kubernetes RBAC to create Jobs | Required for `k8s_job_executor` only | **Required** in the target namespace |
| Message channel provisioned | Not applicable | **Required** — pod log stream or object-storage reader |
| Control-plane reporting | Automatic — worker writes events directly | Via pipes messages; **fails silently if the channel breaks** |

### 2.4 What to document before the first execution

Record the following at workflow level, in the repository holding the workflow,
before it is run for the first time. Section 5.4 explains where each item lives.

1. **Chosen integration pattern** — tightly or loosely coupled, with a one-line
   rationale referencing section 2.1 or 2.2.
2. **Executor and its configuration** — including `max_concurrent` where relevant.
3. **Target namespace** for run pods and, for `k8s_job_executor`, step pods.
4. **Runtime dependencies** the execution pods must reach, as host/port pairs.
   This is the input to the NetworkPolicy review.
5. **Secrets consumed**, by Vault path — not by value.
6. **Expected evidence of a healthy run** — which of the checks in the
   [readiness checklist](readiness-checklist.md) apply, and what a passing run
   looks like in the UI.
7. **For loosely coupled only:** the payload image reference and the message
   channel type.

---

## 3. Decision support and cluster-configuration inputs

### 3.1 Side-by-side comparison

| Dimension | Tightly coupled | Loosely coupled |
|---|---|---|
| What runs the work | A Dagster process importing your code location | Any container image |
| Language constraint | Python, matching the code location's environment | None |
| Dependency on `dagster` package | Full package in the execution image | Only `dagster-pipes` in the payload image |
| Metadata DB connectivity from execution pod | **Required** | **Not required** |
| Runtime information flow | Direct writes to run storage | Structured messages → run worker → run storage |
| Log capture | Native, per step | Via the pipes message stream |
| Step-level retries | Native | Manual, at the dispatching op |
| Asset materialisation | Native, from any step | Reported by the payload, materialised by the launching op |
| Failure granularity | Per step | Per dispatched workload |
| Secret distribution | To every execution pod | To the launching pod, plus whatever the payload needs |
| Blast radius of a compromised execution pod | Orchestration runtime credentials | Message channel only |
| Cross-cluster execution | Not supported | Supported |

### 3.2 Environment-fit indicators

Use these as a fast triage. If several point the same way, that is your answer.

| Observation about your environment | Points to |
|---|---|
| Workflow is pure Python and shares a repository with existing code locations | Tightly coupled |
| Steps are short and numerous, and you want native retries | Tightly coupled |
| NetworkPolicy review for the execution namespace is already approved | Tightly coupled |
| Compute must run in a cluster you do not administer | Loosely coupled |
| Security review objects to distributing DB credentials to execution pods | Loosely coupled |
| Payload is an existing vendor or third-party image | Loosely coupled |
| Workload needs GPUs or a base image incompatible with the code location image | Loosely coupled |
| Team owning the payload is different from the team owning the workflow | Loosely coupled |

### 3.3 Trade-offs

**Tightly coupled**

- *Gains:* the richest observability with no extra work — every step's logs,
  metadata, retries and asset materialisations are native. Simplest mental model.
- *Costs:* the widest connectivity surface. Every execution pod is a credential
  holder and a database client. Scaling step pods scales database connections,
  which is the usual first bottleneck. Execution image and code location image are
  coupled, so a dependency conflict in one workflow can affect others sharing the
  location.

**Loosely coupled**

- *Gains:* a narrow connectivity surface and genuine dependency isolation. The
  payload team can ship on its own cadence. Cross-cluster and cross-trust-boundary
  execution become possible.
- *Costs:* observability is only as good as the payload's instrumentation — an
  uninstrumented payload appears in Dagster as one opaque step. Retry and
  cancellation semantics must be implemented at the dispatching op. The message
  channel becomes a new failure mode, and when it breaks the run can appear to
  succeed while reporting nothing. Two images to build, version and scan instead
  of one.

### 3.4 Cluster-configuration inputs

Once the setup is chosen, the container cluster has to be described. The table
below is the full input surface, split into what a run cannot start without and
what only applies under a stated condition. "Mandatory" means the run fails, or
reports nothing, if the input is absent or wrong.

| Input | Tightly coupled | Loosely coupled | Notes |
|---|---|---|---|
| Cluster / kube context hosting the run worker | Mandatory | Mandatory | The launcher's cluster. Loosely coupled may dispatch to a *second* cluster |
| Namespace for run pods | Mandatory | Mandatory | Supplied by the code location's `DAGSTER_CONTAINER_CONTEXT`; the launcher's `job_namespace` is only a fallback |
| Code location image reference and tag | Mandatory | Mandatory | Never `latest`; check C3 |
| Executor selection | Mandatory | Mandatory | Bounds step granularity; loosely coupled it bounds only dispatch concurrency |
| Service account for run pods | Mandatory | Mandatory | Must have `automountServiceAccountToken: true` |
| Run-pod egress to the metadata database | **Mandatory** | Not applicable | The pod that writes run events |
| Namespace for step pods | Conditional — `k8s_job_executor` only | Not applicable | Defaults to the run pod's namespace |
| RBAC to create and watch Jobs | Conditional — `k8s_job_executor` only | **Mandatory**, in the payload namespace | Same verbs, different namespace |
| Shared RWX volume plus an I/O manager `base_dir` under it | Conditional — `k8s_job_executor` only | Optional | Without it step pods cannot read each other's outputs |
| Per-step resource requests and limits | Optional | Optional | Multiply by step or dispatch concurrency |
| Payload image reference and tag | Not applicable | **Mandatory** | Must share the code location's commit tag |
| Namespace for payload workloads | Not applicable | **Mandatory** | May be in another cluster or trust boundary |
| Message channel and its reader | Not applicable | **Mandatory** | Pod log stream by default; RBAC on `pods/log` *is* the channel |
| Payload wait timeout | Not applicable | Optional | Defaults to 24 h — lower it so scheduling failures surface |
| Vault role and secret paths | Conditional — if the workflow reads secrets | Conditional — launching pod only | Payloads should receive only `extras` and explicit `env` |
| Object-storage endpoint and credentials | Conditional — if the workflow uses S3 | Conditional — if the *payload* uses S3 | |
| Image pull secret | Conditional — non-anonymous registry | Conditional — non-anonymous registry | Applies to the SA that runs the pod, which for dispatched payload Jobs is `default` |

### 3.5 Minimum input set

The list below is the reusable baseline: the smallest set of values a team has to
agree on before a workflow can be configured against a cluster. Record it once
per workflow, in the workflow's own repository, and carry it into whatever
configuration format the team already uses.

**Both setups**

1. `cluster` — the cluster the run worker lands in.
2. `run_namespace` — namespace for run pods.
3. `code_location_image` — image reference including an immutable tag.
4. `executor` — executor name plus its concurrency bound.
5. `service_account` — the identity run pods assume.

**Tightly coupled adds**

6. `metadata_db` — host and port the run and step pods must reach.
7. `step_namespace` — only when `k8s_job_executor` is used.
8. `shared_output_path` — RWX mount path, and the I/O manager `base_dir` pinned
   under it, again only for `k8s_job_executor`.

**Loosely coupled adds**

6. `payload_image` — image reference sharing the code location's commit tag.
7. `payload_namespace` — where dispatched workloads are created; may be in
   another cluster.
8. `message_channel` — the reader in use, and the RBAC verb that carries it.

This guide deliberately does **not** prescribe a configuration schema or key
names. The names above are descriptive labels for the values, not a required
format; the examples in section 3.6 express the same baseline in two different
shapes, and both are correct.

### 3.6 Worked input examples

The reference configuration in [yaml/](../../yaml) shows the baseline expressed
in the formats each setup already uses. Neither is normative.

**Tightly coupled** — the inputs land in Helm values and in the job's executor
config, because that is where the platform chart and Dagster expect them:

```yaml
# yaml/tightly-coupled/values-run-launcher.yaml  (shape, not a fixed schema)
dagster:
  runLauncher:
    type: K8sRunLauncher
    config:
      k8sRunLauncher:
        jobNamespace: dagster              # run_namespace
        failPodOnRunFailure: true
        runK8sConfig:
          containerConfig:
            env:                           # metadata_db, S3 and Vault reachability
              - name: S3_ENDPOINT_URL
                value: https://s3.dev.simpl-europe.eu
```

```python
# step_namespace, shared_output_path and the executor travel with the job
tightly_coupled_k8s_job = distributed_execution_reference.to_job(
    executor_def=k8s_job_executor,
    resource_defs={"io_manager": fs_io_manager.configured({"base_dir": SHARED_IO_BASE_DIR})},
    config={"execution": {"config": {"step_k8s_config": STEP_K8S_CONFIG}}},
)
```

**Loosely coupled** — the same baseline, but the cluster-facing half is the
payload image and its namespace. Here they are environment values set on both the
code location and the run launcher, because the dispatching pod is a run pod:

```yaml
# yaml/loosely-coupled/values-pipes-payload.yaml  (shape, not a fixed schema)
dagster:
  dagster-user-deployments:
    deployments:
      - name: distributed-execution
        env:
          - name: PIPES_PAYLOAD_IMAGE                # payload_image, tag-locked
            value: <registry>/distributed-execution/payload:0.0.0
          - name: PIPES_PAYLOAD_NAMESPACE            # payload_namespace
            value: dagster
```

```python
PAYLOAD_IMAGE = os.environ.get("PIPES_PAYLOAD_IMAGE", ...)
PAYLOAD_NAMESPACE = os.environ.get("PIPES_PAYLOAD_NAMESPACE", "dagster")
```

The `message_channel` input has no key in either file. The default reader is the
pod log stream, so it is configured by granting `pods/log` in
[yaml/loosely-coupled/rbac-pipes-dispatch.yaml](../../yaml/loosely-coupled/rbac-pipes-dispatch.yaml).
An input can be mandatory and still not be a configuration field.

Note what is absent from the loosely coupled example: no metadata database, no
Vault role, no object-storage credentials. Those inputs are not optional there —
they do not exist, which is the operational consequence of the choice.

---

## 4. Readiness checks

Pre-run validation items, their expected evidence, and the symptoms and
corrections for common misconfigurations are maintained separately, in
[readiness-checklist.md](readiness-checklist.md).

---

## 5. Configuring execution-target linkage in code

### 5.1 Configuration constructs

Five constructs bind a workflow to an execution target. Three live in code and
travel with the workflow; two live in deployment configuration.

| Construct | Lives in | Binds |
|---|---|---|
| `executor_def` on `@job` / `to_job()` | Workflow code | Step granularity and placement |
| `tags` on `@job` | Workflow code | Run-level metadata; `dagster-k8s/config` overrides pod shape |
| Job topology — direct ops vs. a dispatching op using a pipes client | Workflow code | Integration pattern |
| `runLauncher` | Helm values | Run worker placement and reachable dependencies |
| `dagster-user-deployments` entry | Helm values | Which image the code location runs |

The first three are the answer to "which configuration constructs bind the
execution target to a workflow" — and they are all versioned alongside the
workflow, which is why setup selection is a code review concern and not a runtime
toggle.

### 5.2 Worked example — switching within the tightly coupled setup

The reference jobs share one graph and differ only by `executor_def`. This is the
smallest possible demonstration that granularity is a code-level decision.

**Before — steps run as subprocesses inside the run worker:**

```python
tightly_coupled_local_job = distributed_execution_reference.to_job(
    name="tightly_coupled_local_job",
    executor_def=multiprocess_executor.configured({"max_concurrent": 2}),
    tags={**COMMON_TAGS, "executor": "multiprocess"},
)
```

**After — one Kubernetes Job per step:**

```python
tightly_coupled_k8s_job = distributed_execution_reference.to_job(
    name="tightly_coupled_k8s_job",
    executor_def=k8s_job_executor,
    resource_defs={
        "io_manager": fs_io_manager.configured({"base_dir": SHARED_IO_BASE_DIR})
    },
    config={
        "execution": {
            "config": {
                "image_pull_policy": "IfNotPresent",
                "step_k8s_config": STEP_K8S_CONFIG,
            }
        }
    },
    tags={**COMMON_TAGS, "executor": "k8s_job"},
)
```

The graph, the ops and the run launcher are all unchanged. What changes
operationally:

- Step pods are now created in the launcher's `jobNamespace`, so that namespace's
  NetworkPolicy must permit egress to Postgres, object storage and Vault — not
  just the run worker's namespace.
- The service account needs RBAC to create and watch Jobs. See
  [yaml/tightly-coupled/rbac-step-executor.yaml](../../yaml/tightly-coupled/rbac-step-executor.yaml).
- Per-step resource requests apply per pod, so the aggregate request for a
  fan-out step is the per-step request multiplied by concurrency.
- **Step outputs must land on storage every step pod can read.** With the default
  filesystem I/O manager each pod writes to its own container filesystem, so a
  downstream step opens a path that does not exist there. `STEP_K8S_CONFIG` mounts
  the RWX `dagster-shared-pvc` at `/dagster/shared`, and the job pins
  `fs_io_manager`'s `base_dir` underneath it.
- Pin `base_dir` directly rather than relying on `DAGSTER_HOME`. The default
  `base_dir` is `$DAGSTER_HOME/storage`, but the Dagster chart already injects
  `DAGSTER_HOME=/tmp/dagster` into the step container; a second entry appended by
  `step_k8s_config` does not displace it, so the override is silently ignored and
  outputs keep going to `/tmp/dagster/storage`.

The executor's defaults go through `config=` rather than
`k8s_job_executor.configured(...)`. `.configured()` collapses the executor's
config schema to `Any`, which makes a run-config `execution:` block accepted and
then discarded, and makes `job_image` and `job_namespace` impossible to supply at
launch time.

The evidence that the switch took effect is in `summarise_results` output
metadata, and the two fields say different things:

| Executor | `contributing_workers` | `contributing_hosts` |
|---|---|---|
| `in_process_executor` | 1 | 1 |
| `multiprocess_executor` | one per unit | 1 — same machine |
| `k8s_job_executor` | one per unit | one per unit — separate pods |

This only works because `generate_work_units` is a `DynamicOut` and the graph
does `units.map(process_work_unit).collect()`. A single op looping over all units
internally would report one worker under *every* executor, because one step
cannot span processes or pods.

### 5.3 Worked example — switching to loosely coupled

The graphs for the two patterns differ by **exactly one node**. This is not a
simplification for the guide; it is how the reference implementations are
written, so the diff below is the real one.

**Before — tightly coupled. The work happens in a Dagster process:**

```python
@graph
def distributed_execution_reference():
    target_report = report_execution_target()
    units = generate_work_units()
    results = units.map(process_work_unit).collect()          # runs in Dagster
    return summarise_results(results, target_report)
```

**After — loosely coupled. The work is dispatched to an external image:**

```python
@graph
def loosely_coupled_k8s_reference():
    target_report = report_execution_target()
    units = generate_work_units()
    results = units.map(dispatch_external_work_k8s).collect() # dispatches, then listens
    return summarise_results(results, target_report)
```

The dispatching op replaces direct computation with a pipes client call:

```python
@op(out=Out(dict))
def dispatch_external_work_k8s(
    context: OpExecutionContext,
    unit: int,
    pipes_k8s_client: PipesK8sClient,
) -> dict:
    completed = pipes_k8s_client.run(
        context=context,
        image=PAYLOAD_IMAGE,
        command=["python", "/app/work.py"],
        namespace=PAYLOAD_NAMESPACE,
        extras={"units": [unit]},
    )
    return _result_from_pipes(context, completed)
```

and the job supplies the client as a resource:

```python
loosely_coupled_k8s_job = loosely_coupled_k8s_reference.to_job(
    name="loosely_coupled_k8s_job",
    executor_def=multiprocess_executor.configured({"max_concurrent": 2}),
    resource_defs={"pipes_k8s_client": PipesK8sClient()},
    tags={**COMMON_TAGS, "transport": "k8s_pod_logs"},
)
```

The executor is still present, but its meaning has changed. It no longer decides
where the work runs — the payload image and `namespace` do that. It only bounds
how many dispatching ops wait on the message channel at once. The two decisions
that were fused in the tightly coupled setup are now independent.

#### What changes operationally

| | Tightly coupled | Loosely coupled |
|---|---|---|
| Images to build | 1 (code location) | 2 (code location + payload) |
| Payload dependency surface | Full `dagster` package | `dagster-pipes` only |
| Execution pod → Postgres | Egress rule **required** | Egress rule can be **removed** |
| Execution pod → Vault / S3 | Credentials required | None needed by the payload |
| RBAC | Only for `k8s_job_executor` | Required on the **dispatching** pod |
| New failure mode | — | Broken message path |

The NetworkPolicy rules you can *remove* are the point of the exercise. See
[yaml/loosely-coupled/rbac-pipes-dispatch.yaml](../../yaml/loosely-coupled/rbac-pipes-dispatch.yaml)
for what replaces them — note that `pods/log` is the message channel, so
removing that verb silently breaks reporting without failing the workload.

#### The payload contract

[payload/work.py](../../payload/work.py) is the whole contract:

```python
with open_dagster_pipes() as pipes:
    units = pipes.get_extra("units")
    pipes.log.info(f"External payload started on {host}")
    pipes.report_custom_message({"results": results, "host": host, ...})
```

`pipes.log` entries appear in the Dagster run logs; `report_custom_message` is
read back by the dispatching op. The payload imports `dagster_pipes` and nothing
else from the Dagster ecosystem — a test asserts this, because the moment the
payload imports `dagster` the isolation argument for choosing this pattern
collapses.

Because the graph maps over the dynamic output, one external workload is
dispatched **per unit**. A test asserts that the four units come back from four
distinct external workers, which is the loosely coupled equivalent of the
`contributing_workers` evidence in section 5.2.

#### Message channel choice

`PipesK8sClient` defaults to `PipesK8sPodLogsMessageReader`, which is what the
reference uses. It needs no extra infrastructure, but it couples message
delivery to the pod log stream — a log shipper that intercepts or truncates
stdout will break reporting while the workload still exits 0. That is why
`_result_from_pipes` raises on an empty message list rather than returning an
empty result. If your cluster's logging setup makes that transport unreliable,
switch to an object-storage message reader, which needs a bucket plus
credentials on both sides.

> **CLUSTER-VERIFIED 2026-08-31.** Run `cff9b348…` of `loosely_coupled_k8s_job`
> reached `RUN_SUCCESS` on sandbox-cat-dat. Four payload Jobs were created, one
> per work unit, and `contributing_hosts` held four distinct payload pod names —
> so the work demonstrably ran off the orchestrator. The payload's log lines
> reached the run log, which is the pod log stream serving as the message
> channel exactly as described above.

### 5.4 Where setup selection is persisted

Setup selection is persisted in configuration artifacts, never as a runtime-only
setting. There is no UI control that switches a workflow between the two patterns.

| Artifact | Holds |
|---|---|
| `src/<package>/**/jobs.py` | `executor_def`, pipes client resource, tags, job topology |
| `payload/work.py` + `payload/Dockerfile` | The external payload contract and its image |
| `yaml/tightly-coupled/values-run-launcher.yaml` | Run launcher type, namespace, pod env |
| `yaml/tightly-coupled/rbac-step-executor.yaml` | Permissions required by `k8s_job_executor` |
| `yaml/loosely-coupled/values-pipes-payload.yaml` | Payload image reference and target namespace |
| `yaml/sandbox/values-sandbox-gitea.yaml` | Sandbox registry coordinates, pull secret, and the SHA both images share |
| `yaml/loosely-coupled/rbac-pipes-dispatch.yaml` | Permissions required by the dispatching pod |
| `yaml/loosely-coupled/probe-pipes-k8s.yaml` | Standalone cluster probe for the pipes transport (checklist L4–L9) |
| `yaml/sandbox/probe-pipes-k8s-sandbox.yaml` | The same probe for sandbox-cat-dat, where project-scoped access forbids creating namespaces and RBAC |
| `yaml/values-dagster-distributed-execution.yaml` | Code location image and entry point |

The Launchpad can override *run configuration* — op config, resource config, tags
— for a single run. It cannot change the executor or the integration pattern.
Those take effect only when a new image is built and the code location reloads.

### 5.5 Reference implementations in this repository

| Job | Executor / transport | Runs locally | Demonstrates |
|---|---|---|---|
| `tightly_coupled_in_process_job` | `in_process_executor` | Yes | Baseline; steps inside the run worker |
| `tightly_coupled_local_job` | `multiprocess_executor` | Linux/macOS | Subprocess fan-out bounded by the run pod |
| `tightly_coupled_k8s_job` | `k8s_job_executor` | No — needs a cluster | One Kubernetes Job per step |
| `loosely_coupled_subprocess_job` | `PipesSubprocessClient` | Yes | The pipes contract end to end, no cluster |
| `loosely_coupled_k8s_job` | `PipesK8sClient` | No — needs a cluster | External Job dispatch, messages over pod logs |

The last two dispatch the same [payload/work.py](../../payload/work.py). The
subprocess variant exists so the pipes contract can be exercised, tested and
demonstrated without any infrastructure.

> **Windows note.** `tightly_coupled_local_job` did not complete on a Windows
> development machine during authoring: the `multiprocess_executor` spawned step
> subprocesses that never terminated, and required manual cleanup. This was not
> reproduced on Linux and is not expected to affect cluster deployments, where
> run workers are Linux pods. For a laptop demonstration on Windows, prefer
> `tightly_coupled_in_process_job` or `loosely_coupled_subprocess_job`, both of
> which are covered by the test suite.

Run the local variants with:

```bash
uv sync --dev
uv run dagster dev -f src/distributed_execution/repository.py
```

`report_execution_target` emits pod identity and the visible orchestration
environment variables as output metadata on every run, which is the evidence the
readiness checklist refers to.

### 5.6 What a verified run actually produced

The two locally runnable jobs were launched from the Dagster UI, so the evidence
below comes from the full daemon → run launcher → executor path rather than from
a unit test. Hostnames are redacted; process IDs are real.

`tightly_coupled_in_process_job`:

```text
succeeded steps: 7
  generate_work_units, report_execution_target,
  process_work_unit[unit_0..unit_3], summarise_results

units                = 4
contributing_hosts   = ['dev-host']
contributing_workers = ['dev-host#25264']
```

`loosely_coupled_subprocess_job`:

```text
succeeded steps: 7
  generate_work_units, report_execution_target,
  dispatch_external_work_subprocess[unit_0..unit_3], summarise_results

units                = 4
contributing_hosts   = ['dev-host']
contributing_workers = ['dev-host#13532', 'dev-host#27116',
                        'dev-host#32984', 'dev-host#31224']
```

Read the two blocks together. The step lists are the same shape, which is the
point of section 5.3 — the graphs differ by one node. `contributing_workers` is
what separates them: one entry means the work happened inside the Dagster
process; four means four external processes did it and reported back over the
pipes channel. `contributing_hosts` stays at one because a laptop is one machine;
on a cluster it is what distinguishes `k8s_job_executor` from
`multiprocess_executor`.

`launcher_namespace` reads `<not-in-kubernetes>` in both. In a cluster run it
holds the namespace the run pod landed in, which is the placement half of the
execution target.

### 5.7 Dagster UI reference

The screenshots below were captured from the local reference service on
2026-09-02. They show the surfaces participants use to inspect job linkage and
run evidence. Cluster placement is evidenced separately by the sandbox run IDs
in the readiness checklist.

All five reference jobs are registered under one code location:

![Distributed execution job list](images/distributed-execution-job-list.png)

The job overview exposes the graph whose executor or dispatch op defines the
integration pattern:

![Tightly coupled reference graph](images/distributed-execution-job-graph.png)

The Launchpad shows that `execution_target` and `executor` are job tags persisted
by code configuration, not a separate runtime-only selector:

![Execution-target tags in Launchpad](images/distributed-execution-launchpad.png)

A successful local run displays the step events and final status used alongside
output metadata as retained evidence:

![Successful tightly coupled reference run](images/distributed-execution-successful-run.png)

---

## 6. Upstream documentation

- Dagster — [run launchers](https://docs.dagster.io/deployment/execution/run-launchers)
- Dagster — [executors](https://docs.dagster.io/deployment/execution/executors)
- Dagster — [customizing your Kubernetes deployment](https://docs.dagster.io/deployment/oss/deployment-options/kubernetes/customizing-your-deployment)
- Dagster — [Dagster Pipes](https://docs.dagster.io/guides/build/external-pipelines)
- Dagster — [Dagster Pipes with Kubernetes](https://docs.dagster.io/guides/build/external-pipelines/kubernetes-pipeline)
- Kubernetes — [Jobs](https://kubernetes.io/docs/concepts/workloads/controllers/job/)
- Kubernetes — [network policies](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
- Kubernetes — [RBAC authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/)

## 7. Related Simpl documentation

- [dagster/documents/user-manual/Orchestration.md](../../../dagster/documents/user-manual/Orchestration.md) — workflow development lifecycle
- [dagster/documents/installation-guide/Installation Guide.md](../../../dagster/documents/installation-guide/Installation%20Guide.md) — full run launcher parameter reference
- [dagster/documents/user-manual/Observability.md](../../../dagster/documents/user-manual/Observability.md) — logging and monitoring

## 8. Outstanding work

Tracked under SIMPL-30451 and SIMPL-30787.

| Item | Status |
|---|---|
| Execution-target choices, prerequisites, workflow-level documentation | Complete |
| Comparison, environment-fit indicators, trade-offs | Complete |
| Cluster-configuration inputs, mandatory vs optional, minimum input set | Complete |
| Readiness checklist and evidence mapping | Complete; both Kubernetes paths cluster-verified |
| Misconfiguration symptoms and corrections | Complete; corrected with observed cluster failure modes |
| Configuration constructs and persistence | Complete |
| Before-and-after example, tightly coupled | Complete |
| Before-and-after example, switching to loosely coupled | Complete |
| Runnable tightly coupled reference implementations | Complete |
| Runnable loosely coupled reference implementation | Complete (subprocess and Kubernetes transports both verified) |
| Both images build; payload image passes the isolation check (L11) | Complete |
| Locally runnable jobs launched from the Dagster UI, evidence recorded in section 5.6 | Complete |
| Payload image published to a container registry | Complete on the sandbox Gitea registry; the GitLab registry still pending |
| GitLab pipeline builds both images | **Pending** — the shared `ds.gitlab-ci.yml` template builds one image from the root Dockerfile |
| End-to-end run of `loosely_coupled_k8s_job` on a cluster | Complete — sandbox-cat-dat, 2026-08-31, run `cff9b348…`; see the readiness checklist |
| `distributed-execution` registered as a code location on a platform Dagster | Complete — tightly coupled run `1d8cb167…` launched through the platform webserver |
| Screenshots of UI surfaces | Complete — job list, graph, Launchpad linkage and successful run captured in section 5.7 |
| Platform architecture document update | Complete — root `deployment_diagram.md` includes the service and both runtime-information paths |

---

## 9. Requirement traceability

The sections above are organised by subject, not by ticket. Two stories share
this material; the mapping below is authoritative for both.

### SIMPL-30787 — defining the execution container cluster

| AC | Requirement | Where it is met |
|---|---|---|
| AC1 | Prerequisites for both setups | Section 2.3 |
| AC1 | What must be documented before execution | Section 2.4 |
| AC1 | Where cluster-runtime connectivity is required, where decoupled reporting is expected | Sections 1.2 and 2.3; the asymmetry is drawn in the section 1.2 diagram |
| AC2 | Mandatory vs optional inputs per setup | Section 3.4 |
| AC2 | Examples for both setups, no fixed config naming model | Section 3.6, with the closing note in 3.5 |
| AC2 | Reusable minimum input set | Section 3.5 |
| AC3 | Checklist separates tightly from loosely coupled checks | [readiness-checklist.md](readiness-checklist.md) sections 2 and 3; section 1 holds the checks common to both |
| AC3 | Each item mapped to expected evidence | The *Expected evidence* column of every checklist table |
| AC4 | Configuration constructs attaching the pattern to a workflow | Section 5.1 |
| AC4 | Worked before-and-after example across setups | Sections 5.2 and 5.3 |
| AC4 | At least one runnable reference implementation per setup | Section 5.5 — three tightly coupled, two loosely coupled |
| AC4 | Selection persisted in configuration artifacts, not runtime-only | Section 5.4 |
| Tech | Diagrams, screenshots, working config files, upstream links | Sections 1.2, 5.7, [yaml/](../../yaml), section 6 |

### SIMPL-30451 — execution targets and integration patterns

| AC | Subject | Where it is met |
|---|---|---|
| AC1 | Choosing an execution target | Section 2 |
| AC2 | Decision support | Sections 3.1 to 3.3 |
| AC3 | Readiness checks | Section 4 and the readiness checklist |
| AC4 | Linkage configured in code | Section 5 |