distributed-execution/documents/user-guide/distributed-execution-guide.md

# Distributed Execution: Execution Targets and Integration Patterns

**Audience:** Participants with permission to configure workflows.
**Scope:** How to select, document and configure the execution target and
integration pattern for a Dagster workflow on the Simpl orchestration platform.

> **Document status.** Both setups are backed by runnable reference
> implementations in this repository. The tightly coupled setup and the loosely
> coupled **subprocess** transport are verified end to end, including runs
> launched from the Dagster UI — see [section 5.6](#56-what-a-verified-run-actually-produced).
> The loosely coupled **Kubernetes** transport is implemented and reviewed but
> has not yet been run against a cluster; statements specific to it are marked
> **NOT CLUSTER-VERIFIED**. See [Outstanding work](#8-outstanding-work).

---

## 1. Concepts

### 1.1 What an execution target is

The **execution target** is where the compute for a workflow actually runs. It is
not a single Dagster setting — it is the combination of three decisions.

| # | Decision | Where it is configured | Scope |
|---|---|---|---|
| 1 | **Placement** — which cluster and namespace the run lands in | Run launcher, in the Dagster instance / Helm values | Per deployment |
| 2 | **Granularity** — one process for the whole run, or one per step | `executor_def`, in workflow code | Per job |
| 3 | **Integration pattern** — whether the compute participates in Dagster's internals or reports back over a message channel | Job topology in workflow code | Per job |

Decisions 1 and 2 are always present. Decision 3 is what this guide calls the
*integration pattern*, and it is the one with the largest operational
consequences, because it determines what the execution pod must be able to reach.

### 1.2 The two integration patterns

**Tightly coupled.** The process doing the work *is* a Dagster process. It imports
the code location, opens a connection to the metadata database and writes run
events directly. Runtime information reaches the control plane because the worker
*is* part of the control plane's data path.

**Loosely coupled.** Dagster only dispatches. It creates a workload on an external
cluster running an arbitrary image, then acts as a listener. The external process
writes structured messages over the dagster-pipes protocol; Dagster reads them and
materialises them as logs and metadata. The execution pod never connects to the
metadata database.

```mermaid
flowchart LR
    subgraph TC["Tightly coupled"]
        direction TB
        D1[Dagster daemon] -->|K8sRunLauncher| RW1[Run worker pod]
        RW1 -->|k8s_job_executor| SP1[Step pods]
        RW1 -.->|events| DB1[(Metadata DB)]
        SP1 -.->|events| DB1
    end
    subgraph LC["Loosely coupled"]
        direction TB
        D2[Dagster daemon] -->|K8sRunLauncher| RW2[Run worker pod]
        RW2 -->|PipesK8sClient| EXT[External workload<br/>any image]
        EXT -.->|pipes messages| RW2
        RW2 -.->|events| DB2[(Metadata DB)]
    end
```

Note the asymmetry in the diagram: in the loosely coupled pattern only the run
worker touches the metadata database. That single arrow is the whole point of the
pattern.

---

## 2. AC1 — Choosing an execution target

### 2.1 Choose tightly coupled when

- The workflow logic is written in Python and lives in a Dagster code location.
- Execution pods can be granted network egress to the metadata database, object
  storage and Vault.
- You want per-step retries, step-level concurrency limits and asset
  materialisation handled natively by Dagster.
- The compute belongs to the same trust boundary as the orchestration platform.

### 2.2 Choose loosely coupled when

- The workload is not Python, or cannot take a dependency on the `dagster`
  package (different runtime, vendor image, GPU base image).
- The target cluster is administratively separate and **must not** be granted
  connectivity to the orchestration metadata database.
- The compute belongs to a different trust boundary, and you want the blast
  radius of a compromised execution pod limited to the message channel.
- The workload already exists as a container and you want to orchestrate it
  without rewriting it.

### 2.3 Prerequisites

| Prerequisite | Tightly coupled | Loosely coupled |
|---|---|---|
| Run launcher configured | Required — `K8sRunLauncher` | Required — `K8sRunLauncher` |
| Code location image contains workflow code | Required | Required for the *launching* job only |
| Execution pod → metadata database (Postgres 5432) | **Required** | Not required |
| Execution pod → object storage | Required if the workflow reads/writes objects | Required only if the payload itself does |
| Execution pod → Vault | Required for secret injection | Only for the launching pod |
| Kubernetes RBAC to create Jobs | Required for `k8s_job_executor` only | **Required** in the target namespace |
| Message channel provisioned | Not applicable | **Required** — pod log stream or object-storage reader |
| Control-plane reporting | Automatic — worker writes events directly | Via pipes messages; **fails silently if the channel breaks** |

### 2.4 What to document before the first execution

Record the following at workflow level, in the repository holding the workflow,
before it is run for the first time. Section 5.4 explains where each item lives.

1. **Chosen integration pattern** — tightly or loosely coupled, with a one-line
   rationale referencing section 2.1 or 2.2.
2. **Executor and its configuration** — including `max_concurrent` where relevant.
3. **Target namespace** for run pods and, for `k8s_job_executor`, step pods.
4. **Runtime dependencies** the execution pods must reach, as host/port pairs.
   This is the input to the NetworkPolicy review.
5. **Secrets consumed**, by Vault path — not by value.
6. **Expected evidence of a healthy run** — which of the checks in the
   [readiness checklist](readiness-checklist.md) apply, and what a passing run
   looks like in the UI.
7. **For loosely coupled only:** the payload image reference and the message
   channel type.

---

## 3. AC2 — Decision support

### 3.1 Side-by-side comparison

| Dimension | Tightly coupled | Loosely coupled |
|---|---|---|
| What runs the work | A Dagster process importing your code location | Any container image |
| Language constraint | Python, matching the code location's environment | None |
| Dependency on `dagster` package | Full package in the execution image | Only `dagster-pipes` in the payload image |
| Metadata DB connectivity from execution pod | **Required** | **Not required** |
| Runtime information flow | Direct writes to run storage | Structured messages → run worker → run storage |
| Log capture | Native, per step | Via the pipes message stream |
| Step-level retries | Native | Manual, at the dispatching op |
| Asset materialisation | Native, from any step | Reported by the payload, materialised by the launching op |
| Failure granularity | Per step | Per dispatched workload |
| Secret distribution | To every execution pod | To the launching pod, plus whatever the payload needs |
| Blast radius of a compromised execution pod | Orchestration runtime credentials | Message channel only |
| Cross-cluster execution | Not supported | Supported |

### 3.2 Environment-fit indicators

Use these as a fast triage. If several point the same way, that is your answer.

| Observation about your environment | Points to |
|---|---|
| Workflow is pure Python and shares a repository with existing code locations | Tightly coupled |
| Steps are short and numerous, and you want native retries | Tightly coupled |
| NetworkPolicy review for the execution namespace is already approved | Tightly coupled |
| Compute must run in a cluster you do not administer | Loosely coupled |
| Security review objects to distributing DB credentials to execution pods | Loosely coupled |
| Payload is an existing vendor or third-party image | Loosely coupled |
| Workload needs GPUs or a base image incompatible with the code location image | Loosely coupled |
| Team owning the payload is different from the team owning the workflow | Loosely coupled |

### 3.3 Trade-offs

**Tightly coupled**

- *Gains:* the richest observability with no extra work — every step's logs,
  metadata, retries and asset materialisations are native. Simplest mental model.
- *Costs:* the widest connectivity surface. Every execution pod is a credential
  holder and a database client. Scaling step pods scales database connections,
  which is the usual first bottleneck. Execution image and code location image are
  coupled, so a dependency conflict in one workflow can affect others sharing the
  location.

**Loosely coupled**

- *Gains:* a narrow connectivity surface and genuine dependency isolation. The
  payload team can ship on its own cadence. Cross-cluster and cross-trust-boundary
  execution become possible.
- *Costs:* observability is only as good as the payload's instrumentation — an
  uninstrumented payload appears in Dagster as one opaque step. Retry and
  cancellation semantics must be implemented at the dispatching op. The message
  channel becomes a new failure mode, and when it breaks the run can appear to
  succeed while reporting nothing. Two images to build, version and scan instead
  of one.

---

## 4. AC3 — Readiness checks

Pre-run validation items, their expected evidence, and the symptoms and
corrections for common misconfigurations are maintained separately, in
[readiness-checklist.md](readiness-checklist.md).

---

## 5. AC4 — Configuring execution-target linkage in code

### 5.1 Configuration constructs

Five constructs bind a workflow to an execution target. Three live in code and
travel with the workflow; two live in deployment configuration.

| Construct | Lives in | Binds |
|---|---|---|
| `executor_def` on `@job` / `to_job()` | Workflow code | Step granularity and placement |
| `tags` on `@job` | Workflow code | Run-level metadata; `dagster-k8s/config` overrides pod shape |
| Job topology — direct ops vs. a dispatching op using a pipes client | Workflow code | Integration pattern |
| `runLauncher` | Helm values | Run worker placement and reachable dependencies |
| `dagster-user-deployments` entry | Helm values | Which image the code location runs |

The first three are the answer to "which configuration constructs bind the
execution target to a workflow" — and they are all versioned alongside the
workflow, which is why setup selection is a code review concern and not a runtime
toggle.

### 5.2 Worked example — switching within the tightly coupled setup

The reference jobs share one graph and differ only by `executor_def`. This is the
smallest possible demonstration that granularity is a code-level decision.

**Before — steps run as subprocesses inside the run worker:**

```python
tightly_coupled_local_job = distributed_execution_reference.to_job(
    name="tightly_coupled_local_job",
    executor_def=multiprocess_executor.configured({"max_concurrent": 2}),
    tags={**COMMON_TAGS, "executor": "multiprocess"},
)
```

**After — one Kubernetes Job per step:**

```python
tightly_coupled_k8s_job = distributed_execution_reference.to_job(
    name="tightly_coupled_k8s_job",
    executor_def=k8s_job_executor.configured(
        {
            "image_pull_policy": "IfNotPresent",
            "step_k8s_config": STEP_K8S_CONFIG,
        }
    ),
    tags={**COMMON_TAGS, "executor": "k8s_job"},
)
```

The graph, the ops and the run launcher are all unchanged. What changes
operationally:

- Step pods are now created in the launcher's `jobNamespace`, so that namespace's
  NetworkPolicy must permit egress to Postgres, object storage and Vault — not
  just the run worker's namespace.
- The service account needs RBAC to create and watch Jobs. See
  [yaml/tightly-coupled/rbac-step-executor.yaml](../../yaml/tightly-coupled/rbac-step-executor.yaml).
- Per-step resource requests apply per pod, so the aggregate request for a
  fan-out step is the per-step request multiplied by concurrency.

The evidence that the switch took effect is in `summarise_results` output
metadata, and the two fields say different things:

| Executor | `contributing_workers` | `contributing_hosts` |
|---|---|---|
| `in_process_executor` | 1 | 1 |
| `multiprocess_executor` | one per unit | 1 — same machine |
| `k8s_job_executor` | one per unit | one per unit — separate pods |

This only works because `generate_work_units` is a `DynamicOut` and the graph
does `units.map(process_work_unit).collect()`. A single op looping over all units
internally would report one worker under *every* executor, because one step
cannot span processes or pods.

### 5.3 Worked example — switching to loosely coupled

The graphs for the two patterns differ by **exactly one node**. This is not a
simplification for the guide; it is how the reference implementations are
written, so the diff below is the real one.

**Before — tightly coupled. The work happens in a Dagster process:**

```python
@graph
def distributed_execution_reference():
    target_report = report_execution_target()
    units = generate_work_units()
    results = units.map(process_work_unit).collect()          # runs in Dagster
    return summarise_results(results, target_report)
```

**After — loosely coupled. The work is dispatched to an external image:**

```python
@graph
def loosely_coupled_k8s_reference():
    target_report = report_execution_target()
    units = generate_work_units()
    results = units.map(dispatch_external_work_k8s).collect() # dispatches, then listens
    return summarise_results(results, target_report)
```

The dispatching op replaces direct computation with a pipes client call:

```python
@op(out=Out(dict))
def dispatch_external_work_k8s(
    context: OpExecutionContext,
    unit: int,
    pipes_k8s_client: PipesK8sClient,
) -> dict:
    completed = pipes_k8s_client.run(
        context=context,
        image=PAYLOAD_IMAGE,
        command=["python", "/app/work.py"],
        namespace=PAYLOAD_NAMESPACE,
        extras={"units": [unit]},
    )
    return _result_from_pipes(context, completed)
```

and the job supplies the client as a resource:

```python
loosely_coupled_k8s_job = loosely_coupled_k8s_reference.to_job(
    name="loosely_coupled_k8s_job",
    executor_def=multiprocess_executor.configured({"max_concurrent": 2}),
    resource_defs={"pipes_k8s_client": PipesK8sClient()},
    tags={**COMMON_TAGS, "transport": "k8s_pod_logs"},
)
```

The executor is still present, but its meaning has changed. It no longer decides
where the work runs — the payload image and `namespace` do that. It only bounds
how many dispatching ops wait on the message channel at once. The two decisions
that were fused in the tightly coupled setup are now independent.

#### What changes operationally

| | Tightly coupled | Loosely coupled |
|---|---|---|
| Images to build | 1 (code location) | 2 (code location + payload) |
| Payload dependency surface | Full `dagster` package | `dagster-pipes` only |
| Execution pod → Postgres | Egress rule **required** | Egress rule can be **removed** |
| Execution pod → Vault / S3 | Credentials required | None needed by the payload |
| RBAC | Only for `k8s_job_executor` | Required on the **dispatching** pod |
| New failure mode | — | Broken message path |

The NetworkPolicy rules you can *remove* are the point of the exercise. See
[yaml/loosely-coupled/rbac-pipes-dispatch.yaml](../../yaml/loosely-coupled/rbac-pipes-dispatch.yaml)
for what replaces them — note that `pods/log` is the message channel, so
removing that verb silently breaks reporting without failing the workload.

#### The payload contract

[payload/work.py](../../payload/work.py) is the whole contract:

```python
with open_dagster_pipes() as pipes:
    units = pipes.get_extra("units")
    pipes.log.info(f"External payload started on {host}")
    pipes.report_custom_message({"results": results, "host": host, ...})
```

`pipes.log` entries appear in the Dagster run logs; `report_custom_message` is
read back by the dispatching op. The payload imports `dagster_pipes` and nothing
else from the Dagster ecosystem — a test asserts this, because the moment the
payload imports `dagster` the isolation argument for choosing this pattern
collapses.

Because the graph maps over the dynamic output, one external workload is
dispatched **per unit**. A test asserts that the four units come back from four
distinct external workers, which is the loosely coupled equivalent of the
`contributing_workers` evidence in section 5.2.

#### Message channel choice

`PipesK8sClient` defaults to `PipesK8sPodLogsMessageReader`, which is what the
reference uses. It needs no extra infrastructure, but it couples message
delivery to the pod log stream — a log shipper that intercepts or truncates
stdout will break reporting while the workload still exits 0. That is why
`_result_from_pipes` raises on an empty message list rather than returning an
empty result. If your cluster's logging setup makes that transport unreliable,
switch to an object-storage message reader, which needs a bucket plus
credentials on both sides.

> **NOT CLUSTER-VERIFIED.** The Kubernetes transport above is implemented and
> its RBAC and configuration are recorded, but it has not yet been run against a
> Simpl cluster. The subprocess transport, which exercises the identical payload
> and the identical message-parsing path, is verified by the test suite.

### 5.4 Where setup selection is persisted

Setup selection is persisted in configuration artifacts, never as a runtime-only
setting. There is no UI control that switches a workflow between the two patterns.

| Artifact | Holds |
|---|---|
| `src/<package>/**/jobs.py` | `executor_def`, pipes client resource, tags, job topology |
| `payload/work.py` + `payload/Dockerfile` | The external payload contract and its image |
| `yaml/tightly-coupled/values-run-launcher.yaml` | Run launcher type, namespace, pod env |
| `yaml/tightly-coupled/rbac-step-executor.yaml` | Permissions required by `k8s_job_executor` |
| `yaml/loosely-coupled/values-pipes-payload.yaml` | Payload image reference and target namespace |
| `yaml/loosely-coupled/rbac-pipes-dispatch.yaml` | Permissions required by the dispatching pod |
| `yaml/values-dagster-distributed-execution.yaml` | Code location image and entry point |

The Launchpad can override *run configuration* — op config, resource config, tags
— for a single run. It cannot change the executor or the integration pattern.
Those take effect only when a new image is built and the code location reloads.

### 5.5 Reference implementations in this repository

| Job | Executor / transport | Runs locally | Demonstrates |
|---|---|---|---|
| `tightly_coupled_in_process_job` | `in_process_executor` | Yes | Baseline; steps inside the run worker |
| `tightly_coupled_local_job` | `multiprocess_executor` | Linux/macOS | Subprocess fan-out bounded by the run pod |
| `tightly_coupled_k8s_job` | `k8s_job_executor` | No — needs a cluster | One Kubernetes Job per step |
| `loosely_coupled_subprocess_job` | `PipesSubprocessClient` | Yes | The pipes contract end to end, no cluster |
| `loosely_coupled_k8s_job` | `PipesK8sClient` | No — needs a cluster | External Job dispatch, messages over pod logs |

The last two dispatch the same [payload/work.py](../../payload/work.py). The
subprocess variant exists so the pipes contract can be exercised, tested and
demonstrated without any infrastructure.

> **Windows note.** `tightly_coupled_local_job` did not complete on a Windows
> development machine during authoring: the `multiprocess_executor` spawned step
> subprocesses that never terminated, and required manual cleanup. This was not
> reproduced on Linux and is not expected to affect cluster deployments, where
> run workers are Linux pods. For a laptop demonstration on Windows, prefer
> `tightly_coupled_in_process_job` or `loosely_coupled_subprocess_job`, both of
> which are covered by the test suite.

Run the local variants with:

```bash
uv sync --dev
uv run dagster dev -f src/distributed_execution/repository.py
```

`report_execution_target` emits pod identity and the visible orchestration
environment variables as output metadata on every run, which is the evidence the
readiness checklist refers to.

### 5.6 What a verified run actually produced

The two locally runnable jobs were launched from the Dagster UI, so the evidence
below comes from the full daemon → run launcher → executor path rather than from
a unit test. Hostnames are redacted; process IDs are real.

`tightly_coupled_in_process_job`:

```text
succeeded steps: 7
  generate_work_units, report_execution_target,
  process_work_unit[unit_0..unit_3], summarise_results

units                = 4
contributing_hosts   = ['dev-host']
contributing_workers = ['dev-host#25264']
```

`loosely_coupled_subprocess_job`:

```text
succeeded steps: 7
  generate_work_units, report_execution_target,
  dispatch_external_work_subprocess[unit_0..unit_3], summarise_results

units                = 4
contributing_hosts   = ['dev-host']
contributing_workers = ['dev-host#13532', 'dev-host#27116',
                        'dev-host#32984', 'dev-host#31224']
```

Read the two blocks together. The step lists are the same shape, which is the
point of section 5.3 — the graphs differ by one node. `contributing_workers` is
what separates them: one entry means the work happened inside the Dagster
process; four means four external processes did it and reported back over the
pipes channel. `contributing_hosts` stays at one because a laptop is one machine;
on a cluster it is what distinguishes `k8s_job_executor` from
`multiprocess_executor`.

`launcher_namespace` reads `<not-in-kubernetes>` in both. In a cluster run it
holds the namespace the run pod landed in, which is the placement half of the
execution target.

---

## 6. Upstream documentation

- Dagster — [run launchers](https://docs.dagster.io/deployment/execution/run-launchers)
- Dagster — [executors](https://docs.dagster.io/deployment/execution/executors)
- Dagster — [customizing your Kubernetes deployment](https://docs.dagster.io/deployment/oss/deployment-options/kubernetes/customizing-your-deployment)
- Dagster — [Dagster Pipes](https://docs.dagster.io/guides/build/external-pipelines)
- Dagster — [Dagster Pipes with Kubernetes](https://docs.dagster.io/guides/build/external-pipelines/kubernetes-pipeline)
- Kubernetes — [Jobs](https://kubernetes.io/docs/concepts/workloads/controllers/job/)
- Kubernetes — [network policies](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
- Kubernetes — [RBAC authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/)

## 7. Related Simpl documentation

- [dagster/documents/user-manual/Orchestration.md](../../../dagster/documents/user-manual/Orchestration.md) — workflow development lifecycle
- [dagster/documents/installation-guide/Installation Guide.md](../../../dagster/documents/installation-guide/Installation%20Guide.md) — full run launcher parameter reference
- [dagster/documents/user-manual/Observability.md](../../../dagster/documents/user-manual/Observability.md) — logging and monitoring

## 8. Outstanding work

Tracked under SIMPL-30451.

| Item | AC | Status |
|---|---|---|
| Execution-target choices, prerequisites, workflow-level documentation | AC1 | Complete |
| Comparison, environment-fit indicators, trade-offs | AC2 | Complete |
| Readiness checklist and evidence mapping | AC3 | Complete; K8s pipes checks not cluster-verified |
| Misconfiguration symptoms and corrections | AC3 | Complete; K8s pipes symptoms not cluster-verified |
| Configuration constructs and persistence | AC4 | Complete |
| Before-and-after example, tightly coupled | AC4 | Complete |
| Before-and-after example, switching to loosely coupled | AC4 | Complete |
| Runnable tightly coupled reference implementations | AC4 | Complete |
| Runnable loosely coupled reference implementation | AC4 | Complete (subprocess verified, K8s not cluster-run) |
| Both images build; payload image passes the isolation check (L11) | Tech details | Complete |
| Locally runnable jobs launched from the Dagster UI, evidence recorded in section 5.6 | Tech details | Complete |
| Payload image published to the container registry | Tech details | **Pending** — needs the CI build job |
| End-to-end run of `loosely_coupled_k8s_job` on a cluster | Tech details | **Pending** |
| Screenshots of UI surfaces | Tech details | **Pending** — needs a deployed platform instance, not a local dev server |
| Platform architecture document update | Tech details | **Pending** |