distributed-execution/documents/user-guide/distributed-execution-guide.md

Distributed Execution: Execution Targets and Integration Patterns

Audience: Participants with permission to configure workflows. Scope: How to select, document and configure the execution target and integration pattern for a Dagster workflow on the Simpl orchestration platform.

Document status. Both setups are backed by runnable reference implementations in this repository. The tightly coupled setup and the loosely coupled subprocess transport are verified end to end, including runs launched from the Dagster UI — see section 5.6. The loosely coupled Kubernetes transport is implemented and reviewed but has not yet been run against a cluster; statements specific to it are marked NOT CLUSTER-VERIFIED. See Outstanding work.

---

1. Concepts

1.1 What an execution target is

The execution target is where the compute for a workflow actually runs. It is not a single Dagster setting — it is the combination of three decisions.

#	Decision	Where it is configured	Scope
1	Placement — which cluster and namespace the run lands in	Run launcher, in the Dagster instance / Helm values	Per deployment
2	Granularity — one process for the whole run, or one per step	executor_def, in workflow code	Per job
3	Integration pattern — whether the compute participates in Dagster's internals or reports back over a message channel	Job topology in workflow code	Per job

Decisions 1 and 2 are always present. Decision 3 is what this guide calls the integration pattern, and it is the one with the largest operational consequences, because it determines what the execution pod must be able to reach.

1.2 The two integration patterns

Tightly coupled. The process doing the work is a Dagster process. It imports the code location, opens a connection to the metadata database and writes run events directly. Runtime information reaches the control plane because the worker is part of the control plane's data path.

Loosely coupled. Dagster only dispatches. It creates a workload on an external cluster running an arbitrary image, then acts as a listener. The external process writes structured messages over the dagster-pipes protocol; Dagster reads them and materialises them as logs and metadata. The execution pod never connects to the metadata database.

flowchart LR
    subgraph TC["Tightly coupled"]
        direction TB
        D1[Dagster daemon] -->|K8sRunLauncher| RW1[Run worker pod]
        RW1 -->|k8s_job_executor| SP1[Step pods]
        RW1 -.->|events| DB1[(Metadata DB)]
        SP1 -.->|events| DB1
    end
    subgraph LC["Loosely coupled"]
        direction TB
        D2[Dagster daemon] -->|K8sRunLauncher| RW2[Run worker pod]
        RW2 -->|PipesK8sClient| EXT[External workload<br/>any image]
        EXT -.->|pipes messages| RW2
        RW2 -.->|events| DB2[(Metadata DB)]
    end

Note the asymmetry in the diagram: in the loosely coupled pattern only the run worker touches the metadata database. That single arrow is the whole point of the pattern.

---

2. AC1 — Choosing an execution target

2.1 Choose tightly coupled when

• The workflow logic is written in Python and lives in a Dagster code location.
• Execution pods can be granted network egress to the metadata database, object storage and Vault.
• You want per-step retries, step-level concurrency limits and asset materialisation handled natively by Dagster.
• The compute belongs to the same trust boundary as the orchestration platform.

2.2 Choose loosely coupled when

• The workload is not Python, or cannot take a dependency on the dagster package (different runtime, vendor image, GPU base image).
• The target cluster is administratively separate and must not be granted connectivity to the orchestration metadata database.
• The compute belongs to a different trust boundary, and you want the blast radius of a compromised execution pod limited to the message channel.
• The workload already exists as a container and you want to orchestrate it without rewriting it.

2.3 Prerequisites

Prerequisite	Tightly coupled	Loosely coupled
Run launcher configured	Required — K8sRunLauncher	Required — K8sRunLauncher
Code location image contains workflow code	Required	Required for the launching job only
Execution pod → metadata database (Postgres 5432)	Required	Not required
Execution pod → object storage	Required if the workflow reads/writes objects	Required only if the payload itself does
Execution pod → Vault	Required for secret injection	Only for the launching pod
Kubernetes RBAC to create Jobs	Required for k8s_job_executor only	Required in the target namespace
Message channel provisioned	Not applicable	Required — pod log stream or object-storage reader
Control-plane reporting	Automatic — worker writes events directly	Via pipes messages; fails silently if the channel breaks

2.4 What to document before the first execution

Record the following at workflow level, in the repository holding the workflow, before it is run for the first time. Section 5.4 explains where each item lives.

1. Chosen integration pattern — tightly or loosely coupled, with a one-line rationale referencing section 2.1 or 2.2.
2. Executor and its configuration — including max_concurrent where relevant.
3. Target namespace for run pods and, for k8s_job_executor, step pods.
4. Runtime dependencies the execution pods must reach, as host/port pairs. This is the input to the NetworkPolicy review.
5. Secrets consumed, by Vault path — not by value.
6. Expected evidence of a healthy run — which of the checks in the readiness checklist apply, and what a passing run looks like in the UI.
7. For loosely coupled only: the payload image reference and the message channel type.

---

3. AC2 — Decision support

3.1 Side-by-side comparison

Dimension	Tightly coupled	Loosely coupled
What runs the work	A Dagster process importing your code location	Any container image
Language constraint	Python, matching the code location's environment	None
Dependency on dagster package	Full package in the execution image	Only dagster-pipes in the payload image
Metadata DB connectivity from execution pod	Required	Not required
Runtime information flow	Direct writes to run storage	Structured messages → run worker → run storage
Log capture	Native, per step	Via the pipes message stream
Step-level retries	Native	Manual, at the dispatching op
Asset materialisation	Native, from any step	Reported by the payload, materialised by the launching op
Failure granularity	Per step	Per dispatched workload
Secret distribution	To every execution pod	To the launching pod, plus whatever the payload needs
Blast radius of a compromised execution pod	Orchestration runtime credentials	Message channel only
Cross-cluster execution	Not supported	Supported

3.2 Environment-fit indicators

Use these as a fast triage. If several point the same way, that is your answer.

Observation about your environment	Points to
Workflow is pure Python and shares a repository with existing code locations	Tightly coupled
Steps are short and numerous, and you want native retries	Tightly coupled
NetworkPolicy review for the execution namespace is already approved	Tightly coupled
Compute must run in a cluster you do not administer	Loosely coupled
Security review objects to distributing DB credentials to execution pods	Loosely coupled
Payload is an existing vendor or third-party image	Loosely coupled
Workload needs GPUs or a base image incompatible with the code location image	Loosely coupled
Team owning the payload is different from the team owning the workflow	Loosely coupled

3.3 Trade-offs

Tightly coupled

• Gains: the richest observability with no extra work — every step's logs, metadata, retries and asset materialisations are native. Simplest mental model.
• Costs: the widest connectivity surface. Every execution pod is a credential holder and a database client. Scaling step pods scales database connections, which is the usual first bottleneck. Execution image and code location image are coupled, so a dependency conflict in one workflow can affect others sharing the location.

Loosely coupled

• Gains: a narrow connectivity surface and genuine dependency isolation. The payload team can ship on its own cadence. Cross-cluster and cross-trust-boundary execution become possible.
• Costs: observability is only as good as the payload's instrumentation — an uninstrumented payload appears in Dagster as one opaque step. Retry and cancellation semantics must be implemented at the dispatching op. The message channel becomes a new failure mode, and when it breaks the run can appear to succeed while reporting nothing. Two images to build, version and scan instead of one.

---

4. AC3 — Readiness checks

Pre-run validation items, their expected evidence, and the symptoms and corrections for common misconfigurations are maintained separately, in readiness-checklist.md.

---

5. AC4 — Configuring execution-target linkage in code

5.1 Configuration constructs

Five constructs bind a workflow to an execution target. Three live in code and travel with the workflow; two live in deployment configuration.

Construct	Lives in	Binds
executor_def on @job / to_job()	Workflow code	Step granularity and placement
tags on @job	Workflow code	Run-level metadata; dagster-k8s/config overrides pod shape
Job topology — direct ops vs. a dispatching op using a pipes client	Workflow code	Integration pattern
runLauncher	Helm values	Run worker placement and reachable dependencies
dagster-user-deployments entry	Helm values	Which image the code location runs

The first three are the answer to "which configuration constructs bind the execution target to a workflow" — and they are all versioned alongside the workflow, which is why setup selection is a code review concern and not a runtime toggle.

5.2 Worked example — switching within the tightly coupled setup

The reference jobs share one graph and differ only by executor_def. This is the smallest possible demonstration that granularity is a code-level decision.

Before — steps run as subprocesses inside the run worker:

tightly_coupled_local_job = distributed_execution_reference.to_job(
    name="tightly_coupled_local_job",
    executor_def=multiprocess_executor.configured({"max_concurrent": 2}),
    tags={**COMMON_TAGS, "executor": "multiprocess"},
)

After — one Kubernetes Job per step:

tightly_coupled_k8s_job = distributed_execution_reference.to_job(
    name="tightly_coupled_k8s_job",
    executor_def=k8s_job_executor.configured(
        {
            "image_pull_policy": "IfNotPresent",
            "step_k8s_config": STEP_K8S_CONFIG,
        }
    ),
    tags={**COMMON_TAGS, "executor": "k8s_job"},
)

The graph, the ops and the run launcher are all unchanged. What changes operationally:

• Step pods are now created in the launcher's jobNamespace, so that namespace's NetworkPolicy must permit egress to Postgres, object storage and Vault — not just the run worker's namespace.
• The service account needs RBAC to create and watch Jobs. See yaml/tightly-coupled/rbac-step-executor.yaml.
• Per-step resource requests apply per pod, so the aggregate request for a fan-out step is the per-step request multiplied by concurrency.

The evidence that the switch took effect is in summarise_results output metadata, and the two fields say different things:

Executor	contributing_workers	contributing_hosts
in_process_executor	1	1
multiprocess_executor	one per unit	1 — same machine
k8s_job_executor	one per unit	one per unit — separate pods

This only works because generate_work_units is a DynamicOut and the graph does units.map(process_work_unit).collect(). A single op looping over all units internally would report one worker under every executor, because one step cannot span processes or pods.

5.3 Worked example — switching to loosely coupled

The graphs for the two patterns differ by exactly one node. This is not a simplification for the guide; it is how the reference implementations are written, so the diff below is the real one.

Before — tightly coupled. The work happens in a Dagster process:

@graph
def distributed_execution_reference():
    target_report = report_execution_target()
    units = generate_work_units()
    results = units.map(process_work_unit).collect()          # runs in Dagster
    return summarise_results(results, target_report)

After — loosely coupled. The work is dispatched to an external image:

@graph
def loosely_coupled_k8s_reference():
    target_report = report_execution_target()
    units = generate_work_units()
    results = units.map(dispatch_external_work_k8s).collect() # dispatches, then listens
    return summarise_results(results, target_report)

The dispatching op replaces direct computation with a pipes client call:

@op(out=Out(dict))
def dispatch_external_work_k8s(
    context: OpExecutionContext,
    unit: int,
    pipes_k8s_client: PipesK8sClient,
) -> dict:
    completed = pipes_k8s_client.run(
        context=context,
        image=PAYLOAD_IMAGE,
        command=["python", "/app/work.py"],
        namespace=PAYLOAD_NAMESPACE,
        extras={"units": [unit]},
    )
    return _result_from_pipes(context, completed)

and the job supplies the client as a resource:

loosely_coupled_k8s_job = loosely_coupled_k8s_reference.to_job(
    name="loosely_coupled_k8s_job",
    executor_def=multiprocess_executor.configured({"max_concurrent": 2}),
    resource_defs={"pipes_k8s_client": PipesK8sClient()},
    tags={**COMMON_TAGS, "transport": "k8s_pod_logs"},
)

The executor is still present, but its meaning has changed. It no longer decides where the work runs — the payload image and namespace do that. It only bounds how many dispatching ops wait on the message channel at once. The two decisions that were fused in the tightly coupled setup are now independent.

What changes operationally

	Tightly coupled	Loosely coupled
Images to build	1 (code location)	2 (code location + payload)
Payload dependency surface	Full dagster package	dagster-pipes only
Execution pod → Postgres	Egress rule required	Egress rule can be removed
Execution pod → Vault / S3	Credentials required	None needed by the payload
RBAC	Only for k8s_job_executor	Required on the dispatching pod
New failure mode	—	Broken message path

The NetworkPolicy rules you can remove are the point of the exercise. See yaml/loosely-coupled/rbac-pipes-dispatch.yaml for what replaces them — note that pods/log is the message channel, so removing that verb silently breaks reporting without failing the workload.

The payload contract

payload/work.py is the whole contract:

with open_dagster_pipes() as pipes:
    units = pipes.get_extra("units")
    pipes.log.info(f"External payload started on {host}")
    pipes.report_custom_message({"results": results, "host": host, ...})

pipes.log entries appear in the Dagster run logs; report_custom_message is read back by the dispatching op. The payload imports dagster_pipes and nothing else from the Dagster ecosystem — a test asserts this, because the moment the payload imports dagster the isolation argument for choosing this pattern collapses.

Because the graph maps over the dynamic output, one external workload is dispatched per unit. A test asserts that the four units come back from four distinct external workers, which is the loosely coupled equivalent of the contributing_workers evidence in section 5.2.

Message channel choice

PipesK8sClient defaults to PipesK8sPodLogsMessageReader, which is what the reference uses. It needs no extra infrastructure, but it couples message delivery to the pod log stream — a log shipper that intercepts or truncates stdout will break reporting while the workload still exits 0. That is why _result_from_pipes raises on an empty message list rather than returning an empty result. If your cluster's logging setup makes that transport unreliable, switch to an object-storage message reader, which needs a bucket plus credentials on both sides.

NOT CLUSTER-VERIFIED. The Kubernetes transport above is implemented and its RBAC and configuration are recorded, but it has not yet been run against a Simpl cluster. The subprocess transport, which exercises the identical payload and the identical message-parsing path, is verified by the test suite.

5.4 Where setup selection is persisted

Setup selection is persisted in configuration artifacts, never as a runtime-only setting. There is no UI control that switches a workflow between the two patterns.

Artifact	Holds
src/<package>/**/jobs.py	executor_def, pipes client resource, tags, job topology
payload/work.py + payload/Dockerfile	The external payload contract and its image
yaml/tightly-coupled/values-run-launcher.yaml	Run launcher type, namespace, pod env
yaml/tightly-coupled/rbac-step-executor.yaml	Permissions required by k8s_job_executor
yaml/loosely-coupled/values-pipes-payload.yaml	Payload image reference and target namespace
yaml/loosely-coupled/rbac-pipes-dispatch.yaml	Permissions required by the dispatching pod
yaml/values-dagster-distributed-execution.yaml	Code location image and entry point

The Launchpad can override run configuration — op config, resource config, tags — for a single run. It cannot change the executor or the integration pattern. Those take effect only when a new image is built and the code location reloads.

5.5 Reference implementations in this repository

Job	Executor / transport	Runs locally	Demonstrates
tightly_coupled_in_process_job	in_process_executor	Yes	Baseline; steps inside the run worker
tightly_coupled_local_job	multiprocess_executor	Linux/macOS	Subprocess fan-out bounded by the run pod
tightly_coupled_k8s_job	k8s_job_executor	No — needs a cluster	One Kubernetes Job per step
loosely_coupled_subprocess_job	PipesSubprocessClient	Yes	The pipes contract end to end, no cluster
loosely_coupled_k8s_job	PipesK8sClient	No — needs a cluster	External Job dispatch, messages over pod logs

The last two dispatch the same payload/work.py. The subprocess variant exists so the pipes contract can be exercised, tested and demonstrated without any infrastructure.

Windows note. tightly_coupled_local_job did not complete on a Windows development machine during authoring: the multiprocess_executor spawned step subprocesses that never terminated, and required manual cleanup. This was not reproduced on Linux and is not expected to affect cluster deployments, where run workers are Linux pods. For a laptop demonstration on Windows, prefer tightly_coupled_in_process_job or loosely_coupled_subprocess_job, both of which are covered by the test suite.

Run the local variants with:

uv sync --dev
uv run dagster dev -f src/distributed_execution/repository.py

report_execution_target emits pod identity and the visible orchestration environment variables as output metadata on every run, which is the evidence the readiness checklist refers to.

5.6 What a verified run actually produced

The two locally runnable jobs were launched from the Dagster UI, so the evidence below comes from the full daemon → run launcher → executor path rather than from a unit test. Hostnames are redacted; process IDs are real.

tightly_coupled_in_process_job:

succeeded steps: 7
  generate_work_units, report_execution_target,
  process_work_unit[unit_0..unit_3], summarise_results

units                = 4
contributing_hosts   = ['dev-host']
contributing_workers = ['dev-host#25264']

loosely_coupled_subprocess_job:

succeeded steps: 7
  generate_work_units, report_execution_target,
  dispatch_external_work_subprocess[unit_0..unit_3], summarise_results

units                = 4
contributing_hosts   = ['dev-host']
contributing_workers = ['dev-host#13532', 'dev-host#27116',
                        'dev-host#32984', 'dev-host#31224']

Read the two blocks together. The step lists are the same shape, which is the point of section 5.3 — the graphs differ by one node. contributing_workers is what separates them: one entry means the work happened inside the Dagster process; four means four external processes did it and reported back over the pipes channel. contributing_hosts stays at one because a laptop is one machine; on a cluster it is what distinguishes k8s_job_executor from multiprocess_executor.

launcher_namespace reads <not-in-kubernetes> in both. In a cluster run it holds the namespace the run pod landed in, which is the placement half of the execution target.

---

6. Upstream documentation

• Dagster — run launchers
• Dagster — executors
• Dagster — customizing your Kubernetes deployment
• Dagster — Dagster Pipes
• Dagster — Dagster Pipes with Kubernetes
• Kubernetes — Jobs
• Kubernetes — network policies
• Kubernetes — RBAC authorization

7. Related Simpl documentation

• dagster/documents/user-manual/Orchestration.md — workflow development lifecycle
• dagster/documents/installation-guide/Installation Guide.md — full run launcher parameter reference
• dagster/documents/user-manual/Observability.md — logging and monitoring

8. Outstanding work

Tracked under SIMPL-30451.

Item	AC	Status
Execution-target choices, prerequisites, workflow-level documentation	AC1	Complete
Comparison, environment-fit indicators, trade-offs	AC2	Complete
Readiness checklist and evidence mapping	AC3	Complete; K8s pipes checks not cluster-verified
Misconfiguration symptoms and corrections	AC3	Complete; K8s pipes symptoms not cluster-verified
Configuration constructs and persistence	AC4	Complete
Before-and-after example, tightly coupled	AC4	Complete
Before-and-after example, switching to loosely coupled	AC4	Complete
Runnable tightly coupled reference implementations	AC4	Complete
Runnable loosely coupled reference implementation	AC4	Complete (subprocess verified, K8s not cluster-run)
Both images build; payload image passes the isolation check (L11)	Tech details	Complete
Locally runnable jobs launched from the Dagster UI, evidence recorded in section 5.6	Tech details	Complete
Payload image published to a container registry	Tech details	Complete on the sandbox Gitea registry; the GitLab registry still pending
End-to-end run of loosely_coupled_k8s_job on a cluster	Tech details	Pending
Screenshots of UI surfaces	Tech details	Pending — needs a deployed platform instance, not a local dev server
Platform architecture document update	Tech details	Pending