distributed-execution/documents/user-guide/distributed-execution-guide.md

Distributed Execution: Execution Targets and Integration Patterns

Audience: Participants with permission to configure workflows. Scope: How to select, document and configure the execution target and integration pattern for a Dagster workflow on the Simpl orchestration platform.

Document status. Both setups are backed by runnable reference implementations and verified end to end. The loosely coupled Kubernetes path reached RUN_SUCCESS on sandbox-cat-dat on 2026-08-31; the tightly coupled k8s_job_executor path reached RUN_SUCCESS through the platform launcher on 2026-09-01. See the readiness checklist for the recorded evidence.

---

1. Concepts

1.1 What an execution target is

The execution target is where the compute for a workflow actually runs. It is not a single Dagster setting — it is the combination of three decisions.

#	Decision	Where it is configured	Scope
1	Placement — which cluster and namespace the run lands in	Run launcher, in the Dagster instance / Helm values	Per deployment
2	Granularity — one process for the whole run, or one per step	executor_def, in workflow code	Per job
3	Integration pattern — whether the compute participates in Dagster's internals or reports back over a message channel	Job topology in workflow code	Per job

Decisions 1 and 2 are always present. Decision 3 is what this guide calls the integration pattern, and it is the one with the largest operational consequences, because it determines what the execution pod must be able to reach.

1.2 The two integration patterns

Tightly coupled. The process doing the work is a Dagster process. It imports the code location, opens a connection to the metadata database and writes run events directly. Runtime information reaches the control plane because the worker is part of the control plane's data path.

Loosely coupled. Dagster only dispatches. It creates a workload on an external cluster running an arbitrary image, then acts as a listener. The external process writes structured messages over the dagster-pipes protocol; Dagster reads them and materialises them as logs and metadata. The execution pod never connects to the metadata database.

flowchart LR
    subgraph TC["Tightly coupled"]
        direction TB
        D1[Dagster daemon] -->|K8sRunLauncher| RW1[Run worker pod]
        RW1 -->|k8s_job_executor| SP1[Step pods]
        RW1 -.->|events| DB1[(Metadata DB)]
        SP1 -.->|events| DB1
    end
    subgraph LC["Loosely coupled"]
        direction TB
        D2[Dagster daemon] -->|K8sRunLauncher| RW2[Run worker pod]
        RW2 -->|PipesK8sClient| EXT[External workload<br/>any image]
        EXT -.->|pipes messages| RW2
        RW2 -.->|events| DB2[(Metadata DB)]
    end

Note the asymmetry in the diagram: in the loosely coupled pattern only the run worker touches the metadata database. That single arrow is the whole point of the pattern.

---

2. Choosing an execution target and cluster setup

2.1 Choose tightly coupled when

• The workflow logic is written in Python and lives in a Dagster code location.
• Execution pods can be granted network egress to the metadata database, object storage and Vault.
• You want per-step retries, step-level concurrency limits and asset materialisation handled natively by Dagster.
• The compute belongs to the same trust boundary as the orchestration platform.

2.2 Choose loosely coupled when

• The workload is not Python, or cannot take a dependency on the dagster package (different runtime, vendor image, GPU base image).
• The target cluster is administratively separate and must not be granted connectivity to the orchestration metadata database.
• The compute belongs to a different trust boundary, and you want the blast radius of a compromised execution pod limited to the message channel.
• The workload already exists as a container and you want to orchestrate it without rewriting it.

2.3 Prerequisites

Prerequisite	Tightly coupled	Loosely coupled
Run launcher configured	Required — K8sRunLauncher	Required — K8sRunLauncher
Code location image contains workflow code	Required	Required for the launching job only
Execution pod → metadata database (Postgres 5432)	Required	Not required
Execution pod → object storage	Required if the workflow reads/writes objects	Required only if the payload itself does
Execution pod → Vault	Required for secret injection	Only for the launching pod
Kubernetes RBAC to create Jobs	Required for k8s_job_executor only	Required in the target namespace
Message channel provisioned	Not applicable	Required — pod log stream or object-storage reader
Control-plane reporting	Automatic — worker writes events directly	Via pipes messages; fails silently if the channel breaks

2.4 What to document before the first execution

Record the following at workflow level, in the repository holding the workflow, before it is run for the first time. Section 5.4 explains where each item lives.

1. Chosen integration pattern — tightly or loosely coupled, with a one-line rationale referencing section 2.1 or 2.2.
2. Executor and its configuration — including max_concurrent where relevant.
3. Target namespace for run pods and, for k8s_job_executor, step pods.
4. Runtime dependencies the execution pods must reach, as host/port pairs. This is the input to the NetworkPolicy review.
5. Secrets consumed, by Vault path — not by value.
6. Expected evidence of a healthy run — which of the checks in the readiness checklist apply, and what a passing run looks like in the UI.
7. For loosely coupled only: the payload image reference and the message channel type.

---

3. Decision support and cluster-configuration inputs

3.1 Side-by-side comparison

Dimension	Tightly coupled	Loosely coupled
What runs the work	A Dagster process importing your code location	Any container image
Language constraint	Python, matching the code location's environment	None
Dependency on dagster package	Full package in the execution image	Only dagster-pipes in the payload image
Metadata DB connectivity from execution pod	Required	Not required
Runtime information flow	Direct writes to run storage	Structured messages → run worker → run storage
Log capture	Native, per step	Via the pipes message stream
Step-level retries	Native	Manual, at the dispatching op
Asset materialisation	Native, from any step	Reported by the payload, materialised by the launching op
Failure granularity	Per step	Per dispatched workload
Secret distribution	To every execution pod	To the launching pod, plus whatever the payload needs
Blast radius of a compromised execution pod	Orchestration runtime credentials	Message channel only
Cross-cluster execution	Not supported	Supported

3.2 Environment-fit indicators

Use these as a fast triage. If several point the same way, that is your answer.

Observation about your environment	Points to
Workflow is pure Python and shares a repository with existing code locations	Tightly coupled
Steps are short and numerous, and you want native retries	Tightly coupled
NetworkPolicy review for the execution namespace is already approved	Tightly coupled
Compute must run in a cluster you do not administer	Loosely coupled
Security review objects to distributing DB credentials to execution pods	Loosely coupled
Payload is an existing vendor or third-party image	Loosely coupled
Workload needs GPUs or a base image incompatible with the code location image	Loosely coupled
Team owning the payload is different from the team owning the workflow	Loosely coupled

3.3 Trade-offs

Tightly coupled

• Gains: the richest observability with no extra work — every step's logs, metadata, retries and asset materialisations are native. Simplest mental model.
• Costs: the widest connectivity surface. Every execution pod is a credential holder and a database client. Scaling step pods scales database connections, which is the usual first bottleneck. Execution image and code location image are coupled, so a dependency conflict in one workflow can affect others sharing the location.

Loosely coupled

• Gains: a narrow connectivity surface and genuine dependency isolation. The payload team can ship on its own cadence. Cross-cluster and cross-trust-boundary execution become possible.
• Costs: observability is only as good as the payload's instrumentation — an uninstrumented payload appears in Dagster as one opaque step. Retry and cancellation semantics must be implemented at the dispatching op. The message channel becomes a new failure mode, and when it breaks the run can appear to succeed while reporting nothing. Two images to build, version and scan instead of one.

3.4 Cluster-configuration inputs

Once the setup is chosen, the container cluster has to be described. The table below is the full input surface, split into what a run cannot start without and what only applies under a stated condition. "Mandatory" means the run fails, or reports nothing, if the input is absent or wrong.

Input	Tightly coupled	Loosely coupled	Notes
Cluster / kube context hosting the run worker	Mandatory	Mandatory	The launcher's cluster. Loosely coupled may dispatch to a second cluster
Namespace for run pods	Mandatory	Mandatory	Supplied by the code location's DAGSTER_CONTAINER_CONTEXT; the launcher's job_namespace is only a fallback
Code location image reference and tag	Mandatory	Mandatory	Never latest; check C3
Executor selection	Mandatory	Mandatory	Bounds step granularity; loosely coupled it bounds only dispatch concurrency
Service account for run pods	Mandatory	Mandatory	Must have automountServiceAccountToken: true
Run-pod egress to the metadata database	Mandatory	Not applicable	The pod that writes run events
Namespace for step pods	Conditional — k8s_job_executor only	Not applicable	Defaults to the run pod's namespace
RBAC to create and watch Jobs	Conditional — k8s_job_executor only	Mandatory, in the payload namespace	Same verbs, different namespace
Shared RWX volume plus an I/O manager base_dir under it	Conditional — k8s_job_executor only	Optional	Without it step pods cannot read each other's outputs
Per-step resource requests and limits	Optional	Optional	Multiply by step or dispatch concurrency
Payload image reference and tag	Not applicable	Mandatory	Must share the code location's commit tag
Namespace for payload workloads	Not applicable	Mandatory	May be in another cluster or trust boundary
Message channel and its reader	Not applicable	Mandatory	Pod log stream by default; RBAC on pods/log is the channel
Payload wait timeout	Not applicable	Optional	Defaults to 24 h — lower it so scheduling failures surface
Vault role and secret paths	Conditional — if the workflow reads secrets	Conditional — launching pod only	Payloads should receive only extras and explicit env
Object-storage endpoint and credentials	Conditional — if the workflow uses S3	Conditional — if the payload uses S3
Image pull secret	Conditional — non-anonymous registry	Conditional — non-anonymous registry	Applies to the SA that runs the pod, which for dispatched payload Jobs is default

3.5 Minimum input set

The list below is the reusable baseline: the smallest set of values a team has to agree on before a workflow can be configured against a cluster. Record it once per workflow, in the workflow's own repository, and carry it into whatever configuration format the team already uses.

Both setups

1. cluster — the cluster the run worker lands in.
2. run_namespace — namespace for run pods.
3. code_location_image — image reference including an immutable tag.
4. executor — executor name plus its concurrency bound.
5. service_account — the identity run pods assume.

Tightly coupled adds

6. metadata_db — host and port the run and step pods must reach.
7. step_namespace — only when k8s_job_executor is used.
8. shared_output_path — RWX mount path, and the I/O manager base_dir pinned under it, again only for k8s_job_executor.

Loosely coupled adds

6. payload_image — image reference sharing the code location's commit tag.
7. payload_namespace — where dispatched workloads are created; may be in another cluster.
8. message_channel — the reader in use, and the RBAC verb that carries it.

This guide deliberately does not prescribe a configuration schema or key names. The names above are descriptive labels for the values, not a required format; the examples in section 3.6 express the same baseline in two different shapes, and both are correct.

3.6 Worked input examples

The reference configuration in yaml/ shows the baseline expressed in the formats each setup already uses. Neither is normative.

Tightly coupled — the inputs land in Helm values and in the job's executor config, because that is where the platform chart and Dagster expect them:

# yaml/tightly-coupled/values-run-launcher.yaml  (shape, not a fixed schema)
dagster:
  runLauncher:
    type: K8sRunLauncher
    config:
      k8sRunLauncher:
        jobNamespace: dagster              # run_namespace
        failPodOnRunFailure: true
        runK8sConfig:
          containerConfig:
            env:                           # metadata_db, S3 and Vault reachability
              - name: S3_ENDPOINT_URL
                value: https://s3.dev.simpl-europe.eu

# step_namespace, shared_output_path and the executor travel with the job
tightly_coupled_k8s_job = distributed_execution_reference.to_job(
    executor_def=k8s_job_executor,
    resource_defs={"io_manager": fs_io_manager.configured({"base_dir": SHARED_IO_BASE_DIR})},
    config={"execution": {"config": {"step_k8s_config": STEP_K8S_CONFIG}}},
)

Loosely coupled — the same baseline, but the cluster-facing half is the payload image and its namespace. Here they are environment values set on both the code location and the run launcher, because the dispatching pod is a run pod:

# yaml/loosely-coupled/values-pipes-payload.yaml  (shape, not a fixed schema)
dagster:
  dagster-user-deployments:
    deployments:
      - name: distributed-execution
        env:
          - name: PIPES_PAYLOAD_IMAGE                # payload_image, tag-locked
            value: <registry>/distributed-execution/payload:0.0.0
          - name: PIPES_PAYLOAD_NAMESPACE            # payload_namespace
            value: dagster

PAYLOAD_IMAGE = os.environ.get("PIPES_PAYLOAD_IMAGE", ...)
PAYLOAD_NAMESPACE = os.environ.get("PIPES_PAYLOAD_NAMESPACE", "dagster")

The message_channel input has no key in either file. The default reader is the pod log stream, so it is configured by granting pods/log in yaml/loosely-coupled/rbac-pipes-dispatch.yaml. An input can be mandatory and still not be a configuration field.

Note what is absent from the loosely coupled example: no metadata database, no Vault role, no object-storage credentials. Those inputs are not optional there — they do not exist, which is the operational consequence of the choice.

---

4. Readiness checks

Pre-run validation items, their expected evidence, and the symptoms and corrections for common misconfigurations are maintained separately, in readiness-checklist.md.

---

5. Configuring execution-target linkage in code

5.1 Configuration constructs

Five constructs bind a workflow to an execution target. Three live in code and travel with the workflow; two live in deployment configuration.

Construct	Lives in	Binds
executor_def on @job / to_job()	Workflow code	Step granularity and placement
tags on @job	Workflow code	Run-level metadata; dagster-k8s/config overrides pod shape
Job topology — direct ops vs. a dispatching op using a pipes client	Workflow code	Integration pattern
runLauncher	Helm values	Run worker placement and reachable dependencies
dagster-user-deployments entry	Helm values	Which image the code location runs

The first three are the answer to "which configuration constructs bind the execution target to a workflow" — and they are all versioned alongside the workflow, which is why setup selection is a code review concern and not a runtime toggle.

5.2 Worked example — switching within the tightly coupled setup

The reference jobs share one graph and differ only by executor_def. This is the smallest possible demonstration that granularity is a code-level decision.

Before — steps run as subprocesses inside the run worker:

tightly_coupled_local_job = distributed_execution_reference.to_job(
    name="tightly_coupled_local_job",
    executor_def=multiprocess_executor.configured({"max_concurrent": 2}),
    tags={**COMMON_TAGS, "executor": "multiprocess"},
)

After — one Kubernetes Job per step:

tightly_coupled_k8s_job = distributed_execution_reference.to_job(
    name="tightly_coupled_k8s_job",
    executor_def=k8s_job_executor,
    resource_defs={
        "io_manager": fs_io_manager.configured({"base_dir": SHARED_IO_BASE_DIR})
    },
    config={
        "execution": {
            "config": {
                "image_pull_policy": "IfNotPresent",
                "step_k8s_config": STEP_K8S_CONFIG,
            }
        }
    },
    tags={**COMMON_TAGS, "executor": "k8s_job"},
)

The graph, the ops and the run launcher are all unchanged. What changes operationally:

• Step pods are now created in the launcher's jobNamespace, so that namespace's NetworkPolicy must permit egress to Postgres, object storage and Vault — not just the run worker's namespace.
• The service account needs RBAC to create and watch Jobs. See yaml/tightly-coupled/rbac-step-executor.yaml.
• Per-step resource requests apply per pod, so the aggregate request for a fan-out step is the per-step request multiplied by concurrency.
• Step outputs must land on storage every step pod can read. With the default filesystem I/O manager each pod writes to its own container filesystem, so a downstream step opens a path that does not exist there. STEP_K8S_CONFIG mounts the RWX dagster-shared-pvc at /dagster/shared, and the job pins fs_io_manager's base_dir underneath it.
• Pin base_dir directly rather than relying on DAGSTER_HOME. The default base_dir is $DAGSTER_HOME/storage, but the Dagster chart already injects DAGSTER_HOME=/tmp/dagster into the step container; a second entry appended by step_k8s_config does not displace it, so the override is silently ignored and outputs keep going to /tmp/dagster/storage.

The executor's defaults go through config= rather than k8s_job_executor.configured(...). .configured() collapses the executor's config schema to Any, which makes a run-config execution: block accepted and then discarded, and makes job_image and job_namespace impossible to supply at launch time.

The evidence that the switch took effect is in summarise_results output metadata, and the two fields say different things:

Executor	contributing_workers	contributing_hosts
in_process_executor	1	1
multiprocess_executor	one per unit	1 — same machine
k8s_job_executor	one per unit	one per unit — separate pods

This only works because generate_work_units is a DynamicOut and the graph does units.map(process_work_unit).collect(). A single op looping over all units internally would report one worker under every executor, because one step cannot span processes or pods.

5.3 Worked example — switching to loosely coupled

The graphs for the two patterns differ by exactly one node. This is not a simplification for the guide; it is how the reference implementations are written, so the diff below is the real one.

Before — tightly coupled. The work happens in a Dagster process:

@graph
def distributed_execution_reference():
    target_report = report_execution_target()
    units = generate_work_units()
    results = units.map(process_work_unit).collect()          # runs in Dagster
    return summarise_results(results, target_report)

After — loosely coupled. The work is dispatched to an external image:

@graph
def loosely_coupled_k8s_reference():
    target_report = report_execution_target()
    units = generate_work_units()
    results = units.map(dispatch_external_work_k8s).collect() # dispatches, then listens
    return summarise_results(results, target_report)

The dispatching op replaces direct computation with a pipes client call:

@op(out=Out(dict))
def dispatch_external_work_k8s(
    context: OpExecutionContext,
    unit: int,
    pipes_k8s_client: PipesK8sClient,
) -> dict:
    completed = pipes_k8s_client.run(
        context=context,
        image=PAYLOAD_IMAGE,
        command=["python", "/app/work.py"],
        namespace=PAYLOAD_NAMESPACE,
        extras={"units": [unit]},
    )
    return _result_from_pipes(context, completed)

and the job supplies the client as a resource:

loosely_coupled_k8s_job = loosely_coupled_k8s_reference.to_job(
    name="loosely_coupled_k8s_job",
    executor_def=multiprocess_executor.configured({"max_concurrent": 2}),
    resource_defs={"pipes_k8s_client": PipesK8sClient()},
    tags={**COMMON_TAGS, "transport": "k8s_pod_logs"},
)

The executor is still present, but its meaning has changed. It no longer decides where the work runs — the payload image and namespace do that. It only bounds how many dispatching ops wait on the message channel at once. The two decisions that were fused in the tightly coupled setup are now independent.

What changes operationally

	Tightly coupled	Loosely coupled
Images to build	1 (code location)	2 (code location + payload)
Payload dependency surface	Full dagster package	dagster-pipes only
Execution pod → Postgres	Egress rule required	Egress rule can be removed
Execution pod → Vault / S3	Credentials required	None needed by the payload
RBAC	Only for k8s_job_executor	Required on the dispatching pod
New failure mode	—	Broken message path

The NetworkPolicy rules you can remove are the point of the exercise. See yaml/loosely-coupled/rbac-pipes-dispatch.yaml for what replaces them — note that pods/log is the message channel, so removing that verb silently breaks reporting without failing the workload.

The payload contract

payload/work.py is the whole contract:

with open_dagster_pipes() as pipes:
    units = pipes.get_extra("units")
    pipes.log.info(f"External payload started on {host}")
    pipes.report_custom_message({"results": results, "host": host, ...})

pipes.log entries appear in the Dagster run logs; report_custom_message is read back by the dispatching op. The payload imports dagster_pipes and nothing else from the Dagster ecosystem — a test asserts this, because the moment the payload imports dagster the isolation argument for choosing this pattern collapses.

Because the graph maps over the dynamic output, one external workload is dispatched per unit. A test asserts that the four units come back from four distinct external workers, which is the loosely coupled equivalent of the contributing_workers evidence in section 5.2.

Message channel choice

PipesK8sClient defaults to PipesK8sPodLogsMessageReader, which is what the reference uses. It needs no extra infrastructure, but it couples message delivery to the pod log stream — a log shipper that intercepts or truncates stdout will break reporting while the workload still exits 0. That is why _result_from_pipes raises on an empty message list rather than returning an empty result. If your cluster's logging setup makes that transport unreliable, switch to an object-storage message reader, which needs a bucket plus credentials on both sides.

CLUSTER-VERIFIED 2026-08-31. Run cff9b348… of loosely_coupled_k8s_job reached RUN_SUCCESS on sandbox-cat-dat. Four payload Jobs were created, one per work unit, and contributing_hosts held four distinct payload pod names — so the work demonstrably ran off the orchestrator. The payload's log lines reached the run log, which is the pod log stream serving as the message channel exactly as described above.

5.4 Where setup selection is persisted

Setup selection is persisted in configuration artifacts, never as a runtime-only setting. There is no UI control that switches a workflow between the two patterns.

Artifact	Holds
src/<package>/**/jobs.py	executor_def, pipes client resource, tags, job topology
payload/work.py + payload/Dockerfile	The external payload contract and its image
yaml/tightly-coupled/values-run-launcher.yaml	Run launcher type, namespace, pod env
yaml/tightly-coupled/rbac-step-executor.yaml	Permissions required by k8s_job_executor
yaml/loosely-coupled/values-pipes-payload.yaml	Payload image reference and target namespace
yaml/sandbox/values-sandbox-gitea.yaml	Sandbox registry coordinates, pull secret, and the SHA both images share
yaml/loosely-coupled/rbac-pipes-dispatch.yaml	Permissions required by the dispatching pod
yaml/loosely-coupled/probe-pipes-k8s.yaml	Standalone cluster probe for the pipes transport (checklist L4–L9)
yaml/sandbox/probe-pipes-k8s-sandbox.yaml	The same probe for sandbox-cat-dat, where project-scoped access forbids creating namespaces and RBAC
yaml/values-dagster-distributed-execution.yaml	Code location image and entry point

The Launchpad can override run configuration — op config, resource config, tags — for a single run. It cannot change the executor or the integration pattern. Those take effect only when a new image is built and the code location reloads.

5.5 Reference implementations in this repository

Job	Executor / transport	Runs locally	Demonstrates
tightly_coupled_in_process_job	in_process_executor	Yes	Baseline; steps inside the run worker
tightly_coupled_local_job	multiprocess_executor	Linux/macOS	Subprocess fan-out bounded by the run pod
tightly_coupled_k8s_job	k8s_job_executor	No — needs a cluster	One Kubernetes Job per step
loosely_coupled_subprocess_job	PipesSubprocessClient	Yes	The pipes contract end to end, no cluster
loosely_coupled_k8s_job	PipesK8sClient	No — needs a cluster	External Job dispatch, messages over pod logs

The last two dispatch the same payload/work.py. The subprocess variant exists so the pipes contract can be exercised, tested and demonstrated without any infrastructure.

Windows note. tightly_coupled_local_job did not complete on a Windows development machine during authoring: the multiprocess_executor spawned step subprocesses that never terminated, and required manual cleanup. This was not reproduced on Linux and is not expected to affect cluster deployments, where run workers are Linux pods. For a laptop demonstration on Windows, prefer tightly_coupled_in_process_job or loosely_coupled_subprocess_job, both of which are covered by the test suite.

Run the local variants with:

uv sync --dev
uv run dagster dev -f src/distributed_execution/repository.py

report_execution_target emits pod identity and the visible orchestration environment variables as output metadata on every run, which is the evidence the readiness checklist refers to.

5.6 What a verified run actually produced

The two locally runnable jobs were launched from the Dagster UI, so the evidence below comes from the full daemon → run launcher → executor path rather than from a unit test. Hostnames are redacted; process IDs are real.

tightly_coupled_in_process_job:

succeeded steps: 7
  generate_work_units, report_execution_target,
  process_work_unit[unit_0..unit_3], summarise_results

units                = 4
contributing_hosts   = ['dev-host']
contributing_workers = ['dev-host#25264']

loosely_coupled_subprocess_job:

succeeded steps: 7
  generate_work_units, report_execution_target,
  dispatch_external_work_subprocess[unit_0..unit_3], summarise_results

units                = 4
contributing_hosts   = ['dev-host']
contributing_workers = ['dev-host#13532', 'dev-host#27116',
                        'dev-host#32984', 'dev-host#31224']

Read the two blocks together. The step lists are the same shape, which is the point of section 5.3 — the graphs differ by one node. contributing_workers is what separates them: one entry means the work happened inside the Dagster process; four means four external processes did it and reported back over the pipes channel. contributing_hosts stays at one because a laptop is one machine; on a cluster it is what distinguishes k8s_job_executor from multiprocess_executor.

launcher_namespace reads <not-in-kubernetes> in both. In a cluster run it holds the namespace the run pod landed in, which is the placement half of the execution target.

5.7 Dagster UI reference

The screenshots below were captured from the local reference service on 2026-09-02. They show the surfaces participants use to inspect job linkage and run evidence. Cluster placement is evidenced separately by the sandbox run IDs in the readiness checklist.

All five reference jobs are registered under one code location:

The job overview exposes the graph whose executor or dispatch op defines the integration pattern:

The Launchpad shows that execution_target and executor are job tags persisted by code configuration, not a separate runtime-only selector:

A successful local run displays the step events and final status used alongside output metadata as retained evidence:

---

6. Upstream documentation

• Dagster — run launchers
• Dagster — executors
• Dagster — customizing your Kubernetes deployment
• Dagster — Dagster Pipes
• Dagster — Dagster Pipes with Kubernetes
• Kubernetes — Jobs
• Kubernetes — network policies
• Kubernetes — RBAC authorization

7. Related Simpl documentation

• dagster/documents/user-manual/Orchestration.md — workflow development lifecycle
• dagster/documents/installation-guide/Installation Guide.md — full run launcher parameter reference
• dagster/documents/user-manual/Observability.md — logging and monitoring

8. Outstanding work

Tracked under SIMPL-30451 and SIMPL-30787.

Item	Status
Execution-target choices, prerequisites, workflow-level documentation	Complete
Comparison, environment-fit indicators, trade-offs	Complete
Cluster-configuration inputs, mandatory vs optional, minimum input set	Complete
Readiness checklist and evidence mapping	Complete; both Kubernetes paths cluster-verified
Misconfiguration symptoms and corrections	Complete; corrected with observed cluster failure modes
Configuration constructs and persistence	Complete
Before-and-after example, tightly coupled	Complete
Before-and-after example, switching to loosely coupled	Complete
Runnable tightly coupled reference implementations	Complete
Runnable loosely coupled reference implementation	Complete (subprocess and Kubernetes transports both verified)
Both images build; payload image passes the isolation check (L11)	Complete
Locally runnable jobs launched from the Dagster UI, evidence recorded in section 5.6	Complete
Payload image published to a container registry	Complete on the sandbox Gitea registry; the GitLab registry still pending
GitLab pipeline builds both images	Pending — the shared ds.gitlab-ci.yml template builds one image from the root Dockerfile
End-to-end run of loosely_coupled_k8s_job on a cluster	Complete — sandbox-cat-dat, 2026-08-31, run cff9b348…; see the readiness checklist
distributed-execution registered as a code location on a platform Dagster	Complete — tightly coupled run 1d8cb167… launched through the platform webserver
Screenshots of UI surfaces	Complete — job list, graph, Launchpad linkage and successful run captured in section 5.7
Platform architecture document update	Complete — root deployment_diagram.md includes the service and both runtime-information paths

---

9. Requirement traceability

The sections above are organised by subject, not by ticket. Two stories share this material; the mapping below is authoritative for both.

SIMPL-30787 — defining the execution container cluster

AC	Requirement	Where it is met
AC1	Prerequisites for both setups	Section 2.3
AC1	What must be documented before execution	Section 2.4
AC1	Where cluster-runtime connectivity is required, where decoupled reporting is expected	Sections 1.2 and 2.3; the asymmetry is drawn in the section 1.2 diagram
AC2	Mandatory vs optional inputs per setup	Section 3.4
AC2	Examples for both setups, no fixed config naming model	Section 3.6, with the closing note in 3.5
AC2	Reusable minimum input set	Section 3.5
AC3	Checklist separates tightly from loosely coupled checks	readiness-checklist.md sections 2 and 3; section 1 holds the checks common to both
AC3	Each item mapped to expected evidence	The Expected evidence column of every checklist table
AC4	Configuration constructs attaching the pattern to a workflow	Section 5.1
AC4	Worked before-and-after example across setups	Sections 5.2 and 5.3
AC4	At least one runnable reference implementation per setup	Section 5.5 — three tightly coupled, two loosely coupled
AC4	Selection persisted in configuration artifacts, not runtime-only	Section 5.4
Tech	Diagrams, screenshots, working config files, upstream links	Sections 1.2, 5.7, yaml/, section 6

SIMPL-30451 — execution targets and integration patterns

AC	Subject	Where it is met
AC1	Choosing an execution target	Section 2
AC2	Decision support	Sections 3.1 to 3.3
AC3	Readiness checks	Section 4 and the readiness checklist
AC4	Linkage configured in code	Section 5