Distributed Execution

Canonical location for documentation, example workflows and reference service implementations covering distributed execution patterns in the Simpl orchestration platform.

The service answers one question for every workflow: where does the compute actually run, and how does runtime information get back to the control plane?

Contents

Area	Location	Status
User guide	documents/user-guide/distributed-execution-guide.md	Complete
Readiness checklist	documents/user-guide/readiness-checklist.md	Complete
Tightly coupled reference	src/distributed_execution/tightly_coupled/jobs.py	Runnable
Loosely coupled reference	src/distributed_execution/loosely_coupled/jobs.py	Runnable and cluster-verified
External payload	payload/work.py	Runnable
Example configuration	yaml/	Complete

Project structure

distributed-execution/
├── src/
│   └── distributed_execution/
│       ├── repository.py           # Dagster definitions (entry point)
│       ├── ops.py                  # Shared ops used by both patterns
│       ├── preflight.py            # Readiness checks backing the checklist
│       ├── tightly_coupled/
│       │   └── jobs.py             # in_process / multiprocess / k8s_job_executor
│       └── loosely_coupled/
│           └── jobs.py             # PipesSubprocessClient / PipesK8sClient
├── payload/                        # External workload: dagster-pipes ONLY
│   ├── work.py
│   ├── requirements.txt
│   └── Dockerfile
├── documents/user-guide/           # Guide, checklist and requirement traceability
├── yaml/                           # Working example configuration
├── tests/
├── Dockerfile
├── pyproject.toml
└── workspace.yaml

Getting started

Prerequisites: Python 3.12+ and uv.

uv sync --dev
uv run dagster dev -f src/distributed_execution/repository.py

The Dagster UI is then available at http://localhost:3000. Two jobs run end to end on a laptop with no cluster: tightly_coupled_in_process_job and loosely_coupled_subprocess_job. The Kubernetes variants of each require a cluster and are documented in the user guide.

tightly_coupled_local_job uses multiprocess_executor. It did not complete on a Windows development machine during authoring — see the Windows note in the user guide's section 5.5.

Running tests

uv run pytest

Building the images

Two images, deliberately: the code location and the external payload are versioned and scanned independently.

docker build -t distributed-execution:0.1.0 .
docker build -f payload/Dockerfile -t distributed-execution-payload:0.1.0 payload/

Both images build and have been smoke tested locally: the code location image loads its definitions, and the payload image contains dagster_pipes without dagster — check L11 in the readiness checklist.

Both must be tagged from the same commit. That shared tag is what keeps a code location and the payload it dispatches on the same version, and nothing at runtime checks the pairing — see the Outstanding work section for what the pipeline still has to learn.

Status

Both execution targets are implemented. Local variants are verified by the test suite, tightly_coupled_k8s_job has run end to end through the platform launcher, and loosely_coupled_k8s_job has run end to end through the standalone cluster probe. See the guide and readiness checklist for the recorded evidence.

Licence

European Union Public Licence v1.2 — see LICENSE.