distributed-execution/README.md

Distributed Execution

Canonical location for documentation, example workflows and reference service implementations covering distributed execution patterns in the Simpl orchestration platform.

The service answers one question for every workflow: where does the compute actually run, and how does runtime information get back to the control plane?

Contents

Area	Location	Status
User guide	documents/user-guide/distributed-execution-guide.md	Complete
Readiness checklist	documents/user-guide/readiness-checklist.md	Complete
Tightly coupled reference	src/distributed_execution/tightly_coupled/jobs.py	Runnable
Loosely coupled reference	src/distributed_execution/loosely_coupled/jobs.py	Runnable (subprocess verified)
External payload	payload/work.py	Runnable
Example configuration	yaml/	Complete

Project structure

distributed-execution/
├── src/
│   └── distributed_execution/
│       ├── repository.py           # Dagster definitions (entry point)
│       ├── ops.py                  # Shared ops used by both patterns
│       ├── preflight.py            # Readiness checks backing the checklist
│       ├── tightly_coupled/
│       │   └── jobs.py             # in_process / multiprocess / k8s_job_executor
│       └── loosely_coupled/
│           └── jobs.py             # PipesSubprocessClient / PipesK8sClient
├── payload/                        # External workload: dagster-pipes ONLY
│   ├── work.py
│   ├── requirements.txt
│   └── Dockerfile
├── documents/user-guide/           # AC1-AC4 documentation
├── yaml/                           # Working example configuration
├── tests/
├── Dockerfile
├── pyproject.toml
└── workspace.yaml

Getting started

Prerequisites: Python 3.12+ and uv.

uv sync --dev
uv run dagster dev -f src/distributed_execution/repository.py

The Dagster UI is then available at http://localhost:3000. Two jobs run end to end on a laptop with no cluster: tightly_coupled_in_process_job and loosely_coupled_subprocess_job. The Kubernetes variants of each require a cluster and are documented in the user guide.

tightly_coupled_local_job uses multiprocess_executor. It did not complete on a Windows development machine during authoring — see the Windows note in the user guide's section 5.5.

Running tests

uv run pytest

Building the images

Two images, deliberately: the code location and the external payload are versioned and scanned independently.

docker build -t distributed-execution:0.1.0 .
docker build -f payload/Dockerfile -t distributed-execution-payload:0.1.0 payload/

Both images build and have been smoke tested locally: the code location image loads its definitions, and the payload image contains dagster_pipes without dagster — check L11 in the readiness checklist.

.gitea/workflows/docker-publish.yml runs the same two builds on every push to main, applies those checks as gates, and tags both images with the same short commit SHA — that shared tag is what keeps a code location and the payload it dispatches on the same version.

Status

Both execution targets are implemented. The tightly coupled jobs and the loosely coupled subprocess transport are verified by the test suite. The loosely coupled Kubernetes transport is implemented but has not yet been run against a cluster; see the guide's Outstanding work section.

Licence

European Union Public Licence v1.2 — see LICENSE.