# Distributed Execution

Canonical location for documentation, example workflows and reference service
implementations covering **distributed execution patterns** in the Simpl
orchestration platform.

The service answers one question for every workflow: *where does the compute
actually run, and how does runtime information get back to the control plane?*

## Contents

| Area | Location | Status |
|---|---|---|
| User guide | [documents/user-guide/distributed-execution-guide.md](documents/user-guide/distributed-execution-guide.md) | Complete |
| Readiness checklist | [documents/user-guide/readiness-checklist.md](documents/user-guide/readiness-checklist.md) | Complete |
| Tightly coupled reference | [src/distributed_execution/tightly_coupled/jobs.py](src/distributed_execution/tightly_coupled/jobs.py) | Runnable |
| Loosely coupled reference | [src/distributed_execution/loosely_coupled/jobs.py](src/distributed_execution/loosely_coupled/jobs.py) | Runnable and cluster-verified |
| External payload | [payload/work.py](payload/work.py) | Runnable |
| Example configuration | [yaml/](yaml/) | Complete |

## Project structure

```text
distributed-execution/
├── src/
│   └── distributed_execution/
│       ├── repository.py           # Dagster definitions (entry point)
│       ├── ops.py                  # Shared ops used by both patterns
│       ├── preflight.py            # Readiness checks backing the checklist
│       ├── tightly_coupled/
│       │   └── jobs.py             # in_process / multiprocess / k8s_job_executor
│       └── loosely_coupled/
│           └── jobs.py             # PipesSubprocessClient / PipesK8sClient
├── payload/                        # External workload: dagster-pipes ONLY
│   ├── work.py
│   ├── requirements.txt
│   └── Dockerfile
├── documents/user-guide/           # Guide, checklist and requirement traceability
├── yaml/                           # Working example configuration
├── tests/
├── Dockerfile
├── pyproject.toml
└── workspace.yaml
```

## Getting started

Prerequisites: Python 3.12+ and `uv`.

```bash
uv sync --dev
uv run dagster dev -f src/distributed_execution/repository.py
```

The Dagster UI is then available at <http://localhost:3000>. Two jobs run end to
end on a laptop with no cluster: `tightly_coupled_in_process_job` and
`loosely_coupled_subprocess_job`. The Kubernetes variants of each require a
cluster and are documented in the user guide.

> `tightly_coupled_local_job` uses `multiprocess_executor`. It did not complete on
> a Windows development machine during authoring — see the Windows note in the
> user guide's section 5.5.

### Running tests

```bash
uv run pytest
```

### Building the images

Two images, deliberately: the code location and the external payload are
versioned and scanned independently.

```bash
docker build -t distributed-execution:0.1.0 .
docker build -f payload/Dockerfile -t distributed-execution-payload:0.1.0 payload/
```

Both images build and have been smoke tested locally: the code location image
loads its definitions, and the payload image contains `dagster_pipes` without
`dagster` — check L11 in the readiness checklist.

Both must be tagged from the same commit. That shared tag is what keeps a code
location and the payload it dispatches on the same version, and nothing at
runtime checks the pairing — see the *Outstanding work* section for what the
pipeline still has to learn.

## Status

Both execution targets are implemented. Local variants are verified by the test
suite, `tightly_coupled_k8s_job` has run end to end through the platform launcher,
and `loosely_coupled_k8s_job` has run end to end through the standalone cluster
probe. See the guide and readiness checklist for the recorded evidence.

## Licence

European Union Public Licence v1.2 — see [LICENSE](LICENSE).