1ce332fa1b
The rename of process_work_units and _results_from_pipes during the fan-out refactor left three references behind, in the guide, the checklist and the loosely coupled module docstring. The guide also showed loosely_coupled_k8s_job without its executor_def and claimed the pipes client replaced the executor, which stopped being true once the graph fanned out. Both images were built and smoke tested: the code location image loads its definitions, and the payload image ships dagster_pipes without dagster. That check is now recorded as L11 in the readiness checklist, so the isolation argument is verified at image level and not only in the source. Changelog: fixed
94 lines
3.6 KiB
Markdown
94 lines
3.6 KiB
Markdown
# Distributed Execution
|
|
|
|
Canonical location for documentation, example workflows and reference service
|
|
implementations covering **distributed execution patterns** in the Simpl
|
|
orchestration platform.
|
|
|
|
The service answers one question for every workflow: *where does the compute
|
|
actually run, and how does runtime information get back to the control plane?*
|
|
|
|
## Contents
|
|
|
|
| Area | Location | Status |
|
|
|---|---|---|
|
|
| User guide | [documents/user-guide/distributed-execution-guide.md](documents/user-guide/distributed-execution-guide.md) | Complete |
|
|
| Readiness checklist | [documents/user-guide/readiness-checklist.md](documents/user-guide/readiness-checklist.md) | Complete |
|
|
| Tightly coupled reference | [src/distributed_execution/tightly_coupled/jobs.py](src/distributed_execution/tightly_coupled/jobs.py) | Runnable |
|
|
| Loosely coupled reference | [src/distributed_execution/loosely_coupled/jobs.py](src/distributed_execution/loosely_coupled/jobs.py) | Runnable (subprocess verified) |
|
|
| External payload | [payload/work.py](payload/work.py) | Runnable |
|
|
| Example configuration | [yaml/](yaml/) | Complete |
|
|
|
|
## Project structure
|
|
|
|
```text
|
|
distributed-execution/
|
|
├── src/
|
|
│ └── distributed_execution/
|
|
│ ├── repository.py # Dagster definitions (entry point)
|
|
│ ├── ops.py # Shared ops used by both patterns
|
|
│ ├── preflight.py # Readiness checks backing the checklist
|
|
│ ├── tightly_coupled/
|
|
│ │ └── jobs.py # in_process / multiprocess / k8s_job_executor
|
|
│ └── loosely_coupled/
|
|
│ └── jobs.py # PipesSubprocessClient / PipesK8sClient
|
|
├── payload/ # External workload: dagster-pipes ONLY
|
|
│ ├── work.py
|
|
│ ├── requirements.txt
|
|
│ └── Dockerfile
|
|
├── documents/user-guide/ # AC1-AC4 documentation
|
|
├── yaml/ # Working example configuration
|
|
├── tests/
|
|
├── Dockerfile
|
|
├── pyproject.toml
|
|
└── workspace.yaml
|
|
```
|
|
|
|
## Getting started
|
|
|
|
Prerequisites: Python 3.12+ and `uv`.
|
|
|
|
```bash
|
|
uv sync --dev
|
|
uv run dagster dev -f src/distributed_execution/repository.py
|
|
```
|
|
|
|
The Dagster UI is then available at <http://localhost:3000>. Two jobs run end to
|
|
end on a laptop with no cluster: `tightly_coupled_in_process_job` and
|
|
`loosely_coupled_subprocess_job`. The Kubernetes variants of each require a
|
|
cluster and are documented in the user guide.
|
|
|
|
> `tightly_coupled_local_job` uses `multiprocess_executor`. It did not complete on
|
|
> a Windows development machine during authoring — see the Windows note in the
|
|
> user guide's section 5.5.
|
|
|
|
### Running tests
|
|
|
|
```bash
|
|
uv run pytest
|
|
```
|
|
|
|
### Building the images
|
|
|
|
Two images, deliberately: the code location and the external payload are
|
|
versioned and scanned independently.
|
|
|
|
```bash
|
|
docker build -t distributed-execution:0.1.0 .
|
|
docker build -f payload/Dockerfile -t distributed-execution-payload:0.1.0 payload/
|
|
```
|
|
|
|
Both images build and have been smoke tested locally: the code location image
|
|
loads its definitions, and the payload image contains `dagster_pipes` without
|
|
`dagster` — check L11 in the readiness checklist.
|
|
|
|
## Status
|
|
|
|
Both execution targets are implemented. The tightly coupled jobs and the loosely
|
|
coupled **subprocess** transport are verified by the test suite. The loosely
|
|
coupled **Kubernetes** transport is implemented but has not yet been run against
|
|
a cluster; see the guide's *Outstanding work* section.
|
|
|
|
## Licence
|
|
|
|
European Union Public Licence v1.2 — see [LICENSE](LICENSE).
|