ccc2e94c2a
The reference graph processed every work unit inside a single op, so exactly one worker was ever reported regardless of executor. That made the guide's claim that k8s_job_executor yields several distinct contributing_hosts false, and left the reference implementations unable to demonstrate the executor choice at all. generate_work_units is now a DynamicOut and both graphs map over it, so one step is created per unit and the loosely coupled pattern dispatches one external workload per unit. Evidence is split into contributing_workers (host and pid, differs per process) and contributing_hosts (differs only across machines), because the previous single field could not distinguish multiprocess fan-out from no fan-out. Tests now assert the mapped step keys rather than a host count, since execute_in_process ignores executor_def and cannot prove executor behaviour on its own. Adds a Windows note: multiprocess_executor did not complete during authoring and left orphaned processes. Changelog: fixed
90 lines
3.4 KiB
Markdown
90 lines
3.4 KiB
Markdown
# Distributed Execution
|
|
|
|
Canonical location for documentation, example workflows and reference service
|
|
implementations covering **distributed execution patterns** in the Simpl
|
|
orchestration platform.
|
|
|
|
The service answers one question for every workflow: *where does the compute
|
|
actually run, and how does runtime information get back to the control plane?*
|
|
|
|
## Contents
|
|
|
|
| Area | Location | Status |
|
|
|---|---|---|
|
|
| User guide | [documents/user-guide/distributed-execution-guide.md](documents/user-guide/distributed-execution-guide.md) | Complete |
|
|
| Readiness checklist | [documents/user-guide/readiness-checklist.md](documents/user-guide/readiness-checklist.md) | Complete |
|
|
| Tightly coupled reference | [src/distributed_execution/tightly_coupled/jobs.py](src/distributed_execution/tightly_coupled/jobs.py) | Runnable |
|
|
| Loosely coupled reference | [src/distributed_execution/loosely_coupled/jobs.py](src/distributed_execution/loosely_coupled/jobs.py) | Runnable (subprocess verified) |
|
|
| External payload | [payload/work.py](payload/work.py) | Runnable |
|
|
| Example configuration | [yaml/](yaml/) | Complete |
|
|
|
|
## Project structure
|
|
|
|
```text
|
|
distributed-execution/
|
|
├── src/
|
|
│ └── distributed_execution/
|
|
│ ├── repository.py # Dagster definitions (entry point)
|
|
│ ├── ops.py # Shared ops used by both patterns
|
|
│ ├── preflight.py # Readiness checks backing the checklist
|
|
│ ├── tightly_coupled/
|
|
│ │ └── jobs.py # in_process / multiprocess / k8s_job_executor
|
|
│ └── loosely_coupled/
|
|
│ └── jobs.py # PipesSubprocessClient / PipesK8sClient
|
|
├── payload/ # External workload: dagster-pipes ONLY
|
|
│ ├── work.py
|
|
│ ├── requirements.txt
|
|
│ └── Dockerfile
|
|
├── documents/user-guide/ # AC1-AC4 documentation
|
|
├── yaml/ # Working example configuration
|
|
├── tests/
|
|
├── Dockerfile
|
|
├── pyproject.toml
|
|
└── workspace.yaml
|
|
```
|
|
|
|
## Getting started
|
|
|
|
Prerequisites: Python 3.12+ and `uv`.
|
|
|
|
```bash
|
|
uv sync --dev
|
|
uv run dagster dev -f src/distributed_execution/repository.py
|
|
```
|
|
|
|
The Dagster UI is then available at <http://localhost:3000>. Two jobs run end to
|
|
end on a laptop with no cluster: `tightly_coupled_in_process_job` and
|
|
`loosely_coupled_subprocess_job`. The Kubernetes variants of each require a
|
|
cluster and are documented in the user guide.
|
|
|
|
> `tightly_coupled_local_job` uses `multiprocess_executor`. It did not complete on
|
|
> a Windows development machine during authoring — see the Windows note in the
|
|
> user guide's section 5.5.
|
|
|
|
### Running tests
|
|
|
|
```bash
|
|
uv run pytest
|
|
```
|
|
|
|
### Building the images
|
|
|
|
Two images, deliberately: the code location and the external payload are
|
|
versioned and scanned independently.
|
|
|
|
```bash
|
|
docker build -t distributed-execution:0.1.0 .
|
|
docker build -f payload/Dockerfile -t distributed-execution-payload:0.1.0 payload/
|
|
```
|
|
|
|
## Status
|
|
|
|
Both execution targets are implemented. The tightly coupled jobs and the loosely
|
|
coupled **subprocess** transport are verified by the test suite. The loosely
|
|
coupled **Kubernetes** transport is implemented but has not yet been run against
|
|
a cluster; see the guide's *Outstanding work* section.
|
|
|
|
## Licence
|
|
|
|
European Union Public Licence v1.2 — see [LICENSE](LICENSE).
|