[SIMPL-30451] Add distributed-execution service with guide and reference implementations
Canonical location for documentation, example workflows and reference service implementations covering distributed execution patterns. Covers AC1-AC4: execution-target selection, decision support, readiness checks and code-level linkage. Tightly coupled jobs and the loosely coupled subprocess transport are verified by the test suite; the Kubernetes pipes transport is implemented but not yet cluster-run and is marked as such in the guide. Changelog: added
This commit is contained in:
85
README.md
Normal file
85
README.md
Normal file
@@ -0,0 +1,85 @@
|
||||
# Distributed Execution
|
||||
|
||||
Canonical location for documentation, example workflows and reference service
|
||||
implementations covering **distributed execution patterns** in the Simpl
|
||||
orchestration platform.
|
||||
|
||||
The service answers one question for every workflow: *where does the compute
|
||||
actually run, and how does runtime information get back to the control plane?*
|
||||
|
||||
## Contents
|
||||
|
||||
| Area | Location | Status |
|
||||
|---|---|---|
|
||||
| User guide | [documents/user-guide/distributed-execution-guide.md](documents/user-guide/distributed-execution-guide.md) | Complete |
|
||||
| Readiness checklist | [documents/user-guide/readiness-checklist.md](documents/user-guide/readiness-checklist.md) | Complete |
|
||||
| Tightly coupled reference | [src/distributed_execution/tightly_coupled/jobs.py](src/distributed_execution/tightly_coupled/jobs.py) | Runnable |
|
||||
| Loosely coupled reference | [src/distributed_execution/loosely_coupled/jobs.py](src/distributed_execution/loosely_coupled/jobs.py) | Runnable (subprocess verified) |
|
||||
| External payload | [payload/work.py](payload/work.py) | Runnable |
|
||||
| Example configuration | [yaml/](yaml/) | Complete |
|
||||
|
||||
## Project structure
|
||||
|
||||
```text
|
||||
distributed-execution/
|
||||
├── src/
|
||||
│ └── distributed_execution/
|
||||
│ ├── repository.py # Dagster definitions (entry point)
|
||||
│ ├── ops.py # Shared ops used by both patterns
|
||||
│ ├── preflight.py # Readiness checks backing the checklist
|
||||
│ ├── tightly_coupled/
|
||||
│ │ └── jobs.py # in_process / multiprocess / k8s_job_executor
|
||||
│ └── loosely_coupled/
|
||||
│ └── jobs.py # PipesSubprocessClient / PipesK8sClient
|
||||
├── payload/ # External workload: dagster-pipes ONLY
|
||||
│ ├── work.py
|
||||
│ ├── requirements.txt
|
||||
│ └── Dockerfile
|
||||
├── documents/user-guide/ # AC1-AC4 documentation
|
||||
├── yaml/ # Working example configuration
|
||||
├── tests/
|
||||
├── Dockerfile
|
||||
├── pyproject.toml
|
||||
└── workspace.yaml
|
||||
```
|
||||
|
||||
## Getting started
|
||||
|
||||
Prerequisites: Python 3.12+ and `uv`.
|
||||
|
||||
```bash
|
||||
uv sync --dev
|
||||
uv run dagster dev -f src/distributed_execution/repository.py
|
||||
```
|
||||
|
||||
The Dagster UI is then available at <http://localhost:3000>. Two jobs run end to
|
||||
end on a laptop with no cluster: `tightly_coupled_local_job` and
|
||||
`loosely_coupled_subprocess_job`. The Kubernetes variants of each are documented
|
||||
in the user guide.
|
||||
|
||||
### Running tests
|
||||
|
||||
```bash
|
||||
uv run pytest
|
||||
```
|
||||
|
||||
### Building the images
|
||||
|
||||
Two images, deliberately: the code location and the external payload are
|
||||
versioned and scanned independently.
|
||||
|
||||
```bash
|
||||
docker build -t distributed-execution:0.1.0 .
|
||||
docker build -f payload/Dockerfile -t distributed-execution-payload:0.1.0 payload/
|
||||
```
|
||||
|
||||
## Status
|
||||
|
||||
Both execution targets are implemented. The tightly coupled jobs and the loosely
|
||||
coupled **subprocess** transport are verified by the test suite. The loosely
|
||||
coupled **Kubernetes** transport is implemented but has not yet been run against
|
||||
a cluster; see the guide's *Outstanding work* section.
|
||||
|
||||
## Licence
|
||||
|
||||
European Union Public Licence v1.2 — see [LICENSE](LICENSE).
|
||||
Reference in New Issue
Block a user