899429b10a
This repository is built by the shared ds.gitlab-ci.yml template; carrying a second CI system alongside it invited the reader to wonder which one is authoritative. The Gitea Actions workflow that builds both images stays in the sandbox Gitea mirror, which is where the sandbox images are produced. The README, the pipes payload values and the sandbox values no longer point at a file this repository does not contain. Changelog: removed
99 lines
3.8 KiB
Markdown
99 lines
3.8 KiB
Markdown
# Distributed Execution
|
|
|
|
Canonical location for documentation, example workflows and reference service
|
|
implementations covering **distributed execution patterns** in the Simpl
|
|
orchestration platform.
|
|
|
|
The service answers one question for every workflow: *where does the compute
|
|
actually run, and how does runtime information get back to the control plane?*
|
|
|
|
## Contents
|
|
|
|
| Area | Location | Status |
|
|
|---|---|---|
|
|
| User guide | [documents/user-guide/distributed-execution-guide.md](documents/user-guide/distributed-execution-guide.md) | Complete |
|
|
| Readiness checklist | [documents/user-guide/readiness-checklist.md](documents/user-guide/readiness-checklist.md) | Complete |
|
|
| Tightly coupled reference | [src/distributed_execution/tightly_coupled/jobs.py](src/distributed_execution/tightly_coupled/jobs.py) | Runnable |
|
|
| Loosely coupled reference | [src/distributed_execution/loosely_coupled/jobs.py](src/distributed_execution/loosely_coupled/jobs.py) | Runnable (subprocess verified) |
|
|
| External payload | [payload/work.py](payload/work.py) | Runnable |
|
|
| Example configuration | [yaml/](yaml/) | Complete |
|
|
|
|
## Project structure
|
|
|
|
```text
|
|
distributed-execution/
|
|
├── src/
|
|
│ └── distributed_execution/
|
|
│ ├── repository.py # Dagster definitions (entry point)
|
|
│ ├── ops.py # Shared ops used by both patterns
|
|
│ ├── preflight.py # Readiness checks backing the checklist
|
|
│ ├── tightly_coupled/
|
|
│ │ └── jobs.py # in_process / multiprocess / k8s_job_executor
|
|
│ └── loosely_coupled/
|
|
│ └── jobs.py # PipesSubprocessClient / PipesK8sClient
|
|
├── payload/ # External workload: dagster-pipes ONLY
|
|
│ ├── work.py
|
|
│ ├── requirements.txt
|
|
│ └── Dockerfile
|
|
├── documents/user-guide/ # AC1-AC4 documentation
|
|
├── yaml/ # Working example configuration
|
|
├── tests/
|
|
├── Dockerfile
|
|
├── pyproject.toml
|
|
└── workspace.yaml
|
|
```
|
|
|
|
## Getting started
|
|
|
|
Prerequisites: Python 3.12+ and `uv`.
|
|
|
|
```bash
|
|
uv sync --dev
|
|
uv run dagster dev -f src/distributed_execution/repository.py
|
|
```
|
|
|
|
The Dagster UI is then available at <http://localhost:3000>. Two jobs run end to
|
|
end on a laptop with no cluster: `tightly_coupled_in_process_job` and
|
|
`loosely_coupled_subprocess_job`. The Kubernetes variants of each require a
|
|
cluster and are documented in the user guide.
|
|
|
|
> `tightly_coupled_local_job` uses `multiprocess_executor`. It did not complete on
|
|
> a Windows development machine during authoring — see the Windows note in the
|
|
> user guide's section 5.5.
|
|
|
|
### Running tests
|
|
|
|
```bash
|
|
uv run pytest
|
|
```
|
|
|
|
### Building the images
|
|
|
|
Two images, deliberately: the code location and the external payload are
|
|
versioned and scanned independently.
|
|
|
|
```bash
|
|
docker build -t distributed-execution:0.1.0 .
|
|
docker build -f payload/Dockerfile -t distributed-execution-payload:0.1.0 payload/
|
|
```
|
|
|
|
Both images build and have been smoke tested locally: the code location image
|
|
loads its definitions, and the payload image contains `dagster_pipes` without
|
|
`dagster` — check L11 in the readiness checklist.
|
|
|
|
Both must be tagged from the same commit. That shared tag is what keeps a code
|
|
location and the payload it dispatches on the same version, and nothing at
|
|
runtime checks the pairing — see the *Outstanding work* section for what the
|
|
pipeline still has to learn.
|
|
|
|
## Status
|
|
|
|
Both execution targets are implemented. The tightly coupled jobs and the loosely
|
|
coupled **subprocess** transport are verified by the test suite. The loosely
|
|
coupled **Kubernetes** transport is implemented but has not yet been run against
|
|
a cluster; see the guide's *Outstanding work* section.
|
|
|
|
## Licence
|
|
|
|
European Union Public Licence v1.2 — see [LICENSE](LICENSE).
|