9b1b7056b0
Check the chart's actual DAGSTER_PG_PASSWORD injection in the tightly coupled preflight instead of DAGSTER_POSTGRES_*, which the Simpl chart never sets; the old check warned about a misconfiguration on a correctly configured run pod. Evidence reports the variable name only, never its value. Record the platform k8s_job_executor run as cluster-verified in the guide and readiness checklist, add the Dagster UI screenshots covering job list, graph, code-configured tags and a successful run. Changelog: fixed
99 lines
3.8 KiB
Markdown
99 lines
3.8 KiB
Markdown
# Distributed Execution
|
|
|
|
Canonical location for documentation, example workflows and reference service
|
|
implementations covering **distributed execution patterns** in the Simpl
|
|
orchestration platform.
|
|
|
|
The service answers one question for every workflow: *where does the compute
|
|
actually run, and how does runtime information get back to the control plane?*
|
|
|
|
## Contents
|
|
|
|
| Area | Location | Status |
|
|
|---|---|---|
|
|
| User guide | [documents/user-guide/distributed-execution-guide.md](documents/user-guide/distributed-execution-guide.md) | Complete |
|
|
| Readiness checklist | [documents/user-guide/readiness-checklist.md](documents/user-guide/readiness-checklist.md) | Complete |
|
|
| Tightly coupled reference | [src/distributed_execution/tightly_coupled/jobs.py](src/distributed_execution/tightly_coupled/jobs.py) | Runnable |
|
|
| Loosely coupled reference | [src/distributed_execution/loosely_coupled/jobs.py](src/distributed_execution/loosely_coupled/jobs.py) | Runnable and cluster-verified |
|
|
| External payload | [payload/work.py](payload/work.py) | Runnable |
|
|
| Example configuration | [yaml/](yaml/) | Complete |
|
|
|
|
## Project structure
|
|
|
|
```text
|
|
distributed-execution/
|
|
├── src/
|
|
│ └── distributed_execution/
|
|
│ ├── repository.py # Dagster definitions (entry point)
|
|
│ ├── ops.py # Shared ops used by both patterns
|
|
│ ├── preflight.py # Readiness checks backing the checklist
|
|
│ ├── tightly_coupled/
|
|
│ │ └── jobs.py # in_process / multiprocess / k8s_job_executor
|
|
│ └── loosely_coupled/
|
|
│ └── jobs.py # PipesSubprocessClient / PipesK8sClient
|
|
├── payload/ # External workload: dagster-pipes ONLY
|
|
│ ├── work.py
|
|
│ ├── requirements.txt
|
|
│ └── Dockerfile
|
|
├── documents/user-guide/ # AC1-AC4 documentation
|
|
├── yaml/ # Working example configuration
|
|
├── tests/
|
|
├── Dockerfile
|
|
├── pyproject.toml
|
|
└── workspace.yaml
|
|
```
|
|
|
|
## Getting started
|
|
|
|
Prerequisites: Python 3.12+ and `uv`.
|
|
|
|
```bash
|
|
uv sync --dev
|
|
uv run dagster dev -f src/distributed_execution/repository.py
|
|
```
|
|
|
|
The Dagster UI is then available at <http://localhost:3000>. Two jobs run end to
|
|
end on a laptop with no cluster: `tightly_coupled_in_process_job` and
|
|
`loosely_coupled_subprocess_job`. The Kubernetes variants of each require a
|
|
cluster and are documented in the user guide.
|
|
|
|
> `tightly_coupled_local_job` uses `multiprocess_executor`. It did not complete on
|
|
> a Windows development machine during authoring — see the Windows note in the
|
|
> user guide's section 5.5.
|
|
|
|
### Running tests
|
|
|
|
```bash
|
|
uv run pytest
|
|
```
|
|
|
|
### Building the images
|
|
|
|
Two images, deliberately: the code location and the external payload are
|
|
versioned and scanned independently.
|
|
|
|
```bash
|
|
docker build -t distributed-execution:0.1.0 .
|
|
docker build -f payload/Dockerfile -t distributed-execution-payload:0.1.0 payload/
|
|
```
|
|
|
|
Both images build and have been smoke tested locally: the code location image
|
|
loads its definitions, and the payload image contains `dagster_pipes` without
|
|
`dagster` — check L11 in the readiness checklist.
|
|
|
|
Both must be tagged from the same commit. That shared tag is what keeps a code
|
|
location and the payload it dispatches on the same version, and nothing at
|
|
runtime checks the pairing — see the *Outstanding work* section for what the
|
|
pipeline still has to learn.
|
|
|
|
## Status
|
|
|
|
Both execution targets are implemented. Local variants are verified by the test
|
|
suite, `tightly_coupled_k8s_job` has run end to end through the platform launcher,
|
|
and `loosely_coupled_k8s_job` has run end to end through the standalone cluster
|
|
probe. See the guide and readiness checklist for the recorded evidence.
|
|
|
|
## Licence
|
|
|
|
European Union Public Licence v1.2 — see [LICENSE](LICENSE).
|