2bbecb1bdd
The previous revision claimed C6 failed on the sandbox because job_namespace is dagster while the launcher's ConfigMap, Secret and PVC live in dataprovider01. That read job_namespace in isolation. Each code location ships a DAGSTER_CONTAINER_CONTEXT whose k8s.namespace is dataprovider01, and the code location context is merged over the launcher's, so it wins - dagster_k8s/container_context.py resolves namespace as 'other.namespace if other.namespace else self.namespace'. Both deployed code locations carry it, so run pods land where their dependencies are. Still true: no run has ever executed on that cluster. That now looks like nothing having been launched rather than anything being broken. The residual risk is a code location reaching the launcher without a container context namespace, which the chart always sets, so it bites only hand-rolled pods. C5 now warns that job_namespace is a default rather than the effective namespace. Changelog: fixed
Distributed Execution
Canonical location for documentation, example workflows and reference service implementations covering distributed execution patterns in the Simpl orchestration platform.
The service answers one question for every workflow: where does the compute actually run, and how does runtime information get back to the control plane?
Contents
| Area | Location | Status |
|---|---|---|
| User guide | documents/user-guide/distributed-execution-guide.md | Complete |
| Readiness checklist | documents/user-guide/readiness-checklist.md | Complete |
| Tightly coupled reference | src/distributed_execution/tightly_coupled/jobs.py | Runnable |
| Loosely coupled reference | src/distributed_execution/loosely_coupled/jobs.py | Runnable (subprocess verified) |
| External payload | payload/work.py | Runnable |
| Example configuration | yaml/ | Complete |
Project structure
distributed-execution/
├── src/
│ └── distributed_execution/
│ ├── repository.py # Dagster definitions (entry point)
│ ├── ops.py # Shared ops used by both patterns
│ ├── preflight.py # Readiness checks backing the checklist
│ ├── tightly_coupled/
│ │ └── jobs.py # in_process / multiprocess / k8s_job_executor
│ └── loosely_coupled/
│ └── jobs.py # PipesSubprocessClient / PipesK8sClient
├── payload/ # External workload: dagster-pipes ONLY
│ ├── work.py
│ ├── requirements.txt
│ └── Dockerfile
├── documents/user-guide/ # AC1-AC4 documentation
├── yaml/ # Working example configuration
├── tests/
├── Dockerfile
├── pyproject.toml
└── workspace.yaml
Getting started
Prerequisites: Python 3.12+ and uv.
uv sync --dev
uv run dagster dev -f src/distributed_execution/repository.py
The Dagster UI is then available at http://localhost:3000. Two jobs run end to
end on a laptop with no cluster: tightly_coupled_in_process_job and
loosely_coupled_subprocess_job. The Kubernetes variants of each require a
cluster and are documented in the user guide.
tightly_coupled_local_jobusesmultiprocess_executor. It did not complete on a Windows development machine during authoring — see the Windows note in the user guide's section 5.5.
Running tests
uv run pytest
Building the images
Two images, deliberately: the code location and the external payload are versioned and scanned independently.
docker build -t distributed-execution:0.1.0 .
docker build -f payload/Dockerfile -t distributed-execution-payload:0.1.0 payload/
Both images build and have been smoke tested locally: the code location image
loads its definitions, and the payload image contains dagster_pipes without
dagster — check L11 in the readiness checklist.
Both must be tagged from the same commit. That shared tag is what keeps a code location and the payload it dispatches on the same version, and nothing at runtime checks the pairing — see the Outstanding work section for what the pipeline still has to learn.
Status
Both execution targets are implemented. The tightly coupled jobs and the loosely coupled subprocess transport are verified by the test suite. The loosely coupled Kubernetes transport is implemented but has not yet been run against a cluster; see the guide's Outstanding work section.
Licence
European Union Public Licence v1.2 — see LICENSE.