j.r/distributed-execution
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[SIMPL-30451] Add distributed-execution service with guide and reference implementations

Browse Source 
Canonical location for documentation, example workflows and reference service implementations covering distributed execution patterns. Covers AC1-AC4: execution-target selection, decision support, readiness checks and code-level linkage. Tightly coupled jobs and the loosely coupled subprocess transport are verified by the test suite; the Kubernetes pipes transport is implemented but not yet cluster-run and is marked as such in the guide.

Changelog: added
This commit is contained in:
ILay
2026-08-26 18:07:06 +02:00
commit 43dbb81a95
32 changed files with 4071 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
CHANGELOG.md Normal file
View File

@@ -0,0 +1,18 @@
## 0.1.0 (unreleased)
### added
- Repository skeleton for the `distributed-execution` supporting data service.
- User guide covering execution-target selection, decision support and readiness checks.
- Runnable tightly coupled reference implementations (`in_process`, `multiprocess`, `k8s_job_executor`).
- Runnable loosely coupled reference implementations via dagster-pipes
(`PipesSubprocessClient`, `PipesK8sClient`) with a standalone external payload image.
- Working example configuration and RBAC for both execution targets.
### pending
- Publish the payload image to the container registry.
- End-to-end run of `loosely_coupled_k8s_job` against a cluster, and confirmation
of the Kubernetes rows in the readiness checklist.
- Screenshots of UI surfaces and integration-pattern diagrams.
- Platform architecture document update.