[SIMPL-30451] Add distributed-execution service with guide and reference implementations
Canonical location for documentation, example workflows and reference service implementations covering distributed execution patterns. Covers AC1-AC4: execution-target selection, decision support, readiness checks and code-level linkage. Tightly coupled jobs and the loosely coupled subprocess transport are verified by the test suite; the Kubernetes pipes transport is implemented but not yet cluster-run and is marked as such in the guide. Changelog: added
This commit is contained in:
32
yaml/loosely-coupled/values-pipes-payload.yaml
Normal file
32
yaml/loosely-coupled/values-pipes-payload.yaml
Normal file
@@ -0,0 +1,32 @@
|
||||
# Loosely coupled execution target - code location overrides.
|
||||
#
|
||||
# Merge into the dagster-user-deployments entry for this code location. These
|
||||
# values point the pipes client at the payload image and target namespace.
|
||||
#
|
||||
# Contrast with yaml/tightly-coupled/values-run-launcher.yaml: there, every
|
||||
# execution pod needed S3 and Vault credentials. Here the payload needs none -
|
||||
# only the dispatching pod does.
|
||||
|
||||
dagster:
|
||||
dagster-user-deployments:
|
||||
deployments:
|
||||
- name: distributed-execution
|
||||
envSecrets: []
|
||||
env:
|
||||
- name: PIPES_PAYLOAD_IMAGE
|
||||
value: code.europa.eu:4567/simpl/simpl-open/data/supporting-data-services/distributed-execution/payload:0.1.0
|
||||
- name: PIPES_PAYLOAD_NAMESPACE
|
||||
value: dagster
|
||||
|
||||
# The dispatching RUN pod creates the payload Job, so the image reference and
|
||||
# namespace must also be visible to run pods, not just the code server.
|
||||
runLauncher:
|
||||
config:
|
||||
k8sRunLauncher:
|
||||
runK8sConfig:
|
||||
containerConfig:
|
||||
env:
|
||||
- name: PIPES_PAYLOAD_IMAGE
|
||||
value: code.europa.eu:4567/simpl/simpl-open/data/supporting-data-services/distributed-execution/payload:0.1.0
|
||||
- name: PIPES_PAYLOAD_NAMESPACE
|
||||
value: dagster
|
||||
Reference in New Issue
Block a user