43dbb81a95
Canonical location for documentation, example workflows and reference service implementations covering distributed execution patterns. Covers AC1-AC4: execution-target selection, decision support, readiness checks and code-level linkage. Tightly coupled jobs and the loosely coupled subprocess transport are verified by the test suite; the Kubernetes pipes transport is implemented but not yet cluster-run and is marked as such in the guide. Changelog: added
33 lines
1.2 KiB
YAML
33 lines
1.2 KiB
YAML
# Loosely coupled execution target - code location overrides.
|
|
#
|
|
# Merge into the dagster-user-deployments entry for this code location. These
|
|
# values point the pipes client at the payload image and target namespace.
|
|
#
|
|
# Contrast with yaml/tightly-coupled/values-run-launcher.yaml: there, every
|
|
# execution pod needed S3 and Vault credentials. Here the payload needs none -
|
|
# only the dispatching pod does.
|
|
|
|
dagster:
|
|
dagster-user-deployments:
|
|
deployments:
|
|
- name: distributed-execution
|
|
envSecrets: []
|
|
env:
|
|
- name: PIPES_PAYLOAD_IMAGE
|
|
value: code.europa.eu:4567/simpl/simpl-open/data/supporting-data-services/distributed-execution/payload:0.1.0
|
|
- name: PIPES_PAYLOAD_NAMESPACE
|
|
value: dagster
|
|
|
|
# The dispatching RUN pod creates the payload Job, so the image reference and
|
|
# namespace must also be visible to run pods, not just the code server.
|
|
runLauncher:
|
|
config:
|
|
k8sRunLauncher:
|
|
runK8sConfig:
|
|
containerConfig:
|
|
env:
|
|
- name: PIPES_PAYLOAD_IMAGE
|
|
value: code.europa.eu:4567/simpl/simpl-open/data/supporting-data-services/distributed-execution/payload:0.1.0
|
|
- name: PIPES_PAYLOAD_NAMESPACE
|
|
value: dagster
|