[SIMPL-30451] Add distributed-execution service with guide and reference implementations
Canonical location for documentation, example workflows and reference service implementations covering distributed execution patterns. Covers AC1-AC4: execution-target selection, decision support, readiness checks and code-level linkage. Tightly coupled jobs and the loosely coupled subprocess transport are verified by the test suite; the Kubernetes pipes transport is implemented but not yet cluster-run and is marked as such in the guide. Changelog: added
This commit is contained in:
41
yaml/loosely-coupled/rbac-pipes-dispatch.yaml
Normal file
41
yaml/loosely-coupled/rbac-pipes-dispatch.yaml
Normal file
@@ -0,0 +1,41 @@
|
||||
# Kubernetes RBAC for the loosely coupled execution target.
|
||||
#
|
||||
# Granted to the service account running the Dagster RUN pods - the pod that
|
||||
# calls PipesK8sClient.run() is the one that creates and watches the payload Job.
|
||||
#
|
||||
# Note what is NOT here: the payload itself needs no RBAC, no database
|
||||
# credentials and no Vault role. Its only channel back to the control plane is
|
||||
# the pod log stream, which the dispatching pod reads.
|
||||
#
|
||||
# If the payload runs in a different namespace from the dispatcher, apply this
|
||||
# Role in the PAYLOAD namespace and keep the RoleBinding subject pointing at the
|
||||
# dispatcher's service account.
|
||||
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: distributed-execution-pipes-dispatcher
|
||||
namespace: dagster
|
||||
rules:
|
||||
- apiGroups: ["batch"]
|
||||
resources: ["jobs", "jobs/status"]
|
||||
verbs: ["create", "get", "list", "watch", "delete"]
|
||||
# pods/log is the message channel for PipesK8sPodLogsMessageReader.
|
||||
# Without it the payload runs but reports nothing.
|
||||
- apiGroups: [""]
|
||||
resources: ["pods", "pods/log", "pods/status"]
|
||||
verbs: ["get", "list", "watch"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: distributed-execution-pipes-dispatcher
|
||||
namespace: dagster
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: dagster-dev
|
||||
namespace: dagster
|
||||
roleRef:
|
||||
kind: Role
|
||||
name: distributed-execution-pipes-dispatcher
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
Reference in New Issue
Block a user