[SIMPL-30451] Distributed execution reference service and container-cluster configuration guide #1

Merged
j.r merged 23 commits from feature/SIMPL-30451-distributed-execution into develop 2026-09-02 17:38:23 +00:00
3 changed files with 60 additions and 1 deletions
Showing only changes of commit c2a49f0971 - Show all commits

View File

@@ -395,6 +395,7 @@ setting. There is no UI control that switches a workflow between the two pattern
| `yaml/tightly-coupled/values-run-launcher.yaml` | Run launcher type, namespace, pod env |
| `yaml/tightly-coupled/rbac-step-executor.yaml` | Permissions required by `k8s_job_executor` |
| `yaml/loosely-coupled/values-pipes-payload.yaml` | Payload image reference and target namespace |
| `yaml/sandbox/values-sandbox-gitea.yaml` | Sandbox registry coordinates, pull secret, and the SHA both images share |
| `yaml/loosely-coupled/rbac-pipes-dispatch.yaml` | Permissions required by the dispatching pod |
| `yaml/values-dagster-distributed-execution.yaml` | Code location image and entry point |
@@ -515,6 +516,7 @@ Tracked under SIMPL-30451.
| Both images build; payload image passes the isolation check (L11) | Tech details | Complete |
| Locally runnable jobs launched from the Dagster UI, evidence recorded in section 5.6 | Tech details | Complete |
| Payload image published to a container registry | Tech details | Complete on the sandbox Gitea registry; the GitLab registry still pending |
| GitLab pipeline builds both images | Tech details | **Pending** — the shared `ds.gitlab-ci.yml` template builds one image from the root Dockerfile |
| End-to-end run of `loosely_coupled_k8s_job` on a cluster | Tech details | **Pending** |
| Screenshots of UI surfaces | Tech details | **Pending** — needs a deployed platform instance, not a local dev server |
| Platform architecture document update | Tech details | **Pending** |

View File

@@ -46,7 +46,7 @@ location image.
| L1 | Payload contract is intact | `uv run pytest tests/test_loosely_coupled.py` | `test_payload_does_not_import_dagster` passes — the payload imports `dagster_pipes` only |
| L2 | Pipes round trip works | Launch `loosely_coupled_subprocess_job` | Run reaches `SUCCESS`; run logs contain the payload's `External payload started on …` line, proving `pipes.log` crossed the channel |
| L3 | Silence is treated as failure | Same test module | `test_silent_message_path_is_treated_as_failure` passes — an empty message list raises rather than yielding an empty result |
| L4 | Payload image is pullable by the target cluster | `kubectl -n <payload-ns> run pull-probe --image=<payload-image> --restart=Never --command -- true` | Pod reaches `Completed`; no `ImagePullBackOff` |
| L4 | Payload image is pullable by the target cluster | `kubectl -n <payload-ns> run pull-probe --image=<payload-image> --restart=Never --command -- true` | Pod reaches `Completed`; no `ImagePullBackOff`. The sandbox Gitea registry rejects anonymous pulls, so the payload namespace needs its own pull secret — see `yaml/sandbox/values-sandbox-gitea.yaml` |
| L5 | Dispatcher can create Jobs in the payload namespace | `kubectl -n <payload-ns> auth can-i create jobs --as=system:serviceaccount:dagster:dagster-dev` | Returns `yes` |
| L6 | Dispatcher can read pod logs — **the message channel** | `kubectl -n <payload-ns> auth can-i get pods/log --as=system:serviceaccount:dagster:dagster-dev` | Returns `yes`. A `no` here breaks reporting *without* failing the workload |
| L7 | Payload Job is actually created | Launch `loosely_coupled_k8s_job`, then `kubectl -n <payload-ns> get jobs -l app.kubernetes.io/name=distributed-execution-payload` | One Job per dispatch, labelled `dagster/execution-target=loosely-coupled` |

View File

@@ -0,0 +1,57 @@
# Sandbox deployment values - Gitea registry.
#
# NOT CLUSTER-VERIFIED. These values are written against images that exist and a
# registry whose auth behaviour was checked, but they have not been applied to a
# Simpl cluster. Confirm and amend after the first deployment.
#
# Use these instead of the code.europa.eu references in
# yaml/values-dagster-distributed-execution.yaml and
# yaml/loosely-coupled/values-pipes-payload.yaml when deploying to the
# dataprovider01 sandbox. Both images are published by
# .gitea/workflows/docker-publish.yml.
#
# The tag below is a short commit SHA and is the same for both images. That is
# the version lock: the code location and the payload it dispatches must come
# from one commit. Bump both together or not at all.
#
# The sandbox Gitea registry requires authentication - an anonymous manifest GET
# returns 401 - so a pull secret is required in every namespace that pulls either
# image. Create it with a Gitea access token that has read:package scope:
#
# kubectl -n <namespace> create secret docker-registry gitea-registry \
# --docker-server=gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu \
# --docker-username=<gitea-user> \
# --docker-password=<gitea-token>
#
# The payload namespace needs it too: PipesK8sClient creates that Job, and a
# missing pull secret there leaves the op waiting on a pod that never starts,
# which surfaces as pod_wait_timeout rather than as an image error.
dagster:
dagster-user-deployments:
deployments:
- name: distributed-execution
image:
repository: gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/j.r/distributed-execution
tag: 5122da4691f9
pullPolicy: IfNotPresent
imagePullSecrets:
- name: gitea-registry
env:
- name: PIPES_PAYLOAD_IMAGE
value: gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/j.r/distributed-execution-payload:5122da4691f9
- name: PIPES_PAYLOAD_NAMESPACE
value: dagster
runLauncher:
config:
k8sRunLauncher:
imagePullSecrets:
- name: gitea-registry
runK8sConfig:
containerConfig:
env:
- name: PIPES_PAYLOAD_IMAGE
value: gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/j.r/distributed-execution-payload:5122da4691f9
- name: PIPES_PAYLOAD_NAMESPACE
value: dagster