From c2a49f097115789cfe3ce550009147100c767762 Mon Sep 17 00:00:00 2001 From: ILay Date: Mon, 31 Aug 2026 16:12:57 +0200 Subject: [PATCH] [SIMPL-30451] Add sandbox deployment values and the pull-secret caveat yaml/sandbox/values-sandbox-gitea.yaml carries the registry coordinates, the pull secret both namespaces need, and the commit SHA the two images share. The sandbox registry rejects anonymous pulls, so a payload namespace without its own pull secret leaves the dispatching op waiting on a pod that never starts - readiness check L4 now says so. Changelog: added --- .../user-guide/distributed-execution-guide.md | 2 + documents/user-guide/readiness-checklist.md | 2 +- yaml/sandbox/values-sandbox-gitea.yaml | 57 +++++++++++++++++++ 3 files changed, 60 insertions(+), 1 deletion(-) create mode 100644 yaml/sandbox/values-sandbox-gitea.yaml diff --git a/documents/user-guide/distributed-execution-guide.md b/documents/user-guide/distributed-execution-guide.md index e81ac61..ab2b94f 100644 --- a/documents/user-guide/distributed-execution-guide.md +++ b/documents/user-guide/distributed-execution-guide.md @@ -395,6 +395,7 @@ setting. There is no UI control that switches a workflow between the two pattern | `yaml/tightly-coupled/values-run-launcher.yaml` | Run launcher type, namespace, pod env | | `yaml/tightly-coupled/rbac-step-executor.yaml` | Permissions required by `k8s_job_executor` | | `yaml/loosely-coupled/values-pipes-payload.yaml` | Payload image reference and target namespace | +| `yaml/sandbox/values-sandbox-gitea.yaml` | Sandbox registry coordinates, pull secret, and the SHA both images share | | `yaml/loosely-coupled/rbac-pipes-dispatch.yaml` | Permissions required by the dispatching pod | | `yaml/values-dagster-distributed-execution.yaml` | Code location image and entry point | @@ -515,6 +516,7 @@ Tracked under SIMPL-30451. | Both images build; payload image passes the isolation check (L11) | Tech details | Complete | | Locally runnable jobs launched from the Dagster UI, evidence recorded in section 5.6 | Tech details | Complete | | Payload image published to a container registry | Tech details | Complete on the sandbox Gitea registry; the GitLab registry still pending | +| GitLab pipeline builds both images | Tech details | **Pending** — the shared `ds.gitlab-ci.yml` template builds one image from the root Dockerfile | | End-to-end run of `loosely_coupled_k8s_job` on a cluster | Tech details | **Pending** | | Screenshots of UI surfaces | Tech details | **Pending** — needs a deployed platform instance, not a local dev server | | Platform architecture document update | Tech details | **Pending** | diff --git a/documents/user-guide/readiness-checklist.md b/documents/user-guide/readiness-checklist.md index d6e5956..f187b43 100644 --- a/documents/user-guide/readiness-checklist.md +++ b/documents/user-guide/readiness-checklist.md @@ -46,7 +46,7 @@ location image. | L1 | Payload contract is intact | `uv run pytest tests/test_loosely_coupled.py` | `test_payload_does_not_import_dagster` passes — the payload imports `dagster_pipes` only | | L2 | Pipes round trip works | Launch `loosely_coupled_subprocess_job` | Run reaches `SUCCESS`; run logs contain the payload's `External payload started on …` line, proving `pipes.log` crossed the channel | | L3 | Silence is treated as failure | Same test module | `test_silent_message_path_is_treated_as_failure` passes — an empty message list raises rather than yielding an empty result | -| L4 | Payload image is pullable by the target cluster | `kubectl -n run pull-probe --image= --restart=Never --command -- true` | Pod reaches `Completed`; no `ImagePullBackOff` | +| L4 | Payload image is pullable by the target cluster | `kubectl -n run pull-probe --image= --restart=Never --command -- true` | Pod reaches `Completed`; no `ImagePullBackOff`. The sandbox Gitea registry rejects anonymous pulls, so the payload namespace needs its own pull secret — see `yaml/sandbox/values-sandbox-gitea.yaml` | | L5 | Dispatcher can create Jobs in the payload namespace | `kubectl -n auth can-i create jobs --as=system:serviceaccount:dagster:dagster-dev` | Returns `yes` | | L6 | Dispatcher can read pod logs — **the message channel** | `kubectl -n auth can-i get pods/log --as=system:serviceaccount:dagster:dagster-dev` | Returns `yes`. A `no` here breaks reporting *without* failing the workload | | L7 | Payload Job is actually created | Launch `loosely_coupled_k8s_job`, then `kubectl -n get jobs -l app.kubernetes.io/name=distributed-execution-payload` | One Job per dispatch, labelled `dagster/execution-target=loosely-coupled` | diff --git a/yaml/sandbox/values-sandbox-gitea.yaml b/yaml/sandbox/values-sandbox-gitea.yaml new file mode 100644 index 0000000..883ee37 --- /dev/null +++ b/yaml/sandbox/values-sandbox-gitea.yaml @@ -0,0 +1,57 @@ +# Sandbox deployment values - Gitea registry. +# +# NOT CLUSTER-VERIFIED. These values are written against images that exist and a +# registry whose auth behaviour was checked, but they have not been applied to a +# Simpl cluster. Confirm and amend after the first deployment. +# +# Use these instead of the code.europa.eu references in +# yaml/values-dagster-distributed-execution.yaml and +# yaml/loosely-coupled/values-pipes-payload.yaml when deploying to the +# dataprovider01 sandbox. Both images are published by +# .gitea/workflows/docker-publish.yml. +# +# The tag below is a short commit SHA and is the same for both images. That is +# the version lock: the code location and the payload it dispatches must come +# from one commit. Bump both together or not at all. +# +# The sandbox Gitea registry requires authentication - an anonymous manifest GET +# returns 401 - so a pull secret is required in every namespace that pulls either +# image. Create it with a Gitea access token that has read:package scope: +# +# kubectl -n create secret docker-registry gitea-registry \ +# --docker-server=gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu \ +# --docker-username= \ +# --docker-password= +# +# The payload namespace needs it too: PipesK8sClient creates that Job, and a +# missing pull secret there leaves the op waiting on a pod that never starts, +# which surfaces as pod_wait_timeout rather than as an image error. + +dagster: + dagster-user-deployments: + deployments: + - name: distributed-execution + image: + repository: gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/j.r/distributed-execution + tag: 5122da4691f9 + pullPolicy: IfNotPresent + imagePullSecrets: + - name: gitea-registry + env: + - name: PIPES_PAYLOAD_IMAGE + value: gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/j.r/distributed-execution-payload:5122da4691f9 + - name: PIPES_PAYLOAD_NAMESPACE + value: dagster + + runLauncher: + config: + k8sRunLauncher: + imagePullSecrets: + - name: gitea-registry + runK8sConfig: + containerConfig: + env: + - name: PIPES_PAYLOAD_IMAGE + value: gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/j.r/distributed-execution-payload:5122da4691f9 + - name: PIPES_PAYLOAD_NAMESPACE + value: dagster