[SIMPL-30451] Add sandbox deployment values and the pull-secret caveat
yaml/sandbox/values-sandbox-gitea.yaml carries the registry coordinates, the pull secret both namespaces need, and the commit SHA the two images share. The sandbox registry rejects anonymous pulls, so a payload namespace without its own pull secret leaves the dispatching op waiting on a pod that never starts - readiness check L4 now says so. Changelog: added
This commit is contained in:
@@ -395,6 +395,7 @@ setting. There is no UI control that switches a workflow between the two pattern
|
||||
| `yaml/tightly-coupled/values-run-launcher.yaml` | Run launcher type, namespace, pod env |
|
||||
| `yaml/tightly-coupled/rbac-step-executor.yaml` | Permissions required by `k8s_job_executor` |
|
||||
| `yaml/loosely-coupled/values-pipes-payload.yaml` | Payload image reference and target namespace |
|
||||
| `yaml/sandbox/values-sandbox-gitea.yaml` | Sandbox registry coordinates, pull secret, and the SHA both images share |
|
||||
| `yaml/loosely-coupled/rbac-pipes-dispatch.yaml` | Permissions required by the dispatching pod |
|
||||
| `yaml/values-dagster-distributed-execution.yaml` | Code location image and entry point |
|
||||
|
||||
@@ -515,6 +516,7 @@ Tracked under SIMPL-30451.
|
||||
| Both images build; payload image passes the isolation check (L11) | Tech details | Complete |
|
||||
| Locally runnable jobs launched from the Dagster UI, evidence recorded in section 5.6 | Tech details | Complete |
|
||||
| Payload image published to a container registry | Tech details | Complete on the sandbox Gitea registry; the GitLab registry still pending |
|
||||
| GitLab pipeline builds both images | Tech details | **Pending** — the shared `ds.gitlab-ci.yml` template builds one image from the root Dockerfile |
|
||||
| End-to-end run of `loosely_coupled_k8s_job` on a cluster | Tech details | **Pending** |
|
||||
| Screenshots of UI surfaces | Tech details | **Pending** — needs a deployed platform instance, not a local dev server |
|
||||
| Platform architecture document update | Tech details | **Pending** |
|
||||
|
||||
@@ -46,7 +46,7 @@ location image.
|
||||
| L1 | Payload contract is intact | `uv run pytest tests/test_loosely_coupled.py` | `test_payload_does_not_import_dagster` passes — the payload imports `dagster_pipes` only |
|
||||
| L2 | Pipes round trip works | Launch `loosely_coupled_subprocess_job` | Run reaches `SUCCESS`; run logs contain the payload's `External payload started on …` line, proving `pipes.log` crossed the channel |
|
||||
| L3 | Silence is treated as failure | Same test module | `test_silent_message_path_is_treated_as_failure` passes — an empty message list raises rather than yielding an empty result |
|
||||
| L4 | Payload image is pullable by the target cluster | `kubectl -n <payload-ns> run pull-probe --image=<payload-image> --restart=Never --command -- true` | Pod reaches `Completed`; no `ImagePullBackOff` |
|
||||
| L4 | Payload image is pullable by the target cluster | `kubectl -n <payload-ns> run pull-probe --image=<payload-image> --restart=Never --command -- true` | Pod reaches `Completed`; no `ImagePullBackOff`. The sandbox Gitea registry rejects anonymous pulls, so the payload namespace needs its own pull secret — see `yaml/sandbox/values-sandbox-gitea.yaml` |
|
||||
| L5 | Dispatcher can create Jobs in the payload namespace | `kubectl -n <payload-ns> auth can-i create jobs --as=system:serviceaccount:dagster:dagster-dev` | Returns `yes` |
|
||||
| L6 | Dispatcher can read pod logs — **the message channel** | `kubectl -n <payload-ns> auth can-i get pods/log --as=system:serviceaccount:dagster:dagster-dev` | Returns `yes`. A `no` here breaks reporting *without* failing the workload |
|
||||
| L7 | Payload Job is actually created | Launch `loosely_coupled_k8s_job`, then `kubectl -n <payload-ns> get jobs -l app.kubernetes.io/name=distributed-execution-payload` | One Job per dispatch, labelled `dagster/execution-target=loosely-coupled` |
|
||||
|
||||
57
yaml/sandbox/values-sandbox-gitea.yaml
Normal file
57
yaml/sandbox/values-sandbox-gitea.yaml
Normal file
@@ -0,0 +1,57 @@
|
||||
# Sandbox deployment values - Gitea registry.
|
||||
#
|
||||
# NOT CLUSTER-VERIFIED. These values are written against images that exist and a
|
||||
# registry whose auth behaviour was checked, but they have not been applied to a
|
||||
# Simpl cluster. Confirm and amend after the first deployment.
|
||||
#
|
||||
# Use these instead of the code.europa.eu references in
|
||||
# yaml/values-dagster-distributed-execution.yaml and
|
||||
# yaml/loosely-coupled/values-pipes-payload.yaml when deploying to the
|
||||
# dataprovider01 sandbox. Both images are published by
|
||||
# .gitea/workflows/docker-publish.yml.
|
||||
#
|
||||
# The tag below is a short commit SHA and is the same for both images. That is
|
||||
# the version lock: the code location and the payload it dispatches must come
|
||||
# from one commit. Bump both together or not at all.
|
||||
#
|
||||
# The sandbox Gitea registry requires authentication - an anonymous manifest GET
|
||||
# returns 401 - so a pull secret is required in every namespace that pulls either
|
||||
# image. Create it with a Gitea access token that has read:package scope:
|
||||
#
|
||||
# kubectl -n <namespace> create secret docker-registry gitea-registry \
|
||||
# --docker-server=gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu \
|
||||
# --docker-username=<gitea-user> \
|
||||
# --docker-password=<gitea-token>
|
||||
#
|
||||
# The payload namespace needs it too: PipesK8sClient creates that Job, and a
|
||||
# missing pull secret there leaves the op waiting on a pod that never starts,
|
||||
# which surfaces as pod_wait_timeout rather than as an image error.
|
||||
|
||||
dagster:
|
||||
dagster-user-deployments:
|
||||
deployments:
|
||||
- name: distributed-execution
|
||||
image:
|
||||
repository: gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/j.r/distributed-execution
|
||||
tag: 5122da4691f9
|
||||
pullPolicy: IfNotPresent
|
||||
imagePullSecrets:
|
||||
- name: gitea-registry
|
||||
env:
|
||||
- name: PIPES_PAYLOAD_IMAGE
|
||||
value: gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/j.r/distributed-execution-payload:5122da4691f9
|
||||
- name: PIPES_PAYLOAD_NAMESPACE
|
||||
value: dagster
|
||||
|
||||
runLauncher:
|
||||
config:
|
||||
k8sRunLauncher:
|
||||
imagePullSecrets:
|
||||
- name: gitea-registry
|
||||
runK8sConfig:
|
||||
containerConfig:
|
||||
env:
|
||||
- name: PIPES_PAYLOAD_IMAGE
|
||||
value: gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/j.r/distributed-execution-payload:5122da4691f9
|
||||
- name: PIPES_PAYLOAD_NAMESPACE
|
||||
value: dagster
|
||||
Reference in New Issue
Block a user