[SIMPL-30451] Verify the Kubernetes pipes transport on the sandbox
Run cff9b348-bfc3-4ac1-ab51-a94892b8e3a0 of loosely_coupled_k8s_job reached RUN_SUCCESS in dataprovider01 on sandbox-cat-dat: four payload Jobs, four distinct payload pod hostnames in contributing_hosts, and the payload log lines in the run log. Checks L4-L10 are now observed rather than derived. Torn down afterwards; the namespace was left as it was found. Corrects two claims that were wrong. The registry does NOT require a pull secret - a bare GET returns 401, but that is the opening move of the Docker token handshake, and completing it anonymously returns the manifest for both images. And PIPES_PAYLOAD_NAMESPACE was set to a namespace named dagster, which does not exist on that cluster; the release is called dagster but runs in dataprovider01. Adds the sandbox probe variant. The generic probe cannot run there: sandbox access is Rancher project-scoped, so a new namespace grants its creator nothing and Role creation is denied everywhere. It is not needed either, since dagster-role already carries the exact pipes permissions. Changelog: added
This commit is contained in:
ILay
@@ -12,33 +12,24 @@
|
||||
#
|
||||
# --- Before applying -------------------------------------------------------
|
||||
#
|
||||
# 1. Pull secret. The registry rejects anonymous pulls, so both this pod and the
|
||||
# payload Jobs it creates need credentials. Use a token with read:package.
|
||||
# Credentials, only if your registry needs them. The Gitea images referenced
|
||||
# below do NOT: a bare GET returns 401, but that is the start of the Docker
|
||||
# token handshake, and completing it anonymously returns the manifest. A probe
|
||||
# pod carrying no credentials pulled and ran on the sandbox on 2026-08-31.
|
||||
#
|
||||
# kubectl -n distexec-probe create secret docker-registry gitea-registry \
|
||||
# --docker-server=gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu \
|
||||
# --docker-username=<gitea-user> \
|
||||
# --docker-password=<gitea-token>
|
||||
#
|
||||
# 2. Payload pods inherit the pull secret from the default service account.
|
||||
# PipesK8sClient does not set serviceAccountName on the Jobs it creates, so
|
||||
# they run as `default`, and this service does not yet pass imagePullSecrets
|
||||
# through base_pod_spec:
|
||||
#
|
||||
# kubectl -n distexec-probe patch serviceaccount default \
|
||||
# -p '{"imagePullSecrets":[{"name":"gitea-registry"}]}'
|
||||
#
|
||||
# Skip this and the payload pod never starts. That surfaces as the op waiting
|
||||
# until pod_wait_timeout, which reads like a hung workload rather than a
|
||||
# missing credential.
|
||||
#
|
||||
# Order does not matter much: apply this file first and the dispatcher pod sits
|
||||
# in ImagePullBackOff until the secret appears, then pulls on the kubelet's next
|
||||
# retry. The default service account only has to be patched before the first
|
||||
# payload Job is created, which is after the dispatcher starts.
|
||||
# If you point this at a registry that does require authentication, remember the
|
||||
# payload Jobs need it too. PipesK8sClient sets no serviceAccountName, so they
|
||||
# run as `default` and inherit nothing from the pod below; patch that account or
|
||||
# pass imagePullSecrets through base_pod_spec in dispatch_external_work_k8s. A
|
||||
# payload pod that cannot pull leaves the op waiting until pod_wait_timeout,
|
||||
# which reads like a hung workload rather than a missing credential.
|
||||
#
|
||||
# kubectl apply -f yaml/loosely-coupled/probe-pipes-k8s.yaml
|
||||
#
|
||||
# On sandbox-cat-dat use yaml/sandbox/probe-pipes-k8s-sandbox.yaml instead: that
|
||||
# cluster's access is Rancher project-scoped, so this file's namespace and RBAC
|
||||
# cannot be created there. The sandbox variant needs neither.
|
||||
#
|
||||
# `--dry-run=server` reports "namespaces distexec-probe not found" for the four
|
||||
# namespaced objects. That is the dry run declining to create the namespace it
|
||||
# would need, not a fault in the manifests; `--dry-run=client` passes clean.
|
||||
@@ -134,6 +125,10 @@ spec:
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
serviceAccountName: pipes-dispatcher
|
||||
# A service account with automountServiceAccountToken: false - which the
|
||||
# platform's dagster-svc-account uses - leaves the pipes client selecting
|
||||
# in-cluster auth and then failing on "Service token file does not exist".
|
||||
automountServiceAccountToken: true
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
|
||||
124
yaml/sandbox/probe-pipes-k8s-sandbox.yaml
Normal file
124
yaml/sandbox/probe-pipes-k8s-sandbox.yaml
Normal file
@@ -0,0 +1,124 @@
|
||||
# Sandbox variant of the pipes cluster probe.
|
||||
#
|
||||
# Runs `loosely_coupled_k8s_job` as a single Job in dataprovider01, reusing the
|
||||
# platform's own dagster-svc-account. Use this instead of
|
||||
# yaml/loosely-coupled/probe-pipes-k8s.yaml on sandbox-cat-dat.
|
||||
#
|
||||
# WHY IT LOOKS DIFFERENT. The generic probe creates its own namespace, service
|
||||
# account and RBAC. That cannot run here: sandbox access is Rancher
|
||||
# project-scoped, so a freshly created namespace belongs to no project and the
|
||||
# creator has no rights inside it, and `create roles` is denied in every
|
||||
# namespace. Verified with `kubectl auth can-i`, not assumed.
|
||||
#
|
||||
# It does not need them anyway. The `dagster-role` bound to dagster-svc-account
|
||||
# already grants batch/jobs (full verbs), jobs/status, pods, events, and
|
||||
# pods/log - exactly the set the pipes client needs. Readiness checks L5 and L6
|
||||
# therefore already pass on this cluster; read the live Role to confirm:
|
||||
#
|
||||
# kubectl -n dataprovider01 get role dagster-role -o yaml
|
||||
#
|
||||
# WHAT THIS COSTS. Unlike the generic probe this is NOT isolated. It borrows a
|
||||
# live service account and leaves payload Jobs in a shared namespace until they
|
||||
# are deleted. Nothing it does is persistent, but it is visible to anyone else
|
||||
# working in dataprovider01.
|
||||
#
|
||||
# --- Credentials: none needed ----------------------------------------------
|
||||
#
|
||||
# Both Gitea images are anonymously pullable. A bare GET to the registry returns
|
||||
# 401, which looks like a refusal but is just the start of the Docker token
|
||||
# handshake; completing it anonymously and re-requesting the manifest returns
|
||||
# 200 for both images. Verify with:
|
||||
#
|
||||
# TOK=$(curl -sS "https://gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/v2/token?service=container_registry&scope=repository:j.r/distributed-execution:pull" | jq -r .token)
|
||||
# curl -sS -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $TOK" \
|
||||
# https://gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/v2/j.r/distributed-execution/manifests/5122da4691f9
|
||||
#
|
||||
# This also means the payload Jobs need nothing: they run as the `default`
|
||||
# service account, which carries no pull secret, and do not need one.
|
||||
#
|
||||
# --- Running ---------------------------------------------------------------
|
||||
#
|
||||
# kubectl -n dataprovider01 apply -f yaml/sandbox/probe-pipes-k8s-sandbox.yaml
|
||||
# kubectl -n dataprovider01 logs -f job/distexec-pipes-probe
|
||||
# kubectl -n dataprovider01 get jobs -l app.kubernetes.io/name=distributed-execution-payload
|
||||
#
|
||||
# --- Teardown --------------------------------------------------------------
|
||||
#
|
||||
# kubectl -n dataprovider01 delete job distexec-pipes-probe
|
||||
# kubectl -n dataprovider01 delete jobs -l app.kubernetes.io/name=distributed-execution-payload
|
||||
#
|
||||
# VERIFIED 2026-08-31 on sandbox-cat-dat / dataprovider01. Run
|
||||
# cff9b348-bfc3-4ac1-ab51-a94892b8e3a0 reached RUN_SUCCESS: four payload Jobs,
|
||||
# four distinct payload pod hostnames in contributing_hosts, messages returned
|
||||
# over the pod log stream. Checks L4-L10 pass. Torn down afterwards; the
|
||||
# namespace was left as it was found.
|
||||
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: distexec-pipes-probe
|
||||
namespace: dataprovider01
|
||||
labels:
|
||||
app.kubernetes.io/name: distributed-execution
|
||||
app.kubernetes.io/component: probe
|
||||
spec:
|
||||
backoffLimit: 0
|
||||
activeDeadlineSeconds: 1800
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: distributed-execution
|
||||
app.kubernetes.io/component: probe
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
# Already holds jobs + pods/log; this probe grants itself nothing.
|
||||
serviceAccountName: dagster-svc-account
|
||||
# dagster-svc-account sets automountServiceAccountToken: false, so every
|
||||
# pod that uses it must opt back in - the chart does this for its own
|
||||
# deployments. Without it the pipes client picks in-cluster auth correctly
|
||||
# and then fails on "Service token file does not exist".
|
||||
automountServiceAccountToken: true
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
runAsGroup: 1000
|
||||
fsGroup: 1000
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: dispatcher
|
||||
image: gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/j.r/distributed-execution:5122da4691f9
|
||||
imagePullPolicy: IfNotPresent
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
command:
|
||||
- dagster
|
||||
- job
|
||||
- execute
|
||||
- -f
|
||||
- src/distributed_execution/repository.py
|
||||
- -j
|
||||
- loosely_coupled_k8s_job
|
||||
env:
|
||||
- name: DAGSTER_HOME
|
||||
value: /dagster-home
|
||||
- name: PIPES_PAYLOAD_IMAGE
|
||||
value: gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/j.r/distributed-execution-payload:5122da4691f9
|
||||
- name: PIPES_PAYLOAD_NAMESPACE
|
||||
value: dataprovider01
|
||||
volumeMounts:
|
||||
- name: dagster-home
|
||||
mountPath: /dagster-home
|
||||
resources:
|
||||
requests:
|
||||
cpu: 200m
|
||||
memory: 512Mi
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: 1Gi
|
||||
volumes:
|
||||
- name: dagster-home
|
||||
emptyDir: {}
|
||||
@@ -1,8 +1,9 @@
|
||||
# Sandbox deployment values - Gitea registry.
|
||||
#
|
||||
# NOT CLUSTER-VERIFIED. These values are written against images that exist and a
|
||||
# registry whose auth behaviour was checked, but they have not been applied to a
|
||||
# Simpl cluster. Confirm and amend after the first deployment.
|
||||
# NOT CLUSTER-VERIFIED. The images and the registry behaviour here are confirmed
|
||||
# - a probe pod ran from them on the sandbox on 2026-08-31 - but these values
|
||||
# have never been applied as a code location. Registering this service against
|
||||
# the sandbox Dagster is still blocked on the version skew below.
|
||||
#
|
||||
# VERSION SKEW, unresolved. These images carry dagster 1.13.19 (requirements.txt
|
||||
# pins no upper bound, so the build took whatever was current). The sandbox
|
||||
@@ -21,39 +22,18 @@
|
||||
# the version lock: the code location and the payload it dispatches must come
|
||||
# from one commit. Bump both together or not at all.
|
||||
#
|
||||
# The sandbox Gitea registry requires authentication - an anonymous manifest GET
|
||||
# returns 401 - so a pull secret is required in every namespace that pulls either
|
||||
# image. Create it with a Gitea access token that has read:package scope:
|
||||
# The Gitea images are anonymously pullable, so NO pull secret is needed. A bare
|
||||
# GET to the registry returns 401, which reads as a refusal but is only the start
|
||||
# of the Docker token handshake; completing it anonymously and re-requesting the
|
||||
# manifest returns 200 for both images. This was confirmed on 2026-08-31, and a
|
||||
# probe pod carrying no credentials pulled and ran on the sandbox.
|
||||
#
|
||||
# kubectl -n <namespace> create secret docker-registry gitea-registry \
|
||||
# --docker-server=gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu \
|
||||
# --docker-username=<gitea-user> \
|
||||
# --docker-password=<gitea-token>
|
||||
#
|
||||
# The payload namespace needs it too: PipesK8sClient creates that Job, and a
|
||||
# missing pull secret there leaves the op waiting on a pod that never starts,
|
||||
# which surfaces as pod_wait_timeout rather than as an image error.
|
||||
#
|
||||
# PULL SECRET PLACEMENT. Checked against the chart templates in dagster 1.12.8,
|
||||
# which is what the sandbox runs. There is no per-deployment imagePullSecrets
|
||||
# field - deployment-user.yaml reads the SUBCHART value and the run launcher
|
||||
# helper reads the PARENT value - so a pull secret nested under a deployments[]
|
||||
# entry is silently discarded, and the pod fails with ImagePullBackOff against a
|
||||
# values file that looks correct.
|
||||
#
|
||||
# Both keys below therefore apply to every code location in the release, not
|
||||
# just this one. Extra pull secrets are ignored for registries they do not match,
|
||||
# so the existing code locations are unaffected.
|
||||
# An earlier revision of this file required a pull secret on both the deployment
|
||||
# and the run launcher. That was wrong, and wrong in an expensive direction: it
|
||||
# sent the reader looking for credentials that do not exist.
|
||||
|
||||
dagster:
|
||||
# Run pods, webserver and daemon.
|
||||
imagePullSecrets:
|
||||
- name: gitea-registry
|
||||
|
||||
dagster-user-deployments:
|
||||
# Code server pods.
|
||||
imagePullSecrets:
|
||||
- name: gitea-registry
|
||||
deployments:
|
||||
- name: distributed-execution
|
||||
image:
|
||||
@@ -63,8 +43,10 @@ dagster:
|
||||
env:
|
||||
- name: PIPES_PAYLOAD_IMAGE
|
||||
value: gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/j.r/distributed-execution-payload:5122da4691f9
|
||||
# The sandbox release is named `dagster` but lives in dataprovider01;
|
||||
# there is no namespace called `dagster` on that cluster.
|
||||
- name: PIPES_PAYLOAD_NAMESPACE
|
||||
value: dagster
|
||||
value: dataprovider01
|
||||
|
||||
runLauncher:
|
||||
config:
|
||||
@@ -75,4 +57,4 @@ dagster:
|
||||
- name: PIPES_PAYLOAD_IMAGE
|
||||
value: gitea.dataprovider01.sandbox-cat-dat.simpl-europe.eu/j.r/distributed-execution-payload:5122da4691f9
|
||||
- name: PIPES_PAYLOAD_NAMESPACE
|
||||
value: dagster
|
||||
value: dataprovider01
|
||||
|
||||
Reference in New Issue
Block a user