[SIMPL-30451] Distributed execution reference service and container-cluster configuration guide #1

j.r · 2026-09-02T17:36:15Z

j.r commented

2026-09-02 17:36:15 +00:00

Brings the completed SIMPL-30451 work and the SIMPL-30787 container-cluster configuration documentation onto develop.

SIMPL-30451: tightly coupled preflight now checks the chart's actual DAGSTER_PG_PASSWORD injection, platform k8s_job_executor run recorded as cluster-verified, Dagster UI screenshots added.
SIMPL-30787: cluster-configuration input reference (mandatory / conditional / optional per setup), reusable minimum input set, worked examples per setup, and requirement traceability.

NOTE: this branch does not carry .gitea/workflows/docker-publish.yml. It is restored on develop in a follow-up commit so sandbox image builds keep working.

Brings the completed SIMPL-30451 work and the SIMPL-30787 container-cluster configuration documentation onto develop. - SIMPL-30451: tightly coupled preflight now checks the chart's actual DAGSTER_PG_PASSWORD injection, platform k8s_job_executor run recorded as cluster-verified, Dagster UI screenshots added. - SIMPL-30787: cluster-configuration input reference (mandatory / conditional / optional per setup), reusable minimum input set, worked examples per setup, and requirement traceability. NOTE: this branch does not carry .gitea/workflows/docker-publish.yml. It is restored on develop in a follow-up commit so sandbox image builds keep working.

j.r added 23 commits 2026-09-02 17:36:16 +00:00

Initial commit 7374db6066

Add new file 0d03e53234

Add new file fe295486f1

Update 2 files cfa6afedd6

- /CODEOWNERS
- /.gitlab/CODEOWNERS

Add new file 0d8078889d

[SIMPL-30451] Add sandbox deployment values and the pull-secret caveat c2a49f0971

yaml/sandbox/values-sandbox-gitea.yaml carries the registry coordinates, the pull secret both namespaces need, and the commit SHA the two images share. The sandbox registry rejects anonymous pulls, so a payload namespace without its own pull secret leaves the dispatching op waiting on a pod that never starts - readiness check L4 now says so.

Changelog: added

[SIMPL-30451] Point image references at the project registry path 6532d640f9

The GitLab project lives at simpl-open/data/supporting-data-services/distributed-execution/distributed-execution, so CI_REGISTRY_IMAGE ends in the project name twice. The pipeline pushes to CI_REGISTRY_IMAGE and nowhere else, so the earlier one-segment path named an image nobody builds - the same mismatch that field-level-pseudo-anonymisation and dataframe-level-anonymisation carry today.

Changelog: fixed

[SIMPL-30451] Merge the distributed-execution service into the GitLab project b625b64df2

The GitLab project was seeded with a scaffold (CODEOWNERS, LICENSE, README, pipeline.variables.sh) unrelated to the local history, so this is an unrelated-histories merge. Kept GitLab's CODEOWNERS - it names the real codeowners group - along with its LICENSE wording and PROJECT_VERSION_NUMBER=0.0.1. Kept the local README, which documents the service rather than GitLab's default template.

The Gitea workflow now derives the registry owner from github.repository_owner instead of hard-coding a personal namespace.

Changelog: added

[SIMPL-30451] Drop the Gitea workflow from the GitLab repository 899429b10a

This repository is built by the shared ds.gitlab-ci.yml template; carrying a second CI system alongside it invited the reader to wonder which one is authoritative. The Gitea Actions workflow that builds both images stays in the sandbox Gitea mirror, which is where the sandbox images are produced.

The README, the pipes payload values and the sandbox values no longer point at a file this repository does not contain.

Changelog: removed

[SIMPL-30451] Put the sandbox pull secrets where the chart actually reads them b47c530d46

dagster 1.12.8 has no per-deployment imagePullSecrets field: deployment-user.yaml reads the dagster-user-deployments subchart value and the run launcher helper reads the parent chart value. The pull secret nested under the deployments[] entry was silently discarded, which would have produced ImagePullBackOff against a values file that reads as correct.

Also records the unresolved 1.13.19 vs 1.12.8 skew between these images and the sandbox control plane.

Changelog: fixed

[SIMPL-30451] Add a standalone cluster probe for the pipes transport 515eef7736

Runs loosely_coupled_k8s_job from one throwaway pod with only the RBAC the pipes client needs, clearing readiness checks L4-L9 without deploying a code location, a webserver or a database. Because the pod is its own control plane it also sidesteps the 1.13.19 vs 1.12.8 skew that blocks registering this service against the sandbox Dagster.

Verified in the container beforehand: the CLI form reaches RUN_SUCCESS, the multiprocess executor fans out to four dynamic steps, and PipesK8sClient selects in-cluster credentials from KUBERNETES_SERVICE_HOST, so no code change is needed to run it in a pod.

Changelog: added

[SIMPL-30451] Harden the probe pod against restricted Pod Security e686df5710

The int cluster did not enforce Pod Security when this was written - no namespace labels, no Kyverno or Gatekeeper. The sandbox was NOT checked; there is no kubeconfig context for it. A cluster that does enforce it rejects the pod outright rather than degrading, which is slow to diagnose from an unrelated namespace.

Records the matching gap it cannot fix: PipesK8sClient builds the payload pod specs, and this service passes no securityContext through base_pod_spec, so a restricted namespace would admit the dispatcher and reject every payload it creates.

Changelog: changed

[SIMPL-30451] Verify the Kubernetes pipes transport on the sandbox 96caa3f81b

Run cff9b348-bfc3-4ac1-ab51-a94892b8e3a0 of loosely_coupled_k8s_job reached RUN_SUCCESS in dataprovider01 on sandbox-cat-dat: four payload Jobs, four distinct payload pod hostnames in contributing_hosts, and the payload log lines in the run log. Checks L4-L10 are now observed rather than derived. Torn down afterwards; the namespace was left as it was found.

Corrects two claims that were wrong. The registry does NOT require a pull secret - a bare GET returns 401, but that is the opening move of the Docker token handshake, and completing it anonymously returns the manifest for both images. And PIPES_PAYLOAD_NAMESPACE was set to a namespace named dagster, which does not exist on that cluster; the release is called dagster but runs in dataprovider01.

Adds the sandbox probe variant. The generic probe cannot run there: sandbox access is Rancher project-scoped, so a new namespace grants its creator nothing and Role creation is denied everywhere. It is not needed either, since dagster-role already carries the exact pipes permissions.

Changelog: added

[SIMPL-30451] Assess the tightly coupled path against the sandbox 18cc46d132

T2, T3, T5 and T8 are confirmed from the live dagster-instance ConfigMap and from dagster-role. None of the T checks could be executed: the tightly coupled path needs this code location registered on the platform Dagster, and unlike the pipes transport it cannot run from a standalone pod, because every step pod connects to the metadata database itself and so they must share instance storage rather than a local SQLite file.

C6 fails on that cluster and explains why no run has ever executed there. The launcher sends run pods to namespace dagster while its instance_config_map, postgres_password_secret and dagster-shared-pvc all live in dataprovider01; those references do not cross namespaces. dataprovider01 holds no pods labelled dagster/run-id and the daemon log shows no launch activity. C6 now tells the reader to check the launcher's namespace-local dependencies, not just the namespace.

Changelog: changed

[SIMPL-30451] Correct the C6 finding: the container context overrides job_namespace 2bbecb1bdd

The previous revision claimed C6 failed on the sandbox because job_namespace is dagster while the launcher's ConfigMap, Secret and PVC live in dataprovider01. That read job_namespace in isolation. Each code location ships a DAGSTER_CONTAINER_CONTEXT whose k8s.namespace is dataprovider01, and the code location context is merged over the launcher's, so it wins - dagster_k8s/container_context.py resolves namespace as 'other.namespace if other.namespace else self.namespace'. Both deployed code locations carry it, so run pods land where their dependencies are.

Still true: no run has ever executed on that cluster. That now looks like nothing having been launched rather than anything being broken. The residual risk is a code location reaching the launcher without a container context namespace, which the chart always sets, so it bites only hand-rolled pods.

C5 now warns that job_namespace is a default rather than the effective namespace.

Changelog: fixed

[SIMPL-30451] Run T1 and T4 on the sandbox 554c92ec92

Executed from a pod in dataprovider01 with a local SQLite instance, deliberately not the platform's Postgres. T1 reachability passes: check_tcp_reachable against pg-cluster.common01.svc.cluster.local:5432 returned True. T4 passes: tightly_coupled_local_job reached RUN_SUCCESS with one entry in contributing_hosts and four distinct PIDs in contributing_workers, which is what the row predicts. Torn down afterwards.

The run exposed a defect in this service. report_execution_target warns that DAGSTER_POSTGRES_HOST, DAGSTER_POSTGRES_USER and DAGSTER_POSTGRES_DB are missing and calls that a tightly coupled misconfiguration, but the Simpl chart never sets them - it injects DAGSTER_PG_PASSWORD and bakes host, user and database into dagster.yaml. TIGHTLY_COUPLED_ENV_VARS therefore reports a misconfiguration that is not one, and would do so on a correctly configured platform run pod.

It also made the T6 obstacle concrete rather than theoretical: step outputs were written to /dagster-home/storage via PickledObjectFilesystemIOManager, which under k8s_job_executor lands on a volume no other step pod can read.

Changelog: added

[SIMPL-30451] Pin dagster to 1.12.8, the control plane version 35ef45de56

The dependency was previously an unbounded 'dagster', so the build took whatever was current and landed on 1.13.19 against a 1.12.8 control plane. A code location is a gRPC server the webserver and daemon call into, and under k8s_job_executor the step pods share a metadata database schema with them, so the version has to match.

Pinned exactly rather than to the 1.12 line: patch releases can carry schema migrations, and 'close enough' is not a safe position against a shared database. Revisit when the platform upgrades.

17 tests pass and definitions validate on the downgraded stack.

Changelog: fixed

[SIMPL-30451] Stop freezing the k8s executor config with .configured() 3f8175a17f

k8s_job_executor.configured({...}) collapses the executor config schema to Any, so a run config execution: block is accepted and then silently discarded. That makes job_image and job_namespace impossible to supply at launch time, which is what a probe - or any deployment that is not the chart - needs. Found on the sandbox: the run reported 'No image included in either executor config or the job' and then tried to clean up in namespace dagster, both values coming from the run launcher because the executor config never applied.

The same defaults now go through to_job(config=...), where they remain overridable from run config and the Launchpad.

Changelog: fixed

[SIMPL-30451] Execute T6 and T7 on the sandbox 64cace2952

Adds the k8s_job_executor probe and records the result. Six step Jobs were created, one per step, and four steps executed and succeeded against the platform's live Postgres. The run then failed before summarise_results: run monitoring polls for a dagster-run-<id> Job that a hand-rolled probe never creates, and with no launcher-supplied container context it looked in the launcher's default namespace, where dagster-svc-account is denied jobs/status. Two step pods had already started and logged 'Skipping step execution' before exiting 0, which is why all six Jobs read Complete while only four steps ran.

Changelog: added

[SIMPL-30451] Mount shared storage in step pods ba93a20ae2

Configure Kubernetes executor step pods with the shared Dagster home and persistent volume used by the code location.

Changelog: fixed

fix(tightly-coupled): pin k8s step IO manager to shared storage 4131288960

[SIMPL-30451] Close the distributed execution reference service 9b1b7056b0

Check the chart's actual DAGSTER_PG_PASSWORD injection in the tightly coupled preflight instead of DAGSTER_POSTGRES_*, which the Simpl chart never sets; the old check warned about a misconfiguration on a correctly configured run pod. Evidence reports the variable name only, never its value.

Record the platform k8s_job_executor run as cluster-verified in the guide and readiness checklist, add the Dagster UI screenshots covering job list, graph, code-configured tags and a successful run.

Changelog: fixed

[SIMPL-30787] Document container-cluster configuration inputs aa49420ca6

Add the cluster-configuration input reference the guide was missing: which inputs are mandatory, conditional or optional in each setup, a reusable minimum input set, and worked examples expressed in the format each setup already uses. No configuration schema or key naming is prescribed.

Retitle the subject sections so they no longer carry one story's AC numbering, and add a traceability section mapping SIMPL-30787 and SIMPL-30451 acceptance criteria onto them.

Changelog: added

j.r merged commit 32b7afa53e into develop

2026-09-02 17:38:23 +00:00

j.r referenced this issue from a commit

2026-09-02 17:38:24 +00:00

Merge pull request '[SIMPL-30451] Distributed execution reference service and container-cluster configuration guide' (#1) from feature/SIMPL-30451-distributed-execution into develop

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: j.r/distributed-execution#1