Commit Graph

9 Commits

Author SHA1 Message Date
ILay
554c92ec92 [SIMPL-30451] Run T1 and T4 on the sandbox
Executed from a pod in dataprovider01 with a local SQLite instance, deliberately not the platform's Postgres. T1 reachability passes: check_tcp_reachable against pg-cluster.common01.svc.cluster.local:5432 returned True. T4 passes: tightly_coupled_local_job reached RUN_SUCCESS with one entry in contributing_hosts and four distinct PIDs in contributing_workers, which is what the row predicts. Torn down afterwards.

The run exposed a defect in this service. report_execution_target warns that DAGSTER_POSTGRES_HOST, DAGSTER_POSTGRES_USER and DAGSTER_POSTGRES_DB are missing and calls that a tightly coupled misconfiguration, but the Simpl chart never sets them - it injects DAGSTER_PG_PASSWORD and bakes host, user and database into dagster.yaml. TIGHTLY_COUPLED_ENV_VARS therefore reports a misconfiguration that is not one, and would do so on a correctly configured platform run pod.

It also made the T6 obstacle concrete rather than theoretical: step outputs were written to /dagster-home/storage via PickledObjectFilesystemIOManager, which under k8s_job_executor lands on a volume no other step pod can read.

Changelog: added
2026-08-31 20:17:45 +02:00
ILay
2bbecb1bdd [SIMPL-30451] Correct the C6 finding: the container context overrides job_namespace
The previous revision claimed C6 failed on the sandbox because job_namespace is dagster while the launcher's ConfigMap, Secret and PVC live in dataprovider01. That read job_namespace in isolation. Each code location ships a DAGSTER_CONTAINER_CONTEXT whose k8s.namespace is dataprovider01, and the code location context is merged over the launcher's, so it wins - dagster_k8s/container_context.py resolves namespace as 'other.namespace if other.namespace else self.namespace'. Both deployed code locations carry it, so run pods land where their dependencies are.

Still true: no run has ever executed on that cluster. That now looks like nothing having been launched rather than anything being broken. The residual risk is a code location reaching the launcher without a container context namespace, which the chart always sets, so it bites only hand-rolled pods.

C5 now warns that job_namespace is a default rather than the effective namespace.

Changelog: fixed
2026-08-31 20:09:50 +02:00
ILay
18cc46d132 [SIMPL-30451] Assess the tightly coupled path against the sandbox
T2, T3, T5 and T8 are confirmed from the live dagster-instance ConfigMap and from dagster-role. None of the T checks could be executed: the tightly coupled path needs this code location registered on the platform Dagster, and unlike the pipes transport it cannot run from a standalone pod, because every step pod connects to the metadata database itself and so they must share instance storage rather than a local SQLite file.

C6 fails on that cluster and explains why no run has ever executed there. The launcher sends run pods to namespace dagster while its instance_config_map, postgres_password_secret and dagster-shared-pvc all live in dataprovider01; those references do not cross namespaces. dataprovider01 holds no pods labelled dagster/run-id and the daemon log shows no launch activity. C6 now tells the reader to check the launcher's namespace-local dependencies, not just the namespace.

Changelog: changed
2026-08-31 19:57:38 +02:00
ILay
96caa3f81b [SIMPL-30451] Verify the Kubernetes pipes transport on the sandbox
Run cff9b348-bfc3-4ac1-ab51-a94892b8e3a0 of loosely_coupled_k8s_job reached RUN_SUCCESS in dataprovider01 on sandbox-cat-dat: four payload Jobs, four distinct payload pod hostnames in contributing_hosts, and the payload log lines in the run log. Checks L4-L10 are now observed rather than derived. Torn down afterwards; the namespace was left as it was found.

Corrects two claims that were wrong. The registry does NOT require a pull secret - a bare GET returns 401, but that is the opening move of the Docker token handshake, and completing it anonymously returns the manifest for both images. And PIPES_PAYLOAD_NAMESPACE was set to a namespace named dagster, which does not exist on that cluster; the release is called dagster but runs in dataprovider01.

Adds the sandbox probe variant. The generic probe cannot run there: sandbox access is Rancher project-scoped, so a new namespace grants its creator nothing and Role creation is denied everywhere. It is not needed either, since dagster-role already carries the exact pipes permissions.

Changelog: added
2026-08-31 19:41:26 +02:00
ILay
515eef7736 [SIMPL-30451] Add a standalone cluster probe for the pipes transport
Runs loosely_coupled_k8s_job from one throwaway pod with only the RBAC the pipes client needs, clearing readiness checks L4-L9 without deploying a code location, a webserver or a database. Because the pod is its own control plane it also sidesteps the 1.13.19 vs 1.12.8 skew that blocks registering this service against the sandbox Dagster.

Verified in the container beforehand: the CLI form reaches RUN_SUCCESS, the multiprocess executor fans out to four dynamic steps, and PipesK8sClient selects in-cluster credentials from KUBERNETES_SERVICE_HOST, so no code change is needed to run it in a pod.

Changelog: added
2026-08-31 17:46:37 +02:00
ILay
c2a49f0971 [SIMPL-30451] Add sandbox deployment values and the pull-secret caveat
yaml/sandbox/values-sandbox-gitea.yaml carries the registry coordinates, the pull secret both namespaces need, and the commit SHA the two images share. The sandbox registry rejects anonymous pulls, so a payload namespace without its own pull secret leaves the dispatching op waiting on a pod that never starts - readiness check L4 now says so.

Changelog: added
2026-08-31 16:12:57 +02:00
ILay
1ce332fa1b [SIMPL-30451] Correct stale references in the guide and verify both images build
The rename of process_work_units and _results_from_pipes during the fan-out refactor left three references behind, in the guide, the checklist and the loosely coupled module docstring. The guide also showed loosely_coupled_k8s_job without its executor_def and claimed the pipes client replaced the executor, which stopped being true once the graph fanned out.

Both images were built and smoke tested: the code location image loads its definitions, and the payload image ships dagster_pipes without dagster. That check is now recorded as L11 in the readiness checklist, so the isolation argument is verified at image level and not only in the source.

Changelog: fixed
2026-08-26 19:21:41 +02:00
ILay
ccc2e94c2a [SIMPL-30451] Fan out work units so the execution target is observable
The reference graph processed every work unit inside a single op, so exactly one worker was ever reported regardless of executor. That made the guide's claim that k8s_job_executor yields several distinct contributing_hosts false, and left the reference implementations unable to demonstrate the executor choice at all. generate_work_units is now a DynamicOut and both graphs map over it, so one step is created per unit and the loosely coupled pattern dispatches one external workload per unit.

Evidence is split into contributing_workers (host and pid, differs per process) and contributing_hosts (differs only across machines), because the previous single field could not distinguish multiprocess fan-out from no fan-out. Tests now assert the mapped step keys rather than a host count, since execute_in_process ignores executor_def and cannot prove executor behaviour on its own. Adds a Windows note: multiprocess_executor did not complete during authoring and left orphaned processes.

Changelog: fixed
2026-08-26 18:50:01 +02:00
ILay
43dbb81a95 [SIMPL-30451] Add distributed-execution service with guide and reference implementations
Canonical location for documentation, example workflows and reference service implementations covering distributed execution patterns. Covers AC1-AC4: execution-target selection, decision support, readiness checks and code-level linkage. Tightly coupled jobs and the loosely coupled subprocess transport are verified by the test suite; the Kubernetes pipes transport is implemented but not yet cluster-run and is marked as such in the guide.

Changelog: added
2026-08-26 18:07:06 +02:00