Commit Graph

7 Commits

Author SHA1 Message Date
ILay
4131288960 fix(tightly-coupled): pin k8s step IO manager to shared storage 2026-09-01 20:03:23 +02:00
ILay
ba93a20ae2 [SIMPL-30451] Mount shared storage in step pods
Configure Kubernetes executor step pods with the shared Dagster home and persistent volume used by the code location.

Changelog: fixed
2026-09-01 19:12:27 +02:00
ILay
3f8175a17f [SIMPL-30451] Stop freezing the k8s executor config with .configured()
k8s_job_executor.configured({...}) collapses the executor config schema to Any, so a run config execution: block is accepted and then silently discarded. That makes job_image and job_namespace impossible to supply at launch time, which is what a probe - or any deployment that is not the chart - needs. Found on the sandbox: the run reported 'No image included in either executor config or the job' and then tried to clean up in namespace dagster, both values coming from the run launcher because the executor config never applied.

The same defaults now go through to_job(config=...), where they remain overridable from run config and the Launchpad.

Changelog: fixed
2026-08-31 20:39:33 +02:00
ILay
6532d640f9 [SIMPL-30451] Point image references at the project registry path
The GitLab project lives at simpl-open/data/supporting-data-services/distributed-execution/distributed-execution, so CI_REGISTRY_IMAGE ends in the project name twice. The pipeline pushes to CI_REGISTRY_IMAGE and nowhere else, so the earlier one-segment path named an image nobody builds - the same mismatch that field-level-pseudo-anonymisation and dataframe-level-anonymisation carry today.

Changelog: fixed
2026-08-31 16:13:05 +02:00
ILay
1ce332fa1b [SIMPL-30451] Correct stale references in the guide and verify both images build
The rename of process_work_units and _results_from_pipes during the fan-out refactor left three references behind, in the guide, the checklist and the loosely coupled module docstring. The guide also showed loosely_coupled_k8s_job without its executor_def and claimed the pipes client replaced the executor, which stopped being true once the graph fanned out.

Both images were built and smoke tested: the code location image loads its definitions, and the payload image ships dagster_pipes without dagster. That check is now recorded as L11 in the readiness checklist, so the isolation argument is verified at image level and not only in the source.

Changelog: fixed
2026-08-26 19:21:41 +02:00
ILay
ccc2e94c2a [SIMPL-30451] Fan out work units so the execution target is observable
The reference graph processed every work unit inside a single op, so exactly one worker was ever reported regardless of executor. That made the guide's claim that k8s_job_executor yields several distinct contributing_hosts false, and left the reference implementations unable to demonstrate the executor choice at all. generate_work_units is now a DynamicOut and both graphs map over it, so one step is created per unit and the loosely coupled pattern dispatches one external workload per unit.

Evidence is split into contributing_workers (host and pid, differs per process) and contributing_hosts (differs only across machines), because the previous single field could not distinguish multiprocess fan-out from no fan-out. Tests now assert the mapped step keys rather than a host count, since execute_in_process ignores executor_def and cannot prove executor behaviour on its own. Adds a Windows note: multiprocess_executor did not complete during authoring and left orphaned processes.

Changelog: fixed
2026-08-26 18:50:01 +02:00
ILay
43dbb81a95 [SIMPL-30451] Add distributed-execution service with guide and reference implementations
Canonical location for documentation, example workflows and reference service implementations covering distributed execution patterns. Covers AC1-AC4: execution-target selection, decision support, readiness checks and code-level linkage. Tightly coupled jobs and the loosely coupled subprocess transport are verified by the test suite; the Kubernetes pipes transport is implemented but not yet cluster-run and is marked as such in the guide.

Changelog: added
2026-08-26 18:07:06 +02:00