Commit Graph

4 Commits

Author SHA1 Message Date
ILay
4131288960 fix(tightly-coupled): pin k8s step IO manager to shared storage 2026-09-01 20:03:23 +02:00
ILay
ba93a20ae2 [SIMPL-30451] Mount shared storage in step pods
Configure Kubernetes executor step pods with the shared Dagster home and persistent volume used by the code location.

Changelog: fixed
2026-09-01 19:12:27 +02:00
ILay
ccc2e94c2a [SIMPL-30451] Fan out work units so the execution target is observable
The reference graph processed every work unit inside a single op, so exactly one worker was ever reported regardless of executor. That made the guide's claim that k8s_job_executor yields several distinct contributing_hosts false, and left the reference implementations unable to demonstrate the executor choice at all. generate_work_units is now a DynamicOut and both graphs map over it, so one step is created per unit and the loosely coupled pattern dispatches one external workload per unit.

Evidence is split into contributing_workers (host and pid, differs per process) and contributing_hosts (differs only across machines), because the previous single field could not distinguish multiprocess fan-out from no fan-out. Tests now assert the mapped step keys rather than a host count, since execute_in_process ignores executor_def and cannot prove executor behaviour on its own. Adds a Windows note: multiprocess_executor did not complete during authoring and left orphaned processes.

Changelog: fixed
2026-08-26 18:50:01 +02:00
ILay
43dbb81a95 [SIMPL-30451] Add distributed-execution service with guide and reference implementations
Canonical location for documentation, example workflows and reference service implementations covering distributed execution patterns. Covers AC1-AC4: execution-target selection, decision support, readiness checks and code-level linkage. Tightly coupled jobs and the loosely coupled subprocess transport are verified by the test suite; the Kubernetes pipes transport is implemented but not yet cluster-run and is marked as such in the guide.

Changelog: added
2026-08-26 18:07:06 +02:00