Commit Graph

3 Commits

Author SHA1 Message Date
ILay
7692891fff [SIMPL-30451] Record real run evidence for both locally runnable jobs
Both jobs were launched from the Dagster UI, so the numbers in the new section 5.6 come from the daemon and run launcher path rather than from execute_in_process. The tightly coupled run reports one contributing worker across seven steps; the loosely coupled subprocess run reports four, which is the difference the two patterns are supposed to make visible. Hostnames are redacted, process IDs are not.

Also ignores a local .dagster_home, which the README's dagster dev instructions will create for anyone who sets DAGSTER_HOME alongside the repo.

Changelog: added
2026-08-26 19:28:19 +02:00
ILay
ccc2e94c2a [SIMPL-30451] Fan out work units so the execution target is observable
The reference graph processed every work unit inside a single op, so exactly one worker was ever reported regardless of executor. That made the guide's claim that k8s_job_executor yields several distinct contributing_hosts false, and left the reference implementations unable to demonstrate the executor choice at all. generate_work_units is now a DynamicOut and both graphs map over it, so one step is created per unit and the loosely coupled pattern dispatches one external workload per unit.

Evidence is split into contributing_workers (host and pid, differs per process) and contributing_hosts (differs only across machines), because the previous single field could not distinguish multiprocess fan-out from no fan-out. Tests now assert the mapped step keys rather than a host count, since execute_in_process ignores executor_def and cannot prove executor behaviour on its own. Adds a Windows note: multiprocess_executor did not complete during authoring and left orphaned processes.

Changelog: fixed
2026-08-26 18:50:01 +02:00
ILay
43dbb81a95 [SIMPL-30451] Add distributed-execution service with guide and reference implementations
Canonical location for documentation, example workflows and reference service implementations covering distributed execution patterns. Covers AC1-AC4: execution-target selection, decision support, readiness checks and code-level linkage. Tightly coupled jobs and the loosely coupled subprocess transport are verified by the test suite; the Kubernetes pipes transport is implemented but not yet cluster-run and is marked as such in the guide.

Changelog: added
2026-08-26 18:07:06 +02:00