[SIMPL-30451] Add distributed-execution service with guide and reference implementations

Canonical location for documentation, example workflows and reference service implementations covering distributed execution patterns. Covers AC1-AC4: execution-target selection, decision support, readiness checks and code-level linkage. Tightly coupled jobs and the loosely coupled subprocess transport are verified by the test suite; the Kubernetes pipes transport is implemented but not yet cluster-run and is marked as such in the guide.

Changelog: added

payload/Dockerfile

@@ -0,0 +1,30 @@
+# External payload image for the loosely coupled execution target.
+#
+# Built and versioned independently of the code location image. It contains no
+# workflow code and no `dagster` package - only the pipes protocol client.
+#
+#   docker build -f payload/Dockerfile -t distributed-execution-payload:0.1.0 payload/
+
+FROM python:3.12-slim-bookworm
+
+RUN python -m pip install --no-cache-dir --upgrade "pip==26.1.2"
+
+WORKDIR /app
+
+RUN addgroup --gid 1000 appgroup && \
+    adduser --uid 1000 --gid 1000 --disabled-password --gecos "" appuser
+
+RUN apt-get update && apt-get upgrade -y \
+    && apt-get clean \
+    && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
+
+COPY requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+
+COPY work.py .
+
+RUN chown -R appuser:appgroup /app
+
+USER appuser
+
+CMD ["python", "/app/work.py"]

payload/requirements.txt

@@ -0,0 +1,3 @@
+# The payload's entire dependency surface. Note the absence of `dagster`:
+# dependency isolation is the point of the loosely coupled pattern.
+dagster-pipes>=1.13.0

payload/work.py

@@ -0,0 +1,55 @@
+"""External payload for the loosely coupled execution target.
+
+This script is deliberately NOT a Dagster code location. It depends only on
+``dagster-pipes``, never on ``dagster``, and it never opens a connection to the
+orchestration metadata database. Everything it reports reaches the control plane
+through the pipes message channel.
+
+Run by ``distributed_execution.loosely_coupled.jobs`` via either
+``PipesSubprocessClient`` (local) or ``PipesK8sClient`` (cluster).
+"""
+
+import os
+import socket
+
+from dagster_pipes import open_dagster_pipes
+
+# Presence of any of these would mean the payload was granted orchestration
+# runtime credentials it has no business holding.
+ORCHESTRATION_ENV_VARS = (
+    "DAGSTER_POSTGRES_HOST",
+    "DAGSTER_POSTGRES_USER",
+    "DAGSTER_POSTGRES_DB",
+)
+
+
+def main() -> None:
+    with open_dagster_pipes() as pipes:
+        units = pipes.get_extra("units")
+        host = socket.gethostname()
+
+        pipes.log.info(f"External payload started on {host} with {len(units)} work units")
+
+        results = [{"unit": unit, "squared": unit * unit, "host": host} for unit in units]
+
+        leaked = [name for name in ORCHESTRATION_ENV_VARS if os.environ.get(name)]
+        if leaked:
+            pipes.log.warning(
+                "Orchestration runtime credentials are visible to this payload: "
+                f"{', '.join(leaked)}. A loosely coupled target should not have them."
+            )
+
+        # The only channel back to the control plane.
+        pipes.report_custom_message(
+            {
+                "results": results,
+                "host": host,
+                "orchestration_env_visible": leaked,
+            }
+        )
+
+        pipes.log.info(f"External payload finished on {host}")
+
+
+if __name__ == "__main__":
+    main()