From 81bb3299a44fa0c7047c5d485de210fee24e726f Mon Sep 17 00:00:00 2001 From: Adrian LODIN Date: Wed, 27 May 2026 17:42:51 +0300 Subject: [PATCH 1/3] [SIMPL-27992](https://jira.simplprogramme.eu/browse/SIMPL-27992) Document Workflow Deletion Process. Changelog: added --- documents/Development Guide.md | 28 +++++++++++++++++++++++++--- 1 file changed, 25 insertions(+), 3 deletions(-) diff --git a/documents/Development Guide.md b/documents/Development Guide.md index 6582768..7f53297 100644 --- a/documents/Development Guide.md +++ b/documents/Development Guide.md @@ -2,7 +2,7 @@ ## 1. Goal and Scope -The purpose of this document is to provide a comprehensive guide for participants to create, manage, and update workflows within the Simpl-Open orchestration platform. +The purpose of this document is to provide a comprehensive guide for participants to create, manage, update and delete workflows within the Simpl-Open orchestration platform. By following a *code-first approach*, developers ensure consistency, traceability, and reliability across all environments. ## 2. Local Development @@ -43,7 +43,10 @@ The heavy-lifting logic lives in separate repositories, pulled in as installable These packages expose their jobs and ops which are then imported and registered in `repository.py`. ### 2.3 Code Examples (Ops, Jobs, and Definitions) -The orchestration logic should be modular. Here is a practical example of how to construct a workflow. +The orchestration logic should be modular. + +### 2.3.1 Workflow creation +Here is a practical example of how to construct a workflow. **1. Defining Ops (`ops/ops.py`)** Ops are the core units of computation. Keep them focused on a single task. @@ -101,6 +104,23 @@ defs = Definitions( ) ``` +### 2.3.2 Workflow deletion +Here is a practical example of how to delete a workflow. + +**1. Pre-delete check** +Before deleting an existing workflow, first check its status (if is needed or referenced): +- **Dagster UI**: Navigate to the "Runs". Identify a given worklog using the "Filter" button or navigation "Newer", "Older" buttons. +- **Details, logs check:** Check workflow status, open it by clicking on it's target or open logs by clicking on it's uuid. + +**2. Delete Ops (`ops/ops.py`)** +If the workflow is defined as an op, delete it's corresponding method from the `ops.py` file. + +**3. Delete Jobs (`jobs/jobs.py`)** +If the workflow is defined as a job, delete it's corresponding method from the `jobs.py` file. + +**4. Delete Registering Definitions (`repository.py`)** +Delete workflow import, and it's definition from the `repository.py` + ### 2.4 Best Practices & Constraints - **Separation of Concerns**: Keep orchestration logic (how ops connect) strictly separate from heavy business logic (which should ideally live in separate Python modules/classes). @@ -145,7 +165,9 @@ The deployment follows these automated steps: To confirm a successful deployment: -- **Dagster UI**: Navigate to the "Deployment" or "Code Locations" tab. Verify that the loaded image tag matches the latest Git commit. +- **Dagster UI**: + - Navigate to the "Deployment" or "Code Locations" tab. Verify that the loaded image tag matches the latest Git commit. + - Navigate to the "Runs" tab. Identify deleted workflow. Click on it's target to open details. "Pipeline not found" message shows that workflow was deleted successfully. Click on it's uuid to open logs. Logs must be available after deletion. - **Health Check**: Trigger a "Test Run" of the job in the production environment using a limited data slice. - **Logs**: Monitor the initialization logs in the Dagster daemon to ensure the code location was loaded without schema or dependency errors. From 96a8a8e4ebcfbea301fd27daf798b3737a4e824a Mon Sep 17 00:00:00 2001 From: Adrian LODIN Date: Thu, 28 May 2026 16:59:00 +0300 Subject: [PATCH 2/3] [SIMPL-27992](https://jira.simplprogramme.eu/browse/SIMPL-27992) Update documentation Workflow Deletion Process. Changelog: changed --- documents/Development Guide.md | 32 +++++++++++++++++++++----------- 1 file changed, 21 insertions(+), 11 deletions(-) diff --git a/documents/Development Guide.md b/documents/Development Guide.md index 7f53297..3056239 100644 --- a/documents/Development Guide.md +++ b/documents/Development Guide.md @@ -105,21 +105,31 @@ defs = Definitions( ``` ### 2.3.2 Workflow deletion -Here is a practical example of how to delete a workflow. +Before deleting an existing workflow, first check its status (if is needed or referenced) in the **Dagster UI**: +- Navigate to the "Runs". Identify a given worklog using the "Filter" button or navigation "Newer", "Older" buttons. +- Check details and status: open workflow details by clicking on the target, or just check the status value. +- Check logs: open workflow logs by clicking on the uuid. -**1. Pre-delete check** -Before deleting an existing workflow, first check its status (if is needed or referenced): -- **Dagster UI**: Navigate to the "Runs". Identify a given worklog using the "Filter" button or navigation "Newer", "Older" buttons. -- **Details, logs check:** Check workflow status, open it by clicking on it's target or open logs by clicking on it's uuid. +Here are two practical examples how to delete a workflow: -**2. Delete Ops (`ops/ops.py`)** -If the workflow is defined as an op, delete it's corresponding method from the `ops.py` file. +**Temporary delete workflow** -**3. Delete Jobs (`jobs/jobs.py`)** -If the workflow is defined as a job, delete it's corresponding method from the `jobs.py` file. +**1. Clean Registering Definitions (`repository.py`)** +Comment or delete workflow definition from the `repository.py`. -**4. Delete Registering Definitions (`repository.py`)** -Delete workflow import, and it's definition from the `repository.py` +**Permanently delete workflow** + +**1. Clean Jobs (`jobs/jobs.py`)** +Delete workflow definition from the `jobs.py` file. + +**2. Clean Ops (`ops/ops.py`)** +If workflow links also ops, delete their definition from the `ops.py` file. + +**3. Clean Tests (`tests/`)** +If jobs or obs have dedicated unittest, check out the `tests/` folder and delete them. + +**3. Clean Scheduler ** +If there is any scheduler for this workflow check and adjust it accordingly. ### 2.4 Best Practices & Constraints From 1b154aa2d710279047443139c9a3632e945f09c9 Mon Sep 17 00:00:00 2001 From: Adrian LODIN Date: Fri, 29 May 2026 15:40:55 +0300 Subject: [PATCH 3/3] [SIMPL-28035](https://jira.simplprogramme.eu/browse/SIMPL-28035) Document Workflow Update Process. Changelog: chenged --- documents/Development Guide.md | 28 ++++++++++++++++++++++------ 1 file changed, 22 insertions(+), 6 deletions(-) diff --git a/documents/Development Guide.md b/documents/Development Guide.md index 3056239..aa4870b 100644 --- a/documents/Development Guide.md +++ b/documents/Development Guide.md @@ -103,10 +103,26 @@ defs = Definitions( loggers={"simpl": simpl_json_logger}, ) ``` +### 2.3.2 Workflow edit -### 2.3.2 Workflow deletion +Updating a workflow, means changing the job name, parameters or definition. +Each modification means getting a new version of that workflow. To make the version visible, the name of the job can include a version tag (eg: `data_processing_job_ver_2`). + +**1. Change Jobs (`jobs/jobs.py`)** +Change workflow name, parameters or definition in the `jobs.py` file. + +**2. Change Ops (`ops/ops.py`)** +If workflow ops are updated, this must be done in `ops.py` file. + +**3. Change Tests (`tests/`)** +If jobs or obs are updated, check out their related `tests` and update them accordingly. + +**4. Update Registering Definitions (`repository.py`)** +If jobs names are updated, make sure to update the import and definitions. + +### 2.3.3 Workflow deletion Before deleting an existing workflow, first check its status (if is needed or referenced) in the **Dagster UI**: -- Navigate to the "Runs". Identify a given worklog using the "Filter" button or navigation "Newer", "Older" buttons. +- Navigate to the "Runs". Identify a given workflow using the "Filter" button or navigation "Newer", "Older" buttons. - Check details and status: open workflow details by clicking on the target, or just check the status value. - Check logs: open workflow logs by clicking on the uuid. @@ -125,11 +141,11 @@ Delete workflow definition from the `jobs.py` file. **2. Clean Ops (`ops/ops.py`)** If workflow links also ops, delete their definition from the `ops.py` file. -**3. Clean Tests (`tests/`)** -If jobs or obs have dedicated unittest, check out the `tests/` folder and delete them. +**3. Clean Tests (`tests/`)** +If jobs or obs have dedicated unittest, check out the `tests/` folder and delete them. -**3. Clean Scheduler ** -If there is any scheduler for this workflow check and adjust it accordingly. +**4. Clean Scheduler** +If the workflow has any scheduler, check and adjust it accordingly. ### 2.4 Best Practices & Constraints