Merge branch 'feature/SIMPL-27992' into 'develop'
[SIMPL-27992](https://jira.simplprogramme.eu/browse/SIMPL-27992) Document... See merge request simpl/simpl-open/development/data-services/template-code-location!9
This commit is contained in:
@@ -2,7 +2,7 @@
|
||||
|
||||
## 1. Goal and Scope
|
||||
|
||||
The purpose of this document is to provide a comprehensive guide for participants to create, manage, and update workflows within the Simpl-Open orchestration platform.
|
||||
The purpose of this document is to provide a comprehensive guide for participants to create, manage, update and delete workflows within the Simpl-Open orchestration platform.
|
||||
By following a *code-first approach*, developers ensure consistency, traceability, and reliability across all environments.
|
||||
|
||||
## 2. Local Development
|
||||
@@ -43,7 +43,10 @@ The heavy-lifting logic lives in separate repositories, pulled in as installable
|
||||
These packages expose their jobs and ops which are then imported and registered in `repository.py`.
|
||||
|
||||
### 2.3 Code Examples (Ops, Jobs, and Definitions)
|
||||
The orchestration logic should be modular. Here is a practical example of how to construct a workflow.
|
||||
The orchestration logic should be modular.
|
||||
|
||||
### 2.3.1 Workflow creation
|
||||
Here is a practical example of how to construct a workflow.
|
||||
|
||||
**1. Defining Ops (`ops/ops.py`)**
|
||||
Ops are the core units of computation. Keep them focused on a single task.
|
||||
@@ -100,6 +103,49 @@ defs = Definitions(
|
||||
loggers={"simpl": simpl_json_logger},
|
||||
)
|
||||
```
|
||||
### 2.3.2 Workflow edit
|
||||
|
||||
Updating a workflow, means changing the job name, parameters or definition.
|
||||
Each modification means getting a new version of that workflow. To make the version visible, the name of the job can include a version tag (eg: `data_processing_job_ver_2`).
|
||||
|
||||
**1. Change Jobs (`jobs/jobs.py`)**
|
||||
Change workflow name, parameters or definition in the `jobs.py` file.
|
||||
|
||||
**2. Change Ops (`ops/ops.py`)**
|
||||
If workflow ops are updated, this must be done in `ops.py` file.
|
||||
|
||||
**3. Change Tests (`tests/`)**
|
||||
If jobs or obs are updated, check out their related `tests` and update them accordingly.
|
||||
|
||||
**4. Update Registering Definitions (`repository.py`)**
|
||||
If jobs names are updated, make sure to update the import and definitions.
|
||||
|
||||
### 2.3.3 Workflow deletion
|
||||
Before deleting an existing workflow, first check its status (if is needed or referenced) in the **Dagster UI**:
|
||||
- Navigate to the "Runs". Identify a given workflow using the "Filter" button or navigation "Newer", "Older" buttons.
|
||||
- Check details and status: open workflow details by clicking on the target, or just check the status value.
|
||||
- Check logs: open workflow logs by clicking on the uuid.
|
||||
|
||||
Here are two practical examples how to delete a workflow:
|
||||
|
||||
**Temporary delete workflow**
|
||||
|
||||
**1. Clean Registering Definitions (`repository.py`)**
|
||||
Comment or delete workflow definition from the `repository.py`.
|
||||
|
||||
**Permanently delete workflow**
|
||||
|
||||
**1. Clean Jobs (`jobs/jobs.py`)**
|
||||
Delete workflow definition from the `jobs.py` file.
|
||||
|
||||
**2. Clean Ops (`ops/ops.py`)**
|
||||
If workflow links also ops, delete their definition from the `ops.py` file.
|
||||
|
||||
**3. Clean Tests (`tests/`)**
|
||||
If jobs or obs have dedicated unittest, check out the `tests/` folder and delete them.
|
||||
|
||||
**4. Clean Scheduler**
|
||||
If the workflow has any scheduler, check and adjust it accordingly.
|
||||
|
||||
### 2.4 Best Practices & Constraints
|
||||
|
||||
@@ -145,7 +191,9 @@ The deployment follows these automated steps:
|
||||
|
||||
To confirm a successful deployment:
|
||||
|
||||
- **Dagster UI**: Navigate to the "Deployment" or "Code Locations" tab. Verify that the loaded image tag matches the latest Git commit.
|
||||
- **Dagster UI**:
|
||||
- Navigate to the "Deployment" or "Code Locations" tab. Verify that the loaded image tag matches the latest Git commit.
|
||||
- Navigate to the "Runs" tab. Identify deleted workflow. Click on it's target to open details. "Pipeline not found" message shows that workflow was deleted successfully. Click on it's uuid to open logs. Logs must be available after deletion.
|
||||
- **Health Check**: Trigger a "Test Run" of the job in the production environment using a limited data slice.
|
||||
- **Logs**: Monitor the initialization logs in the Dagster daemon to ensure the code location was loaded without schema or dependency errors.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user