dataprovider01/template-code-location
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(SIMPL-24642): update Development Guide to reflect consolidated structure

Browse Source
This commit is contained in:
ILay
2026-04-24 18:42:44 +02:00
parent d14b2dfac4
commit 49f3afd6ab
1 changed files with 28 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files

39
documents/Development Guide.md
View File

@@ -9,18 +9,35 @@ By following a *code-first approach*, developers ensure consistency, traceabilit
Development must always begin in a local environment. This allows developers to rapidly iterate, test business logic, and validate DAG (Directed Acyclic Graph) structures without impacting production data. Development must always begin in a local environment. This allows developers to rapidly iterate, test business logic, and validate DAG (Directed Acyclic Graph) structures without impacting production data.
### 2.1 Project Layout ### 2.1 Project Layout
To ensure compatibility with the Simpl-Open platform, every Dagster code location must adhere to the following directory structure: This repository (`template-code-location`) serves as the **single consolidated code location** for all data services workflows. It contains the jobs, ops, and configurations previously spread across `data-processing`, `dataframe-level-anonymisation`, and `field-level-pseudo-anonymisation`.
```text ```text
project-root/ template-code-location/
├── dagster_code_location/ ├── src/
│ ├── jobs/ # Executable workflows │ └── template_code_location/
│ ├── ops/ # Individual functional units (business logic) │ ├── repository.py # Unified entry point (all jobs/sensors/resources)
│ ├── resources/ # External connections (Object storage, APIs, etc...) │ ├── data_processing/ # Data cleaning & transformation ops/jobs
│ └── repository.py # Central entry point for the code location │ │ ├── config_models/
├── tests/ # Unit and integration tests │ │ ├── jobs.py
├── Dockerfile # Containerization instructions │ │ └── ops.py
├── pyproject.toml # Dependency management (Poetry/Pip/UV) │ ├── dataframe_level_anonymisation/ # k-anonymity, l-diversity, t-closeness
└── README.md # Documentation │ │ ├── config_models/
│ │ ├── jobs.py
│ │ ├── ops.py
│ │ └── utils.py
│ ├── field_level_pseudo_anonymisation/ # Field-level encryption/hashing/redaction
│ │ ├── config_models/
│ │ ├── techniques/
│ │ ├── jobs.py
│ │ ├── ops.py
│ │ ├── unstructured_ops.py
│ │ └── utils.py
│ ├── jobs/ # Template example jobs
│ └── ops/ # Template example ops
├── tests/ # All tests (migrated from source repos)
├── Dockerfile
├── pyproject.toml
└── README.md
``` ```
### 2.2 Code Examples (Ops, Jobs, and Definitions) ### 2.2 Code Examples (Ops, Jobs, and Definitions)