docs(SIMPL-24642): update Development Guide to reflect consolidated structure
This commit is contained in:
@@ -9,18 +9,35 @@ By following a *code-first approach*, developers ensure consistency, traceabilit
|
||||
Development must always begin in a local environment. This allows developers to rapidly iterate, test business logic, and validate DAG (Directed Acyclic Graph) structures without impacting production data.
|
||||
|
||||
### 2.1 Project Layout
|
||||
To ensure compatibility with the Simpl-Open platform, every Dagster code location must adhere to the following directory structure:
|
||||
This repository (`template-code-location`) serves as the **single consolidated code location** for all data services workflows. It contains the jobs, ops, and configurations previously spread across `data-processing`, `dataframe-level-anonymisation`, and `field-level-pseudo-anonymisation`.
|
||||
|
||||
```text
|
||||
project-root/
|
||||
├── dagster_code_location/
|
||||
│ ├── jobs/ # Executable workflows
|
||||
│ ├── ops/ # Individual functional units (business logic)
|
||||
│ ├── resources/ # External connections (Object storage, APIs, etc...)
|
||||
│ └── repository.py # Central entry point for the code location
|
||||
├── tests/ # Unit and integration tests
|
||||
├── Dockerfile # Containerization instructions
|
||||
├── pyproject.toml # Dependency management (Poetry/Pip/UV)
|
||||
└── README.md # Documentation
|
||||
template-code-location/
|
||||
├── src/
|
||||
│ └── template_code_location/
|
||||
│ ├── repository.py # Unified entry point (all jobs/sensors/resources)
|
||||
│ ├── data_processing/ # Data cleaning & transformation ops/jobs
|
||||
│ │ ├── config_models/
|
||||
│ │ ├── jobs.py
|
||||
│ │ └── ops.py
|
||||
│ ├── dataframe_level_anonymisation/ # k-anonymity, l-diversity, t-closeness
|
||||
│ │ ├── config_models/
|
||||
│ │ ├── jobs.py
|
||||
│ │ ├── ops.py
|
||||
│ │ └── utils.py
|
||||
│ ├── field_level_pseudo_anonymisation/ # Field-level encryption/hashing/redaction
|
||||
│ │ ├── config_models/
|
||||
│ │ ├── techniques/
|
||||
│ │ ├── jobs.py
|
||||
│ │ ├── ops.py
|
||||
│ │ ├── unstructured_ops.py
|
||||
│ │ └── utils.py
|
||||
│ ├── jobs/ # Template example jobs
|
||||
│ └── ops/ # Template example ops
|
||||
├── tests/ # All tests (migrated from source repos)
|
||||
├── Dockerfile
|
||||
├── pyproject.toml
|
||||
└── README.md
|
||||
```
|
||||
|
||||
### 2.2 Code Examples (Ops, Jobs, and Definitions)
|
||||
|
||||
Reference in New Issue
Block a user