Add initial README.md with project overview and usage instructions
This commit is contained in:
ILay
169
README.md
169
README.md
@@ -0,0 +1,169 @@
|
||||
# Template Code Location
|
||||
|
||||
> Purpose: `template-code-location` is the predefined Application Workflow Template for Simpl-Open providers. It offers a standard Dagster code location that providers can download, extend, and package when building new application workflows, reducing onboarding effort and avoiding configuration drift.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Overview](#overview)
|
||||
2. [What This Template Includes](#what-this-template-includes)
|
||||
3. [Repository Layout](#repository-layout)
|
||||
4. [Prerequisites](#prerequisites)
|
||||
5. [Quick Start](#quick-start)
|
||||
6. [Development Guide](#development-guide)
|
||||
7. [Testing](#testing)
|
||||
8. [Packaging and Deployment](#packaging-and-deployment)
|
||||
9. [Documentation](#documentation)
|
||||
10. [Contributing](#contributing)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
This repository provides a standardized baseline for Dagster code locations in the Simpl-Open ecosystem.
|
||||
|
||||
It is intended for providers who need a ready-made starting point for new application workflows. Instead of creating a code location from scratch, a provider can use this template to:
|
||||
|
||||
- start from an agreed repository structure
|
||||
- reuse the standard Dagster packaging model
|
||||
- follow the same conventions for jobs, ops, resources, and definitions
|
||||
- integrate existing Simpl-Open workflow packages as dependencies
|
||||
- add custom workflow logic with minimal setup effort
|
||||
|
||||
The template is intentionally simple. It includes a small example job and ops that demonstrate the expected pattern for creating new workflows, while the main Dagster entrypoint already shows how to register both local jobs and jobs imported from external packages.
|
||||
|
||||
## What This Template Includes
|
||||
|
||||
- A working Dagster code location package under `src/template_code_location`
|
||||
- A repository entrypoint in `src/template_code_location/repository.py`
|
||||
- Example local ops in `src/template_code_location/ops/ops.py`
|
||||
- Example local job in `src/template_code_location/jobs/jobs.py`
|
||||
- Preconfigured dependencies on shared Simpl-Open workflow packages through `pyproject.toml`
|
||||
- A Dockerfile for container packaging
|
||||
- Supporting guidance in the `documents/` folder
|
||||
|
||||
This template is not meant to contain business-specific logic by default. Providers are expected to replace or extend the sample workflow with their own application-specific jobs and operations.
|
||||
|
||||
## Repository Layout
|
||||
|
||||
```text
|
||||
template-code-location/
|
||||
|- src/
|
||||
| \- template_code_location/
|
||||
| |- __init__.py
|
||||
| |- repository.py
|
||||
| |- jobs/
|
||||
| | |- __init__.py
|
||||
| | \- jobs.py
|
||||
| \- ops/
|
||||
| |- __init__.py
|
||||
| \- ops.py
|
||||
|- documents/
|
||||
|- Dockerfile
|
||||
|- pyproject.toml
|
||||
|- uv.lock
|
||||
\- README.md
|
||||
```
|
||||
|
||||
Key files:
|
||||
|
||||
- `src/template_code_location/repository.py`: Dagster `Definitions` entrypoint used to register jobs, sensors, resources, and loggers
|
||||
- `src/template_code_location/jobs/jobs.py`: place for custom workflow definitions
|
||||
- `src/template_code_location/ops/ops.py`: place for custom operational steps used by local jobs
|
||||
- `pyproject.toml`: package metadata and Git-based dependencies on shared Simpl-Open packages
|
||||
- [documents/Development Guide.md](documents/Development%20Guide.md): workflow lifecycle guidance for creating, editing, and deleting workflows
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Before using this template, make sure the following are available:
|
||||
|
||||
- Python 3.12+
|
||||
- `uv` package manager
|
||||
- Access to the Simpl-Open Git repositories referenced by `pyproject.toml`
|
||||
- A local or shared Dagster environment for validation
|
||||
- Docker, if you plan to build the image locally
|
||||
|
||||
Depending on the workflows you build, you may also need access to supporting platform services such as object storage, PostgreSQL, Vault, or Kubernetes.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Install dependencies
|
||||
|
||||
```powershell
|
||||
uv sync --dev
|
||||
```
|
||||
|
||||
### 2. Start the Dagster code server locally
|
||||
|
||||
```powershell
|
||||
uv run dagster code-server start -h 0.0.0.0 -p 4000 -f src/template_code_location/repository.py
|
||||
```
|
||||
|
||||
This starts the code location using the repository entrypoint already defined in the template.
|
||||
|
||||
### 3. Review the sample workflow
|
||||
|
||||
The template ships with:
|
||||
|
||||
- `fetch_data` and `process_data` ops in `src/template_code_location/ops/ops.py`
|
||||
- `data_processing_job` in `src/template_code_location/jobs/jobs.py`
|
||||
|
||||
These are example building blocks only. Providers should rename, replace, or extend them to match the target application workflow.
|
||||
|
||||
## Development Guide
|
||||
|
||||
Workflow creation, update, deletion, and the expected way to work with external code locations are covered in [documents/Development Guide.md](documents/Development%20Guide.md).
|
||||
|
||||
Use that guide as the primary reference when customizing this template for a provider-specific workflow.
|
||||
|
||||
## Testing
|
||||
|
||||
Custom workflows should always be validated before publication.
|
||||
|
||||
Recommended checks:
|
||||
|
||||
- add unit tests for local jobs and ops
|
||||
- run `pytest` locally after every workflow change
|
||||
- verify that the Dagster code server can load `src/template_code_location/repository.py` without import or definition errors
|
||||
|
||||
Example command:
|
||||
|
||||
```powershell
|
||||
uv run pytest
|
||||
```
|
||||
|
||||
At the moment this template does not include a `tests/` directory, so providers should add one as part of implementing their own workflows.
|
||||
|
||||
## Packaging and Deployment
|
||||
|
||||
This repository is designed to be packaged as a Docker image and promoted through the standard Simpl-Open delivery flow.
|
||||
|
||||
High-level process:
|
||||
|
||||
1. Implement and validate the workflow locally.
|
||||
2. Commit the changes to a feature branch.
|
||||
3. Open a pull request for review.
|
||||
4. Build and publish the container image through CI/CD.
|
||||
5. Update the deployment configuration to use the new image.
|
||||
6. Verify the new code location in Dagster.
|
||||
|
||||
Use immutable image tags such as semantic versions or commit-based identifiers rather than `latest`.
|
||||
|
||||
## Documentation
|
||||
|
||||
Additional guidance is available in the repository documents:
|
||||
|
||||
- [documents/Development Guide.md](documents/Development%20Guide.md): workflow creation, edit, deletion, and deployment process
|
||||
- [documents/Output Separation and Non-Overwrite Principles.md](documents/Output%20Separation%20and%20Non-Overwrite%20Principles.md): output handling expectations for workflow implementations
|
||||
|
||||
## Contributing
|
||||
|
||||
When extending this template:
|
||||
|
||||
- keep the repository structure stable so onboarding stays predictable
|
||||
- preserve the separation between ops, jobs, and repository definitions
|
||||
- avoid hardcoded credentials and environment-specific values
|
||||
- update tests and documentation together with workflow changes
|
||||
|
||||
This repository should remain a reusable baseline for providers, not a dumping ground for unrelated or one-off implementations.
|
||||
|
||||
Reference in New Issue
Block a user