Skip to content

Commit 60ba776

Browse files
feat: Implement initial FlowerPower web application
I've implemented Phase 1 and early Phase 2 of the web application development plan. Features include: - Basic Sanic app setup with Datastar and htpy integration. - FlowerPower project listing and creation. - Project configuration viewing and editing (raw YAML). - Pipeline listing within projects. - New pipeline boilerplate creation. - Pipeline configuration viewing and editing (raw YAML). - Pipeline DAG display (SVG with Graphviz, with fallbacks). You can access all functionality through the web interface starting at the `/` route, with project management under `/projects` and pipeline management nested under specific projects.
1 parent 5d89100 commit 60ba776

File tree

8 files changed

+753
-1
lines changed

8 files changed

+753
-1
lines changed

flowerpower_webapp/README.md

Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
# FlowerPower Web Application
2+
3+
This is a Sanic web application for FlowerPower.
4+
5+
## Running the application
6+
7+
1. Install the necessary dependencies:
8+
```bash
9+
pip install flowerpower[webserver]
10+
```
11+
2. **Project Discovery:**
12+
- The web application looks for FlowerPower projects in a directory named `fp_projects` located at the root of this repository.
13+
- Each subdirectory within `fp_projects` is considered a potential project if it contains a `conf/project.yml` file.
14+
- If the `fp_projects` directory does not exist at the repository root, the application will fall back to searching for projects in the current working directory (`.` when you run `python flowerpower_webapp/main.py`). This allows you to run the web app from within a single project's directory.
15+
- To get started, you can create the `fp_projects` directory in the repository root:
16+
```bash
17+
mkdir fp_projects
18+
```
19+
- Then, place your FlowerPower project folders inside `fp_projects`. For example:
20+
```
21+
fp_projects/
22+
├── my_project_alpha/
23+
│ ├── conf/
24+
│ │ └── project.yml
25+
│ └── ... (other project files)
26+
└── my_project_beta/
27+
├── conf/
28+
│ └── project.yml
29+
└── ... (other project files)
30+
```
31+
- If no projects are found in `fp_projects` (when it's the designated root), the application will automatically create a couple of sample projects (`sample_project_1`, `sample_project_2`) for demonstration purposes when you first run it.
32+
33+
3. Run the application:
34+
```bash
35+
python flowerpower_webapp/main.py
36+
```
37+
38+
The application will be available at `http://localhost:8000`. You can navigate to `/projects` to see the list of discovered projects.
39+
40+
### Creating a New Project
41+
42+
You can create a new FlowerPower project directly from the web interface:
43+
44+
1. Navigate to the main page or the project listing page (`/projects`).
45+
2. Click on the "Create New Project" link/button. This will take you to the `/projects/new` page.
46+
3. Enter a unique name for your new project in the input field. The project name should consist of letters, numbers, underscores, hyphens, or dots.
47+
4. Click the "Create Project" button.
48+
5. If successful, the application will create:
49+
* A new directory for your project under the `fp_projects` directory (e.g., `fp_projects/your_project_name/`).
50+
* Inside the project directory:
51+
* `conf/project.yml` (a default project configuration file).
52+
* `conf/pipelines/` (a directory for your pipeline configuration files).
53+
* `pipelines/` (a directory for your pipeline code modules).
54+
6. You will see a success message, and the newly created project will appear on the `/projects` listing page.
55+
7. If there's an error (e.g., project name already exists, invalid characters), an error message will be displayed.
56+
57+
### Viewing and Editing Project Configuration
58+
59+
Once a project is created or discovered, you can view and edit its `project.yml` configuration:
60+
61+
1. Navigate to the project listing page (`/projects`).
62+
2. Click on the name of the project you wish to view or edit. This will take you to the project's detail page (e.g., `/projects/your_project_name`).
63+
3. The content of the project's `conf/project.yml` file will be displayed in a text area.
64+
4. **Editing:**
65+
* You can directly modify the YAML content in the text area.
66+
* After making your changes, click the "Save Configuration" button.
67+
5. **Validation:**
68+
* The application will first check if the entered text is valid YAML.
69+
* Then, it will validate if the YAML content conforms to the `ProjectConfig` schema (e.g., checking for required fields, correct data types).
70+
6. **Outcome:**
71+
* If the configuration is valid and saved successfully, a success message will be displayed, and the `project.yml` file on the server will be updated.
72+
* If there are errors (e.g., invalid YAML format, schema validation failure, file system issues), an error message will be displayed, and the file will not be saved.
73+
7. **File Not Found:**
74+
* If a project directory exists but its `conf/project.yml` file is missing, the page will display an error message.
75+
* A default `project.yml` template for that project (based on its name) will be loaded into the text area. You can then modify and save this template to create the `project.yml` file.
76+
77+
### Listing Project Pipelines
78+
79+
You can view a list of all available pipelines within a specific FlowerPower project:
80+
81+
1. Navigate to the project's detail page (where you view/edit `project.yml`).
82+
2. Click on the "View Pipelines" link. This will take you to the pipeline listing page for that project (e.g., `/projects/your_project_name/pipelines`).
83+
3. The page will display a list of all pipelines discovered in the project.
84+
* FlowerPower discovers pipelines by looking for Python files (e.g., `my_pipeline.py`) in the project's `pipelines/` directory that have a corresponding configuration file (e.g., `my_pipeline.yml`) in the `conf/pipelines/` directory.
85+
4. Each listed pipeline name will be a link to a (currently placeholder) page for that specific pipeline's details.
86+
5. If no pipelines are found in the project, a "No pipelines found for this project." message will be displayed.
87+
6. If there's an error (e.g., the project's `conf/project.yml` is missing or malformed, which might be needed by the `PipelineManager`), an error message will be shown.
88+
7. From this page, you can also navigate to:
89+
* "Add New Pipeline".
90+
* Back to the project's configuration page.
91+
* Back to the list of all projects.
92+
* Back to the home page.
93+
94+
### Adding a New Pipeline to a Project
95+
96+
You can create the basic file structure for a new pipeline within a project:
97+
98+
1. Navigate to the pipeline listing page for the desired project (e.g., `/projects/your_project_name/pipelines`).
99+
2. Click on the "Add New Pipeline" button/link. This will take you to the form for adding a new pipeline (e.g., `/projects/your_project_name/pipelines/new`).
100+
3. **Enter Pipeline Name:**
101+
* In the "Pipeline Name" input field, enter a name for your new pipeline.
102+
* The name must be a valid Python identifier (e.g., `my_new_data_pipeline`). This generally means it should consist of letters, numbers, and underscores, and cannot start with a number.
103+
4. Click the "Create Pipeline" button.
104+
5. **Outcome:**
105+
* **Success:** If the name is valid and the pipeline doesn't already exist, the application will:
106+
* Create a new Python file: `fp_projects/your_project_name/pipelines/your_pipeline_name.py`.
107+
* Create a new YAML configuration file: `fp_projects/your_project_name/conf/pipelines/your_pipeline_name.yml`.
108+
* A success message will be displayed, and the input field for the pipeline name will be cleared.
109+
* The newly created pipeline will then appear on the pipeline list page for that project.
110+
* **Error:**
111+
* If the pipeline name is invalid (e.g., contains spaces, starts with a number), an error message will be displayed.
112+
* If a pipeline with that name already exists in the project, an error message will indicate this.
113+
* If there are other issues (e.g., problems with the `PipelineManager` or file system permissions), a general error message will be shown.
114+
6. After creating a pipeline, you will need to manually edit its `.py` and `.yml` files to define its functionality and configuration. The web application currently only creates the boilerplate files.
115+
116+
### Viewing and Editing Pipeline Configuration
117+
118+
For each pipeline within a project, you can view and edit its specific YAML configuration file (e.g., `your_pipeline_name.yml`):
119+
120+
1. Navigate to the pipeline listing page for the desired project (e.g., `/projects/your_project_name/pipelines`).
121+
2. Click on the name of the pipeline whose configuration you wish to view or edit. This will take you to that pipeline's detail page (e.g., `/projects/your_project_name/pipelines/your_pipeline_name`).
122+
3. The content of the pipeline's `.yml` file (located in `fp_projects/your_project_name/conf/pipelines/`) will be displayed in a text area.
123+
4. **Editing:**
124+
* You can directly modify the YAML content in the text area.
125+
* After making your changes, click the "Save Pipeline Configuration" button.
126+
5. **Validation:**
127+
* The application will first check if the entered text is valid YAML.
128+
* Then, it will validate if the YAML content conforms to the `PipelineConfig` schema (e.g., checking for expected sections like `params`, `run`, etc.).
129+
6. **Outcome:**
130+
* If the configuration is valid and saved successfully, a success message will be displayed, and the pipeline's `.yml` file on the server will be updated.
131+
* If there are errors (e.g., invalid YAML format, schema validation failure, file system issues), an error message will be displayed, and the file will not be saved.
132+
7. **File Not Found:**
133+
* If a pipeline's `.yml` configuration file is missing (e.g., if only the `.py` file exists or after a new pipeline is created via the UI but before its config is saved for the first time):
134+
* The page will display an error message indicating the file was not found.
135+
* A default `PipelineConfig` template for that pipeline (based on its name) will be loaded into the text area.
136+
* You can then modify and save this template to create the pipeline's `.yml` configuration file.
137+
138+
### Visualizing Pipeline DAG (Directed Acyclic Graph)
139+
140+
You can view a visual representation of a pipeline's structure (its DAG):
141+
142+
1. Navigate to the pipeline's detail/configuration page (e.g., `/projects/your_project_name/pipelines/your_pipeline_name`).
143+
2. Click on the "View DAG" link. This will take you to the DAG visualization page for that pipeline (e.g., `/projects/your_project_name/pipelines/your_pipeline_name/dag`).
144+
3. **DAG Display:**
145+
* If successful and `graphviz` is installed on the server, an SVG image of the DAG will be displayed.
146+
* If `graphviz` is not installed, or if there are errors in the pipeline's Python code or configuration that prevent DAG generation, an error message will be displayed. The page might show a placeholder or a textual representation if SVG generation fails.
147+
4. **Dependencies:**
148+
* DAG visualization as an SVG image requires **Graphviz** to be installed on the server where the FlowerPower web application is running. If Graphviz is not found, the application will attempt to gracefully handle this by showing an error message.
149+
5. **Troubleshooting:**
150+
* If you see an error message instead of a DAG, check the following:
151+
* Ensure Graphviz is installed and in the system's PATH.
152+
* Verify that the pipeline's Python module (`.py` file) and its YAML configuration (`.yml` file) are correctly defined and free of syntax errors.
153+
* Check the application logs for more detailed error messages.
154+
6. From the DAG page, you can navigate back to the pipeline's configuration, the pipeline list for the project, or the home page.

flowerpower_webapp/__init__.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
# This file makes flowerpower_webapp a Python package.

0 commit comments

Comments
 (0)