Complete refactoring #21
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Removed unused imports and cleaned up code in test_cli_integration.py. - Simplified imports in test_registry.py and adjusted PipelineConfig usage. - Updated test_runner.py to reflect changes in PipelineConfig and ProjectAdapterConfig. - Enhanced test_misc.py with better formatting and organization of test cases. - Removed deprecated dependencies from uv.lock and updated flowerpower version to 0.11.6.20.
- Added a new Pipeline class that encapsulates execution logic, configuration, and project context. - Implemented run method with retry logic, input handling, and callback execution. - Introduced private methods for executing the pipeline, managing executors, and setting up adapters. - Enhanced error handling and logging throughout the execution process. refactor(registry): Improve PipelineRegistry initialization and caching - Updated PipelineRegistry to accept base directory and storage options. - Added from_filesystem factory method for creating PipelineRegistry from filesystem parameters. - Implemented caching for loaded pipelines, configurations, and modules to optimize performance. - Enhanced module path management to ensure proper loading of pipeline modules.
…ng the manager. Introduce unit tests for the pipeline and flower power project, ensuring proper functionality and validation of methods.
- Cleaned up import statements across multiple files for better organization. - Removed unnecessary blank lines and adjusted spacing for consistency. - Simplified dictionary and list comprehensions for clarity. - Updated test cases to use consistent string formatting and improved readability. - Ensured all code adheres to PEP 8 style guidelines.
Move job queue related commands (enqueue, schedule, run-job, list-schedules) from pipeline.py to job_queue.py for better organization and consistency. This centralizes job queue functionality in its dedicated module while simplifying the pipeline CLI.
…adability and maintainability
…adability and maintainability
There was a problem hiding this comment.
Pull Request Overview
This pull request focuses on streamlining the project by removing APScheduler support and establishing RQ as the sole job queue backend. It also includes comprehensive cleanup of obsolete files and documentation.
- Removes all APScheduler references from documentation and configuration, making RQ the only supported job queue backend
- Cleans up obsolete configuration files and unused documentation
- Updates Docker Compose setup with service adjustments and new Node-RED service
Reviewed Changes
Copilot reviewed 50 out of 320 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| docs/mkdocs/site/api/init/index.html | Added new API documentation page for the init function |
| docs/mkdocs/site/api/index.html | Added new API overview documentation page |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
| <p>```python | ||
| from flowerpower import init</p> | ||
| <h1 id="initialize-a-new-project">Initialize a new project<a class="headerlink" href="#initialize-a-new-project" title="Permanent link">¶</a></h1> | ||
| <p>project = init(name="my-new-project", job_queue_type="rq")</p> |
There was a problem hiding this comment.
The code example is malformed. The opening triple backticks on line 516 should be properly closed, and the code should not be rendered as HTML paragraph and heading elements. This should be a proper markdown code block.
Suggested change
| <p>project = init(name="my-new-project", job_queue_type="rq")</p> | |
| <pre><code class="language-python">from flowerpower import init | |
| project = init(name="my-new-project", job_queue_type="rq") | |
| </code></pre> |
| <p>This section provides a detailed reference for the FlowerPower API.</p> | ||
| <h2 id="core-components">Core Components<a class="headerlink" href="#core-components" title="Permanent link">¶</a></h2> | ||
| <ul> | ||
| <li><a href="./flowerpowerproject.md">FlowerPowerProject</a></li> |
There was a problem hiding this comment.
The link to FlowerPowerProject points to a non-existent file path './flowerpowerproject.md'. Based on the navigation structure, it should point to 'flowerpower/' to match the existing navigation pattern.
Suggested change
| <li><a href="./flowerpowerproject.md">FlowerPowerProject</a></li> | |
| <li><a href="flowerpower/">FlowerPowerProject</a></li> |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request primarily removes support and documentation for APScheduler as a job queue backend, making RQ (Redis Queue) the only supported option. It also cleans up configuration files, updates the Docker Compose setup, and removes some unused or obsolete files. Below are the most important changes grouped by theme:
Job Queue Backend Simplification:
doc.mdfile and project configuration examples have been updated to reflect RQ as the sole supported option. [1] [2] [3]Configuration and Documentation Cleanup:
alembic.ini,.repomixignore,.roo/mcp.json) and an introductory documentation chapter (docs/01_pipelineregistry_.md) have been removed. [1] [2] [3] [4]CHANGELOG.md) has been updated to reflect recent refactoring and feature changes.Docker Compose and Service Adjustments:
dockge,rq-dashboard), adjusted port mappings for Redis/Valkey, and added a newnoderedservice. [1] [2] [3] [4] [5] [6] [7] [8] [9]Miscellaneous:
anypath.yamlfor configuration or testing purposes.These changes help streamline the project by focusing support on a single job queue backend, cleaning up legacy or unused files, and improving the clarity and maintainability of both documentation and service configuration.