Feature/mra2 44 GitHub actions ci pipeline enhancement - #31
Open
surya-1729 wants to merge 87 commits into
Open
Conversation
surya-1729
commented
Feb 6, 2026
surya-1729
commented
- implemented ci.yml, to carry on linting and code quality checks with github runners in github actions
Updated the README to provide a comprehensive overview of the MLOps Road Accident Prediction project, including project objectives, structure, and technology stack. Added a detailed technical execution plan for Phase 1 in a new document, outlining team roles, tasks, and deadlines. Introduced a roadmap for future phases of the project.
Created .python-version and .tool-versions files to specify Python version 3.11. Added a Makefile with various commands for project setup, dependency management, and testing. Introduced pyproject.toml for project metadata and dependencies, including development tools.
…port Add initial project configuration files
Added .venv to .gitignore to exclude virtual environment files. Updated README to include new development commands using Makefile, installation instructions for UV, and clarified Python version management with .python-version and .tool-versions files.
…port Update .gitignore and enhance README with development commands
MRA2-3: DagsHub and DVC Setup
- Introduced a new `.env.example` file for Dagshub credentials setup. - Enhanced the Makefile with new DVC commands for initialization, remote setup, and status checks. - Updated `README.md` to include instructions for setting up Dagshub and DVC, emphasizing the need for a manually edited `.env` file. - Added a script `setup_dvc_remote.sh` to automate DVC remote configuration using credentials from the `.env` file. - Improved data import and preprocessing scripts for better structure and error handling. - Updated type checking and linting configurations in `pyproject.toml` for improved code quality.
…ync-Basics Add DVC and Dagshub integration
- Added `preprocess.py` with functions for preprocessing input features and aligning them with model expectations. - Updated `predict_model.py` to load model artifacts, preprocess features, and make predictions using the new preprocessing functions. - Enhanced error handling and logging for better traceability during predictions. - Modified `test_features.json` to include additional feature fields required for inference. - Refactored `train_model.py` to save feature metadata alongside the trained model for consistent inference.
- Added support for loading environment variables from a .env file in the Makefile. - Improved error handling in the dvc-setup-remote target to ensure necessary Dagshub credentials are set. - Updated README to inform Windows users about the requirement for a Unix-like environment for running Makefile commands.
…ync-Basics Feature/mra2 8 environment remote sync basics
- Starts from the raw data stored with DVC - Runs stages preprocess, build_features, and train_eval as declared in dvc.yaml - Finally stores the model evaluation result in training_metrics.json
Feature/mra2 2 setup dvc pipeline
Update environment configuration and model parameters for improved tr…
- Updated dvc.yaml to load environment variables from .env file before executing commands. - Added new dependencies (matplotlib, seaborn, mlflow) to requirements.txt for visualization and model tracking. - Introduced manage_model_registry.py script for managing MLflow model registry operations (listing models, transitioning stages, promoting models, etc.). - Modified model_config.yaml to include MLflow model registry configuration options. - Enhanced predict_model.py to support loading models from MLflow Model Registry by stage or version. - Updated train_model.py to log visualizations (confusion matrix, ROC curve, feature importance) to MLflow and improved metric logging. - Refactored save_metrics function to save DVC-compliant metrics and detailed training reports.
Enhance MLflow integration and model management
- Added model trainers for XGBoost, Random Forest, Logistic Regression, and LightGBM. - Created a multi-model orchestrator to train and compare models. - Integrated MLflow for tracking experiments, logging metrics, and visualizations. - Developed utilities for logging models and visualizations to MLflow. - Updated README with usage instructions and configuration details. - Ensured compatibility with existing single model training code.
- Updated dvc.yaml to load environment variables from .env file before executing commands. - Added new dependencies (matplotlib, seaborn, mlflow) to requirements.txt for visualization and model tracking. - Introduced manage_model_registry.py script for managing MLflow model registry operations (listing models, transitioning stages, promoting models, etc.). - Modified model_config.yaml to include MLflow model registry configuration options. - Enhanced predict_model.py to support loading models from MLflow Model Registry by stage or version. - Updated train_model.py to log visualizations (confusion matrix, ROC curve, feature importance) to MLflow and improved metric logging. - Refactored save_metrics function to save DVC-compliant metrics and detailed training reports.
…update configurations
chore: improve docker workflow and docs
- Introduced `docker-compose.test.yml` to override authentication rate limits and Nginx configuration for testing purposes. - Updated `docker-compose.yml` to include environment variables for rate limiting in production. - Enhanced the Makefile to support running tests with the new test configuration. - Added a dedicated Nginx configuration for test runs to prevent hitting rate limits and service unavailability. - Modified test cases to accommodate the new rate limits and ensure robust handling of rate-limiting responses.
feat: enhance training service with model configuration management
- Updated `single_prediction` and `batch_prediction` endpoints to return probability values alongside predictions. - Modified response schemas to include `probability` and `probabilities` fields. - Refactored prediction logic to compute probabilities using `predict_proba` when available. - Enhanced test cases to validate the presence and range of probability values in responses.
- Introduced a new Streamlit dashboard service for the Control Center UI, allowing users to input incident details and receive severity predictions. - Configured Dockerfile and requirements for the dashboard service, ensuring all necessary dependencies are included. - Updated Nginx configuration to route requests to the Streamlit dashboard, enhancing the overall microservices architecture. - Implemented user authentication and role-based access for dashboard features, including admin functionalities for data and user management.
- Added a reusable job runner that executes synchronous functions in a separate thread while capturing logs. - Integrated the job runner into the data and training services to provide real-time logging updates during long-running tasks. - Enhanced job store and response models to include logs and messages for better tracking of job statuses. - Updated the Streamlit dashboard to display job logs and progress, improving user experience in monitoring tasks. - Adjusted dependencies and requirements for the dashboard service to support new features.
- Updated the application to support PostgreSQL as the primary database for user and job data storage. - Enhanced the `.env.example` file to include PostgreSQL configuration options and updated the database connection string. - Implemented SQLAlchemy models for users and jobs, allowing for persistent storage and retrieval. - Added new API endpoints for password reset functionality, including token generation and validation. - Refactored authentication and job management logic to utilize the new database layer. - Updated Docker Compose configuration to include a PostgreSQL service with health checks and persistent storage. - Adjusted dependencies in `requirements.txt` and `pyproject.toml` to include SQLAlchemy and psycopg2-binary for database interactions. - Enhanced job store functionality to support both in-memory and PostgreSQL-backed storage, improving scalability and reliability.
- Updated the `regular_user_credentials` fixture to ensure the password meets minimum length requirements, with a default fallback. - Improved the `user_token` fixture to include user deletion during teardown, ensuring test users are removed after tests run. - Adjusted assertions in the `TestAuthService` class to account for additional response status codes during login attempts and added teardown logic for lockout test users.
- Modified the `pd.read_csv` call in `BaseTrainer` to include the `on_bad_lines="warn"` parameter, allowing for better handling of malformed lines in the features file.
…ization Feature/mra2 46 api security optimization
Feature/mra2 55 dashboard foundation
- Initialized session state for prediction results and features to persist data across reruns. - Updated prediction logic to store results in session state and display them accordingly. - Improved error handling by resetting session state on authentication failure. - Adjusted map rendering logic to utilize stored prediction features for location display.
- Introduced a new geocoding microservice using FastAPI, supporting address geocoding via Nominatim. - Updated Docker configuration to include the geocode service with health checks and environment variables. - Enhanced the Streamlit dashboard to integrate address input and geocoding functionality, allowing users to select locations. - Implemented API routes for geocoding and address suggestions, with compliance to Nominatim usage policies. - Updated requirements and configuration files to support the new geocoding service and its dependencies.
fix: update CSV reading to handle bad lines with a warning
…rvices - Enhanced the health check endpoint in the geocode service to log detailed information about service status and configuration. - Added comprehensive error handling in the health check logic to capture and log exceptions, ensuring better visibility into issues. - Updated the API client to include logging for geocoding service health checks, improving debugging capabilities for service failures.
Feature/mra2 geocoding api & fix
- Fixed Docker build strategy to use Dockerfile.python with proper context - Added build targets for each service (auth_service, data_service, predict_service, train_service) - Removed incorrect hashFiles conditions that were causing silent failures - All services now build from single multi-stage Dockerfile consistently with docker-compose
- Replaced with comprehensive ci.yaml - python-app.yml was basic template with limited testing - New ci.yaml includes multi-version testing, integration tests, security scanning, and Docker builds
…Hub-Actions-CI-Pipeline-Enhancement # Please enter a commit message to explain why this merge is necessary, # especially if it merges an updated upstream into a topic branch. # # Lines starting with '#' will be ignored, and an empty message aborts # the commit.
chrmei
deleted the
feature/MRA2-44-GitHub-Actions-CI-Pipeline-Enhancement
branch
February 11, 2026 14:19
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.