Migrate from bash to modern Python with UV - #2
Open
code-inflation wants to merge 5 commits into
Open
Conversation
Major refactoring to replace the shell script with a modern Python implementation using best practices and tooling: - Replace scrape.sh with Python scraper using httpx and Pydantic - Add UV for fast, reliable dependency management - Add comprehensive type hints for mypy strict mode - Add pytest test suite with 63% code coverage - Add Ruff for linting and formatting - Update GitHub Actions workflow to use Python/UV - Add development documentation to README The new implementation maintains full compatibility with the existing data format while providing better maintainability, testability, and developer experience. Key improvements: - Type-safe data models with Pydantic - Robust error handling and logging - Comprehensive test coverage - Fast dependency resolution with UV - Modern code quality tools (ruff, mypy) - CI/CD integration with linting and testing Technical details: - Python 3.11+ with strict type checking - Project structure: src/scraper/ with proper package layout - Tests in tests/ directory - All JSON output remains identical to bash version
After comprehensive testing, reordered Pydantic model fields to ensure byte-for-byte identical JSON output compared to the original shell script. Testing revealed: - All 7 faction files now serialize identically - Field order matches: id, canton, cantonName, council, faction, factionName, firstName, lastName, number, party, partyName, active, code, gender, officialDenomination, salutationLetter, salutationTitle, factionFunction - Faction top-level order: id, abbreviation, code, members, name, shortName - 2-space indentation preserved - UTF-8 characters handled correctly - Null values lowercase (not Python's None) Verified with round-trip test loading existing faction JSONs, deserializing through models, and re-serializing - all match perfectly.
Fixes issue where workflow would run on every branch push, wasting CI minutes and potentially committing to feature branches. Changes: - Added branch filter to only run on push to master - Scheduled jobs already run on default branch (master) by design - workflow_dispatch still available on all branches for manual testing This prevents the workflow from failing when the cron job runs tonight on master (which still has the shell script), and ensures production scraping only happens on the master branch after the PR is merged.
Created two workflows for better separation of concerns: 1. CI workflow (ci.yml): - Runs on all branches and PRs - Linting (ruff check + format check) - Type checking (mypy) - Tests (pytest with coverage) - Fast feedback for development 2. Scrape workflow (scrape.yml): - Runs on all branches, scheduled, and manual trigger - Always executes the scraper - Shows diff in logs (dry-run preview) - Only commits changes when on master branch - Non-master branches: dry-run mode with detailed diff output Benefits: - Test scraper changes on feature branches without committing - CI runs independently for fast quality feedback - Preview what data would change before merging - Production scraping only happens on master - Scheduled job remains on master (default branch)
Added --all-extras flag to uv sync to install dev dependencies (ruff, mypy, pytest) which are required for linting and testing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Major refactoring to replace the shell script with a modern Python
implementation using best practices and tooling:
The new implementation maintains full compatibility with the existing
data format while providing better maintainability, testability, and
developer experience.
Key improvements:
Technical details: