Feature/config and processing refactor #69
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit enhances the generate_rocky_config.py script with two key improvements:
1. Flexible version matching for RHEL 8/9/10+ compatibility:
- Major-only filtering (e.g., --version 9): Matches any minor version within
that major version (9.0, 9.1, 9.2, 9.6, etc.)
- Full version filtering (e.g., --version 9.6): Requires exact match to the
specified major.minor version
This addresses differences in Red Hat's advisory format across RHEL versions:
- RHEL 8 & 9: Advisories typically don't include minor versions
- RHEL 10+: Advisories now include minor versions (e.g., "RHEL 10.2")
The flexible matching ensures that repository configurations can be generated
with appropriate version matching rules (NULL match_minor_version for RHEL 8/9,
specific match_minor_version for RHEL 10+).
2. Custom mirror naming with --mirror-name-base option:
- Allows specifying a custom base name for generated mirror configurations
- Example: --mirror-name-base "Rocky Linux 9" generates "Rocky Linux 9 x86_64"
instead of "Rocky Linux 9.6 x86_64"
- Useful for creating legacy product entries or custom naming schemes
- Works in combination with --name-suffix for additional flexibility
These changes improve Apollo's ability to generate configurations that align
with Red Hat's advisory matching requirements across different major versions.
- Remove redundant None and empty string checks in mirror name building - Consolidate version filtering logic into single condition block - Eliminate unnecessary ternary operator in version parsing
Any advisory that addresses at least one CVE should be considered a Security Advisory and should returned by the OSV api. Instead of filtering strictly on the advisory "kind" (eg- Security, Bug Fix, Enhancement) we should instead filter based on if there are associated CVEs for the given advisory.
Remove self-explanatory comments that restate what the code does: - Removed obvious filter condition comments - Removed type conversion comment - Removed severity calculation comment
This commit refactors the Red Hat CSAF parser to fix two major issues: 1. Modular Package Extraction Bug - Old code failed to extract modular packages due to ::module:stream suffix - New code extracts NEVRA directly from product_tree product_id field - Strips ::module:stream suffix while preserving full NEVRA with epoch - Fixes 12+ affected advisories (e.g., RHSA-2025:12008 for redis:7) 2. EUS Advisory Filtering - Detects EUS/E4S/AUS/TUS products via CPE and product name - Filters out EUS-only advisories during ingestion - Reduces processed advisories by ~50% - Skips advisories where all products are EUS-related Changes: - apollo/rhcsaf/__init__.py: - Added _is_eus_product() helper for EUS detection - Added _extract_packages_from_product_tree() for product_tree parsing - Updated extract_rhel_affected_products_for_db() to filter EUS products - Updated red_hat_advisory_scraper() to use new extraction and skip EUS-only - apollo/tests/test_rhcsaf.py: - Updated test data to include product_version entries - Added TestEUSDetection class (3 tests) - Added TestModularPackages class (1 test) - Added TestEUSAdvisoryFiltering class (1 test) Validation: - Standalone testing in temp/modular_package_fix/ confirmed: - 18 modular packages extracted (was 0) - Regular packages work identically (no regression) - EUS advisories correctly filtered - All data fields preserved (CVEs, Bugzillas, metadata)
The previous code incorrectly let releases.csv overwrite changes.csv timestamps.
This caused the workflow to miss advisory updates, as changes.csv contains the
most recent modification times while releases.csv contains original publication
dates.
With this fix, when Red Hat updates advisories (like the mass update on
2025-11-07), the workflow will correctly detect and reprocess them.
Changes:
- Reversed merge order: {**releases, **changes} so changes.csv takes precedence
- Updated comment to clarify the intended behavior
- Ensures updated advisories are reprocessed to catch corrections/additions
Add admin interface to view and update the last_indexed_at timestamp that controls which CSAF advisories are processed by the Poll RHCSAF workflow. Changes: - Add DatabaseService methods for getting and updating last_indexed_at - Add admin route handlers for timestamp management - Add UI section with date picker and automatic ISO 8601 conversion - Remove duplicate timestamp display from Poll RHCSAF section - Fix preview results text readability - Add comprehensive unit tests for DatabaseService - Update BUILD.bazel and CI workflow to include new tests
This commit fixes multiple issues in test_csaf_processing.py that caused
CI failures:
1. Missing unittest.main() call
- Added 'if __name__ == "__main__": unittest.main()' block
- Without this, Bazel's py_test runs the file as a script but never
executes the tests, causing false positives
- pytest doesn't need this (auto-discovers tests), but Bazel does
2. Fixed async test lifecycle methods
- Changed 'async def tearDown' to 'async def asyncTearDown'
- Removed incorrect @classmethod decorators from asyncSetUp/asyncTearDown
- These must be instance methods in unittest.IsolatedAsyncioTestCase
- Consolidated setUp logic into asyncSetUp
- Added close_test_db() call to asyncTearDown for proper cleanup
3. Updated test CSAF data structure
- Added product_version entries in product_tree (required by refactored parser)
- Changed from EUS to MAIN product variant (EUS products are filtered out)
- Added proper product_id, purl, and CPE format
- The refactored CSAF parser (commit ccb297e) extracts packages from
product_tree instead of vulnerabilities.product_status.fixed
4. Fixed test assertions
- Changed minor_version expectation from 4 to None (CPE has no minor version)
- Fixed test_no_fixed_packages to remove product_tree entries instead of
just clearing the fixed array
Root cause analysis:
- Bazel tests were never actually running (missing unittest.main())
- GitHub Actions tests were running via pytest in Integration Tests step
- pytest auto-discovers unittest tests without needing __main__ block
- This is why CI showed failures while local Bazel tests appeared to pass
All tests now pass in both Bazel and pytest environments.
Extracted magic constants from _is_eus_product() function to improve maintainability and readability: - EUS_CPE_PRODUCTS: CPE product identifiers for EUS variants - EUS_PRODUCT_NAME_KEYWORDS: Keywords for identifying EUS products Using frozenset for better performance on membership checks.
- Move product_name and cpe declarations closer to usage - Simplify modular package NEVRA extraction using split directly - Remove redundant nevra variable and empty string check
Replace explicit length comparison with truthiness check for red_hat_affected_products set.
Removed comments that simply restated what the code clearly does. Kept only comments that provide non-obvious context such as: - CPE format examples - Product ID format variations - Business logic explanations
- Remove redundant str() calls in f-strings - Use 'raise ... from e' to preserve exception chain
Converted nested helper functions to standalone pure functions: - _traverse_for_eus: Now takes and returns product_eus_map explicitly - _extract_packages_from_branches: Now takes and returns packages explicitly This makes the code more testable, readable, and eliminates hidden state mutations from closure variables.
Check if advisory only affects EUS products immediately after verifying vulnerabilities exist, before extracting packages, CVEs, and other data. This saves processing time for advisories that will be skipped anyway. Also cleaned up redundant product_full_name variable.
This commit addresses two validation issues that prevented importing
configuration files exported from production:
1. Export serializer converting version numbers to floats:
- The _json_serializer in admin_supported_products.py was converting
all Decimal types to float, including version numbers
- Version numbers (match_major_version, match_minor_version) should
be integers, not floats
- Updated serializer to check if Decimal is a whole number and
convert to int, preserving proper type semantics
2. Name validation rejecting parentheses:
- Production database contains legacy products with names like
"Rocky Linux 8.5 x86_64 (Legacy)"
- Validation pattern only allowed: letters, numbers, spaces, dots,
hyphens, and underscores
- Updated NAME_PATTERN to allow parentheses for legacy product naming
- Updated error message to reflect allowed characters
These changes ensure that configurations exported from production can
be successfully imported into development environments without manual
data cleanup.
Update test expectations to match the new behavior where whole number Decimal values are serialized as integers instead of floats. This aligns with the change to _json_serializer that preserves integer types for version numbers and other integer values.
Add boolean active field to supported_products_rh_mirrors table to allow disabling mirrors without deleting them. This preserves historical data and mirror relationships while preventing the mirror from being used in new advisory processing. Changes: - Add active column with default true to supported_products_rh_mirrors - Add database index on active field for query performance - Add migration script for schema change - Update DB model with active field - Add active field to admin UI forms (create and edit) - Update mirror filtering in workflow service to respect active flag - Update configuration import/export to handle active field - Add active field validation in form processing
The active checkbox wasn't saving properly when unchecked because HTML forms don't send unchecked checkbox values. This caused the field to always default to "true" in the backend. Added hidden input with value "false" before each checkbox, so the form always sends a value. Backend now parses all "active" values and takes the last one (which will be "true" if checked, "false" if unchecked). Changes: - Add hidden input to mirror edit and new templates - Update both POST endpoints to manually parse form data for active field - Remove default="true" from Form parameters that was masking the issue
Implemented multi-level sorting and visual status indicators for mirrors in the admin UI to improve usability and organization. Changes: - Sort mirrors by active status (active first), then major version (desc), then name (asc) for logical grouping - Add Status column with green "Active" and gray "Inactive" tags for clear visual differentiation - Update validation to allow parentheses in mirror names for descriptive naming like "Rocky Linux 9 (BaseOS)" - Fetch mirrors with explicit ordering in backend instead of relying on database insertion order
The RHMatcherWorkflow was processing all mirrors regardless of their active status, causing unnecessary fetches from mirrors that should be skipped. This adds a check to skip mirrors where active=False in the match_rh_repos activity.
The block_remaining_rh_advisories function had a nested loop bug where it would iterate over all mirrors from a prefetch, then inside that loop query for active mirrors and iterate over them again. This caused: 1. Redundant database queries (N queries for N total mirrors) 2. Processing each active mirror N times instead of once 3. Variable shadowing with the reused 'mirror' variable name Simplified to a single query for active mirrors and one processing loop.
Removed unnecessary hidden input fields and simplified the form parsing logic for the active checkbox in mirror creation and editing forms. Changes: - Replaced complex list indexing with simple membership check - Removed hidden input fields from both Jinja templates - Updated comments to reflect simpler approach The functionality remains identical, but the code is more readable and maintainable.
Add comprehensive tests for the simplified checkbox parsing logic and active field functionality: - Checkbox parsing for checked/unchecked/missing states - Active field in configuration export (true/false cases) - Active field in configuration import validation - Backwards compatibility for imports without active field All tests pass successfully.
Both admin_supported_product_mirror_repomd_new_post and admin_supported_product_mirror_repomd_post had identical code for building form_data and calling validation. Extracted this into _validate_repomd_form helper that returns validated_data, errors, and the original form_data for use in error templates. This eliminates 14 lines of duplication across the two functions.
Implements new v2 endpoint that uses normalized database relationships
(supported_product_id, major_version) instead of denormalized product_name
strings to enable aggregation across minor versions within a major release.
Key improvements:
- FK-based package filtering prevents cross-product contamination
- Fixes module prefix bug by stripping "module." consistently
- Enables major version aggregation (e.g., all Rocky 8.x advisories)
- Maintains v1 backward compatibility with no functional changes
- Fixes source RPM mapping bug where binaries were incorrectly mapped to
source RPMs from different minor versions (e.g., el8_7 binary mapped to
el8_6 source). The refactored build_source_rpm_mapping() now correctly
matches each binary package to its exact source RPM version.
New endpoint: /{product}/{major_version}/{repo}/updateinfo.xml?arch={arch}
Example: /rocky-linux/8/BaseOS/updateinfo.xml?arch=x86_64
Implements transparent stripping of 'module.' prefix from package_name field at the ORM layer using Python property pattern. The package_name field contains 'module.' prefix for module packages (e.g., 'module.postgresql' instead of 'postgresql'), causing source RPM mapping failures. No database migration required. The ORM property pattern handles the data quality issue transparently. Co-authored-by: Trinity Quirk <[email protected]>
Removes manual module. prefix stripping from business logic now that the AdvisoryPackage ORM model handles it transparently via property getter. Changes: - get_source_package_name(): Removed .removeprefix(module.) call - build_source_rpm_mapping(): Removed manual prefix stripping before comparison - Updated function docstrings to remove references to prefix handling - Updated tests to use clean package names (ORM already stripped prefix) The ORM property pattern ensures pkg.package_name always returns a clean value without the module. prefix, eliminating the need for defensive prefix stripping throughout the codebase. Benefits: - Simpler, more maintainable code - Single source of truth for prefix handling (ORM layer) - No risk of forgetting to strip prefix in future code - Business logic focuses on domain concerns, not data quality issues
Replaces hardcoded architecture list with the Architecture enum from apollo.server.validation module to eliminate duplication and ensure consistency across the codebase. The centralized enum supports all architectures Rocky Linux uses, including riscv64 which was missing from the hardcoded list but exists in the database. Changes: - Import Architecture enum from validation module - Replace hardcoded list check with enum validation - Remove duplicate architecture test (already tested in test_validation.py) Benefits: - Single source of truth for valid architectures - Supports full set of architectures (x86_64, aarch64, i386, i686, ppc64, ppc64le, s390x, riscv64, noarch) - Consistent validation behavior across all Apollo endpoints - No risk of architecture lists diverging between modules
The AdvisoryPackage.__init__() method was dropping all package_name values due to incorrect field attribute handling when using Tortoise ORM source_field mapping. The field _package_name maps to DB column package_name, but __init__ was setting kwargs['package_name'] instead of kwargs['_package_name'], causing values to be silently ignored. Also adds null checks in package extraction logic to skip packages when package_name cannot be determined from source RPM metadata.
trinity-q
approved these changes
Nov 17, 2025
Contributor
trinity-q
left a comment
trinity-q
left a comment
There was a problem hiding this comment.
I've approved all of the PRs which make this one up, so this one is also fine to go.
This was referenced Nov 17, 2025
…ibility Adds optional v2 API endpoint support while maintaining full backwards compatibility with existing v3 API usage patterns. New v2 API enables major version aggregation (e.g., all Rocky 9.x advisories) and optional minor version filtering, providing 80% more advisory coverage compared to v3's single-version matching. Backwards compatibility: - Default behavior unchanged (uses v3 API) - All existing scripts work without modification - New parameters only required when opting into v2
Improves code quality and API ergonomics based on PR feedback. Key changes: - Use aiohttp's params parameter instead of manual query string construction - Remove explicit --api-version flag in favor of auto-detection - API version determined by presence of --major-version parameter - Extract API_BASE_URL as module constant for reusability
trinity-q
approved these changes
Nov 21, 2025
mstg
approved these changes
Nov 22, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Apollo Improvements: Bug Fixes, New Features, and API Enhancements
Overview
This consolidated PR includes six related improvements to Apollo's errata management system, developed and thoroughly reviewed by CIQ across multiple PRs in a staging repository:
Impact: Fixes 230+ missing module advisories, adds major version aggregation, improves operational workflows, and enhances API completeness.
Review Process: Each PR was individually reviewed and approved by CIQ team members, with comprehensive testing and validation at each stage before being merged to the staging repository.
1. Config Generation Flexibility (#15)
Problem
generate_rocky_config.pyrequired exact version matches, preventing generation of configs for RHEL 8/9 (which don't use minor versions in advisories)Solution
Flexible version matching:
--version 9matches all 9.x releases (generates NULLmatch_minor_version)--version 10.2matches only 10.2 (generatesmatch_minor_version: 2)Custom naming:
--mirror-name-base "Rocky Linux 9"produces "Rocky Linux 9 x86_64" instead of "Rocky Linux 9.6 x86_64"Files Changed
scripts/generate_rocky_config.pyFull details in PR #15 →
2. Modular Package Extraction and EUS Filtering (#16)
Problems
releases.csvwas overwritingchanges.csvSolutions
_extract_packages_from_product_tree()to handle modular CSAF format with::module:streamsuffixes_is_eus_product()to detect and skip EUS products via CPE patterns and product nameschanges.csvoverreleases.csv/admin/workflowsinterface for viewing/updating CSAFlast_indexed_attimestampImpact:
Files Changed
apollo/rhcsaf/__init__.py- Parser refactor with modular/EUS handlingapollo/rhworker/poll_rh_activities.py- CSV merge order fixapollo/server/routes/admin_workflows.py- New timestamp management UIapollo/server/services/database_service.py- Database methodsapollo/server/templates/admin_workflows.jinja- UI templateapollo/tests/test_csaf_processing.py- Fixed for Bazelapollo/tests/test_rhcsaf.py- 240 new lines (EUS/modular tests)apollo/tests/test_database_service.py- 226 new linesFull details in PR #16 →
3. OSV API Improvements (#17)
Problem
OSV API filtered by
kind="Security", missing bug fix and enhancement advisories that contain CVEs.Solution
kindfilter from querieslen(advisory.cves) > 0Result: All CVE-containing advisories now discoverable via OSV API, regardless of classification (RLSA, RLBA, RLEA).
Files Changed
apollo/server/routes/api_osv.py- Filter by CVE not kindapollo/tests/test_api_osv.py- 248 new linesFull details in PR #17 →
4. Config Import/Export Fixes (#18)
Problems
Solutions
_json_serializer()to convert whole-number Decimals to intNAME_PATTERNto include parentheses:r"^[a-zA-Z0-9._\s()\-]+$"Result: Configs can be exported from production and imported to staging/dev without manual editing.
Files Changed
apollo/server/routes/admin_supported_products.py- Serializer fixapollo/server/validation.py- Allow parenthesesapollo/tests/test_admin_routes_supported_products.py- Test updatesFull details in PR #18 →
5. Mirror Active Field (#19)
Problem
No way to disable mirrors without deleting them (loses historical data and advisory relationships).
Solution
Added
activeboolean field tosupported_products_rh_mirrors:Use cases:
Files Changed
apollo/migrations/20251104111759_add_mirror_active_field.sql- Migrationapollo/schema.sql- Schema updateapollo/db/__init__.py- ORM modelapollo/rpmworker/rh_matcher_activities.py- Workflow filterapollo/server/routes/admin_supported_products.py- UI handlersapollo/server/services/workflow_service.py- Workflow listapollo/server/templates/admin_supported_product*.jinja- UI components (3 files)apollo/tests/test_admin_routes_supported_products.py- 217 new linesFull details in PR #19 →
6. Updateinfo V2 Endpoint and Module Package Fix (#20)
Problems
Module package source RPM mapping bug: ~8,908 packages with "module." prefix in
package_namefield caused NEVRA mismatchV1 endpoint limitations:
product_name)Solutions
ORM-Level Module Prefix Fix (Trinity Quirk)
Added Python property pattern to
AdvisoryPackagemodel to transparently strip "module." prefix:V2 Updateinfo API Endpoint
New endpoint:
/api/v3/updateinfo/{product_slug}/{major_version}/{repo}/updateinfo.xml?arch={arch}Example:
/api/v3/updateinfo/rocky-linux/8/BaseOS/updateinfo.xml?arch=x86_64Key improvements:
supported_product_idArchitectureenum)Simplified Business Logic
Removed manual
.removeprefix("module.")calls since ORM handles it automatically.Architecture Validation
Replaced hardcoded architecture list with centralized
Architectureenum from validation module.API Comparison
/Rocky%20Linux%208%20x86_64/BaseOS/updateinfo.xml/rocky-linux/8/BaseOS/updateinfo.xml?arch=x86_64product_namesupported_product_idTesting
Files Changed
apollo/db/__init__.py- ORM property pattern for module prefixapollo/server/routes/api_updateinfo.py- V2 endpoint, helpers, architecture validationapollo/tests/test_api_updateinfo.py- 182 new linesapollo/tests/BUILD.bazel- Add test target.github/workflows/test.yaml- Add test to CIFull details in PR #20 →
Combined Statistics
Test Coverage
test_api_osv.py,test_api_updateinfo.py,test_database_service.py)test_csaf_processing.py,test_admin_routes_supported_products.py)Code Changes
Database Changes
add_mirror_active_field)supported_products_rh_mirrorstable (addedactiveboolean)Impact Summary
Deployment Notes
Backward Compatibility
Database Migration Required
Migration file:
apollo/migrations/20251104111759_add_mirror_active_field.sqlOptional Post-Deployment Actions
Breaking Changes
Testing Recommendations
/api/v3/updateinfo/rocky-linux/8/BaseOS/updateinfo.xml?arch=x86_64--version 9(major only)Contributors