Skip to content

[feature:gsoc26] Asynchronous Extraction Pipeline & Multi-Schema fwtool Support #413

@atif09

Description

@atif09

Is your feature request related to a problem? Please describe.

With the extractor class hierarchy and model layer in place, there is no Celery task yet to wire them together. Firmware images are uploaded and saved with extraction_status = unconfirmed but nothing triggers extraction automatically. There is also no handling for the varying fwtool metadata schemas across OpenWrt versions, or for the decompression memory risks introduced by the DTB fallback path

Describe the solution you'd like
I would like to introduce the asynchronous extraction task that drives the full pipeline from upload to confirmed status, along with decompression guardrails

  1. Implement a extract_firmware_metadata Celery task triggered via post_save on FirmwareImage when a new image is created. The task will:
  • It will use the same task timeout used for the current firmware upgrade tasks.
  • Transition extraction_status to in_progress on start
  • The task reads metadata_extractor_class from the category associated with the build, defaulting to OpenWrtMetadataExtractor if not defined
  • Instantiate the extractor and call extract()
  • On success, populate all metadata fields and transition to confirmed
  • On DecompressionLimitExceeded, transition to failed with failure_reason = out_of_memory
  • On UnsupportedImageError, transition to failed with failure_reason = unsupported_format
  • On any unhandled exception, transition to failed with failure_reason = invalid_file
  • Call _update_extraction_status() on the present build after every transition
  1. Handle multi-schema fwtool support via _parse_supported_devices(), ensuring images from targets that migrated to DSA are parsed correctly without hard-failing on older schemas

  2. Implement _check_limits() to enforce two real-time decompression limits during the DTB fallback kernel decompression:

  • A hard size cap via OPENWISP_FIRMWARE_UPGRADER_MAX_DECOMPRESSED_BYTES (default 512MB)
  • A compression ratio limit via OPENWISP_FIRMWARE_UPGRADER_MAX_DECOMPRESSED_RATIO (default 100:1)
  • Exceeding either raises DecompressionLimitExceeded immediately, keeping the Celery worker stable
  1. Implement _compat_blocks_pairing() to prevent automatic device pairing when compat_version > 1.0 , blocking upgrades that could break devices during the swconfig-to-DSA migration

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requestgsoc-ideaIssues part of Google Summer of Code project

Projects

Status

ToDo

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions