[LFX Term 1 2026] Restoring LLM Edge Benchmark Suite Single Task Bench With Compression#409
Conversation
…ompression refactor: adjust algorithm YAML for paradigm type and enhance basemodel.py with logging and model execution improvements Signed-off-by: NishantSinghhhhh <nishantsingh_230137@aitpune.edu.in>
kubeedge-bot
added
kind/bug
kubeedge-bot
added
the
size/L
Signed-off-by: NishantSinghhhhh <nishantsingh_230137@aitpune.edu.in>
|
@MooreZheng sir, after making changes the example works perfectly fine Screencast.from.2026-04-24.00-10-15.webm |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a compression benchmarking suite for LLMs within the Ianvs framework, featuring a comprehensive setup guide, updated dependencies, and a LlamaCppModel implementation that utilizes streaming to measure prefill latency. The review feedback suggests fixing markdown syntax and relative paths in the documentation, transitioning to module-level logging for better practice, and adding safety checks when processing model output chunks to prevent potential runtime errors.
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: hsj576, MooreZheng, NishantSinghhhhh The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
kubeedge-bot
added
the
approved
4 participants
What type of PR is this?
/kind bug
/kind cleanup
/kind feature
What this PR does / why we need it:
This PR refactors and restores the
single_task_bench_with_compressionsuite within the LLM Edge Benchmark to ensure it successfully executes end-to-end within the Ianvs framework.Previously, the compression benchmarking pipeline failed due to unsupported paradigm types, missing pipeline contract methods in the base model script, and relative path resolution errors.
Key improvements and fixes include:
Paradigm & Configuration Alignment: Corrected the
paradigm_typeinalgorithm.yamlfrom the unsupportedsingletasklearning_with_compressionto the recognizedsingletasklearning. Fixed relative pathing bugs by ensuring model and dataset URLs explicitly use absolute paths across all YAML configurations (testenv.yaml,algorithm.yaml, andbenchmarkingjob.yamlhyperparameter overrides).Pipeline Contract Fulfillment: Updated
basemodel.pyto satisfy the strict Ianvs ML lifecycle checks by addingpreprocessandpostprocesspass-through methods with optional**kwargs.