Skip to content

fix(llm_simple_qa): replace hardcoded paths and fix basemodel compatibility#388

Open
rakshaak29 wants to merge 4 commits intokubeedge:mainfrom
rakshaak29:fix/llm-simple-qa-hardcoded-paths
Open

fix(llm_simple_qa): replace hardcoded paths and fix basemodel compatibility#388
rakshaak29 wants to merge 4 commits intokubeedge:mainfrom
rakshaak29:fix/llm-simple-qa-hardcoded-paths

Conversation

@rakshaak29
Copy link
Copy Markdown

What this PR does

Fixes the llm_simple_qa example which was broken due to hardcoded paths and missing methods.
Fixes #387

Changes

  • YAML configs: Replaced all hardcoded absolute paths (/home/icyfeather/...) with relative paths (./examples/..., ./dataset/..., ./workspace)
  • basemodel.py:
    • Use HuggingFace Hub model ID (Qwen/Qwen2.5-0.5B-Instruct) instead of local path
    • Added missing preprocess() method required by sedna's SingleTaskLearning paradigm
    • Added MPS/CPU device detection for cross-platform compatibility

Testing

Verified the benchmark runs end-to-end successfully:

$ ianvs -f examples/llm_simple_qa/benchmarkingjob.yaml

Loading weights: 100%|████████████████████| 290/290 [00:01<00:00, 250.13it/s]
BaseModel doesn't need to preprocess
BaseModel doesn't need to train
BaseModel doesn't need to save
BaseModel load
BaseModel predict
+------+-------------------------------+-----+--------------------+-----------+---------------------+
| rank |           algorithm           | acc |      paradigm      | basemodel |         time        |
+------+-------------------------------+-----+--------------------+-----------+---------------------+
|  1   | simple_qa_singletask_learning | 0.8 | singletasklearning |    gen    | 2026-04-12 05:26:45 |
+------+-------------------------------+-----+--------------------+-----------+---------------------+

@kubeedge-bot
Copy link
Copy Markdown
Collaborator

Welcome @rakshaak29! It looks like this is your first PR to kubeedge/ianvs 🎉

@kubeedge-bot
Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: rakshaak29
To complete the pull request process, please assign jaypume after the PR has been reviewed.
You can assign the PR to them by writing /assign @jaypume in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubeedge-bot kubeedge-bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Apr 12, 2026
Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates configuration paths to relative links, adds local artifacts to .gitignore, and switches the base model to a public Hugging Face repository. Feedback identifies a regression in basemodel.py where CUDA support was removed from device detection, potentially causing runtime errors or performance degradation. Additionally, the new preprocess method returns None, which may break the data pipeline if subsequent stages expect processed data.

@rakshaak29
Copy link
Copy Markdown
Author

/assign @jaypume

rakshaak29 added 4 commits April 12, 2026 06:37
…ix basemodel

- Replace all hardcoded absolute paths with relative paths in YAML configs
- Use HuggingFace Hub model ID (Qwen/Qwen2.5-0.5B-Instruct) instead of local path
- Add missing preprocess() method to BaseModel (required by sedna)
- Add MPS/CPU device detection for macOS compatibility
- Update .gitignore to exclude local dev artifacts

Signed-off-by: rakshaak29 <rakshak29@gmail.com>
Signed-off-by: rakshaak29 <rakshak29@gmail.com>
Signed-off-by: rakshaak29 <rakshak29@gmail.com>
Signed-off-by: rakshaak29 <rakshak29@gmail.com>
@rakshaak29 rakshaak29 force-pushed the fix/llm-simple-qa-hardcoded-paths branch from e0566e3 to b584a15 Compare April 12, 2026 01:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] llm_simple_qa example has hardcoded paths and missing preprocess() method

3 participants