Track of Runnable Examples

[AryanNanda17]

This issue serves to track the runnable examples in the latest release of Ianvs.

Ianvs includes a wide range of examples showcasing its benchmarking capabilities.
So far, I have verified the following examples as runnable and also updated their quick-start documentation guide for easy setup with no errors:

• MOT17/multiedge_inference_bench
• aoa/single_task_bench
• bdd/lifelong_learning_bench
• cifar100/fci_ssl
• cifar100/federated_class_incremental_learning
• cifar100/federated_learning
• cifar100/sedna_federated_learning
• cityscapes/lifelong_learning_bench
• cityscapes/singletask_learning_bench
• cityscapes-synthia/lifelong_learning_bench
• cityscapes-synthia/scene-based-unknown-task-recognition
• cloud-edge-collaborative-inference-for-llm
• government/singletask_learning_bench
• imagenet/multiedge_inference_bench
• llm-agent/singletask_learning_bench
• llm-edge-benchmark-suite/single_task_bench
• llm-edge-benchmark-suite/single_task_bench_with_compression
• llm_simple_qa
• pcb-aoi/incremental_learning_bench
• pcb-aoi/singletask_learning_bench
• robot/lifelong_learning_bench
• robot-cityscapes-synthia/lifelong_learning_bench
• smart_coding/smart_coding_learning_bench
• yaoba/singletask_learning_boost
• yaoba/singletask_learning_yolox_tta

I'll continue updating this list as I verify that the other examples work correctly and update their quick-start documentation as needed.

[AryanNanda17]

I have updated the documentation and verified the installation process for PCB-AOI Single-Task Learning example. The PRs #174, #182, #171 resolve the path errors and dependency conflicts for this example.

[AryanNanda17]

I have updated the quick-start guide of cloud-edge-collaborative-inference-for-llm example in PR #188.
Also, the PR #191, replaces the PCB-AOI related content with cloud-edge-collaborative-inference-for-LLM example.

[NishantSinghhhhh]

@MooreZheng

Hi there,
Thanks for all the updates! Could you help me figure out which benchmark examples are still left to verify? I want to work on the ones that haven’t been tested or documented yet.

[MooreZheng]

@MooreZheng

Hi there, Thanks for all the updates! Could you help me figure out which benchmark examples are still left to verify? I want to work on the ones that haven’t been tested or documented yet.

As far as I am concerned, the list at the top is already the latest one. If there is any news from our members, please let us know @AryanNanda17 @FuryMartin

[NishantSinghhhhh]

@MooreZheng

I’ve done a review of the repo and noticed there are still several documentation and example-related tasks left to complete, including:

• Writing quick-start READMEs for new or untouched example folders (e.g., benchmark_ci_pipeline/, keink_simulation/, kepler_power_bench/, mscoco/lifelong_learning_bench/, t4rec_mllib/, widerface/edge_detection_bench)

• Ensuring ianvs -f testenv.yaml works smoothly on CPU-only hosts for each example, and listing all required datasets/model weights with updated mirror links

• Adding how-to-build-dataset guides for new datasets and troubleshooting FAQs, especially for simulation environments

• Improving environment setup scripts to cover ROCm/CPU-only, adding a matrix of tested OS/Python/CUDA setups, and documenting offline installation for air-gapped nodes

• Creating missing hyper-parameter sweep templates and documentation (examples/_template/hpo)

• Providing clear CI integration steps (e.g., guidelines for adding examples to GitHub Actions workflows)

• Explaining the leaderboard result schema and reporting process in the docs

• Documenting edge-device resource profiling for recent LLM-related demos and including sample power traces

• Adding demo clips or screenshots to video-based example READMEs to clarify expected results

• Translating documentation that's only available in Chinese into English, especially for smart_coding and yaoba/*

• Updating dataset mirrors to avoid timeouts and including backup links or procedures

• Adding version-tag information to clarify which release each example was first verified in

Would it be helpful if I start working on these areas? Please let me know if I should begin tackling any specific tasks from this list, or if there are priorities I should focus on first.

Thanks!

[MooreZheng]

@MooreZheng

I’ve done a review of the repo and noticed there are still several documentation and example-related tasks left to complete, including:

• Writing quick-start READMEs for new or untouched example folders (e.g., benchmark_ci_pipeline/, keink_simulation/, kepler_power_bench/, mscoco/lifelong_learning_bench/, t4rec_mllib/, widerface/edge_detection_bench)
• Ensuring ianvs -f testenv.yaml works smoothly on CPU-only hosts for each example, and lists all required datasets/model weights with updated mirror links
• Adding how-to-build-dataset guides for new datasets and troubleshooting FAQs, especially for simulation environments
• Improving environment setup scripts to cover ROCm/CPU-only, adding a matrix of tested OS/Python/CUDA setups, and documenting offline installation for air-gapped nodes
• Creating missing hyper-parameter sweep templates and documentation (examples/_template/hpo)
• Providing clear CI integration steps (e.g., guidelines for adding examples to GitHub Actions workflows)
• Explaining the leaderboard result schema and reporting process in the docs
• Documenting edge-device resource profiling for recent LLM-related demos and including sample power traces
• Adding demo clips or screenshots to video-based example READMEs to clarify expected results
• Translating documentation that's only available in Chinese into English, especially for smart_coding and yaoba/*
• Updating dataset mirrors to avoid timeouts and including backup links or procedures
• Adding version-tag information to clarify which release each example was first verified in

Would it be helpful if I start working on these areas? Please let me know if I should begin tackling any specific tasks from this list, or if there are priorities I should focus on first.

Thanks!

Good to learn about your progress. My suggestion would be to focus on examples one by one and fix the issue when executing the example. Higher priority should be given to those bugs that would affect the example execution, and make sure examples are runnable.

[NishantSinghhhhh]

@MooreZheng

I will work on the following examples first

• MOT17/multiedge_inference_bench

• aoa/single_task_bench

• bdd/lifelong_learning_bench

DO tell me if I need to have somethig in my mind before I start working on them

[NishantSinghhhhh]

@MooreZheng I just wanted to ask whether I just have to add documentation for the features I have listed above, do I have to implement them too?

[AryanNanda17]

@NishantSinghhhhh, you can also check out this example #106