v0.3.0

What's New in v0.3.0

Ianvs v0.3.0 brings powerful new LLM-related features, including comprehensive (1) LLM testing and benchmarking tools, (2) advanced cloud-edge collaborative inference paradigms, and (3) innovative algorithms tailored for large model optimization.

1. Support for LLM Testing and Benchmarks

Ianvs now supports robust testing for both locally deployed LLMs and public LLM APIs (e.g., OpenAI). This release introduces three specialized benchmarks for evaluating LLM capabilities in diverse scenarios:

• Government-Specific Large Model Benchmark: Designed to assess LLM accuracy and reasoning in government-specific scenarios. using objective (multiple-choice) and subjective (Q&A) tests. Explore the benchmark dataset, try the example.

• Smart Coding Benchmark: This benchmark evaluates the debugging capabilities of LLMs using real-world coding issues from GitHub repositories. Learn more through the example and read the background documentation.

• Large Language Model Edge Benchmark: Focused on testing LLM performance in edge environments, this benchmark evaluates resource efficiency and deployment performance. Access datasets and examples here and check out the detailed documentation.

2. Enhanced Cloud-Edge Collaborative Inference

This release introduces new paradigms and algorithms for collaborative inference to optimize cloud-edge cooperation and improve performance:

• Cloud-Edge Collaborative Inference Paradigm: A new architecture enables efficient cloud-edge collaboration for LLM inference, featuring a baseline algorithm that delivers up to 50% token cost savings without compromising accuracy. Try the example.

• Speculative Decoding Algorithm (EAGLE, ICML'24): Integrated within the collaborative inference framework, this algorithm accelerates inference speeds by 20% or more. Try the example and explore detailed documentation.

• Joint Inference Paradigm for Pedestrian Tracking: A multi-edge inference paradigm for pedestrian tracking utilizing the pretrained ByteTrack model (ECCV'22). See the pedestrian tracking example or refer to the background documentation.

3. Support for New Large Model Algorithms

Ianvs includes new algorithms to improve LLM performance and usability in various scenarios:

• Personalized LLM Agent Algorithm: This algorithm supports single-task learning using the pretrained Bloom model, enabling personalized LLM operations. Explore the example and review the documentation.

• Multimodal Large Model Joint Learning Algorithm: A joint learning algorithm for multimodal understanding with the pretrained RFNet model. Try the example here and learn more in the documentation.

• Unseen Task Processing Algorithm: Supports lifelong learning with pretrained models to handle unseen tasks effectively. Access the example and gain insights from the background documentation.

Detailed Pull Requests:

• Government-Specific Large Model Benchmark by @IcyFeather233 in #144
• Smart Coding Benchmark by @safe-b in #159
• Large Language Model Edge Benchmark by @XueSongTap in #150
• Cloud-Edge Collaborative Inference for LLM by @FuryMartin in #156
• Cloud-Edge Collaborative Speculative Decoding for LLM by @FuryMartin in #179
• Personalized LLM Agent by @Frank-lilinjie in #154
• Multimodal Large Model Joint Learning by @aryan0931 in #167
• Unseen task processing by @nailtu30 in #90
• Documentation refining by @AryanNanda17 in #182

New Contributors

• @IcyFeather233 made their first contribution in #113
• @safe-b made their first contribution in #120
• @XueSongTap made their first contribution in #127
• @FuryMartin made their first contribution in #122
• @aryan0931 made their first contribution in #166
• @AryanNanda17 made their first contribution in #171