|
5 | 5 | - [Why LLM Need Cloud-Edge Collaborative Inference?](#why-llm-need-cloud-edge-collaborative-inference) |
6 | 6 | - [Possible Collaborative Inference Strategy](#possible-collaborative-inference-strategy) |
7 | 7 | - [Details of Design](#details-of-design) |
8 | | -- [Quick Start Guide for Benchmarking with Ianvs](#quick-start-guide-for-benchmarking-with-ianvs) |
| 8 | +- [Quick Start Guide](#quick-start-guide) |
| 9 | + - [Required Resources](#required-resources) |
| 10 | + - [Methods for benchmarking with ianvs](#methods-for-benchmarking-with-ianvs) |
9 | 11 | - [Docker-Based Setup](#docker-based-setup) |
10 | 12 | - [Detailed Setup Guide](#detailed-setup-guide) |
11 | | - - [Required Resources](#required-resources) |
12 | 13 | - [Step 1: Ianvs Preparation](#step-1-ianvs-preparation) |
13 | 14 | - [Step 2: Dataset and Model Preparation](#step-2-dataset-and-model-preparation) |
14 | 15 | - [Dataset Configuration](#dataset-configuration) |
@@ -69,7 +70,27 @@ To save API calls during multi-round testing, this example has designed a result |
69 | 70 |
|
70 | 71 | After all tests are completed, the Test Env Manager will calculate relevant metrics based on selected Metrics and hand over to Story Manager for printing test reports and generating Leader Board. |
71 | 72 |
|
72 | | -## Quick Start Guide for Benchmarking with Ianvs |
| 73 | +## Quick Start Guide |
| 74 | + |
| 75 | +### Required Resources |
| 76 | + |
| 77 | +Before using this example, you need to have the device ready: |
| 78 | + |
| 79 | +- One machine is all you need, i.e., a laptop or a virtual machine is sufficient and a cluster is not necessary |
| 80 | + |
| 81 | +- 2 CPUs or more |
| 82 | + |
| 83 | +- 1 GPU with at least 6GB of memory, depends on the tested model |
| 84 | + |
| 85 | +- 4GB+ free memory, depends on algorithm and simulation setting |
| 86 | + |
| 87 | +- 10GB+ free disk space (depends on your model size) |
| 88 | + |
| 89 | +- Internet connection for GitHub, PyPI, HuggingFace, etc |
| 90 | + |
| 91 | +- Python 3.8+ environment |
| 92 | + |
| 93 | +### Methods for Benchmarking with Ianvs |
73 | 94 |
|
74 | 95 | - To quickly experience benchmarking with Ianvs, proceed with the [Docker-Based Setup](#docker-based-setup). |
75 | 96 | - For a detailed setup process, including creating a custom dataset, refer to the [Detailed Setup Guide](#detailed-setup-guide). |
@@ -112,25 +133,7 @@ ianvs -f examples/cloud-edge-collaborative-inference-for-llm/benchmarkingjob.yam |
112 | 133 |
|
113 | 134 | - If you want to create a custom dataset, proceed to the next section. |
114 | 135 |
|
115 | | -## Detailed Setup Guide |
116 | | - |
117 | | -### Required Resources |
118 | | - |
119 | | -Before using this example, you need to have the device ready: |
120 | | - |
121 | | -One machine is all you need, i.e., a laptop or a virtual machine is sufficient and a cluster is not necessary |
122 | | - |
123 | | -- 2 CPUs or more |
124 | | - |
125 | | -- 1 GPU with at least 6GB of memory, depends on the tested model |
126 | | - |
127 | | -- 4GB+ free memory, depends on algorithm and simulation setting |
128 | | - |
129 | | -- 10GB+ free disk space (depends on your model size) |
130 | | - |
131 | | -- Internet connection for GitHub, PyPI, HuggingFace, etc |
132 | | - |
133 | | -- Python 3.8+ environment |
| 136 | +### Detailed Setup Guide |
134 | 137 |
|
135 | 138 | #### Step 1. Ianvs Preparation |
136 | 139 |
|
@@ -318,7 +321,7 @@ Currently, supported routers include: |
318 | 321 |
|
319 | 322 | You can modify the `router` parameter in `test_queryrouting.yaml` to select the router you want to use. |
320 | 323 |
|
321 | | -For BERT router, you can use [routellm/bert](https://huggingface.co/routellm/bert) or [routellm/bert_mmlu_augmented](https://huggingface.co/routellm/bert_mmlu_augmented) or your own BERT model/ |
| 324 | +For BERT router, you can use [routellm/bert](https://huggingface.co/routellm/bert) or [routellm/bert_mmlu_augmented](https://huggingface.co/routellm/bert_mmlu_augmented) or your own BERT model. |
322 | 325 |
|
323 | 326 | ##### Data Processor Configuration |
324 | 327 | The Data Processor allows you to custom your own data format after the dataset loaded. |
|
0 commit comments