Merge branch 'main' into main

Author: IcyFeather233

.github/workflows/codeql-analysis.yml

@@ -23,7 +23,7 @@ on:
 jobs:
   analyze:
     name: Analyze
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-22.04
 
     strategy:
       fail-fast: false

.github/workflows/fossa.yaml

@@ -10,7 +10,7 @@ on:
 
 jobs:
   build:
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-22.04
     strategy:
       matrix:
         python-version: [ "3.7", "3.8", "3.9" ]

.github/workflows/main-doc.yaml

@@ -11,7 +11,7 @@ on:
 
 jobs:
   pylint:
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-22.04
     name: pylint
     strategy:
       matrix:

.github/workflows/main.yaml

@@ -11,7 +11,7 @@ on:
 
 jobs:
   pylint:
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-22.04
     name: pylint
     strategy:
       matrix:

docs/guides/how-to-build-simulation-env.md

@@ -4,7 +4,7 @@ This document introduces how to build a edge-cloud AI simulation environment(e.g
 
 ## Introduction to `simulation controller`
 
-the `simulation controller` is the core module of system simulation. The simulation controller has been supplemented, which build and deploy local edge-cloud simulation environment with K8s.
+The `simulation controller` is the core module of system simulation. The simulation controller has been supplemented, which build and deploy local edge-cloud simulation environment with K8s.
 
 ![](https://github.com/kubeedge/ianvs/blob/main/docs/proposals/simulation/images/simulation_controller.jpg?raw=true)
 
@@ -36,16 +36,19 @@ Typically, the config file `benchmarkingJob.yaml` is as follows, which represent
 benchmarkingjob:
   # job name of benchmarking; string type;
   name: "benchmarkingjob"
+  
   # the url address of job workspace that will reserve the output of tests; string type;
   # default value: "./workspace"
-  workspace: "./workspace/incremental_learning_bench"
+  workspace: "./workspace-mmlu"
 
   # the url address of test environment configuration file; string type;
   # the file format supports yaml/yml;
-  testenv: "./examples/pcb-aoi/incremental_learning_bench/fault_detection/testenv/testenv.yaml"
+  testenv: "./examples/cloud-edge-collaborative-inference-for-llm/testenv/testenv.yaml"
+  
   # the configuration of test object
   test_object:
     ...
+
   # the configuration of ranking leaderboard
   rank:
     ...

docs/guides/how-to-contribute-algorithms.md

@@ -4,16 +4,17 @@ Ianvs serves as testing tools for test objects, e.g., algorithms. Ianvs does NOT
 
 For algorithm contributors, you can:
 
-1. Release a repo independent of ianvs, but the interface should still follow the SIG AI algorithm interface to launch ianvs. Here are two examples showing how to develop an algorithm for testing in Ianvs.
-   Here are two examples show how to development algorithm for testing in Ianvs.
-    * [incremental-learning]
+1. Release a repo independent of ianvs, but the interface should still follow the SIG AI algorithm interface to launch ianvs. Here are few examples showing how to develop an algorithm for testing in Ianvs:
+    * [cloud-edge-collaborative-inference-for-llm]
     * [single-task-learning]
-2. Integrated the targeted algorithm into sedna so that ianvs can use it directly. in this case, you can connect with sedna owners for help.
+    * [incremental-learning]
+2. Integrate the targeted algorithm into sedna so that ianvs can use it directly. In this case, you can connect with sedna owners for help.
 
 Also, if a new algorithm has already been integrated into Sedna, it can be used in Ianvs directly.
 
 [Sedna Lib]: https://github.com/kubeedge/sedna/tree/main/lib
 [incremental-learning]: ../proposals/algorithms/incremental-learning/basicIL-fpn.md
 [single-task-learning]: ../proposals/algorithms/single-task-learning/fpn.md
 [examples directory]: ../../../../examples
-[Sedna repository]: https://github.com/kubeedge/sedna
+[Sedna repository]: https://github.com/kubeedge/sedna
+[cloud-edge-collaborative-inference-for-llm]: https://github.com/kubeedge/ianvs/tree/main/examples/cloud-edge-collaborative-inference-for-llm

docs/guides/how-to-contribute-test-environments.md

@@ -20,36 +20,40 @@ testenv:
   # dataset configuration
   dataset:
     # the url address of train dataset index; string type;
-    train_index: "./dataset/train_data/index.txt"
+    train_data: "./dataset/mmlu-5-shot/train_data/data.json"
     # the url address of test dataset index; string type;
-    test_index: "./dataset/test_data/index.txt"
-
-  # model eval configuration of incremental learning;
-  model_eval:
-    # metric used for model evaluation
-    model_metric:
-      # metric name; string type;
-      name: "f1_score"
-      # the url address of python file
-      url: "./examples/pcb-aoi/incremental_learning_bench/fault_detection/testenv/f1_score.py"
-
-    # condition of triggering inference model to update
-    # threshold of the condition; types are float/int
-    threshold: 0.01
-    # operator of the condition; string type;
-    # values are ">=", ">", "<=", "<" and "=";
-    operator: ">="
+    test_data_info: "./dataset/mmlu-5-shot/test_data/metadata.json"
 
   # metrics configuration for test case's evaluation; list type;
   metrics:
       # metric name; string type;
-    - name: "f1_score"
+    - name: "Accuracy"
       # the url address of python file
-      url: "./examples/pcb-aoi/incremental_learning_bench/fault_detection/testenv/f1_score.py"
-    - name: "samples_transfer_ratio"
+      url: "./examples/cloud-edge-collaborative-inference-for-llm/testenv/accuracy.py"
+
+    - name: "Edge Ratio"
+      url: "./examples/cloud-edge-collaborative-inference-for-llm/testenv/edge_ratio.py"
+
+    - name: "Cloud Prompt Tokens"
+      url: "./examples/cloud-edge-collaborative-inference-for-llm/testenv/cloud_prompt_tokens.py"
 
-  # incremental rounds setting for incremental learning paradigm.; int type; default value is 2;
-  incremental_rounds: 2
+    - name: "Cloud Completion Tokens"
+      url: "./examples/cloud-edge-collaborative-inference-for-llm/testenv/cloud_completion_tokens.py"
+
+    - name: "Edge Prompt Tokens"
+      url: "./examples/cloud-edge-collaborative-inference-for-llm/testenv/edge_prompt_tokens.py"
+
+    - name: "Edge Completion Tokens"
+      url: "./examples/cloud-edge-collaborative-inference-for-llm/testenv/edge_completion_tokens.py"
+
+    - name: "Time to First Token"
+      url: "./examples/cloud-edge-collaborative-inference-for-llm/testenv/time_to_first_token.py"
+
+    - name: "Throughput"
+      url: "./examples/cloud-edge-collaborative-inference-for-llm/testenv/throughput.py"
+
+    - name: "Internal Token Latency"
+      url: "./examples/cloud-edge-collaborative-inference-for-llm/testenv/internal_token_latency.py"
 ```
 
 It can be found that for a test, we need to set up the three fields:
@@ -62,11 +66,12 @@ That means, if you want to test on a different dataset, different model, or diff
 
 ## Add a new test environment
 
-Please refer to the examples directory, [pcb-aoi](https://github.com/kubeedge/ianvs/tree/main/examples/pcb-aoi) is a scenario for testing.
+Please refer to the examples directory, [cloud-edge-collaborative-inference-for-llm](https://github.com/kubeedge/ianvs/tree/main/examples/cloud-edge-collaborative-inference-for-llm) is a scenario for testing.
 We can regard it as a subject for a student that needs to take an exam, the test env is like an examination paper,
 and the test job is like the student.
 
-For a subject `pcb-aoi`, a new examination paper could be added to the subdirectory, on the same level as a `benchmarking job`.
+For a subject `cloud-edge-collaborative-inference-for-llm`, a new examination paper could be added to the subdirectory, on the same level as a `benchmarking job`.
+
 The detailed steps could be the following:
 
 1. Copy `benchmarking job` and name `benchmarking job_2` or any other intuitive name.

docs/guides/how-to-install-ianvs.md

@@ -10,15 +10,15 @@ This guide covers how to install Ianvs on a Linux environment.
 - 4GB+ free memory, depends on algorithm and simulation setting
 - 10GB+ free disk space
 - Internet connection for GitHub and pip, etc
-- Python 3.6+ installed
+- Python 3.8 installed
 
 you can check the python version by the following command:
 ```
 python -V
 ```
 after doing that, the output will be like this, which means your version fits the bill.
 ```
-Python 3.6.9
+Python 3.8
 ```
 
 ## Install ianvs on Linux