Add llm-benchmarks proposal

[IcyFeather233]

What type of PR is this?

OSPP proposal

What this PR does / why we need it:

Investigated various aspects of implementing the LLM Benchmark in Ianvs, introduced the integration plan of opencompass, and achieved the dataset map.

[IcyFeather233]

By the way, my implementation is in https://github.com/IcyFeather233/ianvs/tree/dev, you can follow this readme to try llm single task learning and opencompass

[hsj576]

The PR should be translated into English to facilitate understanding by people from other countries.

[IcyFeather233]

https://github.com/IcyFeather233/ianvs/tree/dev

Thanks for the advice! I will translate it after the proposal content will not be changed.

[MooreZheng]

This proposal is related to #95.
Great to see a comprehensive proposal. This one is close to the final version. As discussed in the routine meeting, there might be some advices.

1. Though the directories are clear, an architecture is still needed to clarify what is modified in this project. E.g., what is in the TestEnv, TestCase. It seems to me that at least we have made changes on the data format to support NLP. You might want to take a look at https://github.com/kubeedge/ianvs/pull/122/files for an architecture example.
2. Prompts would have different forms, e.g., N shots. How shall we develop a prompt template to adapt all these? It looks interesting and challenging.

[IcyFeather233]

Now I add my changes to ianvs core, including 2 flowcharts showing the architecture. And a more complete benchmark format is updated in the doc.

[hsj576]

/lgtm

[MooreZheng]

Overall looks good to me. Just a few comments would be considered:

1. The revision for index in env manager of ianvs core, which should be highlighted: The proposed design also aims to support both the old and new index.
2. The integration part of OpenCompass should also be highlighted in the structure.png. A tutorial is needed.
3. The prompt template is extendable, including the keys and prompts, which can be also highlighted in the proposal.

[MooreZheng]

/lgtm

[kubeedge-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: MooreZheng

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [MooreZheng]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ojuschugh1]

Hi @IcyFeather233 , @hsj576 , @MooreZheng , I wanted to discuss one thing regarding this project. Is there any way I can chat/discuss this with you? I am unable to find anyone of you on slack channel

[MooreZheng]

Hi @IcyFeather233 , @hsj576 , @MooreZheng , I wanted to discuss one thing regarding this project. Is there any way I can chat/discuss this with you? I am unable to find anyone of you on slack channel

Hi, the slack channel currently has networking issues for Chinese users. You might want to leave your comment here or raise a issue.

[jaypume]

/lgtm

[MooreZheng]

Related issue and PR:
Phase 1 Gov LLM #95 #113 #144
Phase 2 Gov RAG #177 #186 #196