Agent Benchmark for Government Scenarios Based on KubeEdge-Ianvs

[IcyFeather233]

Description:

With the rapid advancement of large model technology, its application potential in government scenarios is becoming increasingly prominent. The intelligent upgrade of government services involves three core scenarios: internal government collaboration, public services, and enterprise services, all of which urgently require large model technology to enhance efficiency and service quality. However, government scenarios demand high levels of professionalism, standardization, and security. The existing evaluation systems for large models lack standardized assessment methods tailored to the vertical domain of government affairs, posing challenges in technology implementation, such as accuracy, compliance, and scenario adaptability.
Therefore, this project aims to leverage the KubeEdge-Ianvs distributed collaborative framework to construct a large model evaluation Pipeline and Benchmark specifically for government scenarios. This will provide quantifiable and reusable capability assessment tools for the intelligent transformation of government services, promoting the secure and efficient application of large model technology in typical scenarios such as official document drafting, smart transportation, and government Q&A.

Expected Outcomes:

1. Introduce datasets from the e-government domain, categorize and reorganize existing datasets according to three standardized task categories: Government Services (e.g., administrative services, hotline support, business facilitation), Government Office Operations (e.g., government knowledge Q&A, document information extraction, document generation), Urban Governance (e.g., urban data analysis, event perception, event dispatching, event analysis)
2. Select at least one of the above scenarios in KubeEdge-Ianvs to provide a standardized test suite, including datasets, test environments, and evaluation metrics, and standardize and organize the datasets into a unified data format.
3. Implement baseline algorithms for e-government agents in KubeEdge-Ianvs based on the standardized test suite.

Recommended Skills:

LLM Agent, LLM Benchmark, VQA

Useful Links:

• https://ianvs.readthedocs.io/en/latest/
• https://github.com/kubeedge/ianvs/tree/main/examples/llm_simple_qa
• https://github.com/kubeedge/ianvs/tree/main/examples/llm-edge-benchmark-suite
• https://github.com/kubeedge/ianvs/tree/main/examples/llm-agent
• https://github.com/kubeedge/ianvs/tree/main/examples/government/singletask_learning_bench
• https://github.com/kubeedge/ianvs/blob/main/docs/proposals/scenarios/llm-benchmarks/llm-benchmarks.md
• https://github.com/kubeedge/ianvs/blob/main/docs/proposals/scenarios/llm-benchmark-suite/llm-edge-benchmark-suite.md
• https://github.com/kubeedge/ianvs/blob/main/docs/proposals/scenarios/Smart_Coding/Smart%20Coding%20benchmark%20suite%20Proposal.md
• https://github.com/kubeedge/ianvs/blob/main/docs/proposals/algorithms/joint-inference/cloud-edge-collaboration-inference-for-llm.md
• https://github.com/kubeedge/ianvs/blob/main/docs/proposals/algorithms/joint-inference/cloud-edge-speculative-decoding-for-llm.md

[ShinjukuZhu]

Common metrics such as BLEU and Exact Match (EM) primarily assess semantic or textual similarity, but they fail to capture critical aspects in government-focused language generation—such as policy terminology correctness, appropriate use of official expressions, and accurate reference to regulations. In high-compliance scenarios like government Q&A, how can we design an evaluation metric system that not only reflects differences in model capabilities, but also ensures adherence to policy language standards and factual accuracy?

[IcyFeather233]

Common metrics such as BLEU and Exact Match (EM) primarily assess semantic or textual similarity, but they fail to capture critical aspects in government-focused language generation—such as policy terminology correctness, appropriate use of official expressions, and accurate reference to regulations. In high-compliance scenarios like government Q&A, how can we design an evaluation metric system that not only reflects differences in model capabilities, but also ensures adherence to policy language standards and factual accuracy?

Hi this is a good question, some advice:

1. Use multiple-choice questions, which only require the ACC (accuracy) metric.
2. Provide reference answers and use a large language model as a judge model to evaluate whether the model's response aligns with the reference answer (refer to the AlignBench paper).
3. Other methods.

In previous ianvs projects, 1 and 2 are used, you can do a survey on LLM Benchmarks and see how other benchmarks evaluate on these situations.

[ShinjukuZhu]

/assign

[MooreZheng]

Related issue and PR:
Phase 1 Gov LLM
#95
#113
#144
Phase 2 Gov RAG
#177
#186
#196
Phase 3 Gov Agent
#199
#223
#258
#262
#268
#279