Skip to content

Commit

Permalink
v0.3.7 test report
Browse files Browse the repository at this point in the history
  • Loading branch information
bigbrother666sh committed Jan 17, 2025
1 parent 43ae4df commit e2f3903
Show file tree
Hide file tree
Showing 27 changed files with 3,467 additions and 1 deletion.
3 changes: 2 additions & 1 deletion test/reports/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
# index of reports

- [extract info from pics test](./extract_info_from_pics_test_20241222_bigbrother666/README.md) by bigbrother666 2024-12-22
- [wiseflow report](./wiseflow_report_v036_bigbrother666/README.md) by bigbrother666 2025-01-04
- [wiseflow V0.36 report](./wiseflow_report_v036_bigbrother666/README.md) by bigbrother666 2025-01-04
- [wiseflow V0.37 report](./wiseflow_report_v037_bigbrother666/README.md) by bigbrother666 2025-01-17
37 changes: 37 additions & 0 deletions test/reports/wiseflow_report_v037_bigbrother666/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# wiseflow testing report

2025-01-17

by bigbrother666

## summary

- task number: 4
- sample number: 8
- Benchmark model: Qwen2.5-72B-Instruct
- involved models: ['Qwen/Qwen2.5-7B-Instruct', 'Qwen/Qwen2.5-14B-Instruct', 'Qwen/Qwen2.5-32B-Instruct', 'deepseek-ai/DeepSeek-V2.5']
- model provider: siliconflow api

## conclusion

- by the new method, 7b size model can achieve similar performance as 32b size model in the links part filtering task.

## Detailed Test Results

| sample | Qwen2.5-14B-Instruct 👍 | Qwen2.5-32B-Instruct| DeepSeek-V2.5| Qwen2.5-7B-Instruct |
|--------|---------------------|---------------------|-----------------|--------------------------|
| **total diff from benchmark in all samples** | **24** | **26** | **58** | **38** |

**note**

Qwen2.5-7B-Instruct model can achieve significant performance improvement by increasing the batch size of single processing, therefore, in V0.3.7 version, we increased the batch size of link processing and reintroduced the concept of Secondary Model, with Qwen2.5-7B-Instruct as the default recommendation.

siliconflow official pricing:

deepseek-ai/DeepSeek-V2.5: ¥1.33 / M Tokens

Qwen/Qwen2.5-32B-Instruct: ¥1.26 / M Tokens

Qwen/Qwen2.5-14B-Instruct: ¥0.7 / M Tokens

Qwen/Qwen2.5-72B-Instruct: ¥4.13 / M Tokens
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
{
"link_dict": {
"[1]": "https://www.gd121.cn/index.shtml",
"[2]": "https://www.gd121.cn/yj/index.shtml",
"[3]": "https://www.gd121.cn/yj/index.shtml?city=广州",
"[4]": "https://www.gd121.cn/yj/index.shtml?city=深圳",
"[5]": "https://www.gd121.cn/yj/index.shtml?city=佛山",
"[6]": "https://www.gd121.cn/yj/index.shtml?city=东莞",
"[7]": "https://www.gd121.cn/yj/index.shtml?city=中山",
"[8]": "https://www.gd121.cn/yj/index.shtml?city=珠海",
"[9]": "https://www.gd121.cn/yj/index.shtml?city=惠州",
"[10]": "https://www.gd121.cn/yj/index.shtml?city=江门",
"[11]": "https://www.gd121.cn/yj/index.shtml?city=肇庆",
"[12]": "https://www.gd121.cn/yj/index.shtml?city=汕头",
"[13]": "https://www.gd121.cn/yj/index.shtml?city=潮州",
"[14]": "https://www.gd121.cn/yj/index.shtml?city=揭阳",
"[15]": "https://www.gd121.cn/yj/index.shtml?city=汕尾",
"[16]": "https://www.gd121.cn/yj/index.shtml?city=湛江",
"[17]": "https://www.gd121.cn/yj/index.shtml?city=茂名",
"[18]": "https://www.gd121.cn/yj/index.shtml?city=阳江",
"[19]": "https://www.gd121.cn/yj/index.shtml?city=云浮",
"[20]": "https://www.gd121.cn/yj/index.shtml?city=韶关",
"[21]": "https://www.gd121.cn/yj/index.shtml?city=清远",
"[22]": "https://www.gd121.cn/yj/index.shtml?city=梅州",
"[23]": "https://www.gd121.cn/yj/index.shtml?city=河源",
"[24]": "https://www.gd121.cn/tq/ld/list.shtml",
"[25]": "https://www.gd121.cn/tq/ld/list.shtml",
"[26]": "https://www.gd121.cn/tq/yt/list.shtml",
"[27]": "https://www.gd121.cn/tq/hy/list.shtml",
"[28]": "https://www.gd121.cn/tq/tf/list.shtml",
"[29]": "https://www.gd121.cn/tq/jdtq_list.shtml",
"[30]": "https://www.gd121.cn/tq/jccz/list.shtml",
"[31]": "https://www.gd121.cn/tq/gqtj/index.shtml",
"[32]": "https://www.gd121.cn/tq/tqsp/list.shtml",
"[33]": "https://www.gd121.cn/fw/index.shtml",
"[34]": "https://www.gd121.cn/kp/index.shtml",
"[35]": "https://www.gd121.cn/hd/index.shtml",
"[36]": "https://www.gd121.cn/hd/yhfk/list.shtml",
"[37]": "https://www.gd121.cn/hd/sjsc/index.shtml",
"[38]": "http://passport.gd121.cn/ucy_center/index",
"[39]": "https://www.gd121.cn/cos/index.shtml",
"[40]": "https://www.gd121.cn/zx/qxzx/list.shtml",
"[41]": "https://www.gd121.cn/zx/qxzx/ff808081941a9bff01943a50301400cb.shtml",
"[42]": "https://www.gd121.cn/zx/qxzx/ff808081941a9bff01943487ebaa00b0.shtml",
"[43]": "https://www.gd121.cn/zx/qxzx/ff808081941a9bff01942fb95f2f0092.shtml",
"[44]": "https://www.gd121.cn/zx/qxzx/ff808081941a9bff01942a71bf0e005d.shtml",
"[45]": "https://www.gd121.cn/zx/qxzx/ff808081941a9bff019425c1ca9f0040.shtml",
"[46]": "https://www.gd121.cn/zx/qxzx/ff808081941a9bff0194203d3eab0023.shtml",
"[47]": "https://www.gd121.cn/zx/qxzx/ff808081941a9bff01941b026a880005.shtml",
"[48]": "https://www.gd121.cn/zx/qxzx/ff8080818a06a656019415daf94a3d5b.shtml",
"[49]": "https://www.gd121.cn/zx/qxzx/ff8080818a06a65601941090da963d47.shtml",
"[50]": "https://www.gd121.cn/zx/qxzx/ff8080818a06a65601940b57ef473d30.shtml",
"[51]": "https://www.gd121.cn/zx/qxzx/ff8080818a06a656019406d5b6493d14.shtml",
"[52]": "https://www.gd121.cn/zx/qxzx/ff8080818a06a656019401b162d13cf9.shtml",
"[53]": "https://www.gd121.cn/zx/qxzx/ff8080818a06a6560193fc4659ea3cd0.shtml",
"[54]": "https://www.gd121.cn/zx/qxzx/ff8080818a06a6560193f727ff503cae.shtml",
"[55]": "https://www.gd121.cn/zx/qxzx/ff8080818a06a6560193f251ce043c95.shtml",
"[56]": "https://www.gd121.cn/zx/qxzx/ff8080818a06a6560193ec938aea3c69.shtml",
"[57]": "https://www.gd121.cn/zx/qxzx/ff8080818a06a6560193e74a0f363c51.shtml",
"[58]": "https://www.gd121.cn/zx/qxzx/ff8080818a06a6560193e21fb3e73c2b.shtml",
"[59]": "https://www.gd121.cn/zx/qxzx/ff8080818a06a6560193dd4610be3c10.shtml",
"[60]": "https://www.gd121.cn/zx/qxzx/ff8080818a06a6560193d825534f3be1.shtml",
"[61]": "https://www.gd121.cn/zx/qxzx/list_2.shtml",
"[62]": "https://www.gd121.cn/zx/qxzx/list_10.shtml",
"[63]": "https://www.gd121.cn/zx/yjxy/list.shtml",
"[64]": "https://www.gd121.cn/zx/yjxy/8a14d310645ec72401645edada382a17.shtml",
"[65]": "https://www.gd121.cn/zx/yjxy/8a14d310645ec72401645edada382a17.shtml",
"[66]": "https://www.gd121.cn/zx/yjxy/8a14d310645ec72401645edadbb42a19.shtml",
"[67]": "https://www.gd121.cn/zx/yjxy/8a14d310645ec72401645edadd2e2a1b.shtml",
"[68]": "https://www.gd121.cn/zx/yjxy/8a14d310645ec72401645edade922a1d.shtml",
"[69]": "https://www.gd121.cn/zx/yjxy/8a14d310645ec72401645edadfff2a1f.shtml",
"[70]": "https://www.gd121.cn/zx/yjxy/8a14d310645ec72401645edae2d42a23.shtml",
"[71]": "https://www.gd121.cn/zx/yjxy/8a14d310645ec72401645edae16f2a21.shtml",
"[72]": "https://www.gd121.cn/zx/yjxy/8a14d310645ec72401645edae4362a25.shtml",
"[73]": "https://www.gd121.cn/kp/rdzt/list.shtml",
"[img74]": "https://www.gd121.cn/upload/image/2022/05/12/1652320917678096536.jpg",
"[75]": "https://tqkp.gd121.cn/pc/index.php/index/index",
"[img76]": "https://www.gd121.cn/upload/image/2021/03/04/1614835718147071414.jpg",
"[77]": "http://gd.weather.com.cn/zt/cbzt/3164789.shtml",
"[78]": "https://www.gd121.cn/zx/zhxx/list.shtml",
"[79]": "https://www.gd121.cn/zx/zhxx/ff8080818a06a656018a6dbc39560290.shtml",
"[80]": "https://www.gd121.cn/zx/zhxx/ff8080817a75a9a2017abcdfd97f01bd.shtml",
"[81]": "https://www.gd121.cn/zx/zhxx/ff808081706d374701718b8ba69306c3.shtml",
"[82]": "https://www.gd121.cn/zx/zhxx/ff808081706d374701716c94668c062a.shtml",
"[83]": "https://www.gd121.cn/zx/zhxx/ff808081706d374701715cf0535405d5.shtml",
"[84]": "https://www.gd121.cn/zx/zhxx/ff808081706d37470171438addf4052e.shtml",
"[85]": "https://www.gd121.cn/kp/yjkp/list.shtml",
"[86]": "https://www.gd121.cn/kp/yjkp/yjcs/zrzhsjcs/8a14d310645ec72401645ec9bdb20626.shtml",
"[87]": "https://www.gd121.cn/kp/yjkp/yjcs/zrzhsjcs/list.shtml",
"[88]": "https://www.gd121.cn/kp/yjkp/yjcs/zrzhsjcs/8a14d310645ec72401645ec9bdb20626.shtml",
"[89]": "https://www.gd121.cn/kp/yjkp/yjcs/ggwssjcs/8a14d310645ec72401645ec9c64f065c.shtml",
"[90]": "https://www.gd121.cn/kp/yjkp/yjcs/ggwssjcs/list.shtml",
"[91]": "https://www.gd121.cn/kp/yjkp/yjcs/ggwssjcs/8a14d310645ec72401645ec9c64f065c.shtml",
"[92]": "https://www.gd121.cn/kp/yjkp/yjcs/shaqsjcs/8a14d310645ec72401645ec9c994066e.shtml",
"[93]": "https://www.gd121.cn/kp/yjkp/yjcs/shaqsjcs/list.shtml",
"[94]": "https://www.gd121.cn/kp/yjkp/yjcs/shaqsjcs/8a14d310645ec72401645ec9c994066e.shtml",
"[95]": "https://www.gd121.cn/kp/yjkp/yjxh/8a14d310645ec72401645ec9c9f70670.shtml",
"[96]": "https://www.gd121.cn/kp/yjkp/yjxh/list.shtml",
"[97]": "https://www.gd121.cn/kp/yjkp/yjxh/8a14d310645ec72401645ec9c9f70670.shtml",
"[img98]": "https://www.gd121.cn/img/blank.gif",
"[99]": "https://www.gd121.cn/share/pluginWea.do",
"[100]": "https://www.gd121.cn/share/pluginWea.do",
"[img101]": "https://www.gd121.cn/img/blank.gif",
"[102]": "https://www.gd121.cn/share/interface.do?id=1",
"[103]": "https://www.gd121.cn/share/interface.do?id=1",
"[img104]": "https://www.gd121.cn/img/blank.gif",
"[105]": "https://www.gd121.cn/fw/yjkhd/8a14d310645ec72401645ecbea730edb.shtml",
"[106]": "https://www.gd121.cn/fw/yjkhd/8a14d310645ec72401645ecbea730edb.shtml",
"[img107]": "https://www.gd121.cn/img/blank.gif",
"[108]": "https://www.gd121.cn/fw/wx/ff8080816f16da59016f176859fa0019.shtml",
"[109]": "https://www.gd121.cn/fw/wx/ff8080816f16da59016f176859fa0019.shtml",
"[110]": "https://www.gd121.cn/yj/index.shtml",
"[111]": "https://www.gd121.cn/fw/index.shtml",
"[112]": "https://www.gd121.cn/kp/yjkp/list.shtml",
"[113]": "https://www.gd121.cn/kp/zcfg/list.shtml",
"[114]": "https://www.gd121.cn/share/",
"[115]": "https://www.gd121.cn/fw/index.shtml",
"[img116]": "https://www.gd121.cn/img/blue.png",
"[117]": "http://bszs.conac.cn/sitename?method=show&id=5A4B4AA9C22F62BEE053022E1AACADE4",
"[118]": "http://gd.cma.gov.cn/",
"[119]": "https://www.gbaweather.net/tc/",
"[120]": "http://beian.miit.gov.cn",
"[121]": "http://www.beian.gov.cn/portal/registerSystemInfo?recordcode=44010402001842"
},
"links_part": [
" \n今天是2025年01月06日 登录/注册\n广东气象 你的冷暖在我心中\n * 首页[1]\n * 预警[2]\n * 珠三角\n广州[3] 深圳[4] 佛山[5] 东莞[6] 中山[7] 珠海[8] 惠州[9] 江门[10] 肇庆[11]\n * 粤东\n汕头[12] 潮州[13] 揭阳[14] 汕尾[15]\n * 粤西\n湛江[16] 茂名[17] 阳江[18] 云浮[19]\n * 粤北\n韶关[20] 清远[21] 梅州[22] 河源[23]\n * 天气[24]\n * 雷达[25]\n * 云图[26]\n * 海洋[27]\n * 台风[28]\n * 旅游天气[29]\n * 机场&车站[30]\n * 高清图集[31]\n * 视频[32]\n * 服务[33]\n * 科普[34]\n * 互动[35]\n * 预约天气\n * 用户反馈[36]\n * 实景上传[37]\n * 用户中心[38]\n * 共享[39]\n\n\n资讯 >气象资讯[40]\n * 又双叒叕有阿冷,晴天还能留住吗?[41]\n2025-01-06 14:32:02\n * 未来几天广东继续维持天晴干燥[42]\n2025-01-05 11:35:11\n * 未来几天广东继续维持天晴干燥[43]\n2025-01-04 13:11:05\n * 广东维持天晴干燥 森林火险预警高[44]\n2025-01-03 12:34:45\n * 弱冷空气持续补充 未来几天广东晴燥维持[45]\n2025-01-02 14:44:05\n * 好天气相伴2025年第一天,元旦快乐![46]\n2025-01-01 13:01:12\n * 元旦广东大部天晴适宜出行[47]\n2024-12-31 12:38:51\n * 广东大部气温缓升 天气干燥[48]\n2024-12-30 12:37:40\n * 广东晴燥天气持续 气温回升[49]\n2024-12-29 11:58:36\n * 未来三天广东大部天晴干燥 早晚寒冷[50]\n2024-12-28 11:38:20\n * 本周末广东晴冷 最低零下2℃[51]\n2024-12-27 14:37:37\n * 新一股冷空气将影响 周末广东转晴冷[52]\n2024-12-26 14:39:50\n * 未来三天,广东有小雨光临![53]\n2024-12-25 13:24:49\n * 新一股冷空气在路上,气温下降3~5℃[54]\n2024-12-24 13:33:34\n * 未来3天广东云量增多气温回升[55]\n2024-12-23 15:01:08\n * 22-25日我省气温小幅回升,早晚寒冷[56]\n2024-12-22 12:15:13\n * 晴燥天气伴冬至 早晚寒冷要保暖[57]\n2024-12-21 11:36:51\n * 21-23日弱冷空气补充 广东晴冷干燥为主[58]\n2024-12-20 11:32:29\n * 冬至又有冷空气入粤[59]\n2024-12-19 12:56:17\n * 冷空气频繁 广东干冷天气持续[60]\n2024-12-18 13:02:25\n\n共200条 下一页[61] 尾页[62] 每页20条,第1/10页 转到第页 跳转 \n--- \n应急响应 更多>>[63]\n广东省气象局结束气象灾害(暴雨)Ⅳ级应急响应 -[64]\n广东省气象局结束气象灾害(暴雨...\n6月12日到14日,珠江三角洲市县、粤西大部分市县出现了暴雨到大暴雨局部特大暴雨。预计,15日...广东省气象局结束气象灾害(暴雨)Ⅳ级应急响应 - [详情][65]\n * 广东省气象局启动气象灾害(暴雨)Ⅳ级应急响应 - 广东省气象局启动气象灾害(暴雨)Ⅳ级...[66]\n * 广东省气象局结束气象灾害(暴雨)Ⅱ级应急响应 - 广东省气象局结束气象灾害(暴雨)Ⅱ级...[67]\n * 广东省气象局结束气象灾害(台风)Ⅲ级应急响应 - 广东省气象局结束气象灾害(台风)Ⅲ级...[68]\n * 广东省气象局启动气象灾害(暴雨)Ⅱ级应急响应 - 广东省气象局启动气象灾害(暴雨)Ⅱ级...[69]\n * 广东省气象局升级气象灾害(台风)应急响应为Ⅲ级 - 广东省气象局升级气象灾害(台风)应急...[70]\n * 广东省气象局启动气象灾害(台风)Ⅳ级应急响应 - 广东省气象局启动气象灾害(台风)Ⅳ级...[71]\n * 广东省气象局结束气象灾害(寒冷)Ⅲ级应急响应 - 广东省气象局结束气象灾害(寒冷)Ⅲ级...[72]\n\n\n热点专题 更多>>[73]\n * 2022年5.12全国防灾减灾日[img74][75]\n * 中国天然氧吧(龙门、揭西、连山、新兴)[img76][77]\n\n\n 1. 1\n 2. 2\n\n\n综合消息 更多>>[78]\n * 7-8日广东大部仍有强降水[79]\n * 7号台风“查帕卡”已生成 广东沿海有风雨 - 7号台风“查帕卡&rdq...[80]\n * 18-20日广东多云有雾 21-24日阴雨寒凉 - 18-20日广东多云有雾 21-24...[81]\n * 12-15日广东大部天晴 早晚寒凉[82]\n * 11日广东将有一次强对流天气过程[83]\n * 三天内广东大部降水频繁 粤北山区需防御地质灾害 - 三天内广东大部降水频繁 粤北山区需防...[84]\n\n\n应急科普 更多>>[85]\n都市中遇到洪水怎么办 -[86]\n自然灾害[87] 都市中遇到洪水怎么办[88]\n中国疾控中心提示:春节期间应做好传染病防御 -[89]\n公共卫生[90] 中国疾控中心提示:春节期间应做好传染病防御[91]\n购买、使用消防产品不容马虎 -[92]\n社会安全[93] 购买、使用消防产品不容马虎[94]\n森林火险预警信号 -[95]\n预警信号[96] 森林火险预警信号[97]\n天气插件[img98][99]\n天气插件[100]\n形式多样,强大的自定义功能,为您的网站量身定做个性化预警插件。\n数据接口[img101][102]\n数据接口[103]\n我们会为您们提供最权威、最真实、最及时、最全面的数据服务与共享。\n停课铃APP[img104][105]\n停课铃APP[106]\n广东省气象局官方客户端,提供全省停课信号、天气预警、天气查询等服务。\n缤纷微天气[img107][108]\n缤纷微天气[109]\n微信小程序,提供基于位置的天气预报、预警服务。\n预警信息[110] | 公众服务[111] | 应急科普[112] | 政策法规[113] | 数据共享[114] | 服务渠道[115]\n1§[img116][117]\n友情链接:广东省气象局网站[118] | 粤港澳大湾区天气网站[119]\n联系电话:020-87664716地址:广东省广州市天河区东莞庄路312号\n§ 粤ICP备05011356号[120] 粤公网安备 44010402001842号[121] ©广东省气象公共服务中心版权所有 | 技术支持:数鹏通(LinkCM)科技"
],
"contents": []
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
[{"focuspoint": "广州市未来一周的冷空气预警信息", "explanation": ""},
{"focuspoint": "广东全省的台风预警", "explanation": "仅限2024年的信息"}]
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
focus statement:
//广州市未来一周的冷空气预警信息//
//广东全省的台风预警//
解释:仅限2024年的信息


raw materials in: ab9447

llm model: Qwen/Qwen2.5-72B-Instruct
process time: 15.076175928115845 s
bad generate times: 0
diff from benchmark: benchmark: 8 results
segments:
https://www.gd121.cn/zx/qxzx/ff8080818a06a6560193d825534f3be1.shtml
https://www.gd121.cn/zx/qxzx/ff8080818a06a6560193fc4659ea3cd0.shtml
https://www.gd121.cn/zx/qxzx/ff8080818a06a6560193e74a0f363c51.shtml
https://www.gd121.cn/zx/qxzx/ff8080818a06a6560193ec938aea3c69.shtml
https://www.gd121.cn/zx/qxzx/ff8080818a06a6560193f251ce043c95.shtml
https://www.gd121.cn/zx/qxzx/ff8080818a06a6560193e21fb3e73c2b.shtml
https://www.gd121.cn/zx/qxzx/ff8080818a06a6560193dd4610be3c10.shtml
https://www.gd121.cn/zx/qxzx/ff8080818a06a6560193f727ff503cae.shtml
************

llm model: Qwen/Qwen2.5-7B-Instruct
process time: 0.6364688873291016 s
bad generate times: 0
diff from benchmark: 差异8个(遗漏8个,多出0个)
segments:

************

llm model: Qwen/Qwen2.5-14B-Instruct
process time: 4.469786167144775 s
bad generate times: 0
diff from benchmark: 差异3个(遗漏2个,多出1个)
segments:
https://www.gd121.cn/zx/qxzx/ff8080818a06a6560193d825534f3be1.shtml
https://www.gd121.cn/zx/qxzx/ff8080818a06a6560193fc4659ea3cd0.shtml
https://www.gd121.cn/zx/qxzx/ff8080818a06a6560193ec938aea3c69.shtml
https://www.gd121.cn/zx/qxzx/ff8080818a06a656019401b162d13cf9.shtml
https://www.gd121.cn/zx/qxzx/ff8080818a06a6560193e21fb3e73c2b.shtml
https://www.gd121.cn/zx/qxzx/ff8080818a06a6560193dd4610be3c10.shtml
https://www.gd121.cn/zx/qxzx/ff8080818a06a6560193f727ff503cae.shtml
************

llm model: Qwen/Qwen2.5-32B-Instruct
process time: 1.2642040252685547 s
bad generate times: 0
diff from benchmark: 差异8个(遗漏8个,多出0个)
segments:

************

llm model: deepseek-ai/DeepSeek-V2.5
process time: 11.476641178131104 s
bad generate times: 0
diff from benchmark: 差异5个(遗漏4个,多出1个)
segments:
https://www.gd121.cn/zx/qxzx/ff8080818a06a6560193d825534f3be1.shtml
https://www.gd121.cn/zx/qxzx/ff8080818a06a656019401b162d13cf9.shtml
https://www.gd121.cn/zx/qxzx/ff8080818a06a6560193e21fb3e73c2b.shtml
https://www.gd121.cn/zx/qxzx/ff8080818a06a6560193dd4610be3c10.shtml
https://www.gd121.cn/zx/qxzx/ff8080818a06a6560193f727ff503cae.shtml
************

Loading

0 comments on commit e2f3903

Please sign in to comment.