Skip to content

Commit 251d6f9

Browse files
wan0gewan0ge
authored
feat: 使用 bangumi-data 数据实现季边界映射，优化跨季请求量与准确性 (huangxd-#429)
* feat: 使用 bangumi-data 数据实现季边界映射，优化跨季请求量与准确性 本提交聚焦于跨季扩展的「请求量」与「命中准确性」两项优化，并以请求级 HTTP 缓存统一解决跨季扩展产生的重复请求（全源受益、零源改动）。 一、TMDB 季边界映射（核心能力，新增 tmdb-util.js） - 新增 buildTmdbSeasonBoundaries / getTmdbSeasonBoundaries，消费 bangumi-data（searchBangumiData）的扁平结果 {matchedSiteKey, siteId}，将 TMDB 的季结构映射为本项目的「季序 → 起始集」边界表。 - dandan-api.js 跨季扩展：在启用 Bangumi Data 且能解析到 TMDB 季边界时，由原先「向后逐季顺序扩展（S+1、S+2…）」收敛为「依据 TMDB 边界直接定位目标集所在季」，跳过其间无关季（如要看 S5、中间 S4 无需拉取），降低无效 API 请求量；未启用 Bangumi Data 或边界未解析时，维持上游原版逐季顺序扩展。 上述 TMDB 季边界增强仅在启用 Bangumi Data 且成功解析到边界时生效；否则跨季扩展与顺延均回落到上游原版逻辑。 二、跨季顺延准确性修正 - 跨季顺延的集数扣减：在能解析到 TMDB 边界时按「TMDB 标准总集数 seasonEpisodeTotal」扣减，否则按原版「某季实际拉到的集数 allEps.length（可能偏短或偏长）」扣减，避免中间季集数统计偏差导致收敛到错误季。 - 原 [spillover] 共有…集 日志同步改用 seasonEpisodeTotal 以与扣减一致。 三、请求级 HTTP 响应复用（全源去重，零源改动） - http-util.js 的 httpGet 新增请求级缓存：同一搜索请求内相同 URL 的重复 GET 直接复用首次响应，不再发网络请求；配套 httpCacheContext 与 runWithHttpCache。 - dandan-api.js 的 searchAnime 用 runWithHttpCache 把整次搜索（主搜索 + 跨季扩展）包进同一请求级缓存上下文。 - 跨季扩展重跑各源 handleAnimes / getEpisodes 时，使用的是与主搜索相同的番剧 id（详情 URL 一致），重复 GET 自动抹平。该机制位于共享 httpGet，对所有约 20 个源自动生效，无需改动任何源文件。 四、改动范围最小化 - 最终仅 3 个文件有分支专属改动：dandan-api.js（跨季扩展逻辑）、tmdb-util.js（季边界映射）、http-util.js（HTTP 缓存）。各源文件（含 dandan.js）经评审已全部回退至上游，未携带任何分支专属改动。 * fix: 修复请求级 HTTP 缓存误杀源级同 URL 重试 请求级 HTTP 缓存（httpCacheContext / runWithHttpCache）对每次成功响应写缓存，读取位于 httpGet 顶部、先于源级重试逻辑。iqiyi 搜索风控（code=-1）与 base_info 异常的源级同 URL 重试重调 httpGet 时命中缓存，复用首次已缓存的失败响应，导致重试被静默吞掉、始终失败。 新增 bypassCache 选项：源级重试显式声明后跳过读/写缓存，刻意重取绕过去重；首次（非重试）调用仍走缓存，跨季扩展的全源去重收益保留。通用请求工具内置的 retries: 重试不受此问题影响（失败尝试不写缓存、循环内重发网络）。本修复仅 iqiyi 有源级同 URL 重试，故根机制（http-util.js）配合其重试接线即覆盖全部受影响源。
1 parent 115fae3 commit 251d6f9

4 files changed

Lines changed: 172 additions & 12 deletions

File tree

‎danmu_api/apis/dandan-api.js‎

Lines changed: 77 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
import { globals } from '../configs/globals.js';
2-
import { getPageTitle, jsonResponse, httpGet, sourceLogContext, toLogSourceName } from '../utils/http-util.js';
2+
import { getPageTitle, jsonResponse, httpGet, sourceLogContext, toLogSourceName, runWithHttpCache, httpCacheContext } from '../utils/http-util.js';
33
import { log } from '../utils/log-util.js'
44
import { simplized } from '../utils/zh-util.js';
55
import { setRedisKey, updateRedisCaches } from "../utils/redis-util.js";
@@ -17,7 +17,7 @@ import {
1717
extractEpisodeTitle, convertChineseNumber, parseFileName, createDynamicPlatformOrder, normalizeSpaces,
1818
extractYear, titleMatches, extractAnimeInfo, extractEpisodeNumberFromTitle, extractSeasonNumberFromAnimeTitle, extractAnimeTitle
1919
} from "../utils/common-util.js";
20-
import { getTMDBChineseTitle } from "../utils/tmdb-util.js";
20+
import { getTMDBChineseTitle, getTmdbSeasonBoundaries } from "../utils/tmdb-util.js";
2121
import { applyMergeLogic, mergeDanmakuList, MERGE_DELIMITER, alignSourceTimelines } from "../utils/merge-util.js";
2222
import { getHanjutvSourceLabel } from "../utils/hanjutv-util.js";
2323
import AIClient from '../utils/ai-util.js';
@@ -481,6 +481,14 @@ async function executeSourceHandlers(resultData, queryTitle, targetAnimesList, r
481481

482482
// Extracted function for GET /api/v2/search/anime
483483
export async function searchAnime(url, preferAnimeId = null, preferSource = null, detailStore = null, targetPlatform = null, forceRefresh = false) {
484+
// 单次搜索请求内启用 HTTP 响应复用缓存: 作为各源通用的请求级复用安全网, 借助 AsyncLocalStorage 做请求级隔离
485+
if (httpCacheContext.getStore()) {
486+
return searchAnimeBody(url, preferAnimeId, preferSource, detailStore, targetPlatform, forceRefresh);
487+
}
488+
return runWithHttpCache(() => searchAnimeBody(url, preferAnimeId, preferSource, detailStore, targetPlatform, forceRefresh));
489+
}
490+
491+
async function searchAnimeBody(url, preferAnimeId = null, preferSource = null, detailStore = null, targetPlatform = null, forceRefresh = false) {
484492
let queryTitle = url.searchParams.get("keyword");
485493

486494
// 搜索词杂音清理：移除画质/配音/版本等杂音词后再提交源站搜索
@@ -492,6 +500,7 @@ export async function searchAnime(url, preferAnimeId = null, preferSource = null
492500
querySeason = querySeason ? parseInt(querySeason, 10) : null;
493501
let queryEpisode = url.searchParams.get("episode");
494502
queryEpisode = queryEpisode ? parseInt(queryEpisode, 10) : null;
503+
let tmdbSeasonBoundaries = null;
495504
log("info", `[system] [searchAnime] Search anime with keyword: ${queryTitle}, target season: ${querySeason}, target episode: ${queryEpisode}`);
496505

497506
// 关键字为空直接返回，不用多余查询
@@ -848,6 +857,24 @@ export async function searchAnime(url, preferAnimeId = null, preferSource = null
848857

849858
if (maxSeason > querySeason) {
850859
log("info", `[system] [LogVar-API] Episode ${queryEpisode} not satisfied in Season ${querySeason}. Parallel mapping to S${querySeason + 1}~S${maxSeason}...`);
860+
// 依据 bangumi-data 的 TMDB 季边界定位目标集所在季, 跨季扩展直接收敛至目标季并跳过无关中间季, 集数扣减交由 findCrossSeasonEpisodeMap 借 TMDB 边界完成
861+
let targetSeasons = [];
862+
if (globals.useBangumiData && queryEpisode) {
863+
tmdbSeasonBoundaries = await getTmdbSeasonBoundaries(queryTitle);
864+
if (tmdbSeasonBoundaries && tmdbSeasonBoundaries.length >= 2) {
865+
for (let i = tmdbSeasonBoundaries.length - 1; i >= 0; i--) {
866+
const b = tmdbSeasonBoundaries[i];
867+
if (queryEpisode >= b.startEpisode) {
868+
targetSeasons = [b.order];
869+
break;
870+
}
871+
}
872+
}
873+
}
874+
875+
const expansionStart = targetSeasons.length > 0 ? Math.min(...targetSeasons) : querySeason + 1;
876+
const expansionEnd = targetSeasons.length > 0 ? Math.max(...targetSeasons) : maxSeason;
877+
851878
const expandPromises = [];
852879
for (const source of globals.sourceOrderArr) {
853880
if (!resultData[source]) continue;
@@ -856,7 +883,7 @@ export async function searchAnime(url, preferAnimeId = null, preferSource = null
856883
// 源间并发、源内顺序，防止同源并发导致模块级缓存竞态
857884
expandPromises.push((async () => {
858885
const sourceResults = [];
859-
for (let s = querySeason + 1; s <= maxSeason; s++) {
886+
for (let s = expansionStart; s <= expansionEnd; s++) {
860887
const seasonAnimes = [];
861888
await executeSourceHandlers({ [source]: resultData[source] }, queryTitle, seasonAnimes, requestAnimeDetailsMap, s, preferAnimeId, preferSource);
862889
if (seasonAnimes.length > 0) {
@@ -953,6 +980,7 @@ export async function searchAnime(url, preferAnimeId = null, preferSource = null
953980
success: true,
954981
errorMessage: "",
955982
animes: responseAnimes,
983+
tmdbSeasonBoundaries,
956984
});
957985

958986
}
@@ -1260,13 +1288,40 @@ function findCrossSeasonEpisodeMap(searchData, title, year, season, episode, pla
12601288
}
12611289
}
12621290

1291+
// 依据 TMDB 季边界推导「季号 -> 该季标准集数」, 使跨季顺延按剧集标准结构扣减而非依赖单源实际集数
1292+
const boundaries = searchData.tmdbSeasonBoundaries;
1293+
const seasonBoundaryCount = new Map();
1294+
if (Array.isArray(boundaries) && boundaries.length >= 2) {
1295+
for (let i = 0; i < boundaries.length; i++) {
1296+
const cur = boundaries[i];
1297+
const next = boundaries[i + 1];
1298+
seasonBoundaryCount.set(cur.order, next ? next.startEpisode - cur.startEpisode : null);
1299+
}
1300+
}
1301+
1302+
12631303
let currentTargetEpisode = episode;
12641304
let currentSeason = season;
12651305
let bestRes = { anime: null, episode: null, score: 0 };
12661306

1267-
while (seasonMap.has(currentSeason)) {
1307+
while (seasonMap.has(currentSeason) || seasonBoundaryCount.has(currentSeason)) {
12681308
const seasonData = seasonMap.get(currentSeason);
1309+
// 该季标准集数: 优先取 TMDB 边界推导值, 末季或边界缺失时回退至实际过滤后集数
1310+
const boundaryCount = seasonBoundaryCount.get(currentSeason);
1311+
1312+
// 中间季未拉取详情(已被 TMDB 边界跳过)时, 仅按其标准集数扣减以推进到目标季, 不参与实际集标题匹配
1313+
if (!seasonData) {
1314+
if (boundaryCount && boundaryCount > 0) {
1315+
currentTargetEpisode -= boundaryCount;
1316+
currentSeason++;
1317+
continue;
1318+
}
1319+
break;
1320+
}
1321+
12691322
const allEps = seasonData.episodes;
1323+
const seasonEpisodeTotal = (boundaryCount && boundaryCount > 0) ? boundaryCount : allEps.length;
1324+
12701325

12711326
let absoluteMatch = null;
12721327
for (const ep of allEps) {
@@ -1293,6 +1348,22 @@ function findCrossSeasonEpisodeMap(searchData, title, year, season, episode, pla
12931348

12941349
if (currentTargetEpisode <= allEps.length) {
12951350
const targetEp = allEps[currentTargetEpisode - 1];
1351+
if (platform && getPlatformMatchScore(extractEpisodeTitle(targetEp.episodeTitle), platform) === 0) {
1352+
currentSeason++;
1353+
continue;
1354+
}
1355+
log("info", `[system] [spillover] 跨季溢出查找命中 (按相对排位计算) -> 所在季：S${currentSeason} 集标题：${targetEp.episodeTitle}`);
1356+
bestRes = {
1357+
anime: seasonData.anime,
1358+
episode: targetEp,
1359+
score: platform ? getPlatformMatchScore(seasonData.actualPlatform, platform) : 1
1360+
};
1361+
break;
1362+
}
1363+
1364+
// 目标集号超出实际集数但仍落在该季 TMDB 标准区间内: 真实集数偏短时返回该季最后一集, 避免无谓顺延至后续季
1365+
if (currentTargetEpisode <= seasonEpisodeTotal) {
1366+
const targetEp = allEps[allEps.length - 1];
12961367
if (platform && getPlatformMatchScore(extractEpisodeTitle(targetEp.episodeTitle), platform) === 0) {
12971368
currentSeason++;
12981369
continue;
@@ -1306,8 +1377,8 @@ function findCrossSeasonEpisodeMap(searchData, title, year, season, episode, pla
13061377
break;
13071378
}
13081379

1309-
log("info", `[system] [spillover] S${currentSeason} 共有 ${allEps.length} 集(已过滤番外)，剩余目标集数为 ${currentTargetEpisode}，映射至 S${currentSeason + 1} 继续查找`);
1310-
currentTargetEpisode -= allEps.length;
1380+
log("info", `[system] [spillover] S${currentSeason} 共有 ${seasonEpisodeTotal} 集(已过滤番外)，剩余目标集数为 ${currentTargetEpisode}，映射至 S${currentSeason + 1} 继续查找`);
1381+
currentTargetEpisode -= seasonEpisodeTotal;
13111382
currentSeason++;
13121383
}
13131384

‎danmu_api/sources/iqiyi.js‎

Lines changed: 9 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -56,14 +56,15 @@ export default class IqiyiSource extends BaseSource {
5656
const queryString = buildQueryString(params);
5757
const url = `https://mesh.if.iqiyi.com/portal/lw/search/homePageV3?${queryString}`;
5858

59-
const doSearch = async () => {
59+
const doSearch = async (bypassCache = false) => {
6060
const resp = await httpGet(url, {
6161
headers: {
6262
'accept': '*/*',
6363
'origin': 'https://www.iqiyi.com',
6464
'referer': 'https://www.iqiyi.com/',
6565
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36'
66-
}
66+
},
67+
bypassCache
6768
});
6869
if (!resp || !resp.data) return null;
6970
return typeof resp.data === "string" ? JSON.parse(resp.data) : resp.data;
@@ -76,7 +77,7 @@ export default class IqiyiSource extends BaseSource {
7677
const reason = !data ? "搜索响应为空" : `搜索接口风控 (code=${data.code})`;
7778
log("info", `[iqiyi] ${reason}，等待 3 秒后重试 (${attempt + 1}/${MAX_RETRIES})`);
7879
await new Promise(r => setTimeout(r, 3000));
79-
data = await doSearch();
80+
data = await doSearch(true);
8081
}
8182

8283
if (!data) {
@@ -522,13 +523,14 @@ export default class IqiyiSource extends BaseSource {
522523

523524
// base_info 接口可能返回空响应或结构残缺的响应，此处最多重试两次再放弃
524525
const MAX_EPISODE_RETRIES = 2;
525-
const fetchEpisodeData = async () => {
526+
const fetchEpisodeData = async (bypassCache = false) => {
526527
try {
527528
const resp = await httpGet(url, {
528529
headers: {
529530
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
530531
'Referer': 'https://www.iqiyi.com/'
531-
}
532+
},
533+
bypassCache
532534
});
533535
if (!resp || !resp.data) return null;
534536
return typeof resp.data === "string" ? JSON.parse(resp.data) : resp.data;
@@ -542,7 +544,7 @@ export default class IqiyiSource extends BaseSource {
542544
for (let attempt = 0; attempt < MAX_EPISODE_RETRIES && (!data || data.status_code !== 0 || !data.data || !data.data.template); attempt++) {
543545
log("info", `[iqiyi] 分集接口返回异常${data ? ` (status_code: ${data.status_code})` : " (响应为空或解析失败)"}，等待 3 秒后重试 (${attempt + 1}/${MAX_EPISODE_RETRIES})`);
544546
await new Promise(r => setTimeout(r, 3000));
545-
data = await fetchEpisodeData();
547+
data = await fetchEpisodeData(true);
546548
}
547549

548550
if (!data || data.status_code !== 0 || !data.data || !data.data.template) {
@@ -972,6 +974,7 @@ export default class IqiyiSource extends BaseSource {
972974
"Content-Type": "application/json",
973975
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36",
974976
},
977+
bypassCache: true
975978
});
976979
const retryData = typeof retryRes.data === "string" ? JSON.parse(retryRes.data) : retryRes.data;
977980
const retryDuration = Number(retryData.data.durationSec) || 0;

‎danmu_api/utils/http-util.js‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,13 @@ import { AsyncLocalStorage } from 'node:async_hooks';
55
// 跨异步生命周期链路的日志上下文追踪器
66
export const sourceLogContext = new AsyncLocalStorage();
77

8+
// 单次搜索请求内的 HTTP 响应复用缓存: 相同 URL 的重复 GET 直接复用, 借助 AsyncLocalStorage 实现请求级隔离
9+
export const httpCacheContext = new AsyncLocalStorage();
10+
11+
export function runWithHttpCache(fn) {
12+
return httpCacheContext.run(new Map(), fn);
13+
}
14+
815
// 源调度键名（sourceOrderArr）到日志标签规范名称的映射
916
// sourceOrderArr 中部分键名与对应源文件的标签命名不一致（如 360→360kan, imgo→mango）
1017
// 此映射表统一转换，确保 HTTP 日志标签与源文件内部标签一致
@@ -53,6 +60,16 @@ function linkSignal(externalSignal, internalController) {
5360
}
5461

5562
export async function httpGet(url, options = {}) {
63+
// 单次搜索请求内 HTTP 响应复用: 若当前请求上下文已激活复用缓存且本 URL 已缓存, 直接返回克隆结果, 跳过重复网络请求
64+
const requestHttpCache = httpCacheContext.getStore();
65+
// 重试调用传入 bypassCache 时跳过复用，避免复用首次已缓存的失败响应而令重试被静默吞掉
66+
const bypassCache = options.bypassCache === true;
67+
if (requestHttpCache && !bypassCache && requestHttpCache.has(url)) {
68+
const cached = requestHttpCache.get(url);
69+
log("info", `[${sourceLogContext.getStore() || 'system'}] [请求复用] 复用请求内已缓存的 HTTP 响应, 跳过重复请求: ${url}`);
70+
return { data: structuredClone(cached.data), status: cached.status, headers: { ...cached.headers } };
71+
}
72+
5673
// 从 options 中获取重试次数，默认为 0
5774
const maxRetries = parseInt(options.retries || '0', 10) || 0;
5875
// 提取允许放行的特定状态码白名单
@@ -205,6 +222,11 @@ export async function httpGet(url, options = {}) {
205222
log("info", `[${currentSource}] [请求模拟] 重试成功`);
206223
}
207224

225+
// 将本次响应记入请求内复用缓存, 供同请求内相同 URL 的后续请求直接复用
226+
if (requestHttpCache && !bypassCache) {
227+
requestHttpCache.set(url, { data: structuredClone(parsedData), status: response.status, headers });
228+
}
229+
208230
// 模拟 iOS 环境：返回 { data: ... } 结构
209231
return {
210232
data: parsedData,

‎danmu_api/utils/tmdb-util.js‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -759,3 +759,67 @@ export function smartTitleReplace(animes, cnAlias) {
759759
}
760760
}
761761
}
762+
763+
// =====================
764+
// TMDB 季边界映射
765+
// =====================
766+
767+
/**
768+
* 将 bangumi-data 的 TMDB 扁平检索结果转换为跨季边界序列.
769+
* 每个 matchedSiteKey 为 'tmdb' 的结果, 其 siteId 形如 "tv/{id}" 或
770+
* "tv/{id}/season/N/episode/{M}", M 为该条目在 TMDB 绝对编号中的起始集数;
771+
* 收集同一 tv/{id} 下各条目起始集数即可推算跨季映射边界
772+
* @param {Array<Object>} matches searchBangumiData 返回的扁平结果数组
773+
* @returns {Array<{order:number, startEpisode:number, title:string, tmdbId:string}>|null}
774+
*/
775+
export function buildTmdbSeasonBoundaries(matches) {
776+
const baseMaps = new Map();
777+
778+
for (const m of matches) {
779+
if (!m || m.matchedSiteKey !== 'tmdb' || !m.siteId) continue;
780+
781+
const epMatch = m.siteId.match(/^(tv\/\d+)(?:\/season\/\d+\/episode\/(\d+))?$/);
782+
if (!epMatch) continue;
783+
784+
const baseId = epMatch[1];
785+
const startEp = epMatch[2] ? parseInt(epMatch[2], 10) : 1;
786+
787+
const entry = { order: 0, startEpisode: startEp, title: m.title || '', tmdbId: baseId };
788+
if (!baseMaps.has(baseId)) {
789+
baseMaps.set(baseId, [entry]);
790+
} else {
791+
baseMaps.get(baseId).push(entry);
792+
}
793+
}
794+
795+
if (baseMaps.size === 0) return null;
796+
797+
const best = [...baseMaps.entries()]
798+
.sort((a, b) => b[1].length - a[1].length)[0];
799+
800+
if (best[1].length < 2) return null;
801+
802+
best[1].sort((a, b) => a.startEpisode - b.startEpisode);
803+
for (let i = 0; i < best[1].length; i++) {
804+
best[1][i].order = i + 1;
805+
}
806+
return best[1];
807+
}
808+
809+
/**
810+
* 依据番剧标题从 bangumi-data 推导 TMDB 跨季边界, 用于在目标集不在首季时跳过无关季号
811+
* @param {string} title 检索标题 (直接传给 searchBangumiData, 内部已做季剥离处理)
812+
* @param {(title: string, siteKeys: string[]) => Promise<Array<Object>>} [searchFn]
813+
* @returns {Promise<Array<{order:number, startEpisode:number, title:string, tmdbId:string}>|null>}
814+
*/
815+
export async function getTmdbSeasonBoundaries(title, searchFn = searchBangumiData) {
816+
if (!globals.useBangumiData) return null;
817+
818+
try {
819+
const matches = await searchFn(title, ['tmdb']);
820+
return buildTmdbSeasonBoundaries(matches);
821+
} catch (e) {
822+
log('warn', `[system] [tmdb] Bangumi-Data 本地获取TMDB季边界失败: ${e.message}（检索词：${title}）`);
823+
return null;
824+
}
825+
}

0 commit comments

Comments
 (0)