Skip to content

Use build/routes.json instead of scraping many docs pages for /rsshub/routes #20520

@black-backdoor

Description

@black-backdoor

Routes

/rsshub/routes/:lang?

Full routes

/rsshub/routes/en

Related documentation

https://docs.rsshub.app/routes/program-update#new-routes

What is expected?

The current route implementation for /rsshub/routes scrapes many documentation pages and uses a hard-coded types list. This is inefficient and brittle. I propose reading the pre-built build/routes.json (see possible alternative below) instead of scraping each category page. I couldn't open a PR for this because I don't yet understand the full RSSHub build process.

Possible alternative
https://raw.githubusercontent.com/DIYgod/RSSHub/refs/heads/gh-pages/build/routes.json

What is actually happening?

Source:
https://github.com/DIYgod/RSSHub/blob/d5cd095599177ac48c774719ad4094967f074030/lib/routes/rsshub/routes.ts

Existing Code:

async function handler(ctx) {
    const isEnglish = ctx.req.param('lang') !== 'zh';

    const lang = isEnglish ? '' : 'zh/';
    const types = [
        'social-media',
        'new-media',
        // ...
        'journal',
        'finance',
        'other',
    ];
    const all = await Promise.all(
        types.map(async (type) => {
            const response = await ofetch(`https://docs.rsshub.app/${lang}routes/${type}`);

            const $ = cheerio.load(response);
            const page = $('.page').toArray();
            const item = $('.routeBlock').toArray();
            return { page, item, type };
        })
    );
    const list = all.flatMap(({ page, item, type }) => item.map((item) => ({ page, item, type })));

    return {
        title: isEnglish ? 'RSSHub has new routes' : 'RSSHub 有新路由啦',
        link: 'https://docs.rsshub.app',
        // ...
    };
}

Deployment information

Self-hosted

Deployment information (for self-hosted)

No response

Additional info

-

This is not a duplicated issue

  • I have searched existing issues to ensure this bug has not already been reported

Metadata

Metadata

Assignees

No one assigned

    Labels

    RSS bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions