Skip to content

feat: Provide shared YouTube video data lookup helper (with Shorts support) - #4443

Open
hkzo wants to merge 4 commits into
code-charity:masterfrom
hkzo:feat/video-metadata
Open

hkzo wants to merge 4 commits into
code-charity:masterfrom
hkzo:feat/video-metadata

Conversation

@hkzo

@hkzo hkzo commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Currently, multiple player features inside player.js maintain their own YouTube video metadata lookup logic. This leads to code duplication and makes individual features harder to follow.

Since we made significant feature enhancements and architectural overhauls from the previous PR (#4154), we are resetting this as a brand-new PR rather than continuing the old one.

Key Changes

  • Cross-video information lookup: Expanded functionality to allow fetching and referencing information for any specified videoId, not just the currently playing video.
  • YouTube Shorts support: Added support for fetching and handling metadata for YouTube Shorts alongside standard videos.
  • Per-videoId caching: Refactored to maintain an independent in-memory cache (Map) for each videoId. Old data is removed via gc().
  • Expanded metadata support: Increased the variety of handled metadata fields, including channel info, author, and uploadDate.
  • Enhanced debug logging: Implemented lightweight detailed debug logging to aid investigations into YouTube spec changes, which can be enabled via development options.

Design / Implementation Direction

  • Decouple features: VideoMetadata provides metadata and marker inputs; it does not decide whether a video is music, where to seek, or how Smart Speed should behave.
  • Cache management per videoId: Caches resolved metadata per videoId to avoid repeated DOM lookups and fallback fetches.
  • Prioritize current-page data: Prioritize existing on-page data, falling back to a watch-page fetch only when the required data is missing from the current context.
  • Use async APIs: Expose async boundaries because fallback paths may require network requests.
  • Encapsulate implementation details: Keep DOM parsing, fetched watch-page parsing, SPA navigation handling, and heatmap marker lookup internal so feature code does not need to know which source produced the data.
  • Keep the public API minimal: Provide both bundled and focused access patterns (e.g., getDataAsync(), etc.).

Testing / Current Status

  • Verified behavior on actual browsers.
  • Manually tested jumpToKeyScene() by using this feature for reference.
  • playerPlaybackSpeed will be submitted in a separate PR (due to existing issues on the current master branch).

LLM disclosure

  • I use LLMs to assist with the translation of this description, as well as for creating short English phrases in code comments or log messages.
  • Writing and refining test code / test cases.

@ImprovedTube2
ImprovedTube2 requested review from AryaXDG and removed request for AryaXDG November 4, 2026 04:52

if (trustedFetchedDocument || this._isCurrentDomVerified(doc, expectedVideoId)) {
// Do not read meta[name="keywords"]; YouTube may populate it with generic
// defaults that are not reliable video-specific keywords.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

defaultKeywords = "video,sharing,camera,phone,video phone,free,upload";

if (DATA.genre && DATA.title && DATA.keywords && DATA.lengthSeconds) { if (DATA.keywords.toString() === defaultKeywords ) {DATA.keywords = ''}

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand that logic, but since default keywords are localized, they would become noise outside of English-speaking regions, so I decided not to adopt them from meta. For example, in Japanese, the default keywords become "動画,ビデオ,共有,カメラ付き携帯電話,動画機能付き携帯電話,無料,アップロード,チャンネル,コミュニティ,YouTube,ユーチューブ".

@ImprovedTube2

Copy link
Copy Markdown
Member

unverified LLM comment:

  • Any redundancy? fetchStatus sourceStatus state.source
  • Full HTML retention? rawFetchedWatchPageHtml
  • Network failure considered as not existing? _getEntryFieldAsync() _setNotFound().

@hkzo

hkzo commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator Author
  1. Any redundancy? fetchStatus sourceStatus state.source
  • state.fetchStatus: Overall status of the watch page fetch. Has idle / pending / done / failed
  • state.sourceStatus: Preparation and processing status per data source. Tracks unhandled, in-progress, completed, and failed states for currentPlayerResponse, currentDom, fetchedDom, fetchedWatchPage, etc. Used to share in-flight fetches and reuse processed sources. Currently, fetchedWatchPage in state.sourceStatus redundantly tracks the same status as state.fetchStatus. A bit redundant.
  • state.source: Source of the value for each metadata item. Records the origin of the adopted value or a not-found result, rather than the data source processing status. Examples: title: fetchedPlayerResponse, genre: currentDom, keywords: not-found.

  1. Full HTML retention? rawFetchedWatchPageHtml

Yes. Retained for lazy parsing of the "fetched DOM" used as the final fallback for title, genre, duration, and lengthSeconds.

  • Maintain lazy parsing: Limits DOMParser's CPU and temporary memory usage exclusively to videos that actually require the fetched DOM. In exchange, the HTML string is kept in the cache for its lifetime on entries where the DOM fallback is not invoked.
  • Parse DOM on fetch: Generates DOM metadata at the same time as JSON extraction, allowing the HTML string to be released immediately. Ensures the DOM fallback is reliably available, but incurs parsing overhead even when the player response alone is sufficient or only ytInitialData is read, and causes HTML and DOM to coexist temporarily during parsing.

Consider earlier release.


  1. Network failure considered as not existing? _getEntryFieldAsync() _setNotFound().

That's correct. If fetching fails and no values can be obtained from other sources, it is currently cached as "not-found". I'll consider revising the state management to distinguish between a state where the data genuinely does not exist and an unverified state due to a fetch failure.

hkzo added 2 commits November 6, 2026 00:41
- Distinguish between 'not-found' and source-specific errors (network/DOM parsing failures) to improve observability.
- Stop storing raw HTML in memory by default to reduce memory footprint (retained only in debug mode).
- Rename SOURCE_PRIORITY_WITH_DOM to SOURCE_PRIORITY_PLAYER_AND_DOM to clarify source composition.
- Minor cleanup and adjustments.
- Move maxEntries and maxAgeMs default values from `gc()` to `ImprovedTube.VideoMetadata`
- Update default maxEntries from 50 to 100
- Update JSDoc for `gc` method to reflect new default value
@hkzo

hkzo commented Nov 8, 2026

Copy link
Copy Markdown
Collaborator Author

Summary of Changes

  • Differentiated error handling for missing data vs. execution errors:
    Previously, "missing specified values" and "errors during DOM parsing or network requests" were treated identically.
    Now, these are handled separately. This allows subsequent logic to distinguish between returning "no data" from the cache and re-triggering a fetch operation.
  • Optimized memory efficiency by shifting from lazy parsing to immediate parsing:
    • Instead of storing HTML for later parsing, the system now parses HTML immediately upon fetching and releases the HTML resource as early as possible.
    • For debugging purposes, a developer-facing internal flag has been added to retain HTML for investigation.
  • Added test cases (drafted with assistance from LLM).
  • Bug fix: Fixed an issue in the reference implementation of jumpToKeyScene().
  • Minor refinements:
    • Fixed const declarations.
    • Moved GC-related default values to properties.
    • Increased the default limit for the cache.-

@hkzo
hkzo force-pushed the feat/video-metadata branch from 92b159b to bac4723 Compare November 11, 2026 01:47
@hkzo

hkzo commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator Author

Updated the PR description to reflect the added support for YouTube Shorts.
With this change, all intended work for this PR is complete.

@hkzo hkzo changed the title feat: Extract shared YouTube video data lookup helper feat: Provide shared YouTube video data lookup helper (with Shorts support) Nov 11, 2026
@hkzo
hkzo force-pushed the feat/video-metadata branch from bac4723 to 0319e33 Compare November 12, 2026 03:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants