Exploring the feasibilty of AI Upscaling.

[sathish-a]

Hello Shaka Player maintainers,

I am exploring the possibility of building (or extending) an open-source web video player to support video upscaling using a super-resolution model, and I am evaluating whether Shaka Player would be a suitable foundation for this work.

At a high level, the idea is to expose user-selectable options (for example, 2× and 4×) within the resolution/quality settings. Based on the selected option, a super-resolution model would be invoked to upscale the decoded video frames. The workflow I am considering involves buffering decoded frames, performing GPU-based inference for upscaling, and then rendering the enhanced frames for playback.

To maintain good startup performance and responsiveness, the player would initially start playback at the native resolution to avoid any cold-start overhead from the model. In parallel, buffered frames could be pre-emptively upscaled in the background. Similarly, during seek operations, the player would temporarily fall back to native resolution playback to ensure low latency and smooth user interaction.

At this stage, I am not looking for implementation details or code. I would greatly appreciate guidance and high-level feedback from the maintainers on:
• Whether Shaka Player’s architecture can accommodate such a workflow
• Which extension points or components would be most relevant for experimenting with decoded-frame processing
• Any known constraints or design considerations I should be aware of before proceeding

I’m hoping to brainstorm and understand the feasibility and best approach before moving forward. Thank you for your time and insights.

[avelad]

At a high level, Shaka Player is not designed to perform post-decode frame processing or to hook into the rendering pipeline. Its current architecture focuses on:

• adaptive streaming logic (DASH / HLS),
• buffering and playback control,
• delegating decoding to the browser via MediaSource Extensions,
• and handing off decoded frames directly to the native <video> element.

Because of that:

1. Decoded frame access

Shaka Player does not expose extension points to intercept or modify decoded video frames before rendering. Achieving GPU-based super-resolution inside the player would require deep changes to its internals or non-standard manipulation of HTMLVideoElement / MediaSource, which is outside the intended scope of the project.

2. Architectural limitations

• The player does not expose raw frame buffers that could be transformed or re-encoded.
• AI upscaling typically operates after decoding, often by copying frames from a <video> element into a <canvas>, running inference, and then re-rendering.
• That workflow would live outside Shaka Player rather than inside its playback pipeline.

3. What is feasible

A more practical approach would be:

• Use Shaka Player strictly for streaming, buffering, and decoding.
• Render video to a <video> element.
• Copy frames to a <canvas> (or WebGL/WebGPU surface).
• Apply AI super-resolution there.
• Display the processed output separately.

This allows AI upscaling alongside Shaka Player without requiring invasive changes to it.

4. Conclusion

In short:

• Shaka Player is not built to support decoded-frame processing or AI upscaling within its render pipeline.
• It can serve as a solid foundation in a broader architecture where upscaling happens at the rendering layer using browser APIs.
• Native integration would likely require a substantial fork or major architectural changes.