Implement the Restore Shader & Userscript

[Tama47]

Can you implement the Restore Shader? Since it's the shader that does the sharpening, Anime4K_Restore_CNN_S.glsl would be a good start. But I would love to see the complete Anime4K: Mode A (Fast) implemented someday. This is a very promising project!

Also, could this be implement into a userscript for web videos upscaling?
https://github.com/net2cn/Bilibili_Anime4K

[sb2702]

Absolutely. I will add some more shaders soon (in the next few weeks?), and the restore shaders seem easy to do.

By script, do you mean a plugin that will automatically add the upscaling or de-noising behavior into an HTML video player, so it upscales the video as you watch it?

I also plan to add an a script to intake a video and upscale it in the browser, and let you download it for viewing later, though that would come later.

First, I would like to re-write these shaders using WebGPU Compute Shaders, which will allow running these neural network operations 2 times or 3 times faster.

[Tama47]

Absolutely. I will add some more shaders soon (in the next few weeks?), and the restore shaders seem easy to do.

That's great to hear! I've actually tried retrofitting the restore shaders using your upscale shader. The output seems to be correct?

Now we just have to combine the restore and the upscale shaders to replicate the Anime4K effect, preferably with the M variant (Anime4K_Restore_CNN_M.glsl and Anime4K_Upscale_CNN_x2_M.glsl).

By script, do you mean a plugin that will automatically add the upscaling or de-noising behavior into an HTML video player, so it upscales the video as you watch it?

Yes, as it would be awesome to watch YouTube or Crunchyroll with Anime4K shaders applied directly from the web!

[Tama47]

Hi, has there been any update yet?

[sb2702]

Hi, I will implement these improvements in December, I have not had much time to work on this recently due to pressure at work. Thanks, Sam

…

On Tue, Nov 21, 2023, 5:30 AM Tama ***@***.***> wrote: Hi, has there been any update yet? — Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABLKLJTIYDD2DKDVPSR4CPTYFSGE3AVCNFSM6AAAAAA5EOMFW2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRQG42DANZZGE> . You are receiving this because you commented.Message ID: ***@***.***>

[sb2702]

@Tama47 - Sorry for the delay in replying to you. I think the restore Shader for the small model seems to be correct. I also recently implemented 2X Upscaling Medium and Large networks recently. They use mostly the same architecture as the Restore networks. If you can give me a few days, I can get the Small, Medium and Large Restore networks implemented.

[Tama47]

Thanks for getting back to me! I'm not in a hurry at all, so please take your time. I just think that smaller shaders will be helpful for most low-end laptops and PCs that most people use. Of course, this would be most helpful for the eventual goal of being able to use it to upscale and watch web videos in real-time. I've already tried drawing YouTube videos to canvas and was successful, but I had troubles implementing the shaders, so development is paused there. Hopefully, you'll be able to get something working. I want to be able to watch anime4k straight from the browser 😅

[sb2702]

@Tama47 I just added the restore shaders. You should be able to use them as follows:

const websr = new WebSR({
    resolution: {
        width: 640,
        height: 360
    }
    network_name: "anime4k/cnn-restore-s",
    weights: await (await fetch('./cnn-restore-s-an.json')).json() //found in weights/anime4k folder
    gpu,
    canvas: //A canvas, with 2x the width and height of your input video
});

await websr.render(source); // ImageBitmap, HTML5VideoElement or HTML5Image element

The weights are here: https://github.com/sb2702/websr/blob/main/weights/anime4k/cnn-restore-s-an.json

There's a small, medium and large network (cnn-restore-s, cnn-restore-m and cnn-restore-l) which correspond Anime 4K's Small, Medium and Large restore networks.

[Tama47]

Thanks for the update! Any news on the extension/userscript? I would really love to see a working prototype on YouTube for example.