MAXIM models and "blocky" output

[tmtisfree]

I implemented all MAXIM models according to the documentation (which is great btw, thank you). I reused the code from the upscaler side of the app which works fine in browsers.

To test the models I used the pictures from the documentation. All models output images with blocks whatever the size of the input or patch size (64, 128) or padding (from 2 to 5).

Patch size of 128 exhibits less blocks, but still visible in small size input. Padding has not much effect (at least on small size input, say width/height < 200 px).

The issue seems similar to this one, but with all models exhibiting blocks.

Tests with medium size input

Deblur output with patch size 64 and padding 2 (blocks visible by zooming in):

Derain output with patch size 64 and padding 4 (blocks visible by zooming in):
Derain model loaded from https://cdn.jsdelivr.net/npm/@upscalerjs/maxim-deraining@0.1.0/models/model.json

Tests of patch size and padding in Low light enhancement model with small size input

Output with patch size 64 and padding 2:

Output with patch size 64 and padding 4:

Output with patch size 128 and padding 2:

Output with patch size 128 and padding 4:

[thekevinscott]

Thank you for the report, @tmtisfree. (It's been a while since I published these models, so I'm trying to get back up to speed.)

I was able to reproduce your results above.

Some of these models - for example, low light enhancement - are global context models. They rely on the full image in order to produce a coherent image, so patch sizing fundamentally won't work. The models themselves can accept any sized image without patch sizing, but in the browser you'll run into WebGL limits above a certain size. So the only answer for those, for larger images, is to run them in Node.

Other models - like the deraining and deblurring - should work fine with patch sizes, and I believe they did work when I originally published them, but they no longer work. I wonder if there was a change in the TFJS library that caused this.

Just to confirm, here's my version with patch sizes on Node:

And without patch sizes:

So the blockiness does seem to be a direct consequence of specifying patch sizes.

---

Can you tell me more about your intended use case for this? I haven't been doing much development on UpscalerJS recently and frankly I'm not sure how much more to do.

UpscalerJS is built on Tensorflow.js, which appears to be in maintenance mode if not abandoned.

I've discussed in the past exploring a version 2 of this that would include support for Transformers.js and/or mlc-ai. Either would address the major challenge of this library which has been providing ports of useful ML models to the browser.

However, either would also be a significant rewrite of this library, and it's not clear to me whether bespoke capability models (like derain, deblur) have a role when generalized image generation models have gotten so much better; e.g. if a Nano Banana can upscale as effectively as a bespoke model, why use bespoke models?

[tmtisfree]

Thank you for your response.

While doing some tests in Firefox it reported once (due to an error in code) that the WebGL size limit was over 16k x 16k IIRC, so the limit is somewhat high.

My use case is for a paint and photo processing program accessible from the Web, I can't use Node. I agree that some models have limited usefulness, but deblur is really impressive even with blocks!

Rewriting the library is probably not an option given the pace in this field but there are no so many alternatives integrating so easily for browser consumption in already existing code (my case). Yours is really unique in fact (and so well documented), and still well downloaded according to the npm depot.

It is baffling why these blocks appear if the library was working correctly before because in the package.json, the version for TFJS is limited to updates to minor and backward-compatible version only: "@tensorflow/tfjs": "~4.11.0".

Anyway I will try to find a working TFJS version (if that is indeed the issue) and make additional testing without using patch size to see if it is a viable path.

Again, thanks for this wonderful library!