All models took same time to upscaling an image

[fehmi]

Here is my code. All models I tried took 17 seconds to finish same image. Why?

import Upscaler from 'upscaler';
import * as tf from '@tensorflow/tfjs';
import x2 from '@upscalerjs/esrgan-thick/2x';
import x3 from '@upscalerjs/esrgan-thick/3x';
import x4 from '@upscalerjs/esrgan-thick/4x';
import slim2x from '@upscalerjs/default-model';
import denoise from '@upscalerjs/maxim-denoising';
import retouch from '@upscalerjs/maxim-retouching';

const upscalers = {
    x2: new Upscaler({ model: x2 }),
    x3: new Upscaler({ model: x3 }),
    x4: new Upscaler({ model: x4 }),
    default: new Upscaler({ model: slim2x })
};

onmessage = async (e) => {
    postMessage({ type: 'log', message: 'Worker started!' });

    const [data, shape, modelName, patchSize = 600, padding = 8] = e.data;
    const tensor = tf.tensor(data, shape);
    const upscaler = upscalers[modelName] || upscalers['x4'];
    const upscaled = await upscaler.upscale(tensor, {
        output: 'tensor',
        patchSize,
        padding,
        awaitNextFrame: true,
        progress: async (percent) => {
            postMessage({
                type: 'progress',
                percent
            });
        }
    });

    const upscaledData = await upscaled.data();
    postMessage([upscaledData, upscaled.shape]);

    tensor.dispose();
    upscaled.dispose();
};

And this is how I call it.

     upscale() {
        var factor = dom.querySelector("controls[upscale] select").value;
        console.log(factor);
        data.upscaling = true;
        methods.upscaleImageWorker(data.image.url, factor);
      },

      // imajı worker'a gönderiyor upscale etmesi için
      upscaleImageWorker: async function (imageSrc, modelName = 'x4') {
        //await disable(); // UI bloklama fonksiyonun varsa
        const image = new Image();
        image.src = imageSrc;
        image.crossOrigin = 'anonymous';
        image.onload = async () => {
          console.log("image loaded");
          const pixels = tf.browser.fromPixels(image); 
          await tf.nextFrame(); // UI thread’in donmaması için
          const data = await pixels.data();
          try {
            window.upscaleWorker.postMessage([data, pixels.shape, modelName, 64, 2]);
          }
          catch (e) {
            console.log(e);
          }
        };
      },

[thekevinscott]

It's been a little while since I trained these models, but I believe this is to be expected.

There's three different ESRGAN models - slim, medium, thick - each with different hyper parameters that affect the size (and performance) of the resultant model. However, within each model the scale factor doesn't have a big effect on the model. Scale (I believe) only affects the final output layer, so it's to be expected that all scales for a given model should be roughly the same inference time.

If you compare two different models (i.e., slim vs thick) you should see faster inference for slim.

With that all said, please feel free to open up a codepen or similar and I'd be happy to take a deeper look.