Testing Image-To-Text

[ndrean]

I gave Bumblebee a try today. The idea was to provide predictions on image captioning to classify an image so that a user can use/put pre-filled tags to easily filter his images.

It turns out that the predictions are.....not too bad and quite fast., at least locally.

This is supposed to be a car:

https://dwyl-imgup.s3.eu-west-3.amazonaws.com/40F36F45.webp

Nx.Serving.run(serving, t_img) 
#=>predictions: [
    %{
      label: "beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon",
      score: 0.9203662276268005
    }
  ]

Testing with a new query string: pred=onto run the model prediction:

curl -X GET http://localhost:4000/api?url=https://dwyl-imgup.s3.eu-west-3.amazonaws.com/40F36F45.webp&w=300&pred=on

{"h":205,"w":300,"url":"https://dwyl-imgup.s3.eu-west-3.amazonaws.com/76F195C6.webp","new_size":11642,"predictions":[{"label":"beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon","score":0.9203662276268005}]],"init_size":79294,"w_origin":960,"h_origin":656,"url_origin":"https://dwyl-imgup.s3.eu-west-3.amazonaws.com/40F36F45.webp"}

I tested 3 models: "facebook/deit-base-distilled-patch16-224" and "microsoft/resnet-50" and ""google/vit-base-patch16-224".

I don't know if anyone tested it?

I submit my code in case any reader sees some obvious fault. It runs locally. It is based on this example. I did not try to deploy this, but here is a guide before I forget: you need to set up a temp dir.

#mix.exs
{:bumblebee, "~> 0.4.2"},
{:nx, "~> 0.6.1"},
{:exla, "~> 0.6.1"},
{:axon, "~> 0.6.0"},

I decided to run a GenServer to start the serving with the app to load the model, but you can start an Nx.Serving in the Aplpication level as well, something like {Nx.serving, serving: serve(), name: UpImg.Serving} where the function Application.serve defines what is in the GenServer below.

defmodule UpImg.GsPredict do
   use GenServer

  def start_link(opts) do
    {:ok, model} = Keyword.fetch(opts, :model)
    GenServer.start_link(__MODULE__, model, name: __MODULE__)
  end

  def serve, do: GenServer.call(__MODULE__, :serving)

  @impl true
  def init(model) do
    {:ok, model, {:continue, :load_model}}
  end

  @impl true
  def handle_continue(:load_model, model) do
    {:ok, resnet} = Bumblebee.load_model({:hf, model})
    {:ok, featurizer} = Bumblebee.load_featurizer({:hf, model})

    {:noreply,
     Bumblebee.Vision.image_classification(resnet, featurizer,
       defn_options: [compiler: EXLA],
       top_k: 1,
       compile: [batch_size: 10]
     )}
  end

  @impl true
  def handle_call(:serving, _from, serving) do
    {:reply, serving, serving}
  end
end

and it is started with the app:

children = [
  ...,
  {UpImg.GsPredict, [model: System.fetch_env!("MODEL")]}
]

The model - the repo id - is passed as an env var so I can very simply change it..

In the API, I use predict/1 when I upload an image from the browser and run this task in parallel to the S3 upload. It takes a Vix.Vips.Image, a transformation of a binary file:

[EDITED]

def predict(%Vix.Vips.Image{} = image) do
    serving = UpImg.GsPredict.serve()

    {:ok, %Vix.Tensor{data: data, shape: shape, names: names, type: type}} =
      Vix.Vips.Image.write_to_tensor(image)

    #{width, height, channels} = shape <- wrong, shape should be HWC. Bug corrected.
    t_img = Nx.from_binary(data, type) |> Nx.reshape(shape, names: names)

    Nx.Serving.run(serving, t_img)
    Task.async(fn -> Nx.Serving.run(serving, t_img) end)
  end

and use it in the flow:

prediction_task = predict(my_image)
...
%{predictions: predictions} = Task.await(prediction_task)

[ndrean]

Prediction slows down the process, roughly 1.5s per request. I will try to deploy this horror.

The "GET" endpoint - where you pass in an URL of a pic - works with the query string addition "pred" (no prediction, thus faster is you don't pass one).

curl -X GET http://localhost:4000/api?url=....&w=300&pred=on

The "POST" endpoint - where you submit files from a client via a FormData to the API - also works, but you use a checkbox if you want the prediction (I capture it the same way, via a key "pred", thus there is a constraint on the FormData naming).

[ndrean]

For complteness,

https://github.com/elixir-nx/bumblebee/tree/main/examples/phoenix#user-images

• it seems that when the image is too small that the findings are not so good. After reading a bit, it seems that sizes around 512x512 are recommended for Image-to-Text. The speed of the recognition is also related to the size of the image, the bigger the longer. To speed up the process, if an image is bigger, I resize to this size and run the ML model on it.

• I added the redirection to accept images from "unsplash" for example. If you submit a src="https://source.unsplans.com/<photo_id>"; it will be redirected. To do this, I used and modified a Finch.stream. If it detects a redirection by reading the headers, it takes the received path and makes a recursion. Otherwise, it writes the stream into a file, so the process eventually ends. This way, the body is processed only once, redirection or not, has a low memory footprint and it does not slow down the process.

• I changed to Nx.Serving.batched_run/3 as it seems to give faster results when treating several pictures (uploaded as a POST request).

To use batchde_run, the set up is different. The process Nx.Serving is launched in the Application module.

#Application.ex
children = [
   ...,
    {Nx.Serving, serving: serve(), name: UpImg.Serving, batch_size: 10, batch_timeout: 100}
]

defp serve do
  model = System.fetch_env!("MODEL")
  {:ok, resnet} = Bumblebee.load_model({:hf, model})
  {:ok, featurizer} = Bumblebee.load_featurizer({:hf, model})

  Bumblebee.Vision.image_classification(resnet, featurizer,
      defn_options: [compiler: EXLA],
      top_k: 1,
      compile: [batch_size: 10]
end

and then use instead:

def predict(%Vix.Vips.Image{} = image) do
    # serving = UpImg.GsPredict.serve()

    {:ok, %Vix.Tensor{data: data, shape: shape, names: names, type: type}} =
      Vix.Vips.Image.write_to_tensor(image)

    {width, height, channels} = shape
   # bug in Vix.Vips, with HWC and WHC....
    t_img = Nx.from_binary(data, type) |> Nx.reshape({height, width, channels}, names: names)

    Task.async(fn -> Nx.Serving.batched_run(UpImg.Serving, t_img) end)
    # Task.async(fn -> Nx.Serving.run(serving, t_img) end)
  end

! One must be careful with the async calls. When you run this async task, say %Task{} = task = predict(image), you can only get the result back -Task.await(task) - from the owner process.

[ndrean]

To read an URL and download it with Finch using streams and potentially accept redirects (and write it into a temp file), you can use the 302 and Location header:

{:ok, path} = Plug.upload.random_file("temp-stream")
{:ok, file} = File.open(path, [:binary, :write])

# url = "https://source.unsplash.com/QT-l619id6w"

request = Finch.build(:get, url)
stream_write(request, path)
File.close!(file)

def stream_write(request, file) do
   Finch.stream(UpImg.Finch, nil, fn
      {:status, status}, _acc ->
        status

      {:headers, headers}, status ->
         handle_headers(headers, status)

      {:data, data}, headers ->
          handle_data(file, data, headers)
    end)
end

def handle_headers(headers, 302), do:
  Enum.find(headers, &(elem(&1, 0) == "location"))

def handle_headers( headers, 200), do: headers

def handle_headers(_,_), do:  {:halt, "bad redirection"}

def handle_data(file, _,  {"location", location}), do:
  Finch.build(:get, location) |> stream_write(file)

def handle_data(_, _,  {:halt, "bad redirection"}), do:
   {:halt, "bad redirection"}

def handle_data(file, data, _) do
  case IO.binwrite(file, data) do
        :ok -> :ok
        {:error, reason} -> {:halt, reason}
   end
end

The memory footprint is low, at the expense of writing the body of the request into a file (but one can just append the chunk in memory if needed).

[LuchoTurtle]

Thanks for the excellent write-up, @ndrean , it was super insightful!
I haven't tried Bumblebee but I'm wanting to (when I have more free time). Are you using any specific Hugging Face models in your experiments?

On the topic, you may also find https://github.com/replicate/replicate-elixir as another alternative. Unfortunately, it's tied to their platform, but it might still be fun to tinker with. This is from this AWESOME talk from Charlie Holtz in https://www.youtube.com/watch?v=TfZI5-oQSqI&ab_channel=ElixirConf. It's an awesome video that really highlights how Elixir has great built-in tools to get AI models with LiveView working seamlessly.

[ndrean]

Yes. I used microsoft/resnet model.

Thanks for the "replicate" link. I will give it a try too!

[ndrean]

@LuchoTurtle Thanks! I really enjoyed watching this video, had a lot of fun! 😀
I just realised that you already wrote plenty of good things on this subject before I woke up! So nothing new under the sun for you... 😀

dwyl/learn-elixir#212
dwyl/image-classifier#1

I am just looking at Image Classification - namely a weighted list of predictions - whilst you wanted Image-to-text, more ambitious. I just wondered what you would do with the generated text for an image because you need to further process this response to extract some keys points, if this is what you want.

For example, the Salesforce/BLIP is an I2T. I run it in a Livebook, the easiest way to do this. It downloads 1.7Gb... The generated code is:

{:ok, model_info} = Bumblebee.load_model({:hf, "Salesforce/blip-image-captioning-base"})

{:ok, featurizer} =
  Bumblebee.load_featurizer({:hf, "Salesforce/blip-image-captioning-base"})

{:ok, tokenizer} =
  Bumblebee.load_tokenizer({:hf, "Salesforce/blip-image-captioning-base"})

{:ok, generation_config} =
  Bumblebee.load_generation_config({:hf, "Salesforce/blip-image-captioning-base"})

generation_config = Bumblebee.configure(generation_config, max_new_tokens: 100)

serving =
  Bumblebee.Vision.image_to_text(model_info, featurizer, tokenizer, generation_config,
    compile: [batch_size: 1],
    defn_options: [compiler: EXLA]
  )

image_input = Kino.Input.image("Image", size: {384, 384})
form = Kino.Control.form([image: image_input], submit: "Run")
frame = Kino.Frame.new()

Kino.listen(form, fn %{data: %{image: image}} ->
  if image do
    Kino.Frame.render(frame, Kino.Text.new("Running..."))

    image =
      image.file_ref
      |> Kino.Input.file_path()
      |> File.read!()
      |> Nx.from_binary(:u8)
      |> Nx.reshape({image.height, image.width, 3})
    %{results: [%{text: text}]} = Nx.Serving.run(serving, image)
    Kino.Frame.render(frame, Kino.Text.new(text))
  end
end)

Kino.Layout.grid([form, frame], boxed: true, gap: 16)

I generated an image with Stable Diffusion, and submit it to BLIP. The result is pretty good! 😁, but it is not classification!

This works for me because the model is "deciphered" in some way in Bumblebee. What if I want to use a specific model? That I don't know how to proceed.

I used a small Image Classification model -300Mb downloaded - embeded into the app as this tends to be much smaller. However, the Image Classification is not as good - same image - even for a 1300x1000px image:

Replicate exposes an endpoint. You also need to be careful with the data you submit to get the right balance between speed and accuracy: you might pay too much or pay for nothing if you don't deliver a properly sized image :)

When you read this "official" example, they naturally stress that the navigator should resize pics instead of a/the server. However, In this git repo, the proposed JS code is a bit ... wordy.

So down to earth I tried to follow the repo recommendations - at least for a WebApp version - and I looked at how to do this. In fact, you can get a bunch of resized images from the browser with a Promise.all because the browser is efficient at doing this: a form accepts an image and you "just" inject it into different resizing canvas that you set, and call canvas.toBlob. You can target a thumbnail, or a "ML" sized image (512px) or 3 different sizes to match mobile, pad and full screen 1440px pretty quickly for example.

One point is the naming: you need a unique base identifier for all these files. It turns out that JS can produce a SHA1 easily, no library, so I used this as a unique naming base, modulo some size identifier. You can also convert into WEBP just like this, and this saves a lot.
You can upload directly to a bucket, and pass down to the server the 512px file to do the ML stuff. The bucket does his stuff and returns a response back to the client - a bunch of URLs - and the client forwards the responses to the server where you update the socket. Meanwhile, the server did the ML stuff to produce a caption/prediction. It remains to save all this into the DB. With the SHA1 naming, we have a common identifier, almost collision free, so we can update the DB record easily. All easy async client-side and server-side. The main difficulty is the "hook".

The shinstagram source.
He uses R2 but I did not get the details how he uses the CDN to serve the files.

[LuchoTurtle]

Thanks for the detailed write-up @ndrean , it is really super insightful! Once I'm cleared with other tasks, I want to give dwyl/image-classifier#1 a whirl and, since you've put much more time and effort than I into this, I might ask ya for some pointers!

I am just looking at Image Classification - namely a weighted list of predictions - whilst you wanted Image-to-text, more ambitious. I just wondered what you would do with the generated text for an image because you need to further process this response to extract some keys points, if this is what you want.

Not necessarily. I actually want a list of keywords that describe an image, just like you want. However, I believe that one may yield fair results by using a combination of an image captioning model like BLIP and a regular LLM to better extract keywords. In the same way https://zhaohengyuan1.github.io/image2paragraph.github.io/ uses three models to densely and accurately describe an image, one may do something like:

Use BLIP to describe the image -> feed into an LLM to gather relevant keywords with context from the image.

Then again, this is pure speculation on my part.

I generated an image with Stable Diffusion, and submit it to BLIP. The result is pretty good! 😁, but it is not classification!

Your results are awesome! Though why do you say it's not considered "classification"? Is it because it's not yielding a set of weighted predictions in lieu of a simple phrase?

This works for me because the model is "deciphered" in some way in Bumblebee. What if I want to use a specific model? That I don't know how to proceed.

Apparently, you can't use any HuggingFace transformer with Bumblebee, which is a shame. I don't know the specifics but, according to https://github.com/elixir-nx/bumblebee#model-support, it "has to be implemented in Bumblebee" (whatever that means).
However you can use https://jonatanklosko-bumblebee-tools.hf.space/apps/repository-inspector/36pihlb7tb7rvmovbnvrmjseud5mzdlxhbfaa6xywewlprok to check if a Transformer model from HuggingFace is supported or not.

So it's fair that you don't know how to use other models from HuggingFace because apparently it's not possible :p.

Image sizes

It's interesting, the deal with image sizes, as you pointed, spans to even image generation with Stable Difusion. Even when I'm doing img-to-img or inpainting, I yield much better results with 512px images.

Using multiple canvas with different sizes, injecting images there is a fun way of getting different-sized images, quite creative!

Thanks for the shinstagram source, I'll have to take a look at it! :D

[ndrean]

Yes, I see, a mix. Sometimes I have good predictions, but more often BLIP is superior. For the moment, I don't know.

Thanks for the CanIUseThisModel, I did not know or found. Things are more clear.

Of course, I did not invent the 512px trick, I read it!

See also: https://github.com/elixir-nx/bumblebee/tree/main/examples/phoenix#user-images

[UPDATED] You can consume the data by sending it directly to a bucket when you run an external: presign_upload: it performs a XHR/fetch request to the bucket endpoint with a presigned url and consumes the data. This means you can't run a prediction any more. You may need to upload the data to the server.

1. The HOOK: use this.upload to send the renamed & resized & WEBP converted files from the client.

• I generate a unique name per picture, simply a SHA1, supposed to be unique: calcSHA1. MDN source of non-crypto usage
• I decided (why not!) to produce 3 versions: a "thumbnail" with max size 200px (I might read this from a dataset, itself populated via an ENV VAR Phoenix side), a "machine-learning" version of target size 512px (width or height), and a pseudo full-screen with max 1440px if needed. You get files named .original_extension
• process the entries through a canvas to produce 3 new versions in format WEBP with canvas.drawImage and canvas.toBlob. Each file will produce 3 files renamed "-m-[200/512/1440].webp"
• this.upload is the secret! An undocumented function found here

const SIZES= [200, 512, 1440];

export default {
   /**
   * Renames a File object with its SHA1 hash and keep the extension
   * source: https://developer.mozilla.org/en-US/docs/Web/API/SubtleCrypto/digest#converting_a_digest_to_a_hex_string
   * @param {File} file - the file input
   * @returns {Promise<File>} a promise that resolves with a renamed File object
   */
  async setHashName(file) {
    const ext = file.type.split("/").at(-1);
    const SHA1name = await this.calcSHA1(file);
    return new File([file], `${SHA1name}.${ext}`, {
      type: file.type,
    });
  },
  /**
   * Calculates a SHA1 hash using the native Web Crypto API.
   * @param {File} file - the file to calculate the hash on.
   * @returns {Promise<String>} a promise that resolves to hash as String
   */
  async calcSHA1(file) {
    const arrayBuffer = await file.arrayBuffer();
    const hash = await window.crypto.subtle.digest("SHA-1", arrayBuffer);
    const hashArray = Array.from(new Uint8Array(hash));
    const hashAsString = hashArray
      .map((b) => b.toString(16).padStart(2, "0"))
      .join("");
    return hashAsString;
  },
  /**
   *
   * @param {File} file  - the file
   * @param {number[]} SIZES - un array of sizes to resize to image to
   * @returns {Promise<File[]>} a promise that resolves to an array of resized images
   */
  async processFile(file, SIZES) {
    return Promise.all(SIZES.map((size) => this.fReader(file, size)));
  },
  /**
   * Reads an image file, resizes it to a given max size, and converts into WEBP format et returns it
   * @param {File} file  - the file image
   * @param {number} MAX  - the max size of the image in px
   * @returns {Promise<File>} resolves with the converted file
   */
  fReader(file, MAX) {
    const self = this;

    return new Promise((resolve, reject) => {
      if (file) {
        const img = new Image();
        const newUrl = URL.createObjectURL(file);
        img.src = newUrl;

        img.onload = function () {
          URL.revokeObjectURL(newUrl);
          const { w, h } = self.resizeMax(img.width, img.height, MAX);
          const canvas = document.createElement("canvas");
          if (canvas.getContext) {
            const ctx = canvas.getContext("2d");
            canvas.width = w;
            canvas.height = h;
            ctx.drawImage(img, 0, 0, w, h);
            // convert the image from the canvas into a Blob and convert into WEBP format
            canvas.toBlob(
              (blob) => {
                const name = file.name.split(".")[0];
                const convertedFile = new File([blob], `${name}-m${MAX}.webp`, {
                  type: "image/webp",
                });
                resolve(convertedFile);
              },
              "image/webp",
              0.75
            );
          }
        };
        img.onerror = function () {
          reject("Error loading image");
        };
      } else {
        reject("No file selected");
      }
    });
  },
  resizeMax(w, h, MAX) {
    if (w > h) {
      if (w > MAX) {
        h = h * (MAX / w);
        w = MAX;
      }
    } else {
      if (h > MAX) {
        w = w * (MAX / h);
        h = MAX;
      }
    }
    return { w, h };
  },
  /**
   * Takes a FileList and an array of sizes, 
   * then renames them with the SHA1 hash, 
   * then resizes the images according to a list of given sizes, 
   * and converts them to WEBP format, 
   * and finally uploads them.
   * @param {FileList} files
   * @param {number[]} SIZES
   */
  async handleFiles(files, SIZES) {
    const renamedFiles = await Promise.all(
      [...files].map((file) => this.setHashName(file))
    );

    const fList = await Promise.all(
      renamedFiles.map((file) => this.processFile(file, SIZES))
    );

    // the "secret" to upload to the server. Undocumented Phoenix.JS function
    this.upload("images", fList.flat());
  },
  /*
  inspired by: https://github.com/elixir-nx/bumblebee/blob/main/examples/phoenix/image_classification.exs
  */
  mounted() {
    this.el.style.opacity = 0;

    this.el.addEventListener("change", async (evt) =>
      this.handleFiles(evt.target.files, SIZES)
    );

    // Drag and drop
    this.el.addEventListener("dragover", (evt) => {
      evt.stopPropagation();
      evt.preventDefault();
      evt.dataTransfer.dropEffect = "copy";
    });

    this.el.addEventListener("drop", async (evt) => {
      evt.stopPropagation();
      evt.preventDefault();
      return this.handleFiles(evt.dataTransfer.files, SIZES);
    });
  },
};

[LuchoTurtle]

@ndrean

    t_img = Nx.from_binary(data, type) |> Nx.reshape({height, width, channels}, names: names)

I'm having trouble with this part. I keep stumbling upon this error when trying to reshape the tensor so I can feed it into the resnet-50 model.

** (ArgumentError) cannot reshape, current shape {11708} is not compatible with new shape {224, 224, 3}

I know for sure that the image is resized according to the model's specification (224x224) up until this point.
I don't know what I'm doing wrong, I'm trying to follow Bumblebee's guide to image classification.

Have you gotten this error before? 👀

[ndrean]

@LuchoTurtle ah yes I remember now, I had this too, it was bug, and my code above was good with the bug until the maintainer corrected it, so its wrong now. but I did not correct it above...

The correct shape is a HWC tuple: width and height were inverted , you see what I did?

Make sure to have the latest version. I think this should work.

 {:ok, %Vix.Tensor{data: data, shape: shape, names: names, type: type}} =
          Image.write_to_tensor(image)

        t_img = Nx.from_binary(data, type) |> Nx.reshape(shape, names: names)

        Nx.Serving.batched_run(UpImg.Serving, t_img)

FYI you cannot deploy this on a small machine, you probably need 1GB RAM. I will probably come back to this as I want to finish this little project.

[nelsonic]

A 2GB RAM VPS instance on OVH IS €3.50/month —> dwyl/learn-devops#64 💭

[ndrean]

Probably need a trial to see how to install on bare-metal (modulo Docker but...). I see they provide an IPv4, so you can put plenty of demo apps with subdomains I imagine. If I want to buy a domain say on Cloudfare, I will need to link OVH-Cloudfare. Should not be too complicated stuff.

[ndrean]

About "prompt engineering":
https://prmpts.ai/blog/what-is-prompt-engineering

[LuchoTurtle]

I dispute that "prompt engineering" is engineering at all 😅.

But I do understand that there's an art to it. Refining models' output to get what you want is not easy, per se, but rather a matter of trial and error and specificity. It's definitely a skill but I honestly can't see the "engineering" part of it - it can be boiled down to clarity in communication and having to work with some quirks that GPT or any other LLMs may have. But hey, maybe I'm an idiot and I'm spewing nonsense, I don't know 😅.

Although, I have to admit, I've dabbled with Stable Diffusion much more than LLMs (though I'm keen on biting the bullet and paying 20 quid a month for access to OpenAI's API after their dev day at https://openai.com/blog/new-models-and-developer-products-announced-at-devday).

From what I've tried, I think "prompt engineering" is much harder with diffusion models than LLMs. But even then, you can circumvent issues with inpainting and ControlNet to get more accurate results rather easily (though it's still very much trial and error, you can't ever get exactly what you want, just what's the closest to what you want).

For example, what I found to generate cool Ghibli-style images with Stable Diffusion, I found that I had to work much more than simply using ChatGPT or any other LLM.

• having to try different models and checkpoints from Civit.AI to see the best results (for example, EasyFluff).
• knowing which VAE to use to get your preferred results (for example https://civitai.com/models/23906/kl-f8-anime2-vae).
• reducing noise and improving consistency with extensions like https://github.com/Seshelle/CFG_Rescale_webui.
• trying different embeddings for negative prompts or positive prompts.
• knowing which LORA to use (like https://civitai.com/models/82098/add-more-details-detail-enhancer-tweaker-lora)

This by itself is much more work to just yield fair results with generative art, something that is much more streamlined with LLMs (or downright not present) and prompting.
I found prompting in diffusion models absolutely chaotic. But I've seen patterns, and I've had luck trying to follow imageboard tags and I assume many models are trained with these imageboards in mind, because they perform much better when I use these tags.

For example, I tend to follow a pattern for positive prompts

establish style +  number of characters +  the camera and/or landscape and/or scene properties and using `"BREAK"` between different subjects that I want in the pictures (to prevent getting mixed up).

Adding weight to each tag and you can go from there.

Is this engineering?

So is this workflow engineering? I don't believe so. It's not deterministic by nature. It's just proper concise communication. It's a skill, but I don't think there's anything esoteric about it.

I liked this answer from https://news.ycombinator.com/item?id=36971327.

> Is Prompt Engineering a Thing?

Yes, it's a dumb name for the skill of modifying your prompts and questions to the LLM in a way that produces better results than if you just asked for what you wanted plainly. As language models get better, this might become obsolete.

> I'm trying to research the subject but I don't see much evidence that companies are racing to hire prompt engineers.

Because it's not really a job. Think of it like using the Google search engine - being able to search well is something you can get better at but being a "Google search-er" isn't a career or a job you'll see openings for.

All in all, aside from my obvious ramble and digression, it's still an interesting read @ndrean . Because although I don't think it's engineering, it's a highly valuable skill that I want to get better at!

[ndrean]

Nice @LuchoTurtle , you look pretty advanced!

Do you use only a Livebook to test all this?

There is indeed some vocabulary to ingest to enter this world. Being able to name things that are really useful is a powerful skill 😀 but feels sometimes like much ado about "almost" nothing. Embedding, transformers, tokenizer, prompt-context etc on the other side are "real" concepts to be understood whilst so-called "engineering" is more like noise.

I am starting to watch/read this: https://www.coursera.org/learn/generative-ai-with-llms/lecture/ZVUcF/prompting-and-prompt-engineering

Playing with images gives an immediate wow effect. I highly recommend https://github.com/cbh123/emoji by the same guy who did Shinstagram. By the way, here is how he prompts engineers it.

I still have basic questions: how do you use these tools in practice to run this on production? Api based approach or embedded in your app somehow?

I do more modest down-to-earth things, more on the LLM side. My first step was image captioning. For example, to run this in practice, I embedded the model, ie download the data on a server as Bumblebee does this in fact. Then mount bind into the running container of your app. This is not totally straightforward: I can run the "base" model (1G) but not the large model (2G). I did not dig into this problem.

Another barrier is that few models can be used by the Elixir eco-system. I finally found something:

https://twitter.com/sean_moriarity/status/1715758666001928613

An explanation on how we add models to Bumblebee (@toranb asked on EEF slack and I thought it would be a good write up here).

The first thing to note is that almost all of the models have significant overlap in implementation details. A transformer is a transformer. There are…

— Sean Moriarity (@sean_moriarity) October 21, 2023

Lastly, another barrier IMO is LiveView. Compared to Streamlit, it is far behind. Liveview is still complicated and fragile: navigation, "liveview session" is obscur. I had some errors I still don't understand. For example, I used a separate "html.heex" file that for some reason gave me double renderings. When I put the same markup into the render function of the liveview, it worked. I also have some cache problems: you change the code but it doesn't render. Few headaches...