NullPointerException with Ollama + streaming + image

[habuma]

Given the following code...

            Flux<String> content = chatClientBuilder.build().prompt()
                    .user(userSpec -> {
                        userSpec.text("Write a haiku about this image.")
                                .media(MediaType.IMAGE_JPEG, imageResource);
                    })
                    .stream()
                    .content();
            content.subscribe(s -> {
            System.err.println("Response: " + s);
            });

Where imageResource is defined as...

    @Value("classpath:/oldfaithful.jpeg")
    Resource imageResource;

And using Ollama (Moondream, Llava, and Bakllava models tested), things work fine. But if you give it something like this:

    @Value("classpath:/oldfaithful.webp")
    Resource imageResource;

Where the difference is that it's sending a WEBP instead of a JPEG, you get a NullPointerException.

It's not surprising that you get an error. WEBP isn't a supported media type, after all. But getting a NPE is not as graceful of a failure as one would hope for. Also, it's not just WEBP...any non-image media gives the same error. Again, not surprising that there's an error. Just not as useful of an error as you'd like to see. I believe that this could be improved upon somehow.

Incidentally, if you change the app to use OpenAI instead of Ollama, it actually works fine. I was a bit surprised that it was able to "see" webp.

[habuma]

The NPE is thrown from here: https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-ollama/src/main/java/org/springframework/ai/ollama/OllamaChatModel.java#L213

A couple of things to note:

• Debugging shows that every property in the chunk is null, although it's the message that leads to the NPE.
• In the line right above that there's a check to see if the message is null before pulling content. So the code already acknowledges that the message could be null and has a safeguard around it when getting the content. But perhaps it should have a similar safeguard around pulling tools.

[markpollack]

Thanks for the great explanation. The NPE is fixed - thanks @muthuishere . I've added a test that confirms an exception is thrown if passing webp, though the exception thrown at the moment isn't very clear that it is because ollama doesn't support webp. I think this is something ollama should fix vs. us hardcoding what formats the model does and doesn't support as that is very tricky to do with ollama as it is a gateway for so many models. Even if it was one model, we could never keep up with it's evolution.

[habuma]

Thanks. And yeah, Spring AI deciding what's supported and not wasn't the goal. Just handling that a bit more gracefully than throwing NPE. I'll give it a try a little later.