NullPointerException with Ollama + streaming + image

[habuma]

Given the following code...

            Flux<String> content = chatClientBuilder.build().prompt()
                    .user(userSpec -> {
                        userSpec.text("Write a haiku about this image.")
                                .media(MediaType.IMAGE_JPEG, imageResource);
                    })
                    .stream()
                    .content();
            content.subscribe(s -> {
            System.err.println("Response: " + s);
            });

Where imageResource is defined as...

    @Value("classpath:/oldfaithful.jpeg")
    Resource imageResource;

And using Ollama (Moondream, Llava, and Bakllava models tested), things work fine. But if you give it something like this:

    @Value("classpath:/oldfaithful.webp")
    Resource imageResource;

Where the difference is that it's sending a WEBP instead of a JPEG, you get a NullPointerException.

It's not surprising that you get an error. WEBP isn't a supported media type, after all. But getting a NPE is not as graceful of a failure as one would hope for. Also, it's not just WEBP...any non-image media gives the same error. Again, not surprising that there's an error. Just not as useful of an error as you'd like to see. I believe that this could be improved upon somehow.

Incidentally, if you change the app to use OpenAI instead of Ollama, it actually works fine. I was a bit surprised that it was able to "see" webp.

[habuma]

The NPE is thrown from here: https://github.com/spring-projects/spring-ai/blob/main/models/spring-ai-ollama/src/main/java/org/springframework/ai/ollama/OllamaChatModel.java#L213

A couple of things to note:

• Debugging shows that every property in the chunk is null, although it's the message that leads to the NPE.
• In the line right above that there's a check to see if the message is null before pulling content. So the code already acknowledges that the message could be null and has a safeguard around it when getting the content. But perhaps it should have a similar safeguard around pulling tools.