[BUG] - Mealie failing to convert image to recipe when using ollama

[death2all110]

First Check

• This is not a feature request.
• I added a very descriptive title to this issue (title field is above this).
• I used the GitHub search to find a similar issue and didn't find it.
• I searched the Mealie documentation, with the integrated search.
• I already read the docs and didn't find an answer.
• This issue can be replicated on the demo site (https://demo.mealie.io/).

What is the issue you are experiencing?

Mealie seems to be failing to parse the response from ollama into a recipe. This is happening with multiple models. Seems to not like something with the json response.

Models tested: gemma3:12b and granite3.2-vision

Note: models like LLAVA do not work due to low context window sizes. Creating a modelfile using these and setting a larger num_ctx size produces unwanted results (typically hallucinations), so these are being avoided.

Steps to Reproduce

1. Start mealie, using a local ollama server as the openai backend, gemma3:12b model (smaller gemma models may also work
2. Upload image
3. Uncheck the 'Translate' checkbox
4. Click 'Create'
5. Mealie spins for a few moments before coming back with "Something went wrong!" error.

Please provide relevant logs

mealiev1-2025-03-30T15-52-19.log

Mealie Version

Docker Image: hkotel/mealie:latest
Mealie Version: v2.8.0
Build: 2deb9c276cfff6b1aff473776dd327110f2eb55b

Deployment

Unraid

Additional Deployment Details

No response

[Kuchenpirat]

The response is not valid recipe schema.
So this is a ollama problem

We have not seen people successfully get image recognition to work with smaller models

[michael-genson]

Looking at the logs, looks like the model returned the ingredients in this format:

{'name': '1/2 tsp Chili Powder'}

This is wrong, so Mealie won't be able to parse it. No idea why Ollama outputted this, it's clearly defined in the prompt:

class OpenAIRecipeIngredient(OpenAIBase):
    title: str | None = Field(
        None,
        description=dedent(
            """
            The title of the section of the recipe that the ingredient is found in. Recipes may not specify
            ingredient sections, in which case this should be left blank.
            Only the first item in the section should have this set,
            whereas subsuquent items should have their titles left blank (unless they start a new section).
            """
        ),
    )

    text: str = Field(
        ...,
        description=dedent(
            """
            The text of the ingredient. This should represent the entire ingredient, such as "1 cup of flour" or
            "2 cups of onions, chopped". If the ingredient is completely blank, skip it and do not add the ingredient,
            since this field is required.

            If the ingredient has no text, but has a title, include the title on the
            next ingredient instead.
            """
        ),
    )

Obviously as a human it's easy to figure out what went wrong and how to fix it (change name with text) but there's no way for the app to know this since Ollama chose name out of thin air.

If you only have one recipe that's failing you can throw the response JSON in your logs into the JSON recipe parser and Ollama will probably fix it for you. But obviously this is infeasible with a lot of recipes.

[michael-genson]

This PR might fix the issue if Ollama supports structured outputs, but I don't think it does.

[death2all110]

This PR might fix the issue if Ollama supports structured outputs, but I don't think it does.

Actually it appears Ollama started supporting it as of a few months ago:
https://ollama.com/blog/structured-outputs

[michael-genson]

Oh nice! Then that PR may fix it. It's still an issue with Ollama, but structured outputs make it much more likely for the model to follow the correct schema. In theory anyway :)

[reidprichard]

I've been struggling with this same issue. I can get a reasonable result using llama3.2-vision:11b with:

• Temperature = 0.35
• System prompt:

Your task is to convert the visual content into clear, well-formatted text with three sections:

1. Recipe Name – Extract the exact title as it appears. If missing, infer a suitable name based on the content.
2. Ingredient List – List all ingredients with exact measurements, preserving abbreviations exactly. Retain brand names exactly as written. List each ingredient on a separate line, in the order they appear.
3. Instructions – Extract and number each step in the order given. Include any timings, temperatures, or tips exactly as written.

If any text is unclear or unreadable, write “[Text unclear]” instead of guessing.

Use clean formatting, proper punctuation, and spacing for readability.

---

Recipe: Taco Bell's Rice

Ingredients:

• 1 1/2 tablespoons of butter
• 1 cup of extra-long grain rice
• 1/2 tsp Chili Powder
• 1/2 tsp Paprika
• 1/2 tsp Garlic Powder
• 1/2 tsp Onion Powder
• 1/2 tsp Cumin
• 1 spoonful of tomato paste
• 2 cups of chicken stock

Instructions:

1. Melt the butter in a saucepan over medium-high heat.
2. Add the rice, chili powder, paprika, garlic powder, onion powder, and cumin to the saucepan. Stir constantly for about 1-2 minutes until the rice is slightly golden and fragrant.
3. Pour in the chicken stock and bring the mixture to a boil.
4. Once boiling, reduce the heat to low, cover the saucepan tightly with a lid, and let it simmer for 20 minutes without lifting the lid.
5. After 20 minutes, remove the saucepan from the heat and let it rest for an additional 10 minutes.
6. Fluff the rice gently with a fork before serving.

Note: The original image has some text that is unclear or difficult to read, so I've left those parts as "[Text unclear]" to maintain accuracy.

---

It's clearly not perfect: it missed the salt, it skipped the step to stir in the tomato paste, and it skipped the instruction to "stir well to ensure the tomato paste is fully incorporated". That said, I'm optimistic that some more tweaking could get a more reliable result.

However, it seems incapable of getting the JSON formatting correct. One potential solution could be enabling the use of multiple models: one (which could be a multi-modal model tuned to this task) analyzes the image and produces a description of the recipe, and a second text-only model handles the formatting.

(A simpler approach could be to feed the multi-modal LLM output into the regular Mealie recipe scraper?)

I pasted the recipe text above into phi3 with the instruction "convert this recipe into JSON format" and got an again-reasonable output:

{
  "recipe": {
    "title": "Taco Bell's Rice",

    "ingredients": [
      {"name": "butter", "quantity": "1 1/2 tablespoons"},
      {"name": "extra-long grain rice", "quantity": "1 cup"},
      {"name": "chili powder", "quantity": "1/2 teaspoon"},
      {"name": "paprika", "quantity": "1/2 teaspoon"},
      {"name": "garlic powder", "quantity": "1/2 teaspoon"},
      {"name": "onion powder", "quantity": "1/2 teaspoon"},
      {"name": "cumin", "quantity": "1/2 teaspoon"},
      {"name": "tomato paste", "quantity": "1 spoonful"},
      {"name": "chicken stock", "quantity": "2 cups"}
    ],

    "instructions": [
      {
        "step": 1,
        "description": "Melt the butter in a saucepan over medium-high heat."
      },
      {
        "step": 2,
        "description": "Add rice and spices (chili powder, paprika, garlic powder, onion powder, cumin) to the saucepan. Stir constantly for about 1-2 minutes until the mixture is slightly golden and fragrant."
      },
      {
        "step": 3,
        "description": "Pour in chicken stock and bring the mixture to a boil."
      },
      {
        "step": 4,
        "description": "Reduce heat to low, cover saucepan tightly with lid, and let simmer for 20 minutes without lifting the lid."
      },
      {
        "step": 5,
        "description": "Remove from heat after 20 minutes. Let it rest for an additional 10 minutes before serving."
      },
      {
        "step": 6,
        "description": "Fluff the rice gently with a fork and serve."
      }
    ],
    "notes": [
      "[Text unclear]" // Placeholder for text that is difficult to read in the original image.
    ]
  }
}

Obviously, this isn't the exact schema used by Mealie, and I'm sure getting a precise output format would be more challenging. But I'm again optimistic that I could get a good-enough result.

This would have the added bonus that the (hopefully more accurate and less expensive) text-only model could handle scraping recipes from URL.

Now, I'm sure that OpenAI users are in the minority, and those running LLMs locally are in the vanishing minority, so I'm not sure this would be worth the development effort, but it would be cool. At any rate, hopefully my workflow is useful to someone else.

[JeffBE]

Hi, facing the same/similar issue when connecting to ollama backend. Not trying with OpenAI, since this somehow contradicts with the self hosting approach and they only accept credit card as payment.... Made sure Ollama is up and running and can be reached via curl.