feat: OpenAI-compatible vision model support for screenshot processing by PremKumarAmbrose · Pull Request #190 · AnsahMohammad/shots-studio

PremKumarAmbrose · 2026-04-08T11:37:44Z

Hi Ansah Mohammad, thank you for reviewing this contribution.

This PR adds OpenAI-compatible provider support for screenshot AI workflows, including endpoint/model configuration.
Summary

Added support for OpenAI-compatible vision-capable endpoints (including llama-swap/Ollama-compatible flows).
Added configurable host URL/API key, model fetch, and provider-aware model persistence.
Wired provider config into foreground/background processing.
Added AI Settings UI for model fetch/selection and responsive layout.

Validation

flutter analyze completed (no new blocking issues from this change set)
flutter test completed (all tests passed)
Manual checks completed:
- OpenAI-compatible fetch + select + persist
- Background processing with selected model
- Gemini flow unchanged
- Gemma flow unchanged
- OCR flow unchanged

- Added OpenAI-compatible endpoint configuration UI - Added model list fetch and persisted selection - Added provider-aware model handling across AI flows - Added vision-capable filter with fallback toggle

CLAassistant · 2026-04-08T11:38:11Z

All committers have signed the CLA.

AnsahMohammad · 2026-04-08T12:54:06Z

Hi thanks for the PR,
Can you share screenshot/video of the UI changes that you made.

PremKumarAmbrose · 2026-04-08T13:29:05Z

Hi, thanks for the quick response.
Here are the UI changes made:

PremKumarAmbrose · 2026-04-11T18:00:10Z

To help with testing or for anyone using a local llama-server setup, I’ve found that Gemma 4 E4B (minimum q4km quant) performs best with the following configuration. This setup optimizes GPU offloading and ensures the vision component handles screenshot context correctly within an 8k window even with 8 images in parallel:

set LLAMA_CHAT_TEMPLATE_KWARGS={"enable_thinking":false}
llama-server.exe --model your_model_path.gguf --mmproj your_mmproj_path.gguf --ctx-size 8192 --n-gpu-layers 99 --batch-size 512 --ubatch-size 512 --parallel 1 --cache-type-k q8_0 --cache-type-v q8_0 --image-max-tokens 512 --temp 0.8 --top-p 0.9 --top-k 40 --min-p 0.005 --reasoning off --no-prefill-assistant

AnsahMohammad · 2026-04-12T01:41:19Z

Thanks Prem for the update. I'll review your code today.

AnsahMohammad · 2026-04-12T10:17:30Z

 # The patch version numbering is automatically incremented by the git hook. 
 # Only update major and minor versions manually. 
-version: 1.9.100+10
+version: 1.9.103+10


it should be 1.9.101

AnsahMohammad · 2026-04-12T10:34:35Z

      'gemini-2.5-flash',
      'gemini-2.5-pro',
    ],
+    'openai_compatible': [],


Is this restricted to openai ?
Since we can use any multi-modal AI to process this, a generalised term like ollama_models would be better know ?

It's actually compatible with any model provider which uses the openai api format. We can change it into something like "openai api compatible" or keep as is to generalize it for any provider. Ollama and llama-swap are one of the providers which supports this api format.

I'm trying to understand why are we developiong for openAI api format in the first place. Since we don't have any openai implementation or services in the codebase, native API will be more flexible is what i believe.

I'm trying to understand why are we developiong for openAI api format in the first place. Since we don't have any openai implementation or services in the codebase, native API will be more flexible is what i believe.

The main reason for supporting the OpenAI API format is its status as a de facto industry standard for LLM APIs. While we don’t use OpenAI’s own services, the OpenAI API spec has become the most widely adopted interface for both cloud and local LLM providers. Many popular local serving solutions like Ollama, llama.cpp, LM Studio, llama-swap, and others intentionally implement the OpenAI API format to maximize compatibility with existing tools and clients.

By targeting this API, we make the app instantly compatible with a huge range of providers, both local and remote, without needing to write custom integrations for each one. This means users can run their own models locally (for privacy, cost, or speed) or connect to any compatible cloud service, simply by pointing to the right endpoint. It also future-proofs the app, since new providers almost always add OpenAI API compatibility first.

Here are the latest, valid documentation links for OpenAI API compatibility for each provider:

Ollama:
Ollama OpenAI Compatibility Documentation

llama.cpp:
llama.cpp OpenAI-Compatible Server (GitHub)
llama-cpp-python OpenAI-Compatible Server
LM Studio:
LM Studio OpenAI Compatibility Docs

llama-swap:
llama-swap GitHub (OpenAI API Compatible)

Addresses the request #183

Yeah i see that while this might be useful for fast porting from ollama to open AI.
https://docs.ollama.com/api/openai-compatibility shows some parameters are simply not available for open AI supporting APIs. Which will affect the customisations that we'll be able to bring.

Since ollama is the major provider that we are trying to import. It will be best if we implement in their native API.

I see your point about the missing parameters in the documentation. It is true that Ollama’s native API exposes specific options that the OpenAI specification ignores. But looking at what actually gets dropped, we should weigh if they are worth the effort for Shots Studio.
The missing parameters are almost entirely low-level llama.cpp parameter tuning as ollama is basically a wrapper for llama.cpp. We lose hardware overrides like num_thread, advanced sampling algorithms like mirostat or tfs_z, and explicit memory controls like keep_alive. However, keep_alive already defaults to 5 minutes in Ollama anyway, and the average user likely just needs reliable vision capabilities rather than granular manipulation.
The real trade-off here is depth versus reach. By using the OpenAI format, we get a "1 vs. many" multiplier. We write the code once, and Shots Studio instantly supports Ollama, LM Studio, LocalAI, and any other proxy. For users already running Ollama, the setup is completely frictionless they simply append /v1 to their existing localhost URL.
If the long-term vision for Shots Studio genuinely requires those specific low-level Ollama parameters, then sticking exclusively to their native API is the right call. Feel free to close this PR if you want to maintain a strict Ollama-native architecture. I just wanted to provide the path that gives users the most 'Bring Your Own AI' flexibility out of the box.

I see your point. Yeah okay it makes sense if this allows integration with other proxies aswell. Okay I'll review the Pr in the above order. Thanks

Glad to hear that works for the project's goals. I have a lot of respect for what you've built here; it's a solid project and I'm happy to be a part of it.

I’m looking forward to seeing how this evolves and I’d be glad to support you in any way I can. Feel free to reach out if you want to discuss new features or if you need help with any specific part of the codebase. Excited for what's next.

AnsahMohammad

Great work,

But instead of a single PR, can we reduce this PR into 3 where :

Service implementation of logic required for connection with ollama and various APIs
Integration of this ollama API with rest of the AI services.
UI changes required for this.

Thanks for your contribution

PremKumarAmbrose · 2026-04-12T11:09:48Z

Great work,

But instead of a single PR, can we reduce this PR into 3 where :
1. Service implementation of logic required for connection with ollama and various APIs

2. Integration of this ollama API with rest of the AI services.

3. UI changes required for this.
Thanks for your contribution

Thanks for the review. I’ve split the work into 3 stacked PRs as requested:

Service implementation for connecting to Ollama/OpenAI-compatible APIs
Integration of that provider into the existing AI flows
UI changes for settings and model selection
I also updated the version to 1.9.101 as requested. Please review them in order if that works best.

AnsahMohammad · 2026-04-12T13:40:35Z

Service implementation for connecting to Ollama/OpenAI-compatible APIs Integration of that provider into the existing AI flows UI changes for settings and model selection I also updated the version to 1.9.101 as requested. Please review them in order if that works best.

Thanks for the response, can you specify the order of PR in which they shall be reviewed.

I'm closing this PR as this is stale now.

PremKumarAmbrose · 2026-04-12T14:02:17Z

Service implementation for connecting to Ollama/OpenAI-compatible APIs Integration of that provider into the existing AI flows UI changes for settings and model selection I also updated the version to 1.9.101 as requested. Please review them in order if that works best.

Thanks for the response, can you specify the order of PR in which they shall be reviewed.

I'm closing this PR as this is stale now.

Please review the PR in this order. #193 -> #191 -> #192

feat: support OpenAI-compatible model provider

d8f4fa7

- Added OpenAI-compatible endpoint configuration UI - Added model list fetch and persisted selection - Added provider-aware model handling across AI flows - Added vision-capable filter with fallback toggle

AnsahMohammad self-requested a review April 8, 2026 12:52

AnsahMohammad reviewed Apr 12, 2026

View reviewed changes

AnsahMohammad requested changes Apr 12, 2026

View reviewed changes

AnsahMohammad closed this Apr 13, 2026

Uh oh!

Conversation

PremKumarAmbrose commented Apr 8, 2026

Uh oh!

CLAassistant commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AnsahMohammad commented Apr 8, 2026

Uh oh!

PremKumarAmbrose commented Apr 8, 2026

Uh oh!

PremKumarAmbrose commented Apr 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AnsahMohammad commented Apr 12, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PremKumarAmbrose Apr 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PremKumarAmbrose Apr 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AnsahMohammad left a comment

Choose a reason for hiding this comment

Uh oh!

PremKumarAmbrose commented Apr 12, 2026

Uh oh!

AnsahMohammad commented Apr 12, 2026

Uh oh!

PremKumarAmbrose commented Apr 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

CLAassistant commented Apr 8, 2026 •

edited

Loading

PremKumarAmbrose commented Apr 11, 2026 •

edited

Loading

PremKumarAmbrose Apr 12, 2026 •

edited

Loading

PremKumarAmbrose Apr 12, 2026 •

edited

Loading