Ollama

[lenzenc]

Is there any support for or example of custom code that can be used to call local models via Ollama?

[brynn-code]

@lenzenc Currently, promptflow only support models deployed to AML through model catalog, local model is not supported. + @prakharg-msft

[HariKunapareddy]

Any update on whether Promptflow has support for locally deployed models?

[D-W-]

Hi @HariKunapareddy , we don't have plan to support locally deployed models in the near future. Please create a feature request if there's need for it. cc: @prakharg-msft

[darrkz]

@lenzenc I think you can use some api proxy system like one-api or new-api. I use it now
or ollama new version support openai api format

[NPuhlmann]

For everyone also stumbling across the topic, you could use a simple proxy script. Then use a Custom Connection in Prompt Flow and set the Base URL to the URL of the proxy server (localhost:8080)

from http.server import BaseHTTPRequestHandler, HTTPServer
import json
import requests

class OpenAIToOllamaProxy(BaseHTTPRequestHandler):
    def do_POST(self):
        # Anfrage lesen
        content_length = int(self.headers['Content-Length'])
        post_data = self.rfile.read(content_length)
        openai_request = json.loads(post_data)

        # OpenAI-Anfrage in Ollama-Format konvertieren
        ollama_request = self.convert_to_ollama_format(openai_request)

        # Anfrage an Ollama-API senden
        ollama_response = requests.post("http://localhost:11434/v1/chat/completions", json=ollama_request)

        # Antwort an den Client senden
        self.send_response(ollama_response.status_code)
        self.send_header('Content-Type', 'application/json')
        self.end_headers()
        self.wfile.write(ollama_response.content)

    def convert_to_ollama_format(self, openai_request):
        # Hier konvertierst du die OpenAI-Anfrage in das Ollama-Format
        # Dies ist ein einfaches Beispiel und muss an deine spezifischen Anforderungen angepasst werden
        ollama_request = {
            "model": "llama3.2:latest",
            "messages": openai_request.get("messages", ""),
            "max_tokens": openai_request.get("max_tokens", 1000)
        }
        return ollama_request

def run(server_class=HTTPServer, handler_class=OpenAIToOllamaProxy, port=8080):
    server_address = ('', port)
    httpd = server_class(server_address, handler_class)
    print(f"Starting proxy server on port {port}...")
    httpd.serve_forever()

if __name__ == "__main__":
    run()