Feature request: add vision capabilities to understand images

[tillydray]

I'll work on this. https://platform.openai.com/docs/guides/vision

[tillydray]

see new plan #122 (comment)

So far my design plan is
1. create a new file to hold new functionality: org-ai-vision.el
1. create a new file to hold common image functionality to be used by both org-ai-vision.el and org-ai-openai-image.el: org-ai-image.el
1. extract functions from org-ai-openai-image.el and put them into org-ai-image.el
1. add new functionality to org-ai-vision.el

If anyone has feedback let me know, especially on file naming

[tillydray]

I assumed there would be commonalities to extract but that was wrong. So my new design plan is

1. rename org-ai-openai-image.el to something more specific to image generation with dall-e, like org-ai-generate-iamge.el
2. create org-ai-vision.el and add new functionality there

feedback welcome

[rksm]

Hey that sounds good! Vision capabilities would be super awesome! How do you imagine referring to an image?

[tillydray]

How do you imagine referring to an image?

Either base64 encoded or a link to the hosted image. Per the documentation

Images are made available to the model in two main ways: by passing a link to the image or by passing the base64 encoded image directly in the request

“Two main ways” sounds to me like there are other ways but I didn’t see any others 🤷