Feature Request: Add support for prompt caching

[joe-elliott]

Anthropic allows specific messages to be flagged for cache using a field: "cache_control": {"type": "ephemeral"}.

I'm not 100% sure, but it seems like ChatGPT and Gemini do this automatically?

I would like access to this feature in Anthropic for large, repetitive prompts (in my case TraceQL docs). Does it make sense to add this feature when one only provider requires it to be explicitly added? It is a nice cost and latency reduction. Am I correct that only Anthropic requires this to be explicitly flagged?

[sd2k]

Yeah I think this would be a good addition. Our plugin's backend currently assumes that everything looks roughly like an OpenAI request but we do already customise and transform the model field and we should be able to do the same for messages by defining a custom message struct, embedding openai.ChatCompletionMessage and adding the new cache_control field.

We'd need to make similar changes to the frontend package but that should be easier since Message is defined manually there anyway.

[iamrajiv]

make sense @joe-elliott i have one doubt i think gemini provides both manual and automatic caching so should we allow users to select what they need since we are building this why not make it modular and flexible in general wdyt?

[joe-elliott]

that sounds correct? i think the challenge here is to provide features like this through the abstraction layer in a way that is not confusing. perhaps a supportedFeatures() []Features function? or does that just add confusion/complexity?

generally i think it should "fail open" where if you request a feature on a provider that doesn't support it the plugin just ignores the request. will defer to @sd2k as he maintains this repo.