Add KV-cache contribution to total emissions

[gregoirevilde-se]

Description

Hi, I've been looking for a method to compute emissions linked to cached tokens to apply it to my use case (LLM usage for software development throuhg Cursor/Claude & large context windows) and recently attempted to add KV-cache emissions to the calculation engine in this PR : #200.

The update is according to this paper detailing KV-cache impact on GPU power consumption + model latency. As I'm not an expert in this matter, I'd like your input/advice on this methodology and if it would be possible to add it to Ecologits' main calculation engine.

This feature seems important as cached tokens increasingly represent a large part of developers' token usage (based on monitoring developers' Cursor usage in my team) and I haven't found an accurate way of using this token count thourgh Ecologits yet.

If something is missing, how can I help to efficiently contribute to this feature ?

Brainstorming possible future features :

• Upload Cursor/other tool personal usage log to automatically calculate developers total emissions
• In Ecologits online calculator, provide the option of selecting context window scenarios such as "scan entire codebase/max model context window usage", "one-off agent edit", "25 successive queries to a single agent", ...

[samuelrince]

Hi @gregoirevilde-se,

First, thank you for the feedback, and sorry for replying a bit late. We are very busy with the recent announcement we made related to joining the CodeCarbon non-profit.

As mentioned in #183, we have launched a working group on the topic of input tokens (with and w/o caching). I expect we can release this before this summer.

Quick question for you, when in the PR #200 you mention "Cache Read" what does it mean exactly? Is it not the same as cached input tokens? If you have more details on that, I am very interested!

[gregoirevilde-se]

Hi @samuelrince ,

No worries, as a CodeCarbon and EcoLogits enjoyer I'm excited about the announcement and possible synergies so completely understand your focus.

Based on this Cursor forum entry and cross-referencing with Claude's caching documentation + our team's tokens vs spend logs, I understand "Cache Read" as "kv-cached input tokens". Here we can also note the "Input w/ cache write" that I understand as "write to kv cache".

Also, if aditional contributors are needed for the input token working group, I'm interested :)