Support for input tokens

[leokeba]

Description

Hello,

Thanks for developing EcoLogits, it is a very useful piece of software. However, there are certain use cases where the number of input tokens can be quite variable and sometimes very high compared to the number of output tokens. For example, when doing classification tasks with LLMs, the context can be very large while the output may just be a few tokens. RAG is another example where the context / prompt can be very large compared to the response. In these situations, it feels like EcoLogits might not be adequate yet for estimating carbon impacts.

Is this feature on your roadmap ? Did you already collect data regarding the impact of input tokens ? Are there specific reasons why this might be difficult ? (KV-Cache, different attention mechanisms, model architecture, etc.)

Thanks again

[samuelrince]

Hi @leokeba, sorry to get back to you so late. We have been very busy internally with the recent announcement related to EcoLogits joining the CodeCarbon non-profit.

We recently launched a working group to account for input tokens in EcoLogits. We will certainly also account for cached input tokens at the same time. Release should happen before this summer, hopefully, as this is an important methodology update.