Analytics and tracing ehnancements

[sdryapko]

Name and Version

development

What is the problem this feature will solve?

Refactor our analytics and make sure that we are following industry standards and able to solve most of the cases we have.

What is the feature you are proposing to solve the problem?

We suggest to refactor our analytics flow to cover following requirements:

1. We should try to avoid some excessive calculations on the data upload (like ML) to make analytics really real-time
2. Our process to collect data for analytics should not impact other processes (for example upload audit logs to BLOB storage)
3. We should have the way to add / modify historical data in case of having some issues during data collection (examples of usecases for modify : price recalculation, wrong cost in configs and etc, for add: issues with job which record data)
4. We should be able to configure what specific information can we store in our analytics database and what can be used for some additional processing on top of raw data (for example we should be able to save some subset of this fields or other fields available from audit logs: timestamp, target model/agent, client (application), input tokens, output tokens, cost, user id, claims, request content, response content, feedback messages)
5. We should be able to work with long-range data.
6. We should improve way how we work with non-standard ways to calculate tokens (like caching, strange cost calculation for image generation models and etc.) including different costs for different endpoints
7. We should be able to run some additional jobs / plugins (like sentiment analysis or topic analysis) on top of raw analytics data effectively which can produce additional data in separate tables (for example some aggregated usage for some period of time) or some additional jobs to modify existing tables (like associate feedback messages with original conversation)
8. Add all DIAL entities (like interceptors) usage to analytics.
9. We should be able to provide API (including filters, sort, join, group by, aggregates) to connect other services (like admin panel or MCP server) to data from analytics.
10. We should structure the data inside database the way to be able easily develop or take existing MCP server to work with data from analytics
11. We should be able to do data migration from existing InfluxDB or raw data from audit logs.
12. We should expose industry-standard api for data from analytics database (ODBC, Pandas, Clickhouse).
13. We should be able to track conversations details and dig into it.
14. We should provide SDK (or other easy way) for usecase developers to provide them the way to record some data about their usecases to separate tables of analytics.
15. Solution should be cloud-agnostic and we should not bring some database with inappropriate for us license

What alternatives have you considered?

No response