Description
In this work we would like to add:
observability submodule
Event datatype:
- Data model:
event_id: str
event_type: str # "llm.call" | "retrieval.query" | "evaluator.metric_result" | "session.start" | "evaluator.aggregate_result" ...
timestamp: datetime.timestamp
scope_type: str # "session" | "retrieval" | "evaluation"
scope_id: str # "session_id" | "evaluation_id"
parent_scope_id: str | None
payload: dict
Observer class that handles the event stream queue.
- Writer Protocol
jsonl Writer
- Writer exceptions are caught and logged inside the consumer
- Out of scope:
publish_event api, contextvar reads, emission sites in scopes, and viz integration (DuckDB)
Requirements
- This module should be independent of the rest of the framework
- Observer should be able to separate queues for different writers.
Description
In this work we would like to add:
observabilitysubmoduleEventdatatype:Observerclass that handles the event stream queue.jsonlWriterpublish_eventapi,contextvarreads, emission sites in scopes, and viz integration (DuckDB)Requirements