Skip to content

Required pieces: Event data model, Observer Submodule, and jsonl Writer #1240

Description

@Amir-R25

Description

In this work we would like to add:

  • observability submodule
  • Event datatype:
    • Data model:
      event_id: str
      event_type: str # "llm.call" | "retrieval.query" | "evaluator.metric_result" | "session.start" | "evaluator.aggregate_result" ...
      timestamp: datetime.timestamp
      scope_type: str # "session" | "retrieval" | "evaluation"
      scope_id: str  # "session_id" | "evaluation_id"
      parent_scope_id: str | None
      payload: dict
  • Observer class that handles the event stream queue.
  • Writer Protocol
    • jsonl Writer
    • Writer exceptions are caught and logged inside the consumer
  • Out of scope: publish_event api, contextvar reads, emission sites in scopes, and viz integration (DuckDB)

Requirements

  • This module should be independent of the rest of the framework
  • Observer should be able to separate queues for different writers.

Metadata

Metadata

Assignees

Labels

[Maintainer] featureA well scoped initiative defined by a user facing requirements

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions