Skip to content

feat(reporting): align delegated reporter protocol with ReportingContext and multi-type dispatch - #591

Open
saurabh-net wants to merge 2 commits into
mainfrom
feat/delegated-reporting-context
Open

feat(reporting): align delegated reporter protocol with ReportingContext and multi-type dispatch#591
saurabh-net wants to merge 2 commits into
mainfrom
feat/delegated-reporting-context

Conversation

@saurabh-net

@saurabh-net saurabh-net commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Context & Problem

This PR aligns the delegated remote reporting mechanism (RemoteReporter) with the same structured context model established for delegated scorers:

  1. Structured ReportingContext in Protobuf:
    Previously, ReportingRequest only contained ReporterSpec (name, configuration JSON, timeout), omitting evaluation context (job_id, run_time, store_type, DataFrame records, and database). We introduce ReportingContext in eval_agent.proto:

    message ReportingContext {
      string job_id = 1;
      string run_time = 2;          // Timestamp of evaluation run
      string store_type = 3;        // "CONFIGS", "EVALS", "SCORES", "SUMMARY"
      string results_json = 4;      // Serialized DataFrame records as JSON string (orient="records")
      string database = 5;          // Optional target database name
    }
    
    message ReportingRequest {
      ReporterSpec reporter = 1;
      ReportingContext context = 2;
    }
  2. Multi-Type Dispatch (Option 2):
    Native EvalBench reporters (Reporter.store) are invoked across four distinct store phases: STORETYPE.CONFIGS, STORETYPE.EVALS, STORETYPE.SCORES, and STORETYPE.SUMMARY. Previously, RemoteReporter had hardcoded filtering (if type_name != "EVALS": return), which prevented remote reporters (like CSV or custom sinks) from receiving and persisting other report tables. We now dispatch every store_type across the stream along with its serialized DataFrame rows, allowing the remote worker to decide which types to process.

  3. Logging & Diagnostics:

    • Standardized remote reporting logs with clear [REMOTE_REPORTER] prefixes for easy debugging across stream sessions.
    • Clean handling of timeouts and failure responses without crashing the evaluation runner.

Changes

  • evalbench/evalproto/eval_agent.proto: Added ReportingContext message and updated ReportingRequest to include context.
  • evalbench/reporting/remote_reporter.py: Constructed ReportingContext, serialized DataFrame records with ISO date formatting, dispatched across all store_type calls, and handled response logging.
  • evalbench/test/agent_grpc_proxy_test.py: Added comprehensive unit tests covering ReportingContext field validation, multi-type dispatch across all 4 store phases (CONFIGS, EVALS, SCORES, SUMMARY), timeout handling, and failure response handling.

Verification

  • pytest evalbench/test/agent_grpc_proxy_test.py evalbench/test/agent_grpc_proxy_integration_test.py (9 passed, 0 warnings).
  • pycodestyle clean (0 errors).
  • Zero generated proto stubs in diff against main.

@prernakakkar-google

Copy link
Copy Markdown
Collaborator

/gcbrun

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants