Skip to content

Add anonymous shape-only usage telemetry with stdlib-only sender - #33

Merged
requie merged 6 commits into
mainfrom
feat/telemetry
Jul 25, 2026
Merged

Add anonymous shape-only usage telemetry with stdlib-only sender#33
requie merged 6 commits into
mainfrom
feat/telemetry

Conversation

@requie

@requie requie commented Jul 25, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds anonymous, shape-only usage telemetry to guide development priorities (adapter adoption, evaluation surface usage) without collecting any agent content, prompts, or user data. The implementation uses only stdlib (urllib, json, threading, queue) with zero new dependencies. Telemetry failures are isolated and never propagate to the host process.

Key Changes

  • Core telemetry module (src/agentegrity/core/telemetry.py):

    • Stdlib-only PostHog sender with background batching and graceful degradation
    • Persistent anonymous UUID stored at ~/.agentegrity/id (mode 0600), with fallback to ephemeral per-process IDs
    • Re-entrant scoping via context vars to track when events should be captured
    • Environment classification (local/CI/Docker/Colab/Kaggle) and version tagging on every event
    • Full opt-out support: DO_NOT_TRACK=1, AGENTEGRITY_TELEMETRY_DISABLED=1, or disable_telemetry() at runtime
  • Shape builders (src/agentegrity/core/_telemetry_props.py):

    • Single auditable module where all event properties are constructed
    • Strict allowlist enforcement: only enums, counts, booleans, rounded floats (3 decimals), durations, version strings
    • Never: prompts, model I/O, tool arguments, file paths, agent/profile names, capability strings, policy text, exception messages
    • Composite score bucketing for funnel analysis
  • Client instrumentation (src/agentegrity/sdk/client.py):

    • evaluate(): captures profile shape + per-property scores + composite bucket + duration
    • attest(): captures record kind, signedness, evidence count
    • create_adapter(): captures adapter name + framework availability (even on import failure)
  • Monitor instrumentation (src/agentegrity/core/monitor.py):

    • monitor_violation(): captures action + lowest-scoring property for violation attribution
  • Attestation chain instrumentation (src/agentegrity/core/attestation.py):

    • verify_chain(): captures record count, decision count, verification outcome
  • CLI instrumentation (src/agentegrity/__main__.py):

    • Captures command name (info/doctor/verify-decisions)
  • Test suite (tests/test_telemetry.py):

    • 40+ tests covering opt-out mechanisms, anonymous ID persistence, scoping semantics, exception reporting, tag merging
    • Leak-guard tests verify all shape builders emit only allowlisted keys and no string > 64 chars
    • Sender tests verify queue overflow handling, HTTP failure isolation, batch envelope format
  • Documentation (docs/telemetry.md):

    • Complete event catalog with properties
    • Anonymity mechanism and ID lifecycle
    • Opt-out instructions (env vars + runtime API)
    • Explicit list of what is never collected
  • Public API (src/agentegrity/core/__init__.py):

    • Exports disable_telemetry(), telemetry_capture(), telemetry_run_context(), telemetry_tag(), scoped_telemetry

Implementation Details

  • Isolation: All telemetry code runs in a background daemon thread; exceptions in the sender, HTTP layer, or ID file I/O are caught and dropped silently
  • Scoping: Context vars track nesting depth and per-scope tags; events are only captured when _scope_depth > 0, preventing orphaned events outside a run context
  • Batching: Worker thread lingers 0.5s to batch events, posts with 2s timeout, flushes on process exit via atexit hook
  • Decorator support: @scoped_telemetry wraps both sync and async callables, automatically establishing a run context
  • Conftest integration: Test suite disables telemetry globally to prevent test pollution

Opt-Out

Telemetry is fully disabled

@cursor

cursor Bot commented Jul 25, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@requie
requie requested a review from mansoorcogensec July 25, 2026 17:03
@requie requie self-assigned this Jul 25, 2026
@requie requie added the enhancement New feature or request label Jul 25, 2026
@requie
requie removed the request for review from mansoorcogensec July 25, 2026 17:04
@requie
requie merged commit d765cbb into main Jul 25, 2026
11 checks passed
@requie
requie deleted the feat/telemetry branch July 26, 2026 16:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant