Skip to content

feat(protos): Add proto override system for in-repo proto definitions#115251

Draft
dashed wants to merge 18 commits into
masterfrom
dashed/feat/proto-dev-loader
Draft

feat(protos): Add proto override system for in-repo proto definitions#115251
dashed wants to merge 18 commits into
masterfrom
dashed/feat/proto-dev-loader

Conversation

@dashed

@dashed dashed commented May 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Summary

  • Add a proto override system that temporarily overrides pip-installed sentry-protos modules with local .proto definitions for faster iteration, without waiting for a sentry-protos release
  • Proto files in proto/sentry_protos/ override the pip package transparently. Changes should eventually be published back to sentry-protos
  • Existing imports don't change — from sentry_protos.billing.v1.data_category_pb2 import DataCategory works the same whether the module comes from proto/ or pip. No code changes needed
  • getsentry picks this up automatically — no changes needed there (shared venv, editable install)

Architecture

Two modules with distinct responsibilities:

  • proto_compiler.py — build-time tool that compiles .proto → _pb2.py. Has a CLI: python -m sentry.utils.proto_compiler compile --source proto --output .proto_cache
  • proto_loader.py — lightweight runtime import hook. Intercepts sentry_protos.*_pb2 imports, serves compiled overrides from .proto_cache/, falls back to pip. In dev mode, triggers compilation on demand when sources change. Only intercepts leaf _pb2 imports — intermediate packages are left to pip so non-overridden siblings remain importable

All four paths

Path How it works Where
Dev install() detects proto/ dir, auto-compiles on import via grpcio-tools runner/initializer.py before django.setup()
Test Same as dev — auto-compile on import pytest_configure in testutils/pytest/sentry.py
CI Explicit pre-compile step before pytest, avoids xdist worker race conditions setup-sentry/action.yml after fast_editable
Prod Pre-compiled in getsentry Dockerfile (js-builder stage) — grpcio-tools in builder only, not in final image getsentry#20290

All steps are no-ops when proto/ has no .proto files — zero impact until protos are added.

Related PRs

Included

File Purpose
src/sentry/utils/proto_compiler.py Build-time .proto → _pb2.py compiler with CLI
src/sentry/utils/proto_loader.py Runtime import hook (lightweight, no grpcio-tools in prod)
src/sentry/utils/PROTO_OVERRIDE.md Full documentation
proto/README.md Proto override directory
bin/sync-protos Script for copying protos from sentry-protos
src/sentry/runner/initializer.py install() call before django.setup()
src/sentry/testutils/pytest/sentry.py install() call in pytest_configure
.github/actions/setup-sentry/action.yml CI pre-compile step (avoids xdist race)
pyproject.toml + uv.lock grpcio-tools added as dev dependency
tests/sentry/utils/test_proto_compiler.py 31 compiler tests
tests/sentry/utils/test_proto_loader.py 26 loader tests

Test plan

  • 57 unit tests pass (compiler + loader)
  • Lint + mypy passes
  • Verify bin/sync-protos copies billing protos correctly from sentry-protos
  • Verify from sentry_protos.billing.v1.data_category_pb2 import DataCategory works with overrides
  • Verify non-overridden imports (snuba, seer) still resolve from pip
  • Verify getsentry tests pass without changes

@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label May 9, 2026
@dashed dashed mentioned this pull request May 9, 2026
Merged
noahsmartin
noahsmartin reviewed May 11, 2026
View reviewed changes
Comment thread proto/README.md Outdated

- `billing/` — billing service protos (data categories, usage, contracts, etc.)

Other domains (snuba, seer, taskbroker, etc.) remain in the `sentry-protos` pip package until migrated.

@noahsmartin noahsmartin May 11, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming this was AI generated, we should probably take out the notion of "migrating" and "other domains" since the plan is this is just a temporary version of protos until we decide to switch back to sentry-protos

noahsmartin
noahsmartin reviewed May 11, 2026
View reviewed changes
Comment thread src/sentry/utils/PROTO_OVERRIDE.md Outdated

## Why This Exists

Proto definitions used to live exclusively in the `sentry-protos` repo, with a separate pip release cycle. Iterating on protos required publishing a new `sentry-protos` version before sentry could use the changes. This system moves proto source files into the sentry repo itself (`proto/`), compiling them at import time during development and pre-compiling them during CI/deploy for production.

@noahsmartin noahsmartin May 11, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where is the pre-compiling during CI? I only see the compiling in initializer.py in this PR. That is at runtime not CI time right?

@github-actions github-actions Bot added the Scope: Frontend Automatically applied to PRs that change frontend components label May 11, 2026
@dashed dashed force-pushed the dashed/feat/proto-dev-loader branch from 86d01dd to 14c7de1 Compare May 11, 2026 20:14
@getsentry getsentry deleted a comment from github-actions Bot May 11, 2026
dashed and others added 18 commits May 27, 2026 16:09
@dashed @claude
feat(protos): Add runtime proto compiler for development
b76b1c2 
Install a custom import hook that compiles .proto files to _pb2.py
modules at import time, bypassing the installed sentry-protos pip
package. This eliminates the release cycle friction when iterating
on proto definitions during local development.

Set SENTRY_PROTO_DEV_DIR to a sentry-protos checkout or place
.proto files in {repo_root}/proto/ to activate.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@dashed @claude
ref(protos): Split proto_dev_loader into compiler + loader
8e31304 
Split the combined proto_dev_loader module into two focused modules:

- proto_compiler: build-time tool that compiles .proto files into
  _pb2.py modules. Supports CLI usage for CI/deploy pre-compilation.
- proto_loader: lightweight runtime import hook that serves pre-compiled
  pb2 overrides with transparent pip fallback. Works in both development
  (on-demand compilation) and production (pre-compiled only).

Fix namespace shadowing bug: the original finder claimed intermediate
packages (sentry_protos.billing, etc.) with search paths pointing only
to the cache directory, making pip-installed pb2 modules invisible. The
new loader only intercepts _pb2 leaf imports and delegates intermediate
packages to pip's finder unless they represent new domains not in pip.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@dashed @claude
docs(protos): Add detailed documentation for proto override system
5292f5e 
Add PROTO_OVERRIDE.md with comprehensive guide covering architecture,
quick start for dev and prod, import hook internals, the namespace
shadowing bug and its fix, directory structure, CLI reference, API
reference, supported import patterns, and troubleshooting.

Enhance inline comments in proto_compiler.py and proto_loader.py
explaining design decisions: why grpcio-tools is lazily imported,
why RLock over Lock, why intermediate packages are delegated to pip,
and how _pip_package_has avoids recursion in the MetaPathFinder.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@dashed @claude
feat(protos): Add in-repo proto workflow and sync tooling
55c6c69 
Add proto/ directory for committing .proto source files directly to
the sentry repo. Files here are auto-detected by proto_loader with
highest priority, enabling both local dev (auto-compile on import)
and prod (pre-compile during build) without a sentry-protos release.

Add bin/sync-protos helper script for syncing proto domains from a
sentry-protos checkout into proto/. Supports selective domain sync,
batch compilation, and status reporting. Defaults to the billing
domain and ../sentry-protos.

Update PROTO_OVERRIDE.md with in-repo workflow, priority order docs,
switching guide, and getsentry integration instructions.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@dashed @claude
docs(protos): Reframe docs for in-repo proto ownership
aae8b4b 
Proto sources live in this repo and are edited here directly.
The sync script is a one-time migration tool, not an ongoing
workflow. Update README and PROTO_OVERRIDE.md to reflect that
proto/ is the source of truth, with pip as fallback for
non-migrated domains only.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@dashed @claude
ref(protos): Strip implementation details from docs and comments
33157a3 
Remove the "Namespace Shadowing Bug" section and other internal
implementation narratives from PROTO_OVERRIDE.md. Simplify code
comments to explain what matters, not how we got here. Trim
verbose docstrings throughout both modules.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@dashed @claude
feat(protos): Wire proto_loader.install() into startup and tests
197d6d8 
Call install() in two places so proto overrides are active before
any sentry_protos imports:

- runner/initializer.py: before django.setup(), which triggers
  model imports that reference sentry_protos at module level.
- testutils/pytest/sentry.py: in pytest_configure, before
  initialize_app() and test collection.

Both calls are no-ops when no proto/ directory or cache exists.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@dashed @claude
docs(protos): Simplify getsentry section — no setup required
faa2b5a 
Both repos share one venv with sentry editable-installed, so
proto overrides work automatically in getsentry with no changes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@dashed @claude
docs(protos): Clarify that existing imports don't change
c8f057a 
The override is transparent — same import paths work regardless
of whether the module comes from proto/ or pip.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@dashed @claude
build(protos): Add grpcio-tools as dev dependency
9fa1cce 
Required for proto compilation during development (auto-compile on
import) and CI build steps. Not needed in production where protos
are pre-compiled.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@dashed @claude
revert: build(protos): Add grpcio-tools as dev dependency
0e63ba5 
grpcio-tools is not available on pypi.devinfra.sentry.io yet.
Need to add it to getsentry/pypi first before it can be declared
as a dependency. For now, it's installed manually or via the
existing venv (already present from prior installs).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@dashed @claude
build(protos): Add grpcio-tools as dev dependency
87936a6 
Now available on pypi.devinfra.sentry.io (getsentry/pypi#2138).
Required for proto compilation during development (auto-compile
on import) and CI build steps.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@dashed @claude
docs(protos): Clarify that pre-compilation lives in getsentry Dockerfile
a9f09df 
The production pre-compile step is in getsentry's Dockerfile
(js-builder stage), not in this repo. The initializer.py call
is runtime only — it serves pre-compiled files, not compiles them.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@dashed @claude
ci(protos): Pre-compile proto overrides before test runs
8015a39 
Add a proto compilation step to the setup-sentry CI action, after
fast_editable and before devservices. This pre-compiles protos from
proto/ into .proto_cache/ so xdist workers don't race to compile
the same files at import time.

Conditional on proto/sentry_protos/ existing — no-op until billing
protos are migrated.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@dashed @claude
docs(protos): Add CI path to documentation
876c3ba 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@dashed @claude
fix(protos): Fix mypy errors in proto compiler and tests
14bf970 
Add grpc_tools and mypy_protobuf to the missing stubs list in
pyproject.toml (the sanctioned way to handle untyped third-party
modules). Add -> None return annotations to all test methods.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@dashed @claude
fix(protos): Fix remaining mypy annotation errors
87e04d6 
Fix generator fixture return type (Generator, not None) and add
missing -> None to test methods that were missed by the regex.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@dashed @claude
docs(protos): Reframe as temporary override, not permanent migration
346c6a2 
This is a temporary mechanism for faster iteration on proto
definitions, not a permanent migration of ownership away from
sentry-protos. Changes should eventually be published back to
sentry-protos. Remove "migration" language throughout.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@dashed dashed force-pushed the dashed/feat/proto-dev-loader branch from b9cc2bd to 346c6a2 Compare May 27, 2026 20:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@noahsmartin noahsmartin noahsmartin left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

Scope: Backend Automatically applied to PRs that change backend components Scope: Frontend Automatically applied to PRs that change frontend components

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@dashed @noahsmartin