Skip to content

feat(protos): Add proto override system for in-repo proto definitions#115251

Draft
dashed wants to merge 11 commits intomasterfrom
dashed/feat/proto-dev-loader
Draft

feat(protos): Add proto override system for in-repo proto definitions#115251
dashed wants to merge 11 commits intomasterfrom
dashed/feat/proto-dev-loader

Conversation

@dashed
Copy link
Copy Markdown
Member

@dashed dashed commented May 9, 2026 •
edited
Loading

Summary

  • Add a proto override system that compiles and serves .proto files directly from the sentry repo, replacing the pip-installed sentry-protos package for migrated domains
  • Proto sources live in proto/sentry_protos/ and are the source of truth for migrated domains (starting with billing). Non-migrated domains (snuba, seer, etc.) fall back to the pip package transparently
  • Existing imports don't change — from sentry_protos.billing.v1.data_category_pb2 import DataCategory works the same whether the module comes from proto/ or pip. No code migration needed
  • Works in both development (auto-compile on import) and production (pre-compile during build)
  • getsentry picks this up automatically — no changes needed there (shared venv, editable install)

Architecture

Two modules with distinct responsibilities:

  • proto_compiler.py — build-time tool that compiles .proto → _pb2.py. Has a CLI for CI/deploy: python -m sentry.utils.proto_compiler compile --source proto --output .proto_cache
  • proto_loader.py — lightweight runtime import hook. Intercepts sentry_protos.*_pb2 imports, serves compiled overrides from .proto_cache/, falls back to pip. In dev mode, triggers compilation on demand when sources change. Only intercepts leaf _pb2 imports — intermediate packages are left to pip so non-overridden siblings remain importable

Hookup points

  • runner/initializer.py — install() called before django.setup(), which triggers model imports that reference sentry_protos at module level
  • testutils/pytest/sentry.py — install() called in pytest_configure before initialize_app() and test collection
  • Both calls are no-ops when no proto/ directory or cache exists — zero impact on developers not using overrides

Included

File Purpose
src/sentry/utils/proto_compiler.py Build-time .proto → _pb2.py compiler with CLI
src/sentry/utils/proto_loader.py Runtime import hook (lightweight, no grpcio-tools in prod)
src/sentry/utils/PROTO_OVERRIDE.md Full documentation
proto/README.md Proto source directory (ready for billing migration)
bin/sync-protos One-time migration script for copying domains from sentry-protos
src/sentry/runner/initializer.py install() call before django.setup()
src/sentry/testutils/pytest/sentry.py install() call in pytest_configure
tests/sentry/utils/test_proto_compiler.py 31 compiler tests
tests/sentry/utils/test_proto_loader.py 26 loader tests

Test plan

  • 57 unit tests pass (compiler + loader)
  • Lint passes (ruff, pre-commit hooks)
  • Verify bin/sync-protos copies billing protos correctly from sentry-protos
  • Verify from sentry_protos.billing.v1.data_category_pb2 import DataCategory works with in-repo protos
  • Verify non-overridden imports (snuba, seer) still resolve from pip
  • Verify getsentry tests pass without changes

dashed and others added 8 commits May 8, 2026 19:42
@dashed @claude
feat(protos): Add runtime proto compiler for development
2db7c17 
Install a custom import hook that compiles .proto files to _pb2.py
modules at import time, bypassing the installed sentry-protos pip
package. This eliminates the release cycle friction when iterating
on proto definitions during local development.

Set SENTRY_PROTO_DEV_DIR to a sentry-protos checkout or place
.proto files in {repo_root}/proto/ to activate.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@dashed @claude
ref(protos): Split proto_dev_loader into compiler + loader
cbed4df 
Split the combined proto_dev_loader module into two focused modules:

- proto_compiler: build-time tool that compiles .proto files into
  _pb2.py modules. Supports CLI usage for CI/deploy pre-compilation.
- proto_loader: lightweight runtime import hook that serves pre-compiled
  pb2 overrides with transparent pip fallback. Works in both development
  (on-demand compilation) and production (pre-compiled only).

Fix namespace shadowing bug: the original finder claimed intermediate
packages (sentry_protos.billing, etc.) with search paths pointing only
to the cache directory, making pip-installed pb2 modules invisible. The
new loader only intercepts _pb2 leaf imports and delegates intermediate
packages to pip's finder unless they represent new domains not in pip.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@dashed @claude
docs(protos): Add detailed documentation for proto override system
56a5f48 
Add PROTO_OVERRIDE.md with comprehensive guide covering architecture,
quick start for dev and prod, import hook internals, the namespace
shadowing bug and its fix, directory structure, CLI reference, API
reference, supported import patterns, and troubleshooting.

Enhance inline comments in proto_compiler.py and proto_loader.py
explaining design decisions: why grpcio-tools is lazily imported,
why RLock over Lock, why intermediate packages are delegated to pip,
and how _pip_package_has avoids recursion in the MetaPathFinder.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@dashed @claude
feat(protos): Add in-repo proto workflow and sync tooling
b75ca5a 
Add proto/ directory for committing .proto source files directly to
the sentry repo. Files here are auto-detected by proto_loader with
highest priority, enabling both local dev (auto-compile on import)
and prod (pre-compile during build) without a sentry-protos release.

Add bin/sync-protos helper script for syncing proto domains from a
sentry-protos checkout into proto/. Supports selective domain sync,
batch compilation, and status reporting. Defaults to the billing
domain and ../sentry-protos.

Update PROTO_OVERRIDE.md with in-repo workflow, priority order docs,
switching guide, and getsentry integration instructions.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@dashed @claude
docs(protos): Reframe docs for in-repo proto ownership
4e87a5f 
Proto sources live in this repo and are edited here directly.
The sync script is a one-time migration tool, not an ongoing
workflow. Update README and PROTO_OVERRIDE.md to reflect that
proto/ is the source of truth, with pip as fallback for
non-migrated domains only.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@dashed @claude
ref(protos): Strip implementation details from docs and comments
6229db8 
Remove the "Namespace Shadowing Bug" section and other internal
implementation narratives from PROTO_OVERRIDE.md. Simplify code
comments to explain what matters, not how we got here. Trim
verbose docstrings throughout both modules.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@dashed @claude
feat(protos): Wire proto_loader.install() into startup and tests
c3c953a 
Call install() in two places so proto overrides are active before
any sentry_protos imports:

- runner/initializer.py: before django.setup(), which triggers
  model imports that reference sentry_protos at module level.
- testutils/pytest/sentry.py: in pytest_configure, before
  initialize_app() and test collection.

Both calls are no-ops when no proto/ directory or cache exists.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@dashed @claude
docs(protos): Simplify getsentry section — no setup required
bc1679b 
Both repos share one venv with sentry editable-installed, so
proto overrides work automatically in getsentry with no changes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label May 9, 2026
dashed and others added 3 commits May 8, 2026 20:46
@dashed @claude
docs(protos): Clarify that existing imports don't change
e8c71b8 
The override is transparent — same import paths work regardless
of whether the module comes from proto/ or pip.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@dashed @claude
build(protos): Add grpcio-tools as dev dependency
1769a25 
Required for proto compilation during development (auto-compile on
import) and CI build steps. Not needed in production where protos
are pre-compiled.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@dashed @claude
revert: build(protos): Add grpcio-tools as dev dependency
a4086ea 
grpcio-tools is not available on pypi.devinfra.sentry.io yet.
Need to add it to getsentry/pypi first before it can be declared
as a dependency. For now, it's installed manually or via the
existing venv (already present from prior installs).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@dashed dashed mentioned this pull request May 9, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@dashed