Document how to properly package generated python code

[maxnoe]

What language does this apply to?

Python

Describe the problem you are trying to solve.

Shipping a python module that includes generated python code from proto definitions.

The proto definitions are not under the direct control of the authors of the python package.

Describe the solution you'd like

Documentation, and maybe a real fix for #1491 .

Describe alternatives you've considered

Additional context

#1491 contains everything but a real solution to the issue.

The documentation for how to use protobuf with python does not mention packaging or shipping python packages that include generated protobuf code.

There are no guidelines how this could be done and naïve approaches lead to the many different proto structure and import related issues in #1491.

I try to summarize the situation:

• The import statements in the generated python code depend only on the import statements in the .proto files, which means that the structure of the .proto files necessarily has to be the same as the python package.

  This is a big issue for people wanting to use .proto definition in python packages that are not directly related or have no control over the .proto sources.

• There currently are two hacky solutions for this issue:

  1. Replacing the import statements in the generated python code, e.g. using sed. This was actively discouraged by the protobuf maintainers, as generated code is no supposed to be modified. This is a reasonable stance in general.
  2. Messing with python's import path, e.g. via sys.path.insert(0, os.path.dirname(__file__)), sys.path.append(os.path.dirname(__file___)) or similar.

  This – in python terms – is the more hacky approach and create naming conflicts with other packages / stdlib files.
  E.g. imagine a package that has a random.py module. If this module would do the above, the stdlib random module is shadowed.

Actual solutions for this issue on the protoc level were rejected, e.g. in #7470.

• The only current way to get a working python package, is to directly have the .proto files in the correct structure.

  I created a minimal example here: https://github.com/maxnoe/python-proto-package/

  The structure is:

  .
  ├── protos
  │   └── prototest
  │       ├── bar.proto
  │       └── foo.proto
  ├── pyproject.toml
  ├── README.md
  ├── setup.py
  └── src
      └── prototest
          └── __init__.py
  
  

  calling protoc like protoc -I protos --python_out = src works in this case and creates proper import statements.

  While this works, it is not a solution for projects that need to include external proto definitions beyond their control.

[github-actions]

We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please add a comment.

This issue is labeled inactive because the last activity was over 90 days ago. This issue will be closed and archived after 14 additional days without activity.

[niloc132]

#19212 looks like another attempt to resolve this as it is handled in other languages, and divorce directory structure from language packaging.

We've taken the sed approach here as it seems more reasonable than "just copy your .proto files to a python-specific directory tree before compiling them". It is not without its downsides, and we're reevaluating some assumptions.

#1491 is a useful read, and it seems to "document" the implicit assumptions that should be part of actual documentation (rather than us finding out long after we've started to use protobufs) - namely, proto files should be mixed in py sources or otherwise must follow the expected packages of those py sources regardless of where those proto files started their life (namely, in a not-yet-using-python project). If you ever might add python to your project, the directory structure must start as if you already know what that python hierarchy will look like (or have two copies) - implying that this information probably needs to show up in every lang's documentation?

[github-actions]

We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please add a comment.

This issue is labeled inactive because the last activity was over 90 days ago. This issue will be closed and archived after 14 additional days without activity.

[maxnoe]

Ping