buf generate from local modules without push

[dghubble]

Starting with buf you might have a file structure like this:

├── buf.gen.yaml
├── example
│   ├── buf.lock
│   ├── buf.yaml
│   └── example.proto
├── gen
│   ├── example
│   │   ├── example_grpc.pb.go
│   │   ├── example.pb.go
│   │   └── example.pb.gw.go

version: v1
plugins:
  - name: go
    out: gen
    opt:
      - paths=source_relative
...

When you start to use dependencies (e.g. google/api/annotations.proto), as mentioned in the tour, buf build works ok for the module (after buf mod update).

version: v1
deps:
  - buf.build/googleapis/googleapis

But running buf generate from the top-level won't use those dependencies. Here are some examples,

buf generate
google/type/money.proto: does not exist
google/type/annotations.proto: does not exist

The tour goes into pushing modules ot BSR and generating from those pushed modules (e.g. buf generate buf.build/username/private-repo). But I simply want a tool that resolves these common dependencies. Not pushing our internal modules to BSR.

Why can't buf generate work when the local module is valid (according to buf build)?

I see others are confused too #922, #1570.

Workarounds

Workspaces

Everyone seems to end up switching to using buf workspaces so that dependencies work as expected without the step of needing to push modules to a repository to then be able to generate code.

# buf.work.yaml
version: v1
directories:
  - example

But that causes other problems, like generated code now goes to gen/ directly (example directory is lost).

├── gen
│   ├── example_grpc.pb.go
│   ├── example.pb.go
│   ├── example.pb.gw.go

Where did the example directory go? How can we get the same generated file structure as before?

Generate each directory

It seems if you call buf generate on each directory containing .proto files, you can also workaround the dependencies not being found.

buf generate example

Like the workspace approach, the example directory is gone.

├── gen
│   ├── example_grpc.pb.go
│   ├── example.pb.go
│   ├── example.pb.gw.go

[nrakochy]

Thanks for the great question. I am facing this same scenario and arrived here for this very reason.