Protofetch

A source dependency management tool for Protobuf files.

Motivation

If you use protobuf extensively as a data format for services to communicate with or to share your APIs with the outside world, you need a way to get correct versions of protobuf files for each service and ability to depend on a specific version. This is needed on both server and client side. Without automation, it can quickly become cumbersome, error-prone and overall unmanageable.

To make it bearable, usable and stable, one needs tooling that automates this work and makes it predictable. This is what Protofetch aims to do.

Why Protofetch?

Protofetch aims to tackle the complexity of handling protobuf dependencies in a declarative fashion. It makes it trivial to declare dependencies and to manage them.

It gives you the ability to have:

• dependency on specific version/hash;
• predictable builds/test/CI that depend on protobufs;
• easy to read declarative specification of protobuf dependencies;
• automate fetching of the dependencies themselves with their transitive dependencies.
• caching of dependencies so that they can be shared across multiple projects.

Roadmap

This project is still under development and is subject to change in the future. We aim to achieve at least the following goals before releasing the first stable version.

• Fetch dependencies based on git tag or branch
• Cache dependencies locally by revision
• Fetch transitive dependencies
• Declarative rules per dependency
  • Allow policies
  • Deny policies
  • Dependency pruning (remove proto files that are not needed)
• Prevent circular dependencies

Getting Started

npm:

npm install @coralogix/protofetch

Cargo:

Protofetch is also released to crates.io, so if you have a Rust toolchain installed, you can build Protofetch from source with:

cargo install protofetch

Pre-built binaries:

You can download pre-built binaries from the GitHub Releases page.

Usage

# Fetch proto sources, updating the lock file if needed.
protofetch fetch

# Verify the lock file, and fetch proto sources. Useful for CI.
protofetch fetch --locked

Protofetch module

Each service using protofetch will require a module descriptor which uses toml format. This descriptor is by default called protofetch.toml and is located in the root of the service's repository. This can be changed, but it is heavily discouraged.

Field	Type	Required	Description
name	String	Mandatory	A name of the defined module
description	String	Optional	A description of the module
dependencies	[Dependency]	Optional	Dependencies to fetch

Dependency format

Field	Type	Required	Description	Example
url	String	Mandatory	An address of the repository to checkout protobuf files from	"github.com/coralogix/cx-api-users/"
revision	String	Optional	A revision to checkout, this can either be a tagged version or a commit hash	v0.2
branch	Boolean	Optional	A branch to checkout, fetches last commit	feature/v2
protocol	String	Optional	A protocol to use: [ssh, https]	ssh
allow_policies	[String]	Optional	Allow policy rules (* at the beginning or end matches arbitrary directory depth)	"/prefix/*", "*/subpath/*", "/path/to/file.proto"
deny_policies	[String]	Optional	Deny policy rules (* at the beginning or end matches arbitrary directory depth)	"/prefix/*", "*/subpath/*", "/path/to/file.proto"
prune	bool	Optional	Whether to follow proto imports instead of copying by module dependency alone	true / false
transitive	bool	Optional	Flags this dependency as transitive	true / false
content_roots	[String]	Optional	Which subdirectories to import from	["/myservice", "/com/org/client"]

Allow and deny policies

The patterns in allow_policies and deny_policies are matched against the paths relative to the nearest path in content_roots.

allow_policies apply only to the dependency they are defined on and describe the proto files you want from that dependency. When prune = false, only matching proto files are included. When prune = true, matching proto files are used as traversal roots: protofetch follows their proto import statements across module dependencies, may include additional imported files from the same module, and may include imported files from the dependency subtree.

deny_policies apply to the whole dependency subtree. When prune = false, matching proto files are excluded. When prune = true, matching proto files and their import tree are excluded.

Dependency pruning

The pruning feature makes protofetch follow proto file import statements instead of copying files only from module-level dependency relationships. With pruning enabled, protofetch starts from the files included by allow_policies, recursively follows their imports across module dependencies, and fetches every imported proto file it reaches. This traversal may include additional files from the same module even if they were not directly matched by allow_policies, and it may include imported proto files from the dependency subtree.

Additional transitive dependencies

When pruning is enabled, protofetch looks for imported proto files in the module dependency tree. This relies on each dependency declaring its own module dependencies in a protofetch.toml file. If a dependency does not provide one, you can still use the pruning feature by declaring additional dependencies with transitive = true. Files from transitive dependencies are not fetched directly. They are available as candidates and are copied only if the import traversal from a pruned dependency reaches them.

Protofetch dependency toml example

name = "repository name"
description = "this is a repository"

[dep1]
url = "github.com/org/dep1"
revision = "1.3.0"
prune = true
allow_policies = ["/prefix/*", "*/subpath/*", "/path/to/file.proto"]

[dep2]
url = "github.com/org/dep2"
branch = "feature/v2"

[another-name]
url = "github.com/org/dep3"
revision = "a16f097eab6e64f2b711fd4b977e610791376223"
transitive = true

[scoped-down-dep4]
url = "github.com/org/dep4"
revision = "v1.1"
content_roots = ["/scope/path"]
allow_policies = ["prefix/subpath/scoped_path/*"]

Git protocol

Protofetch supports accessing Git repositories using ssh or https. By default, Protofetch uses ssh. You can configure the default Git protocol with the PROTOFETCH_GIT_PROTOCOL environment variable.

It is also possible to set protocol in the protofetch.toml, but this should be only necessary if the Git server does not support both protocols. Otherwise, it is better to leave this field unset, to let users choose whichever protocol they prefer.

SSH support

You need to have an SSH agent running, with your SSH key loaded:

ssh-add ~/.ssh/your-private-key

HTTPS support

If you want to use https you need to configure git to use a credentials helper.

To support https when 2FA is enabled you must generate a personal access token and set it as the password. The following permissions are sufficient when creating the token.

Scope down multi API repo

In the case of a repo that supports multiple APIs, but only a specific directory is needed, a combination of content_roots and allow_policies can be used.

For example: the dep4 repo contains the following:

dep4
 ├── scope
 │   ├── path1
 │   └── path2
 └── scope2
     └── unrelated

We only need protobuf files from dep4/scope/path1, where path1 is the package name.

[scoped-down-dep4]
url = "github.com/org/dep4"
revision = "v1.1"
content_roots = ["/scope"]
allow_policies = ["path1/*"]