Skip to content

Commit 4b9045a

Browse files
saraiva132saraiva132
authored
WIP Sc 4376 pruning transitive dependencies (#17)
* sc-4376 Add allow_list, content_root and transitive dependency pruning
1 parent 967c9d0 commit 4b9045a

21 files changed

+1222
-223
lines changed

‎Cargo.lock

+37-45
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml

+7-4
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
[package]
22
name = "protofetch"
3-
version = "0.0.10"
3+
version = "0.0.11"
44
edition = "2018"
55
license = "Apache-2.0"
66
description = "A source dependency management tool for Protobuf."
@@ -13,9 +13,8 @@ exclude = [
1313
".github", ".gitignore"
1414
]
1515

16-
1716
[dependencies]
18-
clap = { version = "3.1.15", features = ["derive"] }
17+
clap = { version = "3.1.17", features = ["derive"] }
1918
git2 = { version = "0.14.3", features = ["vendored-openssl"] }
2019
config = "0.13.1"
2120
serde = { version = "1.0.137", features = ["derive"] }
@@ -28,4 +27,8 @@ lazy_static = "1.4.0"
2827
strum = { version = "0.24.0", features = ["derive"] }
2928
derive-new = "0.5.9"
3029
home = "0.5.3"
31-
mio = "0.8.2"
30+
smart-default = "0.6.0"
31+
test-log = "0.2.8"
32+
33+
[dev-dependencies]
34+
project-root = "0.2.2"

‎README.md

+57-18
Original file line numberDiff line numberDiff line change
@@ -29,22 +29,20 @@ It gives you the ability to have:
2929

3030
## Roadmap
3131

32-
3332
This project is still under development and is subject to changes in the future.
3433
We aim to achieve at least the following goals before releasing the first stable version.
3534

3635
- [x] Fetch dependencies based on git tag or branch
3736
- [x] Cache dependencies locally by revision
3837
- [x] Fetch transitive dependencies
3938
- [ ] Declarative rules per dependency
40-
- [ ] Whitelisting
41-
- [ ] Blacklisting
42-
- [ ] Dependency pruning (remove ``proto`` files that are not needed)
39+
- [x] Allow policies
40+
- [ ] Deny policies
41+
- [x] Dependency pruning (remove `proto` files that are not needed)
4342
- [ ] Prevent circular dependencies
4443

4544
## Getting Started
4645

47-
4846
Protofetch is being released to cargo so to use it you can directly download the crate from the [crates.io](https://crates.io/crates/protofetch)
4947
and install it with `cargo install protofetch`.
5048

@@ -53,6 +51,9 @@ and install it with `cargo install protofetch`.
5351
```sh
5452
# -f forces lock file to be generated in every run
5553
protofetch fetch -f
54+
55+
# For just lock file generation
56+
protofetch lock
5657
```
5758

5859
## Protofetch module
@@ -61,21 +62,24 @@ Each service using protofetch will require a module descriptor which uses `toml`
6162
This descriptor is by default called `protofetch.toml` and is located in the root of the service's repository.
6263
This can be changed, but it is heavily discouraged.
6364

64-
| Field | Type | Required | Description |
65-
|---------------|:-----------------|:----------|:---------------------------------------------------------------------------|
66-
| name | String | mandatory | the name of the defined module |
67-
| description | String | Optional | the description of the module |
68-
| proto_out_dir | String | Optional | the path to write the proto files to, relative to where the command is run |
69-
| dependencies | List[Dependency] | Optional | The dependencies to fetch |
65+
| Field | Type | Required | Description |
66+
|---------------|:-------------|:----------|:---------------------------------------------------------------------------|
67+
| name | String | mandatory | the name of the defined module |
68+
| description | String | Optional | the description of the module |
69+
| proto_out_dir | String | Optional | the path to write the proto files to, relative to where the command is run |
70+
| dependencies | [Dependency] | Optional | The dependencies to fetch |
7071

7172
### Dependency format
7273

73-
| Field | Type | Required | Description | Example |
74-
|----------|:--------|:----------|:-----------------------------------------------------------------------------------:|-------------------------------------:|
75-
| url | String | mandatory | the address of the repo to checkout protobuf files from | "github.com/coralogix/cx-api-users/" |
76-
| revision | String | mandatory | the revision to checkout from, this can either be a tagged version or a commit hash | v0.2 |
77-
| branch | Boolean | Optional | branch can be used to override revision for testing purposes, fetches last commit | feature/v2 |
78-
| protocol | String | mandatory | protocol to use: [ssh, https] | ssh |
74+
| Field | Type | Required | Description | Example |
75+
|----------------|:---------|:----------|:-----------------------------------------------------------------------------------:|--------------------------------------------------:|
76+
| url | String | mandatory | the address of the repo to checkout protobuf files from | "github.com/coralogix/cx-api-users/" |
77+
| revision | String | mandatory | the revision to checkout from, this can either be a tagged version or a commit hash | v0.2 |
78+
| branch | Boolean | Optional | branch can be used to override revision for testing purposes, fetches last commit | feature/v2 |
79+
| protocol | String | mandatory | protocol to use: [ssh, https] | ssh |
80+
| allow_policies | [String] | Optional | Allow policy rules. | "/prefix/*", "*/subpath/*", "/path/to/file.proto" |
81+
| prune | bool | Optional | Whether to prune unneded transitive proto files | true /false |
82+
| transitive | bool | Optional | Flags this dependency as transitive | true /false |
7983

8084

8185
### Protofetch dependency toml example
@@ -103,7 +107,7 @@ url = "github.com/org/dep3"
103107
revision = "a16f097eab6e64f2b711fd4b977e610791376223"
104108
```
105109

106-
## HTTPS Support
110+
## HTTPS support
107111

108112
If you want to use https you need to specify credentials using one of the following:
109113

@@ -116,3 +120,38 @@ To support https when `2FA` is enabled you must generate a personal access token
116120
The following permissions are suficient when creating the token.
117121

118122
![GitHub personal access token](readme-images/github-personal-access-token.png)
123+
124+
## Transitive dependency support and pruning
125+
126+
Protofetch supports pulling transitive dependencies for your convenience.
127+
However, there is some manual work involved if the dependencies do not define their own protofetch module.
128+
129+
In a situation where A depends on B, you should flag that dependency as transitive.
130+
131+
This is helpful especially when you take advantage of the pruning feature which allows you to only recursively fetch
132+
the proto files you actually need. With pruning enabled, protofetch will recursively find what protofiles your root
133+
protos depend on and fetch them for as long as they are imported (flag as transitive dependency or fetched from other modules).
134+
135+
Moreover, you can also use the allow_policies to scope down the root proto files you want from a dependency.
136+
As an example, the following module depends only on A's file `/proto/path/example.proto` but since pruning is enabled and
137+
B is flagged as transitive, if the allowed file has any file dependencies it will pull them and its dependencies, recursively.
138+
139+
```toml
140+
name = "repository name"
141+
description = "this is a repository"
142+
proto_out_dir = "proto/src/dir/output"
143+
144+
[A]
145+
protocol = "https"
146+
url = "github.com/org/A"
147+
revision = "1.3.0"
148+
allow_policies = ["/proto/path/example.proto"]
149+
prune = true
150+
151+
[B]
152+
protocol = "ssh"
153+
url = "github.com/org/B"
154+
revision = "5.2.0"
155+
branch = "feature/v2"
156+
transitive = true
157+
```

‎resources/cache/dep1/hash1/proto/example.proto

+11
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
syntax = "proto3";
2+
3+
package proto;
4+
5+
import "proto/example2.proto";
6+
import "google/protobuf/descriptor.proto";
7+
8+
option (scalapb.options) = {
9+
scope: PACKAGE
10+
flat_package: true
11+
};

‎resources/cache/dep2/hash2/proto/example2.proto

+15
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
syntax = "proto3";
2+
3+
package proto;
4+
5+
import "scalapb/scalapb.proto";
6+
import "proto/example3.proto";
7+
import "google/protobuf/descriptor.proto";
8+
import "google/protobuf/struct.proto";
9+
10+
option (scalapb.options) = {
11+
scope: PACKAGE
12+
flat_package: true
13+
};
14+
15+

‎resources/cache/dep2/hash2/proto/example3.proto

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
syntax = "proto3";
2+
3+
package proto;
4+
5+
import "google/protobuf/struct.proto";
6+
import "proto/example5.proto";
7+
8+
option (scalapb.options) = {
9+
scope: PACKAGE
10+
flat_package: true
11+
};
12+

‎resources/cache/dep2/hash2/proto/example4.proto

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
syntax = "proto3";
2+
3+
package proto;
4+
5+
import "google/protobuf/struct.proto";
6+
7+
option (scalapb.options) = {
8+
scope: PACKAGE
9+
flat_package: true
10+
};

‎resources/cache/dep2/hash2/proto/example5.proto

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
syntax = "proto3";
2+
3+
package proto;
4+
5+
import "google/protobuf/struct.proto";
6+
7+
option (scalapb.options) = {
8+
scope: PACKAGE
9+
flat_package: true
10+
};

‎resources/cache/dep3/hash3/proto/example.proto

+11
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
syntax = "proto3";
2+
3+
package proto;
4+
5+
import "proto/example2.proto";
6+
import "google/protobuf/descriptor.proto";
7+
8+
option (scalapb.options) = {
9+
scope: PACKAGE
10+
flat_package: true
11+
};

‎resources/cache/dep3/hash3/root/proto/root.proto

+11
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
syntax = "proto3";
2+
3+
package proto;
4+
5+
import "proto/example2.proto";
6+
import "google/protobuf/descriptor.proto";
7+
8+
option (scalapb.options) = {
9+
scope: PACKAGE
10+
flat_package: true
11+
};

0 commit comments

Comments
 (0)