Skip to content

Combine gem and annotations commands #1613

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 7 commits into from
Closed

Combine gem and annotations commands #1613

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
b277ed2
Deprecate and remove command files
bitwise-aiden Sep 5, 2023
b6e006b
Update RubyGems is a package manager for Ruby.
bitwise-aiden Sep 5, 2023
1d430c1
Add tests for new gem + annotation logic
bitwise-aiden Sep 5, 2023
53a0b42
Update README.md to reflect gem + annotation change
bitwise-aiden Sep 5, 2023
f016318
Address PR issues
bitwise-aiden Sep 7, 2023
746f095
Add code to remove old annotation file
bitwise-aiden Sep 7, 2023
01f3b68
Fix typo and wording
bitwise-aiden Sep 15, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
143 changes: 49 additions & 94 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,10 +40,9 @@ Tapioca makes it easy to work with [Sorbet](https://sorbet.org) in your codebase
* [Excluding a gem from RBI generation](#excluding-a-gem-from-rbi-generation)
* [Changing the strictness level of the RBI for a gem](#changing-the-strictness-level-of-the-rbi-for-a-gem)
* [Keeping RBI files for gems up-to-date](#keeping-rbi-files-for-gems-up-to-date)
* [Pulling RBI annotations from remote sources](#pulling-rbi-annotations-from-remote-sources)
* [Basic authentication](#basic-authentication)
* [Using a .netrc file](#using-a-netrc-file)
* [Changing the typed strictness of annotations files](#changing-the-typed-strictness-of-annotations-files)
* [Pulling RBI annotations from remote sources](#pulling-rbi-annotations-from-remote-sources)
* [Basic authentication](#basic-authentication)
* [Using a .netrc file](#using-a-netrc-file)
* [Generating RBI files for Rails and other DSLs](#generating-rbi-files-for-rails-and-other-dsls)
* [Keeping RBI files for DSLs up-to-date](#keeping-rbi-files-for-dsls-up-to-date)
* [Writing custom DSL compilers](#writing-custom-dsl-compilers)
Expand Down Expand Up @@ -72,7 +71,7 @@ $ tapioca help

Commands:
tapioca --version, -v # Show version
tapioca annotations # Pull gem RBI annotations from remote sources
tapioca annotations # [removed] Pull gem RBI annotations from remote sources
tapioca check-shims # Check duplicated definitions in shim RBIs
tapioca configure # Initialize folder structure and type checking configuration
tapioca dsl [constant...] # Generate RBIs for dynamic methods
Expand Down Expand Up @@ -192,6 +191,15 @@ Options:
# Default: development
[--halt-upon-load-error], [--no-halt-upon-load-error] # Halt upon a load error while loading the Rails application
# Default: true
[--annotations], [--no-annotations] # Include RBI annotations from remote source
# Default: true
[--annotations-sources=one two three] # URIs of the sources to pull gem RBI annotations from
# Default: ["https://raw.githubusercontent.com/Shopify/rbi-central/main"]
[--annotations-netrc], [--no-annotations-netrc] # Use .netrc to authenticate to private annotation sources
# Default: true
[--annotations-netrc-file=ANNOTATIONS_NETRC_FILE] # Path to .netrc file
[--annotations-auth=ANNOTATIONS_AUTH] # HTTP authorization header for private annotation sources
[--exclude-annotations=gem [gem ...]] # Excludes annotation for gem while generating RBI
-c, [--config=<config file path>] # Path to the Tapioca configuration file
# Default: sorbet/tapioca/config.yml
-V, [--verbose], [--no-verbose] # Verbose output for debugging purposes
Expand Down Expand Up @@ -321,70 +329,31 @@ Nothing to do, all RBIs are up-to-date.

This option can be used on CI to make sure the RBI files are always up-to-date and ensure accurate type checking. **Warning**: doing so will break your normal Dependabot workflow as every pull-request opened to bump a gem version will fail CI since the RBI will be out-of-date and will require you to manually run `bin/tapioca gems` to update them.

### Pulling RBI annotations from remote sources

Since Tapioca does not perform any type inference, the RBI files generated for the gems do not contain any type signatures. Instead, Tapioca relies on the community to provide high-quality, manually written RBI annotations for public gems.

To pull the annotations relevant to your project from the central repository, run the `annotations` command:
#### Pulling RBI annotations from remote sources

```shell
$ bin/tapioca annotations

Retrieving index from central repository... Done
Listing gems from Gemfile.lock... Done
Removing annotations for gems that have been removed... Nothing to do
Fetching gem annotations from central repository...
Since Tapioca does not perform any type inference, the RBI files generated for the gems do not contain any type signatures. Instead, Tapioca relies on the community to provide high-quality, manually written RBI annotations for public gems. These annotations are sourced and combined with the generated RBIs as a step during `tapioca gem`.

Fetched activesupport
created sorbet/rbi/annotations/activesupport.rbi

Done
```
By default, Tapioca will pull the annotations stored in the central repository located at https://github.com/Shopify/rbi-central. It is possible to use a custom repository by changing the value of the `--annotations-sources` option. For example if your repository is stored on GitHub:

<!-- START_HELP_COMMAND_ANNOTATIONS -->
```shell
$ tapioca help annotations

Usage:
tapioca annotations

Options:
[--sources=one two three] # URIs of the sources to pull gem RBI annotations from
# Default: ["https://raw.githubusercontent.com/Shopify/rbi-central/main"]
[--netrc], [--no-netrc] # Use .netrc to authenticate to private sources
# Default: true
[--netrc-file=NETRC_FILE] # Path to .netrc file
[--auth=AUTH] # HTTP authorization header for private sources
--typed, -t, [--typed-overrides=gem:level [gem:level ...]] # Override for typed sigils for pulled annotations
-c, [--config=<config file path>] # Path to the Tapioca configuration file
# Default: sorbet/tapioca/config.yml
-V, [--verbose], [--no-verbose] # Verbose output for debugging purposes

Pull gem RBI annotations from remote sources
```
<!-- END_HELP_COMMAND_ANNOTATIONS -->

By default, Tapioca will pull the annotations stored in the central repository located at https://github.com/Shopify/rbi-central. It is possible to use a custom repository by changing the value of the `--sources` options. For example if your repository is stored on Github:

```shell
$ bin/tapioca annotations --sources https://raw.githubusercontent.com/$USER/$REPO/$BRANCH
$ bin/tapioca gem --annotations-sources https://raw.githubusercontent.com/$USER/$REPO/$BRANCH
```

Tapioca also supports pulling annotations from multiple sources:

```shell
$ bin/tapioca annotations --sources https://raw.githubusercontent.com/$USER/$REPO1/$BRANCH https://raw.githubusercontent.com/$USER/$REPO2/$BRANCH
$ bin/tapioca gem --annotations-sources https://raw.githubusercontent.com/$USER/$REPO1/$BRANCH https://raw.githubusercontent.com/$USER/$REPO2/$BRANCH
```

#### Basic authentication
##### Basic authentication

Private repositories can be used as sources by passing the option `--auth` with an authentication string. For Github, this string is `token $TOKEN` where `$TOKEN` is a [personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token):
Private repositories can be used as sources by passing the option `--auth` with an authentication string. For GitHub, this string is `token $TOKEN` where `$TOKEN` is a [personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token):

```shell
$ bin/tapioca annotations --sources https://raw.githubusercontent.com/$USER/$PRIVATE_REPO/$BRANCH --auth "token $TOKEN"
$ bin/tapioca gem --annotations-sources https://raw.githubusercontent.com/$USER/$PRIVATE_REPO/$BRANCH --annotations-auth "token $TOKEN"
```

#### Using a .netrc file
##### Using a .netrc file

Tapioca supports reading credentials from a [netrc](https://www.gnu.org/software/inetutils/manual/html_node/The-_002enetrc-file.html) file (defaulting to `~/.netrc`).

Expand All @@ -396,38 +365,21 @@ machine raw.githubusercontent.com
password $TOKEN
```

where `$USERNAME` is your Github username and `$TOKEN` is a [personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token), then, if you run Tapioca with the `--netrc` option (enabled by default), your annotation requests should be authenticated properly.
where `$USERNAME` is your GitHub username and `$TOKEN` is a [personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token), then, if you run Tapioca with the `--annotations-netrc` option (enabled by default), your annotation requests should be authenticated properly.

The `--netrc-file` option can be specified to read from a file other than `~/.netrc`:
The `--annotations-netrc-file` option can be specified to read from a file other than `~/.netrc`:

```shell
$ bin/tapioca annotations --netrc-file /path/to/my/netrc/file
$ bin/tapioca gem --annotations-netrc-file /path/to/my/netrc/file
```

Similar to `--netrc-file`, you can also specify an alternative netrc file by using the `TAPIOCA_NETRC_FILE` environment variable:
Similar to `--annotations-netrc-file`, you can also specify an alternative netrc file by using the `TAPIOCA_NETRC_FILE` environment variable:

```shell
$ TAPIOCA_NETRC_FILE=/path/to/my/netrc/file bin/tapioca annotations
$ TAPIOCA_NETRC_FILE=/path/to/my/netrc/file bin/tapioca gem
```

Tapioca will first try to find the netrc file as specified by the `--netrc-file` option. If that option is not supplied, it will try the `TAPIOCA_NETRC_FILE` environment variable value. If that value is not supplied either, it will fallback to `~/.netrc`.

#### Changing the typed strictness of annotations files

Sometimes the annotations files pulled by Tapioca will create type errors in your project because of incompatibilities.
It is possible to ignore such files by switching their strictness level `--typed-overrides` option:

```shell
$ bin/tapioca annotations --typed-overrides gemA:ignore gemB:false
```

Or through the configuration file:

```yaml
annotations:
typed_overrides:
gemA: "ignore"
gemB: "false"
Tapioca will first try to find the netrc file as specified by the `--annotations-netrc-file` option. If that option is not supplied, it will try the `TAPIOCA_NETRC_FILE` environment variable value. If that value is not supplied either, it will fallback to `~/.netrc`.
```

### Generating RBI files for Rails and other DSLs
Expand Down Expand Up @@ -805,7 +757,6 @@ Loading Sorbet payload... Done
Loading shim RBIs from sorbet/rbi/shims... Done
Loading gem RBIs from sorbet/rbi/gems... Done
Loading gem RBIs from sorbet/rbi/dsl... Done
Loading annotation RBIs from sorbet/rbi/annotations... Done
Looking for duplicates... Done

Duplicated RBI for ::MyModel#title:
Expand All @@ -829,22 +780,20 @@ Usage:
tapioca check-shims

Options:
[--gem-rbi-dir=GEM_RBI_DIR] # Path to gem RBIs
# Default: sorbet/rbi/gems
[--dsl-rbi-dir=DSL_RBI_DIR] # Path to DSL RBIs
# Default: sorbet/rbi/dsl
[--shim-rbi-dir=SHIM_RBI_DIR] # Path to shim RBIs
# Default: sorbet/rbi/shims
[--annotations-rbi-dir=ANNOTATIONS_RBI_DIR] # Path to annotations RBIs
# Default: sorbet/rbi/annotations
[--todo-rbi-file=TODO_RBI_FILE] # Path to the generated todo RBI file
# Default: sorbet/rbi/todo.rbi
[--payload], [--no-payload] # Check shims against Sorbet's payload
# Default: true
-w, [--workers=N] # Number of parallel workers (default: auto)
-c, [--config=<config file path>] # Path to the Tapioca configuration file
# Default: sorbet/tapioca/config.yml
-V, [--verbose], [--no-verbose] # Verbose output for debugging purposes
[--gem-rbi-dir=GEM_RBI_DIR] # Path to gem RBIs
# Default: sorbet/rbi/gems
[--dsl-rbi-dir=DSL_RBI_DIR] # Path to DSL RBIs
# Default: sorbet/rbi/dsl
[--shim-rbi-dir=SHIM_RBI_DIR] # Path to shim RBIs
# Default: sorbet/rbi/shims
[--todo-rbi-file=TODO_RBI_FILE] # Path to the generated todo RBI file
# Default: sorbet/rbi/todo.rbi
[--payload], [--no-payload] # Check shims against Sorbet's payload
# Default: true
-w, [--workers=N] # Number of parallel workers (default: auto)
-c, [--config=<config file path>] # Path to the Tapioca configuration file
# Default: sorbet/tapioca/config.yml
-V, [--verbose], [--no-verbose] # Verbose output for debugging purposes

Check duplicated definitions in shim RBIs
```
Expand Down Expand Up @@ -918,11 +867,17 @@ gem:
rbi_max_line_length: 120
environment: development
halt_upon_load_error: true
annotations: true
annotations_sources:
- https://raw.githubusercontent.com/Shopify/rbi-central/main
annotations_netrc: true
annotations_netrc_file: ''
annotations_auth: ''
exclude_annotations: []
check_shims:
gem_rbi_dir: sorbet/rbi/gems
dsl_rbi_dir: sorbet/rbi/dsl
shim_rbi_dir: sorbet/rbi/shims
annotations_rbi_dir: sorbet/rbi/annotations
todo_rbi_file: sorbet/rbi/todo.rbi
payload: true
workers: 1
Expand Down
1 change: 0 additions & 1 deletion lib/tapioca.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,6 @@ class Error < StandardError; end
DEFAULT_GEM_DIR = T.let("#{DEFAULT_RBI_DIR}/gems", String)
DEFAULT_SHIM_DIR = T.let("#{DEFAULT_RBI_DIR}/shims", String)
DEFAULT_TODO_FILE = T.let("#{DEFAULT_RBI_DIR}/todo.rbi", String)
DEFAULT_ANNOTATIONS_DIR = T.let("#{DEFAULT_RBI_DIR}/annotations", String)

DEFAULT_OVERRIDES = T.let(
{
Expand Down
56 changes: 41 additions & 15 deletions lib/tapioca/cli.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,6 @@ def init
# We need to make sure that trackers stay enabled until the `gem` command is invoked
Runtime::Trackers.with_trackers_enabled do
invoke(:configure)
invoke(:annotations)
invoke(:gem)
end

Expand Down Expand Up @@ -250,6 +249,28 @@ def dsl(*constant_or_paths)
type: :boolean,
desc: "Halt upon a load error while loading the Rails application",
default: true
option :annotations,
type: :boolean,
desc: "Include RBI annotations from remote source",
default: true
option :annotations_sources,
type: :array,
default: [CENTRAL_REPO_ROOT_URI],
desc: "URIs of the sources to pull gem RBI annotations from"
option :annotations_netrc,
type: :boolean,
default: true,
desc: "Use .netrc to authenticate to private annotation sources"
option :annotations_netrc_file, type: :string, desc: "Path to .netrc file"
option :annotations_auth,
type: :string,
default: nil,
desc: "HTTP authorization header for private annotation sources"
option :exclude_annotations,
type: :array,
banner: "gem [gem ...]",
desc: "Excludes annotation for gem while generating RBI",
default: []
def gem(*gems)
set_environment(options)

Expand All @@ -267,6 +288,14 @@ def gem(*gems)
raise MalformattedArgumentError, "Option '--verify' must be provided without any other arguments" if verify
end

if !options[:annotations_netrc] && options[:annotations_netrc_file]
raise Thor::Error, set_color(
"Options `--no-annotations-netrc` and `--annotations-netrc-file` can't be used together",
:bold,
:red,
)
end

command_args = {
gem_names: all ? [] : gems,
exclude: options[:exclude],
Expand All @@ -278,12 +307,17 @@ def gem(*gems)
file_header: options[:file_header],
include_doc: options[:doc],
include_loc: options[:loc],
include_annotations: options[:annotations],
include_exported_rbis: options[:exported_gem_rbis],
number_of_workers: options[:workers],
auto_strictness: options[:auto_strictness],
dsl_dir: options[:dsl_dir],
rbi_formatter: rbi_formatter(options),
halt_upon_load_error: options[:halt_upon_load_error],
annotations_sources: options[:annotations_sources],
annotations_auth: options[:annotations_auth],
annotations_netrc_file: netrc_file(options),
excluded_annotations: options[:exclude_annotations],
}

command = if verify
Expand All @@ -302,7 +336,6 @@ def gem(*gems)
option :gem_rbi_dir, type: :string, desc: "Path to gem RBIs", default: DEFAULT_GEM_DIR
option :dsl_rbi_dir, type: :string, desc: "Path to DSL RBIs", default: DEFAULT_DSL_DIR
option :shim_rbi_dir, type: :string, desc: "Path to shim RBIs", default: DEFAULT_SHIM_DIR
option :annotations_rbi_dir, type: :string, desc: "Path to annotations RBIs", default: DEFAULT_ANNOTATIONS_DIR
option :todo_rbi_file, type: :string, desc: "Path to the generated todo RBI file", default: DEFAULT_TODO_FILE
option :payload, type: :boolean, desc: "Check shims against Sorbet's payload", default: true
option :workers, aliases: ["-w"], type: :numeric, desc: "Number of parallel workers (default: auto)"
Expand All @@ -311,7 +344,6 @@ def check_shims
gem_rbi_dir: options[:gem_rbi_dir],
dsl_rbi_dir: options[:dsl_rbi_dir],
shim_rbi_dir: options[:shim_rbi_dir],
annotations_rbi_dir: options[:annotations_rbi_dir],
todo_rbi_file: options[:todo_rbi_file],
payload: options[:payload],
number_of_workers: options[:workers],
Expand All @@ -320,7 +352,7 @@ def check_shims
command.run
end

desc "annotations", "Pull gem RBI annotations from remote sources"
desc "annotations", "[removed] Pull gem RBI annotations from remote sources"
option :sources,
type: :array,
default: [CENTRAL_REPO_ROOT_URI],
Expand All @@ -335,18 +367,12 @@ def check_shims
desc: "Override for typed sigils for pulled annotations",
default: {}
def annotations
if !options[:netrc] && options[:netrc_file]
raise Thor::Error, set_color("Options `--no-netrc` and `--netrc-file` can't be used together", :bold, :red)
end

command = Commands::Annotations.new(
central_repo_root_uris: options[:sources],
auth: options[:auth],
netrc_file: netrc_file(options),
typed_overrides: options[:typed_overrides],
raise Thor::Error, set_color(
"`tapioca annotation` has been removed." \
"The functionality has been merged into `tapioca gem`. Please use that instead.",
:bold,
:red,
)

command.run
end

map ["--version", "-v"] => :__print_version
Expand Down
1 change: 0 additions & 1 deletion lib/tapioca/commands.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@ module Tapioca
module Commands
autoload :Command, "tapioca/commands/command"
autoload :CommandWithoutTracker, "tapioca/commands/command_without_tracker"
autoload :Annotations, "tapioca/commands/annotations"
autoload :CheckShims, "tapioca/commands/check_shims"
autoload :AbstractDsl, "tapioca/commands/abstract_dsl"
autoload :DslCompilerList, "tapioca/commands/dsl_compiler_list"
Expand Down
Loading