[Tracking] New pull request assignment proposal

[apiraino]

This issue tracks the progress of the implementation of a new workflow for assigning pull requests to the Rust project contributors.

2024-02-20 UPDATE: After other opinions came in, a new plan was discussed. Updates below.

2023-12-12 UPDATE: After another meeting with T-infra, we agreed a second plan detailing how to split the work further. The new plan is detailed below.

2023-11-08 UPDATE: After a meeting with T-infra, it was remarked that the first implementation (#1719) was hard to review so we agreed a plan to split into smaller bites (#1745)

• Step 1 (Workflow for tracking PRs assignment #1773 )
  • add the DB table, strip down DB fields to user_id,assigned_prs
  • Initial DB table population with a one-off job
  • no migration/import/sync tooling
• Step 2 (Add Zulip integration to query PRs assignment #1778)
  • Implement a command to query from Zulip the number of assigned PRs.
• Step 3 (Add webhook handler to update PR workload queues #1781)
  • Add webhook handler to keep track of PR assignments
  • This has been enabled on the rust-lang/rust git repo in https://togithub.com/rust-lang/rust/pull/122383 using the triagebot.toml config key [pr_tracking]
• Step 4 (Add work queue tracking in triagebot DB #1879)
  • Add a new nullable field MAX_PR in the DB table
  • The starting value for MAX_PR is "null" i.e. no enforcement on PRs assignment
  • Update documentation on forge.r-l.o
• Step 5 (Implement setting review assignment limit #1919)
  • Modify the zulip triagebot command work show to also return MAX_PR
  • Add a Zulip command work set-pr-limit <num> to allow people updating their MAX_PR
  • Update documentation on forge.r-l.o
• Step 6 (Take review preferences into account when determining reviewers #1947)
  • Change the PRs assignment heuristic: filter out team members that have num_assigned_prs >= MAX_PR (with MAX_PR null will ignore this limit)
  • (?) Enable the new assignment if the a new key [review_prefs] in the triagebot.toml
• Step X
  • MAX_PR will start to be enforced (add the setting it in the rust-lang/rust triagebot.toml)
  • People are then asked to set this value
  • if a PR is not assigned because everyone is at num_assigned_prs == MAX_PR, people can grab that PR manually
  • Self-assign respects MAX_PR either. Post a comment on the PR suggesting the user to increase their MAX_PR and self-assign again
• Further work (some points to be clarified)
  • When an PR is closed/merged it stays assigned to a reviewer. Should be removed from the work queue returned by the Zulip triagebot
  • [Question] Add a filter to enable this feature only for members of a number of teams?
  • HTTP endpoint to sync assignment counts with GitHub
  • Tackle the preferences: HTML backoffice or chat-based? A web backoffice would be (imo) nice but the auth part is tricky to get approved (see comment).
  • how to sync PRs when/if bors webhook fail?
  • backoffice: add a button to the contributor to resync their own situation

---

Summary

The current pull request assignment is basically randomly assigned among team members. The new proposed workflow has the following features:

• each Rust project contributor belonging to a team (I'd start will a small cohort of testers) can set their own review capacity and time off using a web backoffice
• the web backoffice will persiste these data in a DB
• everytime a pull request assignment is invoked with r? <team> or <team_member> the triagebot will check the current availability and workload of the candidates and assign the pull request to the team member less busy, excluding automatically those that are on time off in that moment.

A slightly old design document is at this HackMD link.

Permissions and Authorization

This backoffice could be thought as a simple form listing all team members. It is served by the triagebot (from triagebot.infra.rust-lang.org). Access to this backoffice is gated by a GitHub App. Users allowed to this backoffice:

1. must be logged into GitHub
2. must agree to the share some details with a GitHub App (only to read their profile)
3. must be a Rust project team member
4. the team they belong must be whitelisted in the env var NEW_PR_ASSIGNMENT_TEAMS

Team leaders are considered administrators and can see the preferences of every team member. Normal team members can only see their own preferences and those of team members that agree to share theirs within the team.

DEPLOYMENT

There will a few env vars to enable the new PR assignment and retrict access to this backoffice.

The idea would be to iterate on this pull request and iron out all the wrinkles visible without deployment:

• Deploy this changes but keep the new PR assignment DISABLED. The web backoffice will still be accessible.
• Team members can access that, start playing with with it fill their preferences
• When we are all confident about these changes, we can flip the env variable USE_NEW_PR_ASSIGNMENT and have the team listed in NEW_PR_ASSIGNMENT_TEAMS use it for real.
• After a first period of tests, add more teams