Ability to disable rules for specific repositories

[QuintinWillison]

What?

We should take inspiration from how this achieved with other linting and analysis tools (e.g. ESLint's eslint-disable).

I think it's important that this tool has 'default opinions' around best practice - i.e. what to check and how (fail, warn or pass). However, there are multiple levels at which it would be useful to override the defaults:

• for individual repositories
• for groups of repositories:
  • public vs private
  • topic
• for entire orgs (once we've implemented Audit our other orgs - including ably-forks and ably-labs #6)
• for all orgs (once we've implemented Audit our other orgs - including ably-forks and ably-labs #6)

For Individual Repositories

There may be specific repositories which can't adhere to certain rules. There could be a number of reasons:

• the team working on the repository has a different workflow which means some rules, inferring particular discipline or team practices, can't be enforced
• the repository is in early stages of incubation and therefore unable to conform on all fronts for the moment
• the repository has downstream dependencies whereby conforming to all rules would break behaviours required by the needs of those downstream dependencies

For Groups of Repositories

Public vs Private

There are some rules which we are happier to relax for repositories that aren't in the open source - for example around merge button behaviour, given some teams prefer to rebase often on feature branches including at the stage they land them into their main branch.

Topic

In the ably org we're using topics primarily to assist external visitors who need to make sense of what groups of repositories we're offering (e.g. the client-library topic).

In some cases we should perhaps prefer to configure groups of repository by topic (new topics to be created for this purpose, perhaps especially for private repositories) rather than overriding rules on a per-repository basis.

For Entire Orgs

Once we implement #6 then this tool will run across all of Ably's orgs: ably, ably-forks and ably-labs.

The ably-labs org is a good example of a space where we will likely want to relax many rules by default. It's designed to be a playground, with minimal friction to bias for action.

For All Orgs

In other words - at 'tool level'. The idea with this is that if others decide to adopt or fork this tooling for using with their own orgs then they'll be able to override our the defaults at the highest level. If we (Ably) need the defaults to change then the expectation is probably that we would modify the source code, thus changing the tool defaults (a.k.a. 'default opinions').

How?

We could allow individual repositories to include a magic file somewhere that allows them to override audit tool rules, however that is somewhat counter to the goals of this tool which is to provide global oversight with minimal footprint on the processes of individual teams and the repositories they are working on. Therefore, we will not implement anything that enables rules to be disabled via source code content in the repositories being monitored.

We could bake rule overrides into the source code of this repository (the tool itself). This is not viable once we open this repository up as open source because others who may adopt or fork it will have their own requirements. In other words, this tool and all source code in this repository should be agnostic as to the company and/or GitHub orgs that it is running on behalf of.

We should probably implement the capability for this tool to source the rules from a central location (or, perhaps, locations - e.g. per-org). Perhaps a repository called something like repository-audit-rules. It might contain a single JSON file rules.json with a layout something like:

{
  "organizations": {
    "ably": {
      "repositories": {
        "some-repository-name": {
          "rules": {
            "merge-button/allow-rebase-merging": true,
          },
        },
      },
    },
  },
}

Where alongside the repositories key there could also be keys in that org's map for topics, publicRepositories and privateRepositories. i.e. this would be an extensible, logical structure.