MONGOID-5908 Run upcoming Ruby and Rails builds on a bi-weekly schedule (#6166)

Author: comandeo-mongo

.evergreen/README.md

@@ -0,0 +1,52 @@
+# Evergreen configuration
+
+This directory contains configuration and scripts used to run Mongoid's test
+suite in Evergreen, MongoDB's continuous integration system.
+
+`config.yml` is generated. Do not edit it by hand. Edit the ERB templates under
+`config/` and regenerate with:
+
+```
+ruby .evergreen/update-evergreen-configs
+```
+
+
+## Scheduled upcoming-version builds (MONGOID-5908)
+
+Two build variants run the test suite against the upcoming, in-development
+versions of Ruby and Rails, so that incompatibilities are caught before those
+versions ship:
+
+- `ruby-dev` — the main test suite against `ruby-dev` (provided by
+  `mongo-ruby-toolchain`).
+- `rails-master` — the main test suite against Rails `master`.
+
+Both are defined in `config/variants.yml.erb`. Two properties matter:
+
+- They are not tagged `pr`, so they never run on pull requests. They only run
+  on the `master` waterfall.
+- Their `test` task carries `batchtime: 20160` (14 days), so Evergreen
+  activates each at most once every two weeks instead of on every commit.
+
+### Failure notification
+
+Failures of these scheduled runs should open a Jira issue in the `MONGOID`
+project. This is not expressed in `config.yml`; it is an Evergreen project
+notification, configured once by a project admin in the Evergreen UI:
+
+1. Open the `mongoid` project in Evergreen, go to
+   **Settings -> Notifications** (project subscriptions).
+2. Add a subscription:
+   - **Event**: a task finishes -> the task fails.
+   - **Scope**: limit to the build variants, using a regex that matches the
+     `ruby-dev` and `rails-master` variant display names (they contain
+     `ruby-dev` and `Rails master`).
+   - **Action**: Create a Jira issue.
+   - **Project**: `MONGOID`; pick an appropriate issue type (e.g. `Task`/`Bug`).
+3. Evergreen fills the issue with the failing task, the build variant (which
+   names the Ruby or Rails version), the version/commit, and links to the
+   failed tasks, so the report includes the version, commit hash, and the list
+   of failures.
+
+Because this lives in project settings rather than the repository, it must be
+recreated if the project is reconfigured.

.evergreen/config.yml

@@ -743,6 +743,9 @@ buildvariants:
   tasks:
     - name: "test"
 
+# MONGOID-5908: test master against the upcoming Ruby (ruby-dev). Runs off-PR
+# (no "pr" tag) on a ~2-week cadence. Failures are reported as MONGOID Jira
+# issues via an Evergreen project notification; see .evergreen/README.md.
 - matrix_name: "ruby-dev"
   matrix_spec:
     ruby: ["ruby-dev"]
@@ -753,6 +756,7 @@ buildvariants:
   display_name: "${ruby}, ${driver}, ${mongodb-version}, ${topology}"
   tasks:
     - name: "test"
+      batchtime: 20160 # run at most once every 14 days
 
 - matrix_name: "driver-upcoming"
   matrix_spec:
@@ -811,6 +815,9 @@ buildvariants:
   tasks:
     - name: "test"
 
+# MONGOID-5908: test master against the upcoming Rails (rails master). Runs
+# off-PR (no "pr" tag) on a ~2-week cadence. Failures are reported as MONGOID
+# Jira issues via an Evergreen project notification; see .evergreen/README.md.
 - matrix_name: "rails-master"
   matrix_spec:
     ruby: ["ruby-4.0"]
@@ -823,6 +830,7 @@ buildvariants:
   display_name: "${rails}, ${driver}, ${mongodb-version} (FLE ${fle})"
   tasks:
     - name: "test"
+      batchtime: 20160 # run at most once every 14 days
 
 - matrix_name: "rails-8"
   matrix_spec:

.evergreen/config/variants.yml.erb

@@ -94,6 +94,9 @@ buildvariants:
   tasks:
     - name: "test"
 
+# MONGOID-5908: test master against the upcoming Ruby (ruby-dev). Runs off-PR
+# (no "pr" tag) on a ~2-week cadence. Failures are reported as MONGOID Jira
+# issues via an Evergreen project notification; see .evergreen/README.md.
 - matrix_name: "ruby-dev"
   matrix_spec:
     ruby: ["ruby-dev"]
@@ -104,6 +107,7 @@ buildvariants:
   display_name: "${ruby}, ${driver}, ${mongodb-version}, ${topology}"
   tasks:
     - name: "test"
+      batchtime: 20160 # run at most once every 14 days
 
 - matrix_name: "driver-upcoming"
   matrix_spec:
@@ -162,6 +166,9 @@ buildvariants:
   tasks:
     - name: "test"
 
+# MONGOID-5908: test master against the upcoming Rails (rails master). Runs
+# off-PR (no "pr" tag) on a ~2-week cadence. Failures are reported as MONGOID
+# Jira issues via an Evergreen project notification; see .evergreen/README.md.
 - matrix_name: "rails-master"
   matrix_spec:
     ruby: ["ruby-4.0"]
@@ -174,6 +181,7 @@ buildvariants:
   display_name: "${rails}, ${driver}, ${mongodb-version} (FLE ${fle})"
   tasks:
     - name: "test"
+      batchtime: 20160 # run at most once every 14 days
 
 - matrix_name: "rails-8"
   matrix_spec: