Add Cloud Monitoring alerts for abnormal request volume

[skapadia]

Motivation

The bridge now has rate limiting
(#6),
max-instances=1, and the full stack of application-layer defenses. What
we don't have is visibility: if someone starts hammering /webhook,
we won't know about it until we check logs or notice a Cloud Run bill
larger than expected.

Cloud Run already exports run.googleapis.com/request_count for every
service, and Cloud Monitoring alerting policies against that metric are
free. A pair of well-tuned alerts gives us a low-cost early-warning
signal without needing to move to Cloud Armor (Option 2 in
docs/security-and-network-options.md).

Implementation approach

Two separate alerting policies, each with its own rationale:

Alert	Metric filter	Threshold	What it tells us
Total volume spike	service_name = zoom-recording-bridge	> 500 requests / 24h	Someone is hammering the endpoint — any status
4xx spike	service_name = zoom-recording-bridge, response_code_class = 4xx	> 100 requests / 24h	Specifically forged or rate-limited traffic (real Zoom traffic is near-100% 2xx)

Context for the thresholds: Chariot's real Zoom volume is a handful of
events per day. 500/day is a conservative abuse ceiling that won't false
positive from occasional bot scans; 100/day on 4xx is a tighter early
signal since legitimate traffic almost never produces 4xx.

Definition as code

Capture both policies as YAML under ops/monitoring/ (new directory) so
they're reproducible and committable. Apply via
gcloud alpha monitoring policies create --policy-from-file=....

Proposed file layout:

ops/monitoring/
  README.md                          # How to apply / update
  request-volume-alert.yaml          # Total volume > 500/24h
  request-4xx-alert.yaml             # 4xx > 100/24h
  notification-channel-email.yaml    # Reusable email channel

The email notification channel is created once and referenced by both
policies via its full resource name
(projects/.../notificationChannels/...). YAML can use the channel
display name at apply time; the gcloud CLI resolves it.

Why MQL, not threshold-on-rate

Cloud Monitoring's UI makes "rate per hour" easy, but strict "total per
24h" requires MQL (Monitoring Query Language). Example:

fetch cloud_run_revision
| metric 'run.googleapis.com/request_count'
| filter resource.service_name = 'zoom-recording-bridge'
| align rate(1h)
| group_by 24h, [sum(value)]
| condition val() > 500

Use this for both policies; adjust threshold and add
response_code_class filter for the 4xx variant.

Test plan

Manual trigger (pre-apply smoke test)

Temporarily lower the volume alert threshold to something tiny (e.g. 10)
and curl-loop the production endpoint to generate traffic:

for i in {1..20}; do
  curl -s -o /dev/null \
    https://zoom-recording-bridge-251159997459.us-east1.run.app/webhook
done

Expected: email lands within 2–3 minutes. Reset the threshold back to
500 after confirming. Do this outside a working meeting so real Zoom
webhooks aren't competing with the test.

Absent-data check (optional follow-up)

Consider a third policy that alerts on absence of requests for > 7
days ("is Zoom's webhook still firing at all?"). Not in scope for this
issue, but worth noting as a future follow-up. Out of scope because it
requires care to avoid false positives during quiet weeks.

Tuning period

After apply, run for 2–4 weeks without acting on alerts. If either
policy fires on legitimate traffic, raise the threshold. If neither
fires over a period that includes some observable abuse attempts,
consider lowering.

Documentation updates

• docs/security-and-network-options.md — add a note under "What this
  posture is missing" that volume alerting now exists as a detective
  control even though preventive network-layer controls still don't.
• ops/monitoring/README.md (new) — how to apply the policies, how to
  update thresholds, how to add a new notification channel.

Out of scope

• Slack / PagerDuty channels. Email first; add richer channels when
  we have a clearer signal that the alerts are worth routing to a
  channel that pages someone.
• Absent-data alert. Real but separate concern; file as follow-up.
• Per-endpoint breakdown. There's only one real endpoint.
• Log-based metrics. Request count is sufficient; custom log
  metrics add complexity for no gain here.