Scraper v2

[bbrewington]

Idea with this new scraper version is use the BigQuery SUBSCRIPTION table (fields CaseNumber / PhoneNumber) as our list of what to scrape. We only ever want to scrape case info for cases people have signed up to

Here's the flow:

1. User signs up for subscriptions on a case number
2. Query bigquery table(s) to find case
3. If:
   • Case found --> use info from BigQuery
   • Case not found --> search online site (with large date range) to get hearing info, add to BigQuery, then use info from BigQuery

[BennyJW]

When the scraper runs on Cloud Run collecting 90 days of data it takes 1 minute 20 seconds. That seems like a fairly minimal load. We can add techniques to reduce and spread out the load, like waiting 10 seconds between requests. It does not seem like we need to revamp the architecture and add significant complexity over concerns about bot detection.