Add 2026-01 SLC crawl #399
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lkacenja
commented
Jan 27, 2026
|
|
||
| def get_crawl_status_display | ||
| if document_status == DOCUMENT_STATUS_NEW && last_crawl_date.present? && last_crawl_date.after?(1.week.ago) | ||
| if document_status == DOCUMENT_STATUS_NEW && last_crawl_date.present? && last_crawl_date.after?(1.month.ago) |
Contributor
Author
There was a problem hiding this comment.
This anticipated much more frequent crawling. By the time the SLC folks log in the "new" tag won't be visible anymore.
lkacenja
commented
Jan 27, 2026
|
|
||
|
|
||
| def get_file(url: str, output_path: str, wait_to_retry: int = 1000) -> str: | ||
| def get_file(url: str, output_path: str, wait_to_retry: int = 2) -> str: |
Contributor
Author
There was a problem hiding this comment.
This was absurdly high. I think I thought time.sleep(...) took milliseconds.
allisonmorgan
approved these changes
Jan 27, 2026
Contributor
allisonmorgan
left a comment
allisonmorgan
left a comment
There was a problem hiding this comment.
This looks great to me. I confirmed I could run import the new data and run the app locally. Thanks for tackling these complex crawling issues! 🚀
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
It's time to update SLC's documents. They are going to use the tool to assist in their efforts starting in early February.
When I tried to run the crawler, I got 403 responses from the SLC site on all urls. I added the adaptive requesting technique that I implemented in the document inference component to the crawler. That seems to have fixed it.
This PR includes:
What additional steps are required to test this branch locally?
bundler installandbundler clean --forcerails db:drop ; rails db:migrate; rails db:setupbin/rake documents:import_documents["1", "db/seeds/site_documents_2025_10_07/salt_lake_city.csv", true]bin/rake documents:import_documents["1", "db/seeds/site_documents_2026_01_22/salt_lake_city.csv", true]Are there any areas you would like extra review?
Here is a hex notebook that explores the changes between releases a bit. It may be helpful in creating transparency around the new data. The first table is most reliable.
Are there any rake tasks to run on production?
Yes, the new documents will need to be imported.