feat: Implement background crawling jobs (6/6) - #62
Conversation
yoomlam
left a comment
yoomlam
left a comment
There was a problem hiding this comment.
Wondering what behavior is desired if anything fails
| tasks = [run_pipeline(pipeline, job) for job in jobs] | ||
| results = await asyncio.gather(*tasks) | ||
|
|
||
| # Combine jobs with their results and convert to SupportEntry objects |
There was a problem hiding this comment.
If one of the jobs fail, do we want SupportEntry records from successful jobs to be created?
| for job, support_entries in results: | ||
| logger.info("Saving results for domain: %s", job.domain) | ||
| save_job_results(db_session, job, support_entries.values()) |
There was a problem hiding this comment.
What happens if a job fails? Is the effect desirable?
There was a problem hiding this comment.
Updated: if a job fails, an error is logged but other jobs continue and will save their results to the db.
| with config.db_session() as db_session, db_session.begin(): | ||
| process_all_jobs(db_session) |
There was a problem hiding this comment.
If any exception is raised, nothing should be commited to the DB.
There was a problem hiding this comment.
👍 I double-checked that's the behavior of db_session.begin() in a context manager (if process_all_jobs were to raise an uncaught exception, the transaction would be rolled back rather than committed)
|
@yoomlam thanks for the feedback! I update the job filtering logic to happen in a query, and updated the parallel processing to let individual jobs fail while still saving the results from any jobs that succeed. (I also switched back to ChatPromptBuilder!) |
| continue | ||
|
|
||
| # Deduplicate by name | ||
| support_entries = {entry["name"]: SupportEntry(**entry) for entry in result} |
There was a problem hiding this comment.
Noting that this deduplicates within a job, not across jobs.
3 participants
Ticket
https://navalabs.atlassian.net/browse/DST-1251
https://navalabs.atlassian.net/browse/DST-1252
Changes
Context for reviewers
gpt-5and high reasoning effort is quite slow, so you may want to manually modify the code togpt-5-miniandmedium. (I don't recommend going lower because then the model is unlikely to respond with valid JSON.)Testing
See entries in database:
Re-run
make process-crawl-jobsand confirm it does nothing because the crawl interval is every 24 hours:Preview environment for frontend
♻️ Environment destroyed ♻️
Preview environment for app
♻️ Environment destroyed ♻️