Improved Resource and Cache Updater #248
Draft
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The crux of this PR is to improve response performance and to help improve #177
I'll be focusing on implementing the following:
The following flow is to be implemented for Resources:
graph TD; X[Start] --> A A[Query DB] --> B{Resource Exists?}; B -->|Yes| C{Is Expired?} B -->|No| D{Scrape MAL} C -->|Yes| G[dispatch UpdateResource] C -->|No| Z[Jikan returns Resource] D -->|MAL returns error| F[Jikan returns error] D -->|MAL page exists and parsed data is returned| E[Insert as new resource] E --> A G--> ZThe following flow is to be implemented for Cache:
graph TD; X[Start] --> B{Cache Exists?} B -->|Yes| C{Is Expired?} B -->|No| D{Scrape MAL} C -->|Yes| G[dispatch UpdateCache] C -->|No| Z[Jikan returns Cache] D -->|MAL returns error| F[Jikan returns error] D -->|MAL page exists and parsed data is returned| E[Insert as new cache] E --> B G--> ZThere will be 2 Updater Jobs
mal_idas unique ID and updates through query and scraper - both available in defined in:App\Anime,App\Manga,App\Character, andApp\Personclassesrequest_hashas unique ID and updates by passed table name and scraper closureJobs utilize Queues and queues are stored in MongoDB and utilize Redis.
Pending Jobs table:
jobsFailed Jobs table:
jobs_failedThis PR also adds new migrations for these tables which can be added by
php artisan migrate --path=/database/migrations/2022_06_29_231146_create_jobs_table.phpphp artisan migrate --path=/database/migrations/2022_06_29_231217_create_jobs_failed_table.phpRunning Queues
php artisan queue:work --queue=high,defaultNotable changes
modifiedAttoupdated_atcreatedAttocreated_atTodo