Skip to content

Conversation

@irfan-dahir
Copy link
Contributor

@irfan-dahir irfan-dahir commented Jun 29, 2022

The crux of this PR is to improve response performance and to help improve #177

I'll be focusing on implementing the following:

  • Executing update resource/cache after serving response (so the requester does not have to wait for the server to parse MAL first)
  • Stale response will be served until the updater job completes
  • Improving code flow

The following flow is to be implemented for Resources:

graph TD;
    X[Start] --> A
    A[Query DB] --> B{Resource Exists?};
    B -->|Yes| C{Is Expired?}
    B -->|No| D{Scrape MAL}
    C -->|Yes| G[dispatch UpdateResource]
    C -->|No| Z[Jikan returns Resource]
    D -->|MAL returns error| F[Jikan returns error]
    D -->|MAL page exists and parsed data is returned| E[Insert as new resource]
    E --> A
    G--> Z
Loading

The following flow is to be implemented for Cache:

graph TD;
    X[Start] --> B{Cache Exists?}
    B -->|Yes| C{Is Expired?}
    B -->|No| D{Scrape MAL}
    C -->|Yes| G[dispatch UpdateCache]
    C -->|No| Z[Jikan returns Cache]
    D -->|MAL returns error| F[Jikan returns error]
    D -->|MAL page exists and parsed data is returned| E[Insert as new cache]
    E --> B
    G--> Z
Loading

There will be 2 Updater Jobs

  1. UpdateResource: Uses mal_id as unique ID and updates through query and scraper - both available in defined in: App\Anime, App\Manga, App\Character, and App\Person classes
  2. UpdateCache: Uses request_hash as unique ID and updates by passed table name and scraper closure

Jobs utilize Queues and queues are stored in MongoDB and utilize Redis.
Pending Jobs table: jobs
Failed Jobs table: jobs_failed

This PR also adds new migrations for these tables which can be added by
php artisan migrate --path=/database/migrations/2022_06_29_231146_create_jobs_table.php
php artisan migrate --path=/database/migrations/2022_06_29_231217_create_jobs_failed_table.php


Running Queues

php artisan queue:work --queue=high,default

Notable changes

  • The internal last modified property has been changed from modifiedAt to updated_at
  • The internal created property has been changed from createdAt to created_at

Todo

  • Create UpdateResource
  • Create UpdateCache
  • Configure delay between jobs to prevent rate limiting
  • Implement UpdateResource for Anime, Manga, Character and Person
  • Implement UpdateCache for all cache based requests

⚠️ This will require resolving merge conflicts with #227 to include the new framework

@irfan-dahir irfan-dahir added this to the REST 4.0 milestone Jun 29, 2022
@irfan-dahir irfan-dahir self-assigned this Jun 29, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: On Hold

Development

Successfully merging this pull request may close these issues.

3 participants