Import YouTube history into the rate later list

[sigmike]

Description

Initial draft to import your watch history into your rate later list.

The bulk import is not done yet, so the import is very slow, but it works.

Remaining to do:

• bulk import
• improve the style of the status box while importing
• translate texts
• improve behavior when the user is not connected (currently it marks all imports as failed)

[sigmike]

Also, I added a button to the extension dropdown but it only opens a tab to the watch history, so I'm not sure it's useful.

[GresilleSiffle]

I'm testing the current version. It works well thank you 👍👍

Here are few things that could improve the user experience.

(1) Do not allow the users to click on the import button if they are not connected to Tournesol. For non-authenticated users, all import attempts will fail. For those users, it may look like the process is simply not working : /

Suggestions:

• rename the button "Import history to Tournesol" (maybe it's more engaging than the current title? even if the current one is correct)
• for non-authenticated users this button could open the login modal

(2) Add an indication showing that the process in ongoing. Because the numbers of imported videos are not updated in real time, displaying a message could prevent the users to not interrupt the process by closing the tab or refreshing.

Suggestions:

• a simple title could be enough for now
• later we could add an animation

Exemple with a random title (could be "Processing, please wait..." or similar)

(3) Make the overlay a bit bigger, to clearly show that something is going on. I hope the users will understand it's not a dialogue box expecting an action like confirm/cancel, but more a control panel.

Also, I added a button to the extension dropdown but it only opens a tab to the watch history, so I'm not sure it's useful.

I think it can be good to display the button, it shows that the feature exists. Maybe we could rename it "Import history?".

[GresilleSiffle]

The videos that are already in the rate-later list appear as successfully imported, sounds good to me.

[GresilleSiffle]

The bulk import is not done yet, so the import is very slow, but it works.

Should we add the bulk import before releasing a first version of this feature? What do you think @amatissart ?

Without the bulk import, the API allows authenticated users to make 120 requests per minute. This is very tight for an import process. If we decide to release a first version without the bulk import, we may want to manually control the pace of the import.

I just lock myself out of tournesol.app by clicking five times on the import button. After that all import attempts fail : ]

[sigmike]

rename the button "Import history to Tournesol"

OK

for non-authenticated users this button could open the login modal

OK I'll try that.

(2) Add an indication showing that the process in ongoing.
(3) Make the overlay a bit bigger, to clearly show that something is going on.

The title is indeed needed. A loading animation would really convey that something is happening. Maybe one of the CC0 pure CSS animations at https://loading.io/css/?

Or maybe even a description of the process and a "start" button?

Should we add the bulk import before releasing a first version of this feature?

It doesn't seem very difficult to add the bulk import, so it's probably better to wait for it.

I think I'll make the current API also accept an array of IDs as input. But if you'd rather have a new entry point, tell me.

I don't know how heavy this process is to the backend. If it is, we may want a custom rate limit for this specific call (and that may warrant a specific entry point). The current version of the feature will do a call 1 second after the last request completed.

[sigmike]

I made all the changes. It should work well, with a bulk import.

I ended up using a specific API endpoint, but it can probably be merged into the existing create endpoint.

[GresilleSiffle]

Thank you very much, I'll have a look tomorrow.

[GresilleSiffle]

Hello @sigmike

I created a new PR #2063 that targets this branch to add the following features:

• stop the regular calls to captureStep when the end of the history is reached (your feedback is welcome)
• display a "Import complete!" message to the users when the import is complete

The title is indeed needed. A loading animation would really convey that something is happening. Maybe one of the CC0 pure CSS animations at https://loading.io/css/?

The animation you chose is great.

Or maybe even a description of the process and a "start" button?

I think it's not necessary because the users already need to click on the Import button to trigger the import. One click seems enough to me.

I think I'll make the current API also accept an array of IDs as input. But if you'd rather have a new entry point, tell me.

Adding a new entry point is good for now.

As @amatissart said here #2059 (comment) and there #2059 (comment) we should limit the number of videos that can be added at the same time by the API. This will make the process more robust by avoiding timeout, and also will give us more control on how we manage the calls to the YouTube API.

20 videos per batch sounds good to me.

[sigmike]

Ok, thanks for the suggestions. I'll implement that.

[GresilleSiffle]

Minus the two previous comments, the PR looks good to me, thanks a lot @sigmike

(1) about the YouTube API calls

@amatissart before merging should we consider defining a new strategy to limit the number of YouTube API generated by the history imports?

Random ideas:

• completely prevent YouTube API calls when using "bulk" routes of our back end (for now); the metadata of an entity will be refreshed after its first comparison
• allow a few calls per user per day on our "bulk" routes; this will postpone the problem, but the problem will still be there
• do nothing and consume as much quota as possible per day (we may need to check the back end first to be sure that a depleted quota will not prevent new videos to be added)

In the future we may want to consider using a task queue, to spread the API calls over the whole day while saving some credits for our daily routines like the playlist updates.

(2) about users that will import several times their history

It will possible for a user to import several times his/her history to Tournesol. It will also be possible that subsequent imports will re-add videos to the rate-later list that have been automatically removed after 4 comparisons. This will, for instance, make the suggestions made by the Auto button less relevant, because it could suggest videos the users don't want to compare.

In my opinion it's not a blocking point to merge this PR, but it could be useful to have a solution for that.

Ideas:

• by default do not add to the rate-later list videos that have been compared more that X times (X being the setting rate_later__auto_remove).
• by default import everything but allow the users to "clean" their rate-later lists by removing all videos that have been compared more than X times
• make the import process remember the range of imported videos, to not imported them again (this looks tricky, especially to define what is a "range", two dates?, two UIDs?)
• do nothing

I have a preference for the solution "by default import everything but allow the users to "clean" their rate-later..."

[GresilleSiffle]

We created a new PR that targets this one (here #2064) to address the point (1) of #2059 (comment).

Regarding the point (2) we haven't fully decided yet.

[GresilleSiffle]

Let's not forget this comment #2068 (comment)

I'll have time tomorrow.