Skip to content

[ext] feat: keep imported videos in cache to not import them again #2068

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
GresilleSiffle merged 6 commits into from
Mar 27, 2025
Merged

[ext] feat: keep imported videos in cache to not import them again #2068

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
0976b79
[ext] feat: keep improted videos in cache to not import them again
GresilleSiffle Mar 21, 2025
41f9185
[ext] feat: display a more visible alert when a HTTP 429 is received
GresilleSiffle Mar 22, 2025
df92b57
[ext] refactor: simplify a CSS margin declaration
GresilleSiffle Mar 22, 2025
3fa272e
[ext] chore: remove useless console.log
GresilleSiffle Mar 22, 2025
6c7dab5
fix: wrong english title in the import status box
GresilleSiffle Mar 26, 2025
f1149d9
ui: simplify alert message in the import status box
GresilleSiffle Mar 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 19 additions & 19 deletions browser-extension/src/_locales/en/messages.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,40 +158,40 @@
"description": "Label of the button displayed on the YouTube history page to import the history into your 'rate later' list."
},
"rateLaterHistoryStatusBoxTitle": {
"message": "Videos are being imported, please wait.",
"message": "Import of the YouTube history",
"description": "Title of the the rate later history import status box."
},
"rateLaterHistoryStatusBoxRateLimited": {
"message": "You have imported the maximum number of videos allowed today.",
"description": "Title of the the rate later history import status box after rate limit has been reached."
"rateLaterHistoryStatusBox429AlertP1" : {
"message": "The import is complete.",
"description": "First paragraph of the alert message, displayed when the daily import limit is reached."
},
"rateLaterHistoryStatusBox429AlertP2" : {
"message": "You have imported the maximum number of videos allowed today. You can start a new import in 24 hours.",
"description": "Second paragraph of the alert message, displayed when the daily import limit is reached."
},
"rateLaterHistoryStatusBoxMessage": {
"message": "Press Stop when the visible videos count doesn't increase anymore or when you think enough videos have been imported.",
"description": "Instructions displayed on the rate later history import status box."
},
"rateLaterHistoryStatusBoxMessageRateLimited": {
"message": "You'll be able to start a new import in 24 hours, after having watched new videos.",
"description": "Instructions displayed on the rate later history import status box after rate limist has been reached."
},
"rateLaterHistoryVideoCount": {
"message": "Videos visible in history:",
"description": "Label showing the count of videos found in the YouTube history during import."
},
"rateLaterHistoryAddedCount": {
"message": "Videos added to rate later:",
"description": "Label showing the count of videos successfully added to the rate later list."
"rateLaterHistoryProcessedCount": {
"message": "Videos processed during the import:",
"description": "Label showing the count of videos processed by the import."
},
"rateLaterHistoryAddedNewCount": {
"message": "new videos:",
"description": "Label showing the number of successfully imported videos that were NOT already in the rate later list before the import."
"rateLaterHistorySentCount": {
"message": "imported videos:",
"description": "Label showing the number of sent videos successfully imported in the rate later list (including those already present)."
},
"rateLaterHistoryAddedExistingCount": {
"message": "already in your list:",
"description": "Label showing the number of successfully imported videos that were already in the rate later list before the import."
"rateLaterHistorySkippedCount": {
"message": "skipped videos (previously imported):",
"description": "Label showing the number of videos that have NOT been sent to the Tournesol server, because they were present in the cache of previously imported videos."
},
"rateLaterHistoryFailureCount": {
"message": "Failures:",
"description": "Label showing the count of videos that failed to be added to the rate later list."
"message": "failures:",
"description": "Label showing the count of sent videos that failed to be added to the rate later list, because of a server error."
},
"rateLaterHistoryCloseButtonLabel": {
"message": "Close",
Expand Down
38 changes: 19 additions & 19 deletions browser-extension/src/_locales/fr/messages.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -156,40 +156,40 @@
"description": "Label du bouton affiché sur l'historique YouTube pour l'importer dans les vidéos à comparer plus tard."
},
"rateLaterHistoryStatusBoxTitle": {
"message": "Les vidéos sont en train d'être importées, veuillez patienter.",
"message": "Import de l'historique YouTube",
"description": "Titre de l'écran d'import de l'historique YouTube vers la liste des vidéos à comparer plus tard."
},
"rateLaterHistoryStatusBoxRateLimited": {
"message": "Vous avez importé le nombre maximal de vidéos autorisé aujourd'hui.",
"description": "Titre de l'écran d'import de l'historique YouTube lorsque la limite quotidienne est atteinte."
},
"rateLaterHistoryStatusBoxMessage": {
"message": "Appuyez sur Stop quand le nombre de vidéos visibles n'augmente plus ou que vous pensez que suffisamment de vidéos ont été importées.",
"description": "Instructions affichées sur l'écran d'import de l'historique YouTube vers la liste des vidéos à comparer plus tard."
},
"rateLaterHistoryStatusBoxMessageRateLimited": {
"message": "Vous pourrez relancer un import dans 24 heures, après avoir vu de nouvelles vidéos.",
"description": "Instructions affichées sur l'écran d'import de l'historique YouTube, lorsque la limite quotidienne est atteinte."
"rateLaterHistoryStatusBox429AlertP1" : {
"message": "L'import est terminée.",
"description": "Premier paragraphe du message d'alerte, lorsque la limite d'import quotidienne est atteinte."
},
"rateLaterHistoryStatusBox429AlertP2" : {
"message": "Vous avez importé le nombre maximal de vidéos autorisé aujourd'hui. Vous pourrez lancer un nouvel import dans 24h.",
"description": "Deuxième paragraphe du message d'alerte, lorsque la limite d'import quotidienne est atteinte."
},
"rateLaterHistoryVideoCount": {
"message": "Vidéos visibles dans l'historique :",
"description": "Étiquette indiquant le nombre de vidéos trouvées dans l'historique YouTube pendant l'import."
},
"rateLaterHistoryAddedCount": {
"message": "Vidéos importées dans les vidéos à comparer :",
"description": "Étiquette indiquant le nombre total de vidéos envoyées avec succès à la liste des vidéos à comparer plus tard."
"rateLaterHistoryProcessedCount": {
"message": "Vidéos traitées par le processus d'import :",
"description": "Étiquette indiquant le nombre total de vidéos traitées par le processus d'import."
},
"rateLaterHistoryAddedNewCount": {
"message": "nouvelles vidéos :",
"description": "Étiquette indiquant le nombre de vidéos envoyées avec succès qui n'étaient PAS déjà présentes dans la liste à comparer plus tard avant l'import."
"rateLaterHistorySentCount": {
"message": "vidéos importées :",
"description": "Étiquette indiquant le nombre de vidéos envoyées qui ont été ajoutées avec succès dans la liste à comparer plus tard (inclus les vidéos déjà présentes)."
},
"rateLaterHistoryAddedExistingCount": {
"message": "déjà présentes dans votre liste :",
"description": "Étiquette indiquant le nombre de vidéos envoyées avec succès qui étaient déjà présentes dans la liste à comparer plus tard avant l'import."
"rateLaterHistorySkippedCount": {
"message": "vidéos évitées (importées précédemment) :",
"description": "Étiquette indiquant le nombre de vidéos non envoyées au serveur Tournesol, car déjà présentes dans le cache de vidéos précédement envoyées."
},
"rateLaterHistoryFailureCount": {
"message": "Échecs :",
"description": "Étiquette indiquant le nombre de vidéos qui n'ont pas pu être ajoutées à la liste."
"message": "échecs :",
"description": "Étiquette indiquant le nombre de vidéos envoyées qui n'ont pas pu être ajoutées à la liste à comparer plus tard, à cause d'une erreur du serveur."
},
"rateLaterHistoryCloseButtonLabel": {
"message": "Fermer",
Expand Down
37 changes: 35 additions & 2 deletions browser-extension/src/rateLaterHistory.css
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,12 +71,37 @@ body.tournesol-capturing-history ytd-thumbnail {
gap: 8px;
}

#tournesol-rate-later-history-status-box h3 {
margin-bottom: 8px;
text-decoration-line: underline;
text-decoration-color: rgb(255, 211, 51);
text-decoration-thickness: 0.4em;
text-decoration-skip-ink: none;
text-underline-offset: 0.2em;
}

#tournesol-rate-later-history-status-box .alert-box {
padding: 16px;
border-radius: 4px;

font-weight: 400;
line-height: 1.43;
letter-spacing: 0.02em;

color: rgb(30, 70, 32);
background-color: rgb(237, 247, 237);
}

#tournesol-rate-later-history-status-box .alert-box p:not(:first-child){
margin-top: 8px;
}

#tournesol-rate-later-history-status-box ul {
list-style-position: inside;
}

#tournesol-rate-later-history-status-box ul li:not(:first-child) {
margin-top: 4px;
margin-top: 6px;
}

#tournesol-rate-later-history-status-box .counter-section {
Expand All @@ -97,7 +122,7 @@ body.tournesol-capturing-history ytd-thumbnail {
}

.lds-dual-ring {
margin: auto;
margin: 16px auto;
}

.lds-dual-ring,
Expand Down Expand Up @@ -128,3 +153,11 @@ body.tournesol-capturing-history ytd-thumbnail {
transform: rotate(360deg);
}
}

.display-none {
display: none;
}

.visibility-hidden {
visibility: hidden;
}
142 changes: 101 additions & 41 deletions browser-extension/src/rateLaterHistory.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,18 +35,30 @@ const addOverlay = () => {
title.textContent = chrome.i18n.getMessage('rateLaterHistoryStatusBoxTitle');
statusBox.append(title);

const alertBox = document.createElement('div');
alertBox.classList.add('alert-box', 'display-none');
const alertBoxText1 = document.createElement('p');
alertBoxText1.textContent = chrome.i18n.getMessage(
'rateLaterHistoryStatusBox429AlertP1'
);
const alertBoxText2 = document.createElement('p');
alertBoxText2.textContent = chrome.i18n.getMessage(
'rateLaterHistoryStatusBox429AlertP2'
);
alertBox.append(alertBoxText1);
alertBox.append(alertBoxText2);
statusBox.append(alertBox);

const loader = document.createElement('div');
loader.classList.add('lds-dual-ring');
statusBox.append(loader);

const showTooManyRequests = () => {
loader.classList.add('display-none');
loader.classList.remove('lds-dual-ring');
title.textContent = chrome.i18n.getMessage(
'rateLaterHistoryStatusBoxRateLimited'
);
message.textContent = chrome.i18n.getMessage(
'rateLaterHistoryStatusBoxMessageRateLimited'
);
alertBox.classList.remove('display-none');
message.classList.add('visibility-hidden');

stopButton.textContent = chrome.i18n.getMessage(
'rateLaterHistoryCloseButtonLabel'
);
Expand Down Expand Up @@ -116,21 +128,18 @@ const addOverlay = () => {
label: chrome.i18n.getMessage('rateLaterHistoryVideoCount'),
});

const { displayCount: displaySentVideoCount } = addCounter({
label: chrome.i18n.getMessage('rateLaterHistoryAddedCount'),
const { displayCount: displayProcessedVideoCount } = addCounter({
label: chrome.i18n.getMessage('rateLaterHistoryProcessedCount'),
});

const { displayCounts: displayNewVideosCount } = addCounterList({
const { displayCounts: displayDetailedVideoCounts } = addCounterList({
labels: [
chrome.i18n.getMessage('rateLaterHistoryAddedNewCount'),
chrome.i18n.getMessage('rateLaterHistoryAddedExistingCount'),
chrome.i18n.getMessage('rateLaterHistorySentCount'),
chrome.i18n.getMessage('rateLaterHistorySkippedCount'),
chrome.i18n.getMessage('rateLaterHistoryFailureCount'),
],
});

const { displayCount: displayFailedVideoCount } = addCounter({
label: chrome.i18n.getMessage('rateLaterHistoryFailureCount'),
});

const message = document.createElement('p');
message.id = 'when-to-stop';
message.textContent = chrome.i18n.getMessage(
Expand All @@ -154,9 +163,8 @@ const addOverlay = () => {
return {
removeOverlay,
displayHistoryVideoCount,
displaySentVideoCount,
displayNewVideosCount,
displayFailedVideoCount,
displayProcessedVideoCount,
displayDetailedVideoCounts,
showTooManyRequests,
stopButton,
};
Expand All @@ -178,6 +186,42 @@ const addRateLaterBulk = async (videoIds) =>
);
});

const skipPreviouslyImported = async (videoSet) => {
const newVideos = new Set();
const localStorage = await chrome.storage.local.get('youtubeHistoryImported');
const historyInStorage = localStorage?.youtubeHistoryImported;

for (const video of videoSet) {
if (!historyInStorage?.includes(video)) {
newVideos.add(video);
}
}

return newVideos;
};

const saveImportedToLocalStorage = async (importedVideosSet) => {
const localStorage = await chrome.storage.local.get('youtubeHistoryImported');
const historyInStorage = localStorage?.youtubeHistoryImported;

let videosStr = '';
for (const video of importedVideosSet) {
if (!historyInStorage?.includes(video)) {
videosStr += `,${video}`;
}
Comment on lines +208 to +211
Copy link
Member

@amatissart amatissart Mar 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Loading youtubeHistoryImported in a Set could perform better, instead of using String.includes(). I suppose that is not issue even for thousands of videos?

Copy link
Collaborator Author

@GresilleSiffle GresilleSiffle Mar 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I thought about it. I tried with a string of 10,000 videos and the performance were good enough, but it's probably better to use a Set.

}

if (!historyInStorage) {
videosStr = videosStr.substring(1);
}

await chrome.storage.local.set({
youtubeHistoryImported: historyInStorage
? historyInStorage + videosStr
: videosStr,
});
};

const chunkArray = (array, chunkSize) => {
if (chunkSize <= 0) {
throw new Error('Chunk size must be greater than 0');
Expand Down Expand Up @@ -249,22 +293,23 @@ const loadMoreVideos = () => {

const startHistoryCapture = async () =>
new Promise((resolve) => {
// Videos currently visible in the YT history.
const historyVideoIdSet = new Set();
// Visible videos that have been sent to the Tournesol API.
let sentVideoIdSet = new Set();
// Visible videos that have been skipped, and not sent to the Tournesol API.
let skippedVideoIdSet = new Set();

let addedCount = 0;
let failedCount = 0;

let newlyImportedTotal = 0;
let previouslyImportedTotal = 0;

let skippedCount = 0;
let abort = false;

const {
removeOverlay,
displayHistoryVideoCount,
displaySentVideoCount,
displayNewVideosCount,
displayFailedVideoCount,
displayProcessedVideoCount,
displayDetailedVideoCounts,
showTooManyRequests,
stopButton,
} = addOverlay();
Expand All @@ -287,27 +332,33 @@ const startHistoryCapture = async () =>

displayHistoryVideoCount(historyVideoIdSet.size);

const videosToSend = historyVideoIdSet.difference(sentVideoIdSet);
if (videosToSend.size > 0) {
const {
processedSet,
failedSet,
addedNbr,
alreadyExistingNbr,
tooManyRequests,
} = await addVideoIdsToRateLater(videosToSend);
const videosToSend = historyVideoIdSet
.difference(sentVideoIdSet)
.difference(skippedVideoIdSet);

sentVideoIdSet = sentVideoIdSet.union(videosToSend);
let newVideosToSend = new Set();

addedCount += processedSet.size;
displaySentVideoCount(addedCount);
try {
newVideosToSend = await skipPreviouslyImported(videosToSend);
} catch (error) {
console.error(error);
newVideosToSend = videosToSend;
}

newlyImportedTotal += addedNbr;
previouslyImportedTotal += alreadyExistingNbr;
displayNewVideosCount([newlyImportedTotal, previouslyImportedTotal]);
if (newVideosToSend.size > 0) {
const { processedSet, failedSet, tooManyRequests } =
await addVideoIdsToRateLater(newVideosToSend);

try {
await saveImportedToLocalStorage(processedSet);
} catch (error) {
console.error(error);
}

addedCount += processedSet.size;
failedCount += failedSet.size;
displayFailedVideoCount(failedCount);

sentVideoIdSet = sentVideoIdSet.union(newVideosToSend);

if (tooManyRequests) {
abortCapture();
Expand All @@ -316,6 +367,15 @@ const startHistoryCapture = async () =>
}
}

const skipped = videosToSend.difference(newVideosToSend);
if (skipped.size > 0) {
skippedCount += skipped.size;
skippedVideoIdSet = skippedVideoIdSet.union(skipped);
}

displayProcessedVideoCount(addedCount + skippedCount + failedCount);
displayDetailedVideoCounts([addedCount, skippedCount, failedCount]);

if (abort) return;

loadMoreVideos();
Expand Down