An attempt to download [a YouTube channel] resulted in an accumulation (long list) of "unavailable videos" in Tasks, from all current and previously attempted failed downloads [why is the very same video individually downloadable?]

[EMG70]

Describe the bug/problem
An attempt to download playlists results in an accumulation of long list in Tasks " unavailable videos" from all current and previously attempts.

A clear and concise description of what the bug is. If you are asking for support, please check our Wiki if your question is already answered there.

Unavailable videos for any reason are expected to show as those that have failed for the playlist not cumulative list from previous downloads.

To Reproduce

Download
Steps to reproduce the behavior:

1. Go to 'https://www.youtube.com/@agribusinesswithwadzanaima275' name of playlist
2. Click on 'https://www.youtube.com/watch?v=EiMFnDquvdM' failing video being added to the list of unavailable videos
3. https://www.youtube.com/watch?v=b4RLo3tgpoA failing video being added to the list of unavailable videos
4. Scroll down to '....'
5. See error

Logfile
SUDO IIAB-DIAGNOSTICS - https://paste.centos.org/view/10fc9a97

calibre-web.log

Add content of calibre-web.log file or the relevant error, try to reproduce your problem with "debug" log-level to get more output.

Expected behavior

Unavailable videos for any reason are expected to show as those that have failed for the playlist not cumulative list from previous downloads.

A clear and concise description of what you expected to happen.

Screenshots

If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• OS: [e.g. Windows 10/Raspberry Pi OS]
• Python version: [e.g. python2.7]
• Calibre-Web version: [e.g. 0.6.8 or 087c4c5 (git rev-parse --short HEAD)]:
• Docker container: [None/LinuxServer]:
• Special Hardware: [e.g. Rasperry Pi Zero]
• Browser: [e.g. Chrome 83.0.4103.97, Safari 13.3.7, Firefox 68.0.1 ESR]
  

Additional context
Add any other context about the problem here. [e.g. access via reverse proxy, database background sync, special database location]
This is a members only video https://www.youtube.com/watch?v=MfRdmsbs3R8 but also gets added to list of unavailable accumulative list.

This video https://www.youtube.com/watch?v=EiMFnDquvdM fails to download as part of playlist https://www.youtube.com/@agribusinesswithwadzanaima275 but successfully downloaded as an individual url.(last line of screenshot)

The results are however not consistent ,some individual public urls are still failing. eg https://www.youtube.com/watch?v=AT4khVKBBkg which plays well in youtube.

[deldesir]

Here is why:

calibre-web/scripts/lb-wrapper

Line 82 in 486b7bb

xklb_full_cmd="lb tubeadd '${XKLB_DB_FILE}' '${URL_OR_SEARCH_TERM}' --force ${VERBOSITY}"

This line of code from lb-wrapper ensures, with the use of --force, to update already existed metadata for a given path as explained in https://github.com/chapmanjacobd/library/blob/main/xklb/usage.py#L570-L572

• When a video url is submitted, the path is the video url, the metadata for this url is updated. The download goes through
• When a Youtube Playlist is submitted, the path is the playlist url, the metadate for the playlist url is updated, but the video urls path are not because they are secondary. Previous failures are seen and handled then.

[holta]

@deldesir which below are most relevant? (In helping us explain — and decide what to do next!)

• PR UPDATE paths instead of REPLACE + FOOTNOTE re: Archivists-vs-YouTube having conflicting world views, on how to define "Duplicate Videos" #125 (added --force to tubeadd)
• Subsequent playlists redownloads fail #150
  • PR Enable subsequent [re]downloads of playlists #151

[holta]

@EMG70 just FYI @deldesir is looking into this, but it might not be easy!

Something equivalent to xklb tubeadd's --force might be needed for entire playlists — not just single videos ?

(Sorting out how xklb helps us with download errors & metadata-extract.py for entire playlists — not just single videos — likely needs work!)

[EMG70]

@EMG70 just FYI @deldesir is looking into this, but it might not be easy!

Something equivalent to xklb tubeadd's --force might be needed for entire playlists — not just single videos ?

(Sorting how xklb helps us with download errors & metadata-extract.py for entire playlists — not just single videos — likely needs work!)

Thank you

[holta]

1. Go to https://www.youtube.com/@agribusinesswithwadzanaima275 name of playlist

@EMG70: Were you aware that you are trying to download an entire YouTube Channel here — rather than a standard playlist?

@deldesir is investigating if this might possibly be exacerbating the problem.

1. FWIW this particular channel contains 220 videos and 0 [zero] playlists...)

   

2. Whereas many YouTube channels contain playlists — e.g. https://youtube.com/@bbc has 6 tabs including "Playlists" ...

   

Tangentially, @deldesir is being affected by IP addresses (cloud VPS IP addresses especially?) being banned / blocked by YouTube, as mentioned below...

• YouTube video(s) not downloadable (or viewable?) from certain IP address(es): "Sign in to confirm you’re not a bot. This helps protect our community" #243

[EMG70]

Yes I was aware of her 220 agricultural videos under her channel ,unfortunately she doesn't group them by Playlist.I have downloaded same videos successfully before my calibre-web got corrupted after upgrading to 24.04 in last two weeks .
It could well be the new Youtube restrictions which came up recently.

[holta]

@deldesir suspects the large number of videos (220 in this case) might be causing this problem?

Regardless whether or not contained in a standard playlist.