TED: update all ZIMs by topic

[benoit74]

As discussed in many meetings / issues, we will:

• remove former 6 topics recipes which are not that relevant anymore / pushed forward by TED
• create 355+ topics recipes, on per TED topic

I will assist on this by automating the creation of the recipe.

What we have agreed:

• name: ted_mul_
• title: TED
• description: A collection of TED videos about
• ZIM tags: ted
• we will grab videos in all languages (not only english)
• we will grab all available subtitles for all videos
• we will reencode all videos to a lower quality

In order to grab videos in all languages, since the scrapper does not support it we will use a trick:

• set languages to “eng,fra,esp,deu,chi” (this will probably select a fair amount of the videos)
• activate the “Subtitles enough” option (which says that the video is selected if its language is among the selected ones (above) OR if there is subtitles in one of the selected languages)

We will use the --low-quality option for now since the scraper does not allow to really create high quality videos anyway (see openzim/ted#160)

[benoit74]

I've created the first recipe with an automated tool: https://farm.openzim.org/recipes/ted_topic_street-art

Note that I have created a new Zimfarm tag so that it is now easier to find all Ted recipes which grab a topic: https://farm.openzim.org/recipes?tag=ted-by-topic

All next recipes will be created with the same settings (except topic which will change of course).

@RavanJAltaie could you please:

• review in details the recipe configuration (ALL parameters) to confirm everything is OK up to your knowledge of the scraper
• review the first ZIM produced at https://dev.library.kiwix.org/viewer#ted_mul_street-art_2024-01

Once I have your confirmation, I will create the 354 remaining recipes.

[RavanJAltaie]

@benoit74 I've checked the recipe parameters, they all look fine (AFAIK).
About the zim file, it's working fine, 4 videos are there in English, but when I filter the languages to French, chinese, spanish, nothing happens. Same videos in English are there.
When I filter to German, 1 video disappears and there are 3 videos left.

[benoit74]

Thank you !

About the filter by language, AFAIK this is the "normal" behavior, this is linked to the fact that all videos have voice in English and subtitles in French, Chinese and Spanish, while only 3 videos out of 4 have German subtitles.

I agree this is far from intuitive, but I don't think this is something new.

Please check another TED ZIM created before.

If everything is ok (i.e. there is nothing new here, we know that the UI for TED and youtube videos must be enhanced), please move the recipe to PROD and confirm I can create all remaining recipes.

[benoit74]

We've agreed last thursday that I might proceed.

However I now have two three doubts:

• are we sure that ted_mul_street-art is the proper name? don't we want to add topic somewhere in the name to differentiate them more easily, i.e. ted_mul_topic-street-art
• are we ok to not have the recipe name and topic name aligned? (no a technical or convention issue, more a maintainability question)
• don't we want to add a ZIM tag by_topic, again to differentiate these ZIMs from other TED ZIMs? (not sure this is a good idea)

[RavanJAltaie]

Hello @benoit74

• No, adding topic to the topic won't be necessary. I believe ted_mul_street-art is good enough.
• Yes, we are ok as we can consider TED the exception of matching zimname & titles in recipes.
• No, as later not these newly generated zims will be The TED ZIMs.

[benoit74]

All 354 new recipes (out of 355) have been created, they should start soon.

[benoit74]

I also propose that we create a ted_topic_all recipe which will create one single ZIM with all TED videos (easily doable by setting the topic to "all"). Some people (at least me) might be interested to fetch this big ZIM rather than many smaller ones with overlapping videos across them. WDYT?

[benoit74]

TED scraper 3.0.2 is now ready with new encoder and hopefully mostly everything fixed for proper release of these ZIMs.
I built and tested few ZIMs over the weekend and they looked perfect.

I hence reenabled, reconfigured and enabled all ted recipes by topic with a letter + ted_topic_3d-printing to publish new ZIMs to production.

@RavanJAltaie @Popolechien could you please have a look at these ZIMs as well once they hit production?

If quality is confirmed to be OK, I propose that we go as planned and enable and request all recipes of TED by topic (I can do it programmatically quite easily, no need to click 355 times on the "request" button).

[kelson42]

@benoit74 Please share on or two URL of ZIM with VP9 when the first are online.

[Popolechien]

Yes I was going to ask what it is we should specifically test for (sound on macOs, isn'it?)