Towards using only next endpoint for the major data. (Partial solution to #5003) #5237

mk-pmb · 2025-04-11T13:35:48Z

From my discussion with @unixfox , these are the changes that I was able to salvage from #5003 that either were not critizised in code review or where I was able to apply the suggestions.

absidue · 2025-04-11T16:09:53Z

Looks like some of the commits in this branch are outdated as it is missing changes that are already in the master branch such as #4934.

mk-pmb · 2025-04-11T16:38:15Z

That may well be. I don't understand enough crystal to be able to identify redundant code; I could only transplant changes that @unixfox made. If you tell me which (parts of) commits I shall omit, I can of course do that.

mk-pmb · 2025-04-11T16:47:53Z

There's also this commit that looks important: mk-pmb@108ee36
… but should probably be solved by instead duplicating the unconditional part at the end into the tiny "else" branch and adding early return (then invert the condition to make it a tiny "then" branch), to avoid having to indent the huge original "then" branch.

(If you do decide to indent, please do that in a separate commit or git will make a reeeeeeally confusing diff. It will be bad enough already but at least that way a diff with proper ignore-whitespace options can assure you that no actual code was changed.)

Shouldn't really belong to this PR though.

Update: I learned enough crystal to do the early return transformation, but now I see that the common tail below has grown with companion stuff, so probably now the better choice is to factor out the "then" branch into its own function.

mk-pmb · 2025-04-11T17:05:07Z

Ok yeah, some things seem to have been broken by git's rebase. I now discovered the UTC fallback, and it seems obvious even to me that the rebased commit does the opposite of what it claims. I removed that commit.

syeopite · 2025-06-06T22:15:18Z

src/invidious/yt_backend/extractors_utils.cr

+  return false if badges.nil?
+
+  badges.as_a.each do |badge|
+    icon_type = badge.dig("metadataBadgeRenderer", "icon", "iconType").as_s
+
+    return true if icon_type == "PRIVACY_UNLISTED"
+  end
+
+  return false


Suggested change

return false if badges.nil?

badges.as_a.each do |badge|

icon_type = badge.dig("metadataBadgeRenderer", "icon", "iconType").as_s

return true if icon_type == "PRIVACY_UNLISTED"

end

return false

return false if badges.nil?

badges.as_a.each do |badge|

icon_type = badge.dig("metadataBadgeRenderer", "icon", "iconType").as_s

return true if icon_type == "PRIVACY_UNLISTED"

end

return false

syeopite · 2025-06-06T23:34:27Z

src/invidious/videos/parser.cr

+    published_txt = video_primary_renderer
+      .try &.dig?("dateText", "simpleText")
+
+    if published_txt.try &.as_s.includes?("ago") && !published_txt.nil?
+      published = decode_date(published_txt.as_s.lchop("Started streaming "))
+    elsif published_txt && published_txt.try &.as_s.matches?(/(\w{3} \d{1,2}, \d{4})$/)
+      published = Time.parse(published_txt.as_s.match!(/(\w{3} \d{1,2}, \d{4})$/)[0], "%b %-d, %Y", Time::Location::UTC)
+    else
+      published = Time.utc
+    end


Something like this should be more efficient

video_primary_renderer.try(&.dig?("dateText", "simpleText")).try do | published_txt | published_txt = published_txt.as_s published = if published_txt.includes?("ago") decode_date(published_txt.lchop("Started streaming ")) elsif matched_time = published_txt.match(/(\w{3} \d{1,2}, \d{4})$/) Time.parse(matched_time[0], "%b %-d, %Y", Time::Location::UTC) end end published ||= Time.utc

Please also try using the scheduledStartTime prior to the other attempts

https://github.com/iv-org/invidious/pull/5003/files#r1807376801

Hi, thanks for your feedback! Unfortunately I know almost nothing about crystal so I'd rather not try and implement actual changes. In this PR I just rebased @unixfox's code in hopes to get it merged asap to avoid even more branch divergence and thus even more effort for rebasing. I hope it's not too late already.

If it's important to you that I update the PR, I can try and apply all the changes you requested. However, since those are your changes, wouldn't it be more clear to merge and then apply them, rather than having me apply them in unixfox's or your name? (Evem deciding that for each patch seems needlessly error-prone.)

Except of course if any of the patches introduce a critical bug, then obviously I should fix the offending code before it's merged. If so, please tell me, because I'm unable to classify the kind of changes you made.

For any code that's not actually "bad" but just "could be better" (e.g. "should be more efficient"), let's make improvements a separate PR. After all, this PR is titled "Towards … (Partial solution …)" because I know it's not perfect, it's just a useful step, or at least I hope it is. And since most contributors here have a tendency to only occasionally have time for this project, I think merging acceptable parts as early as possible is a good approach.

syeopite · 2025-06-06T23:35:21Z

src/invidious/videos/parser.cr

+  if live_now.nil?
+    live_now = video_primary_renderer
+      .try &.dig?("viewCount", "videoViewCountRenderer", "isLive").try &.as_bool
+  end


Suggested change

if live_now.nil?

live_now = video_primary_renderer

.try &.dig?("viewCount", "videoViewCountRenderer", "isLive").try &.as_bool

end

live_now ||= video_primary_renderer.try &.dig?("viewCount", "videoViewCountRenderer", "isLive").try &.as_bool

src/invidious/videos/parser.cr

syeopite · 2025-06-06T23:39:44Z

src/invidious/videos/parser.cr

-    "shortDescription" => JSON::Any.new(short_description.try &.as_s || nil),
+    "shortDescription" => JSON::Any.new(short_description.try &.as_s || ""),


Why is shortDescription not allowed to be nil?

I have no idea. @unixfox , do you still remember?

syeopite · 2025-06-07T00:02:30Z

src/invidious/yt_backend/extractors_utils.cr

+  return false if badges.nil?
+
+  badges.as_a.each do |badge|
+    icon_type = badge.dig("metadataBadgeRenderer", "icon", "iconType").as_s
+
+    return true if icon_type == "PRIVACY_UNLISTED"
+  end
+
+  return false
+rescue ex
+  LOGGER.debug("Unable to parse owner badges. Got exception: #{ex.message}")
+  LOGGER.trace("Owner badges data: #{badges.to_json}")
+
+  return false


This entire function could pretty much be reduced down to a single line. I don't know if the error handling + logging is intentional here but it shouldn't even be needed if you just use a dig? instead.

Suggested change

return false if badges.nil?

badges.as_a.each do |badge|

icon_type = badge.dig("metadataBadgeRenderer", "icon", "iconType").as_s

return true if icon_type == "PRIVACY_UNLISTED"

end

return false

rescue ex

LOGGER.debug("Unable to parse owner badges. Got exception: #{ex.message}")

LOGGER.trace("Owner badges data: #{badges.to_json}")

return false

return badges.try(&.as_a.any?(&.dig?("metadataBadgeRenderer", "icon", "iconType").==("PRIVACY_UNLISTED"))) || false

syeopite · 2025-06-07T00:05:05Z

src/invidious/helpers/errors.cr

+  error_message = <<-END_HTML
+    <div class="error_message">
+      <h2>#{translate(locale, "error_processing_data_youtube")}</h2>
+      <p>#{translate(locale, message)}</p>
+      #{error_redirect_helper(env)}
+    </div>
+  END_HTML


Copy and pasted from my review from the original PR:

Your changes here will show the Error while processing the data sent by YouTube message on any InfoException while also removing the next steps message on all InfoExceptions

Example:

And InfoExceptions are raised in places unrelated to YouTube as well such as when a user inserts the wrong password.

Related: #4651

syeopite · 2025-06-07T00:05:54Z

locales/en-US.json

+    "carousel_go_to": "Go to slide `x`",
+    "error_from_youtube_unplayable": "Video unplayable due to an error from YouTube:",
+    "error_processing_data_youtube": "Error while processing the data sent by YouTube",
+    "refresh_page": "Refresh the page"


Suggested change

"refresh_page": "Refresh the page"

syeopite · 2025-06-07T00:06:33Z

src/invidious/views/embed.ecr

+<div id="player-container" class="h-box">
+    <%= rendered "components/player" %>
+</div>
+<% else %>
+<div id="player-error-container" class="h-box">
+    <h3>
+        <%= video.reason %>
+    </h3>
+</div>


Suggested change

<div id="player-container" class="h-box">

<%= rendered "components/player" %>

</div>

<% else %>

<div id="player-error-container" class="h-box">

<h3>

<%= video.reason %>

</h3>

</div>

<div id="player-container" class="h-box">

<%= rendered "components/player" %>

</div>

<% else %>

<div id="player-error-container" class="h-box">

<h3>

<%= video.reason %>

</h3>

</div>

syeopite · 2025-06-07T00:19:08Z

src/invidious/videos/parser.cr

  video_details = player_response.dig?("videoDetails")
+  video_details ||= {} of String => JSON::Any
  if !(microformat = player_response.dig?("microformat", "playerMicroformatRenderer"))
    microformat = {} of String => JSON::Any
  end


https://github.com/iv-org/invidious/pull/5003/files#r1807374623

unixfox added 9 commits April 11, 2025 15:21

Guard video against missing required info.

637aef8

Try to detect an "unlisted" badge for videos.

fe73551

parser: Deal with missing lengthSeconds in video_details.

a9dfc70

parser: publishDate: Add even more fallbacks.

70acf7c

parser: live_now: Add fallback based on viewCount and isLive.

b2f7d07

parser: Assume videos are family-friendly by default.

2a33036

parser: Assume video isListed if it has no badges.

30b5938

parser: Add fallbacks for author and ucid detection.

ed2827b

Watch view: Better error message when video is unplayable.

bcd1f85

mk-pmb requested review from SamantazFox and a team as code owners April 11, 2025 13:35

mk-pmb mentioned this pull request Apr 11, 2025

Rebase the draft for "trying to only use next endpoint for the major data" onto master unixfox/invidious#1

Closed

mk-pmb force-pushed the use-only-next-for-extra-info-250411 branch from 3b761f4 to a9dfc70 Compare April 11, 2025 17:04

This was referenced May 17, 2025

[Bug] videoDetails - inv_sig_helper not installed #5307

Closed

Make base-Invidious video info extraction more resilient #5312

Merged

syeopite requested changes Jun 7, 2025

View reviewed changes

		"shortDescription" => JSON::Any.new(short_description.try &.as_s \|\| nil),
		"shortDescription" => JSON::Any.new(short_description.try &.as_s \|\| ""),

Uh oh!

Towards using only next endpoint for the major data. (Partial solution to #5003) #5237

Are you sure you want to change the base?

Towards using only next endpoint for the major data. (Partial solution to #5003) #5237

Uh oh!

Conversation

mk-pmb commented Apr 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

absidue commented Apr 11, 2025

Uh oh!

mk-pmb commented Apr 11, 2025

Uh oh!

mk-pmb commented Apr 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mk-pmb commented Apr 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mk-pmb Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mk-pmb commented Apr 11, 2025 •

edited

Loading

mk-pmb commented Apr 11, 2025 •

edited

Loading

mk-pmb commented Apr 11, 2025 •

edited

Loading

mk-pmb Jun 9, 2025 •

edited

Loading