Skip to content

initial base_url companion support + proxy companion #5266

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions config/config.example.yml
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ db:
## If you are using a reverse proxy then you will probably need to
## configure the public_url to be the same as the domain used for Invidious.
## Also apply when used from an external IP address (without a domain).
## Examples: https://MYINVIDIOUSDOMAIN or http://192.168.1.100:8282
## Examples: https://MYINVIDIOUSDOMAIN/companion or http://192.168.1.100:8282/companion
##
## Both parameter can have identical URL when Invidious is hosted in
## an internal network or at home or locally (localhost).
Expand All @@ -84,8 +84,8 @@ db:
## Default: <none>
##
#invidious_companion:
# - private_url: "http://localhost:8282"
# public_url: "http://localhost:8282"
# - private_url: "http://localhost:8282/companion"
# public_url: "http://localhost:8282/companion"

##
## API key for Invidious companion, used for securing the communication
Expand Down
37 changes: 37 additions & 0 deletions src/invidious/routes/companion.cr
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
module Invidious::Routes::Companion
Copy link
Member

@syeopite syeopite May 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add the Accept-Encoding header to each companion request as so you aren't decompressing the body. Invidious should just directly proxy the compressed response from companion to the end user.

https://crystal-lang.org/api/1.16.3/HTTP/Client.html

If compress [HTTP::Client instance property] isn't set to false, and no Accept-Encoding header is explicitly specified, an HTTP::Client will add an "Accept-Encoding": "gzip, deflate" header, and automatically decompress the response body/body_io.

# /companion
def self.get_companion(env)
url = env.request.path.lchop("/companion")

begin
COMPANION_POOL.client &.get(url, env.request.header) do |resp|
return self.proxy_companion(env, resp)
end
rescue ex
end
end

def self.options_companion(env)
url = env.request.path.lchop("/companion")

begin
COMPANION_POOL.client &.options(url, env.request.header) do |resp|
return self.proxy_companion(env, resp)
end
rescue ex
end
end

private def self.proxy_companion(env, response)
env.response.status_code = response.status_code
response.headers.each do |key, value|
env.response.headers[key] = value
end

if response.status_code >= 300
return env.response.headers.delete("Transfer-Encoding")
end

return proxy_file(response, env)
Copy link
Member

@syeopite syeopite May 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you applied the suggestion to not decompress the data received from companion then the resulting compressed response will still keep its Content-Encoding header which will cause proxy_file to compress it again.

To avoid a double compression:

Suggested change
return proxy_file(response, env)
return IO.copy response.body_io, env.response

end
end
8 changes: 6 additions & 2 deletions src/invidious/routes/embed.cr
Original file line number Diff line number Diff line change
Expand Up @@ -205,10 +205,14 @@ module Invidious::Routes::Embed

if CONFIG.invidious_companion.present?
invidious_companion = CONFIG.invidious_companion.sample
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replaced with invidious_companion_urls

Suggested change
invidious_companion = CONFIG.invidious_companion.sample

invidious_companion_urls = CONFIG.invidious_companion.map do |companion|
uri =
"#{companion.public_url.scheme}://#{companion.public_url.host}#{companion.public_url.port ? ":#{companion.public_url.port}" : ""}"
end.join(" ")
env.response.headers["Content-Security-Policy"] =
env.response.headers["Content-Security-Policy"]
.gsub("media-src", "media-src #{invidious_companion.public_url}")
.gsub("connect-src", "connect-src #{invidious_companion.public_url}")
.gsub("media-src", "media-src #{invidious_companion_urls}")
.gsub("connect-src", "connect-src #{invidious_companion_urls}")
end

rendered "embed"
Expand Down
8 changes: 6 additions & 2 deletions src/invidious/routes/watch.cr
Original file line number Diff line number Diff line change
Expand Up @@ -194,10 +194,14 @@ module Invidious::Routes::Watch

if CONFIG.invidious_companion.present?
invidious_companion = CONFIG.invidious_companion.sample
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replaced with invidious_companion_urls

Suggested change
invidious_companion = CONFIG.invidious_companion.sample

invidious_companion_urls = CONFIG.invidious_companion.map do |companion|
uri =
"#{companion.public_url.scheme}://#{companion.public_url.host}#{companion.public_url.port ? ":#{companion.public_url.port}" : ""}"
end.join(" ")
env.response.headers["Content-Security-Policy"] =
env.response.headers["Content-Security-Policy"]
.gsub("media-src", "media-src #{invidious_companion.public_url}")
.gsub("connect-src", "connect-src #{invidious_companion.public_url}")
.gsub("media-src", "media-src #{invidious_companion_urls}")
.gsub("connect-src", "connect-src #{invidious_companion_urls}")
end

templated "watch"
Expand Down
9 changes: 8 additions & 1 deletion src/invidious/routing.cr
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,7 @@ module Invidious::Routing
end

# -------------------
# Media proxy routes
# Proxy routes
# -------------------

def register_api_manifest_routes
Expand Down Expand Up @@ -223,6 +223,13 @@ module Invidious::Routing
get "/vi/:id/:name", Routes::Images, :thumbnails
end

def register_companion_routes
if CONFIG.invidious_companion.present?
get "/companion/*", Routes::Companion, :get_companion
options "/companion/*", Routes::Companion, :options_companion
end
end

# -------------------
# API routes
# -------------------
Expand Down
34 changes: 25 additions & 9 deletions src/invidious/yt_backend/connection_pool.cr
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,22 @@ struct YoutubeConnectionPool
end
end

class CompanionWrapper
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think there's a reason for this to be a class.

Suggested change
class CompanionWrapper
struct CompanionWrapper

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think you can also document this type? It wasn't immediately obvious to me what this wrapper was used for. Maybe something like:

# Packages a `HTTP::Client` to an invidious-companion instance alongside the configuration for that instance.
#  
# This is used as the resource for the `CompanionPool` as to allow the ability to
# query companion instances hosted on a subpath
struct CompanionWrapper

property client : HTTP::Client
property companion : Config::CompanionConfig

def initialize(companion : Config::CompanionConfig)
@companion = companion
@client = HTTP::Client.new(companion.private_url)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
@client = HTTP::Client.new(companion.private_url)
@client = make_client(companion.private_url, use_http_proxy: false)

end

def close
@client.close
end
end

struct CompanionConnectionPool
property pool : DB::Pool(HTTP::Client)
property pool : DB::Pool(CompanionWrapper)

def initialize(capacity = 5, timeout = 5.0)
options = DB::Pool::Options.new(
Expand All @@ -57,26 +71,28 @@ struct CompanionConnectionPool
checkout_timeout: timeout
)

@pool = DB::Pool(HTTP::Client).new(options) do
@pool = DB::Pool(CompanionWrapper).new(options) do
companion = CONFIG.invidious_companion.sample
next make_client(companion.private_url, use_http_proxy: false)
client = make_client(companion.private_url, use_http_proxy: false)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is leaking a client

Suggested change
client = make_client(companion.private_url, use_http_proxy: false)

CompanionWrapper.new(companion: companion)
end
end

def client(&)
conn = pool.checkout
wrapper = pool.checkout

begin
response = yield conn
response = yield wrapper
rescue ex
conn.close
wrapper.client.close
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
wrapper.client.close
wrapper.close


companion = CONFIG.invidious_companion.sample
conn = make_client(companion.private_url, use_http_proxy: false)
client = make_client(companion.private_url, use_http_proxy: false)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
client = make_client(companion.private_url, use_http_proxy: false)

wrapper = CompanionWrapper.new(companion: companion)

response = yield conn
response = yield wrapper
ensure
pool.release(conn)
pool.release(wrapper)
end

response
Expand Down
30 changes: 18 additions & 12 deletions src/invidious/yt_backend/youtube_api.cr
Original file line number Diff line number Diff line change
Expand Up @@ -695,22 +695,28 @@ module YoutubeAPI
# Send the POST request

begin
response = COMPANION_POOL.client &.post(endpoint, headers: headers, body: data.to_json)
body = response.body
if (response.status_code != 200)
raise Exception.new(
"Error while communicating with Invidious companion: \
status code: #{response.status_code} and body: #{body.dump}"
)
response_body = ""

COMPANION_POOL.client do |wrapper|
companion_base_url = wrapper.companion.private_url.path
puts "Using companion: #{wrapper.companion.private_url}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Debug puts

Suggested change
puts "Using companion: #{wrapper.companion.private_url}"


response = wrapper.client.post(companion_base_url + endpoint, headers: headers, body: data.to_json)
response_body = response.body
Comment on lines +704 to +705
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

JSON.parse can actually accept an IO as an argument. It'll be more efficient to stream the request and parse the JSON as data comes in.

You should be able to do something like this:

wrapper.client.post("#{companion_base_url}#{endpoint}", headers: headers, body: data.to_json) do | response |
  response_body = JSON.parse(response.body_io)
end


if response.status_code != 200
raise Exception.new(
"Error while communicating with Invidious companion: " \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is duplicating the error message twice when it'd be appended to the InfoException below

Suggested change
"Error while communicating with Invidious companion: " \

"status code: #{response.status_code} and body: #{response_body.dump}"
)
end
end

# Convert result to Hash
return JSON.parse(response_body).as_h
rescue ex
raise InfoException.new("Error while communicating with Invidious companion: " + (ex.message || "no extra info found"))
end

# Convert result to Hash
initial_data = JSON.parse(body).as_h

return initial_data
end

####################################################################
Expand Down
Loading