-
-
Notifications
You must be signed in to change notification settings - Fork 1.9k
initial base_url companion support + proxy companion #5266
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
@@ -0,0 +1,37 @@ | ||||||
module Invidious::Routes::Companion | ||||||
# /companion | ||||||
def self.get_companion(env) | ||||||
url = env.request.path.lchop("/companion") | ||||||
|
||||||
begin | ||||||
COMPANION_POOL.client &.get(url, env.request.header) do |resp| | ||||||
return self.proxy_companion(env, resp) | ||||||
end | ||||||
rescue ex | ||||||
end | ||||||
end | ||||||
|
||||||
def self.options_companion(env) | ||||||
url = env.request.path.lchop("/companion") | ||||||
|
||||||
begin | ||||||
COMPANION_POOL.client &.options(url, env.request.header) do |resp| | ||||||
return self.proxy_companion(env, resp) | ||||||
end | ||||||
rescue ex | ||||||
end | ||||||
end | ||||||
|
||||||
private def self.proxy_companion(env, response) | ||||||
env.response.status_code = response.status_code | ||||||
response.headers.each do |key, value| | ||||||
env.response.headers[key] = value | ||||||
end | ||||||
|
||||||
if response.status_code >= 300 | ||||||
return env.response.headers.delete("Transfer-Encoding") | ||||||
end | ||||||
|
||||||
return proxy_file(response, env) | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If you applied the suggestion to not decompress the data received from companion then the resulting compressed response will still keep its To avoid a double compression:
Suggested change
|
||||||
end | ||||||
end |
Original file line number | Diff line number | Diff line change | ||
---|---|---|---|---|
|
@@ -205,10 +205,14 @@ module Invidious::Routes::Embed | |||
|
||||
if CONFIG.invidious_companion.present? | ||||
invidious_companion = CONFIG.invidious_companion.sample | ||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Replaced with
Suggested change
|
||||
invidious_companion_urls = CONFIG.invidious_companion.map do |companion| | ||||
uri = | ||||
"#{companion.public_url.scheme}://#{companion.public_url.host}#{companion.public_url.port ? ":#{companion.public_url.port}" : ""}" | ||||
end.join(" ") | ||||
env.response.headers["Content-Security-Policy"] = | ||||
env.response.headers["Content-Security-Policy"] | ||||
.gsub("media-src", "media-src #{invidious_companion.public_url}") | ||||
.gsub("connect-src", "connect-src #{invidious_companion.public_url}") | ||||
.gsub("media-src", "media-src #{invidious_companion_urls}") | ||||
.gsub("connect-src", "connect-src #{invidious_companion_urls}") | ||||
end | ||||
|
||||
rendered "embed" | ||||
|
Original file line number | Diff line number | Diff line change | ||
---|---|---|---|---|
|
@@ -194,10 +194,14 @@ module Invidious::Routes::Watch | |||
|
||||
if CONFIG.invidious_companion.present? | ||||
invidious_companion = CONFIG.invidious_companion.sample | ||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Replaced with
Suggested change
|
||||
invidious_companion_urls = CONFIG.invidious_companion.map do |companion| | ||||
uri = | ||||
"#{companion.public_url.scheme}://#{companion.public_url.host}#{companion.public_url.port ? ":#{companion.public_url.port}" : ""}" | ||||
end.join(" ") | ||||
env.response.headers["Content-Security-Policy"] = | ||||
env.response.headers["Content-Security-Policy"] | ||||
.gsub("media-src", "media-src #{invidious_companion.public_url}") | ||||
.gsub("connect-src", "connect-src #{invidious_companion.public_url}") | ||||
.gsub("media-src", "media-src #{invidious_companion_urls}") | ||||
.gsub("connect-src", "connect-src #{invidious_companion_urls}") | ||||
end | ||||
|
||||
templated "watch" | ||||
|
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -46,8 +46,22 @@ struct YoutubeConnectionPool | |||||
end | ||||||
end | ||||||
|
||||||
class CompanionWrapper | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't think there's a reason for this to be a class.
Suggested change
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Do you think you can also document this type? It wasn't immediately obvious to me what this wrapper was used for. Maybe something like: # Packages a `HTTP::Client` to an invidious-companion instance alongside the configuration for that instance.
#
# This is used as the resource for the `CompanionPool` as to allow the ability to
# query companion instances hosted on a subpath
struct CompanionWrapper |
||||||
property client : HTTP::Client | ||||||
property companion : Config::CompanionConfig | ||||||
|
||||||
def initialize(companion : Config::CompanionConfig) | ||||||
@companion = companion | ||||||
@client = HTTP::Client.new(companion.private_url) | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
end | ||||||
|
||||||
def close | ||||||
@client.close | ||||||
end | ||||||
end | ||||||
|
||||||
struct CompanionConnectionPool | ||||||
property pool : DB::Pool(HTTP::Client) | ||||||
property pool : DB::Pool(CompanionWrapper) | ||||||
|
||||||
def initialize(capacity = 5, timeout = 5.0) | ||||||
options = DB::Pool::Options.new( | ||||||
|
@@ -57,26 +71,28 @@ struct CompanionConnectionPool | |||||
checkout_timeout: timeout | ||||||
) | ||||||
|
||||||
@pool = DB::Pool(HTTP::Client).new(options) do | ||||||
@pool = DB::Pool(CompanionWrapper).new(options) do | ||||||
companion = CONFIG.invidious_companion.sample | ||||||
next make_client(companion.private_url, use_http_proxy: false) | ||||||
client = make_client(companion.private_url, use_http_proxy: false) | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is leaking a client
Suggested change
|
||||||
CompanionWrapper.new(companion: companion) | ||||||
end | ||||||
end | ||||||
|
||||||
def client(&) | ||||||
conn = pool.checkout | ||||||
wrapper = pool.checkout | ||||||
|
||||||
begin | ||||||
response = yield conn | ||||||
response = yield wrapper | ||||||
rescue ex | ||||||
conn.close | ||||||
wrapper.client.close | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
||||||
companion = CONFIG.invidious_companion.sample | ||||||
conn = make_client(companion.private_url, use_http_proxy: false) | ||||||
client = make_client(companion.private_url, use_http_proxy: false) | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
wrapper = CompanionWrapper.new(companion: companion) | ||||||
|
||||||
response = yield conn | ||||||
response = yield wrapper | ||||||
ensure | ||||||
pool.release(conn) | ||||||
pool.release(wrapper) | ||||||
end | ||||||
|
||||||
response | ||||||
|
Original file line number | Diff line number | Diff line change | ||
---|---|---|---|---|
|
@@ -695,22 +695,28 @@ module YoutubeAPI | |||
# Send the POST request | ||||
|
||||
begin | ||||
response = COMPANION_POOL.client &.post(endpoint, headers: headers, body: data.to_json) | ||||
body = response.body | ||||
if (response.status_code != 200) | ||||
raise Exception.new( | ||||
"Error while communicating with Invidious companion: \ | ||||
status code: #{response.status_code} and body: #{body.dump}" | ||||
) | ||||
response_body = "" | ||||
|
||||
COMPANION_POOL.client do |wrapper| | ||||
companion_base_url = wrapper.companion.private_url.path | ||||
puts "Using companion: #{wrapper.companion.private_url}" | ||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Debug puts
Suggested change
|
||||
|
||||
response = wrapper.client.post(companion_base_url + endpoint, headers: headers, body: data.to_json) | ||||
response_body = response.body | ||||
Comment on lines
+704
to
+705
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
You should be able to do something like this: wrapper.client.post("#{companion_base_url}#{endpoint}", headers: headers, body: data.to_json) do | response |
response_body = JSON.parse(response.body_io)
end |
||||
|
||||
if response.status_code != 200 | ||||
raise Exception.new( | ||||
"Error while communicating with Invidious companion: " \ | ||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is duplicating the error message twice when it'd be appended to the
Suggested change
|
||||
"status code: #{response.status_code} and body: #{response_body.dump}" | ||||
) | ||||
end | ||||
end | ||||
|
||||
# Convert result to Hash | ||||
return JSON.parse(response_body).as_h | ||||
rescue ex | ||||
raise InfoException.new("Error while communicating with Invidious companion: " + (ex.message || "no extra info found")) | ||||
end | ||||
|
||||
# Convert result to Hash | ||||
initial_data = JSON.parse(body).as_h | ||||
|
||||
return initial_data | ||||
end | ||||
|
||||
#################################################################### | ||||
|
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add the
Accept-Encoding
header to each companion request as so you aren't decompressing the body. Invidious should just directly proxy the compressed response from companion to the end user.https://crystal-lang.org/api/1.16.3/HTTP/Client.html