actual upload speed faster than -ratelimit argument

[Sc0ttmeup]

Seems to be similar / the same issue as #95

The main issue is that the upload rate is a lot higher than the argument provided for -ratelimit

I don't have a progress bar, just numbers, but the lack of progress bar doesn't bother me personally.

• Max upload bandwidth is being hit in the example below
• Specified -ratelimit 1024 but getting closer to 16,150 Kbps upload speed

root@debian:/opt/script# youtubeuploader -cache /opt/config/youtubeuploader/request.token -secrets /opt/config/youtubeuploader/client_secrets.json -ratelimit 1024 -filename /mnt/sdc1/temp/joined/2025-06-11_09-25-26.mov -playlistID #redacted# -privacy unlisted

Uploading file "/mnt/sdc1/temp/joined/2025-06-11_09-25-26.mov"

Progress:  16.15 Mbit/s** ( 1.93 MiB/s), 42304k / 3507658k (1.2%) ETA 29m17s, Elapsed 21s

• Specified -ratelimit 102 but getting closer to 7,000 Kbps upload speed in the example below

 root@debian:/opt/script# youtubeuploader -cache /opt/config/youtubeuploader/request.token -secrets /opt/config/youtubeuploader/client_secrets.json -ratelimit 102 -filename /mnt/sdc1/temp/joined/2025-06-11_09-25-26.mov -playlistID #redacted#  -privacy unlisted
 
Uploading file "/mnt/sdc1/temp/joined/2025-06-11_09-25-26.mov"
 
Progress:   6.99 Mbit/s ( 0.83 MiB/s), 83952k / 3507658k (2.4%) ETA 1h6m52s, Elapsed 1m38s

Youtubeuploader version: 1.25.1
Debian 12 LXC on Proxmox

[porjo]

That's a weird one. I ran a few tests myself with v1.25.1 and was not able to reproduce the issue i.e. the ratelimit I set is what I observed being applied. There is an initial spike before it settles down to the average, but that's to be expected.

If you run it outside of LXC (i.e. directly on the host), do you see the same issue? And what if you run it on another host, connected to the same network?

[Sc0ttmeup]

• On proxmox host, tested as root and non-root user and things behave more as expected.
  No progress meter but -ratelimit is definitely close to where it should be:

• • -ratelimit 102 shows 106 Kbit/s after 36s

user@proxmox_host:/mnt/sdc1/temp/youtubeuploader -cache /mnt/sdc1/temp/ytu_cfg/request.token -secrets /mnt/sdc1/temp/ytu_cfg/client_secrets.json -ratelimit 102 -filename /mnt/sdc1/temp/joined/2025-07-05_09-08-18.mp4 -playlistID #redacted# -privacy unlisted
Uploading file "/mnt/sdc1/temp/joined/2025-07-05_09-08-18.mp4"
Progress:    106 Kbit/s (   13 KiB/s), 464k / 4164823k (0.0%) ETA 89h35m15s, Elapsed 36s

 

• On a QEMU VM of Debian 12 running under proxmox, things were about the same as they were as running on the LXC:

• • -ratelimit 102 shows 6.46 Mbit/s after 33s

user@QEMU_VM:/mnt/sdc1/temp/youtubeuploader -cache /mnt/sdc1/temp/ytu_cfg/request.token -secrets /mnt/sdc1/temp/ytu_cfg/client_secrets.json -ratelimit 102 -filename /mnt/sdc1/temp/joined/2025-07-05_09-06-42_joined.mov -playlistID #redacted# -privacy unlisted
Uploading file "/mnt/sdc1/temp/joined/2025-07-05_09-06-42_joined.mov"
Progress:   6.46 Mbit/s ( 0.77 MiB/s), 25920k / 6982624k (0.4%) ETA 2h27m7s, Elapsed 33s

 

I don't have another non-virtual linux system available just at the moment to test on.

I could try under Windows 11 if you like, although it seems to me that the common link is a virtualized system: the previous issue mentions they're running under ESXi here #95 (comment)

[porjo]

Thanks for the additional info. I have a VPS that runs on kvm/qemu and wasn't able to replicate the issue there. Unfortunately I don't have proxmox or LXC to test against. I don't think this is related to #95 as the fix for that was not related to the virtual environment and also the way the ratelimiting code works has changed significantly since that issue.

I'll leave this issue open for now, and perhaps someone else will encounter the issue and comment. Otherwise, I'm inclined to think this is specific to your particular environment.

[porjo]

@Sc0ttmeup I've just released a new version v1.25.2 which may help with this. Could you give it a try? If the issue persists, could you please add the -debug flag and paste the output here?

[Sc0ttmeup]

Nice! Looks like you got it with that update.

I've only tested a couple of times with 1.25.2, the limiter worked both times.

Version 1.25.1 tried to upload faster than requested every time I tried it on a Proxmox guest.

-ratelimit 102 requested using 1.25.2, settling down and listing 104KBit/s after 55 seconds.

/mnt/sdc1/temp/youtubeuploader -cache /mnt/sdc1/temp/ytu_cfg/request.token -secrets /mnt/sdc1/temp/ytu_cfg/client_secrets.json -ratelimit 102 -filename ./joined/2025-06-30_13-34-48.mp4 -playlistID #redacted# -privacy unlisted
Uploading file "./joined/2025-06-30_13-34-48.mp4"
Progress:    104 Kbit/s (   13 KiB/s), 704k / 6919749k (0.0%) ETA 150h46m34s, Elapsed 55s

[porjo]

Ok, that's great! Would you mind running it with debug flag anyway and pasting the output here - I'm curious to see what values it shows.

[Sc0ttmeup]

No worries, pasted below.

btw, I just noticed that -version isn't outputting anything on 1.25.2

user@proxmox_lxc:/usr/bin/youtubeuploader -debug -cache /mnt/sdc1/temp/ytu_cfg/request.token -secrets /mnt/sdc1/temp/ytu_cfg/client_secrets.json -ratelimit 102 -filename /mnt/sdc1/temp/joined/2025-06-30_13-34-48.mp4 -playlistID #redacted# -privacy unlisted

time=2025-07-16T22:35:32.887+10:00 level=DEBUG msg="youtubeuploader version" version=1.25.2
Uploading file "/mnt/sdc1/temp/joined/2025-06-30_13-34-48.mp4"
time=2025-07-16T22:35:32.934+10:00 level=DEBUG msg="adding file name to request" file=2025-06-30_13-34-48.mp4
time=2025-07-16T22:35:33.196+10:00 level=DEBUG msg="content-Type header" value=application/x-www-form-urlencoded
time=2025-07-16T22:35:33.196+10:00 level=DEBUG msg="requesting URL" url=https://oauth2.googleapis.com/token
time=2025-07-16T22:35:38.751+10:00 level=DEBUG msg="response status" code=200
time=2025-07-16T22:35:38.751+10:00 level=DEBUG msg="response dump" response="HTTP/2.0 200 OK\r\nAlt-Svc: h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000\r\nCache-Control: no-cache, no-store, max-age=0, must-revalidate\r\nContent-Type: application/json; charset=utf-8\r\nDate: Wed, 16 Jul 2025 12:35:38 GMT\r\nExpires: Mon, 01 Jan 1990 00:00:00 GMT\r\nPragma: no-cache\r\nServer: scaffolding on HTTPServer2\r\nVary: Origin\r\nVary: X-Origin\r\nVary: Referer\r\nX-Content-Type-Options: nosniff\r\nX-Frame-Options: SAMEORIGIN\r\nX-Xss-Protection: 0\r\n\r\n{\n  \"access_token\": #redacted#,\n  \"expires_in\": 3599,\n  \"scope\": \"https://www.googleapis.com/auth/youtube https://www.googleapis.com/auth/youtubepartner https://www.googleapis.com/auth/youtube.upload\",\n  \"token_type\": \"Bearer\"\n}"
time=2025-07-16T22:35:38.751+10:00 level=DEBUG msg="content-Type header" value=application/json
time=2025-07-16T22:35:38.751+10:00 level=DEBUG msg="requesting URL" url="https://youtube.googleapis.com/upload/youtube/v3/videos?alt=json&notifySubscribers=true&part=snippet&part=status&part=recordingDetails&prettyPrint=false&uploadType=resumable"
time=2025-07-16T22:35:46.605+10:00 level=DEBUG msg="response status" code=200
time=2025-07-16T22:35:46.611+10:00 level=DEBUG msg="response dump" response="HTTP/2.0 200 OK\r\nContent-Length: 0\r\nAlt-Svc: h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000\r\nContent-Type: text/plain; charset=utf-8\r\nContent-Type: application/json; charset=UTF-8\r\nDate: Wed, 16 Jul 2025 12:35:46 GMT\r\nLocation: https://youtube.googleapis.com/upload/youtube/v3/videos?alt=json&notifySubscribers=true&part=snippet&part=status&part=recordingDetails&prettyPrint=false&uploadType=resumable&upload_id=#redacted#\r\nServer: UploadServer\r\nVary: Origin\r\nVary: X-Origin\r\nVary: Referer\r\nX-Guploader-Uploadid: #redacted#\r\n\r\n"
time=2025-07-16T22:35:46.611+10:00 level=DEBUG msg="content-Type header" value=video/mp4
time=2025-07-16T22:35:46.611+10:00 level=DEBUG msg="requesting URL" url="https://youtube.googleapis.com/upload/youtube/v3/videos?alt=json&notifySubscribers=true&part=snippet&part=status&part=recordingDetails&prettyPrint=false&uploadType=resumable&upload_id=#redacted#"
time=2025-07-16T22:35:46.611+10:00 level=DEBUG msg="limiter: creating limiter" burstlimit=16384 ratelimit=102 "initial buf len"=16384
Progress:    105 Kbit/s (   13 KiB/s), 512k / 6919749k (0.0%) ETA 149h23m44s, Elapsed 40s

[porjo]

Thankyou for sending that output. Unfortunately, I'm not see anything there that explains the previous behaviour you were seeing. But I'm glad it's working now - let's see how it goes.

btw, I just noticed that -version isn't outputting anything on 1.25.2

Thanks, someone else reported that too. It'll be fixed in the next release.