Skip to content

feat: overhaul Smart Speed engine with SponsorBlock, captions, and modular architecture (#4147) - #4355

Merged
ImprovedTube merged 1 commit into
code-charity:masterfrom
AryaXDG:master
Sep 24, 2026
Merged

ImprovedTube merged 1 commit into
code-charity:masterfrom
AryaXDG:master

Conversation

@AryaXDG

@AryaXDG AryaXDG commented Sep 22, 2026

Copy link
Copy Markdown
Member

PR: overhaul Smart Speed engine with SponsorBlock, captions, and modular architecture

Summary

This replaces the earlier proof-of-concept heatmap implementation with a full rewrite of the Smart Speed engine. It keeps the original core idea intact using YouTube's "Most Replayed" heatmap markers to slow down for high-engagement scenes and speed up for low-engagement ones but turns it into a modular, multi-signal system and fixes a number of reliability issues in the old version.

Why

Issue #4147 asks for using the heatmap's 100 intensity markers to auto-adjust playback speed, since view-density per scene tends to correlate with relevance (slow intros/outros, catch the "spikes"). The old code (partly implemented in #3628) proved the concept but was a single 400-line object with global mutable state, no cleanup between videos, and no way to combine the heatmap with other relevance signals the issue also discusses (e.g. captions/transcript, sponsor segments).

What changed

Architecture

  • Rewritten as a self-contained IIFE with clearly separated modules instead of one flat ImprovedTube.heatmap object:
    • SpeechMap timeline of speech blocks with eased lead-in/release
    • SponsorBlockClient k-anonymity SponsorBlock API lookup with caching
    • HeatmapSource tiered heatmap acquisition (active player -> page globals -> local cache -> watch-page fetch)
    • CaptionSource transcript acquisition (TimedText API -> transcript panel scrape)
    • Profiles channel/category/language profile resolution
    • Signals / computeCompositeSpeed weighted blending of multiple speed signals
    • Overrides absolute (non-blended) speed rules, e.g. SponsorBlock segments
    • DurationSolver binary search for target-duration bias
    • Session per-video state, replacing shared global fields
    • Engine state machine driving the whole lifecycle
    • UI all DOM work, so the engine itself never touches the DOM
  • Introduced a Session object created per video instead of resetting fields on a shared object avoids stale state leaking across video changes.
  • Added an explicit state machine (IDLE / LOADING / ACTIVE / WHITELISTED / EXCLUDED / PAUSED / AD) instead of ad-hoc boolean flags (sessionDisabled, etc).

New signals beyond the raw heatmap

  • Captions/transcript signal: pulls captions via the TimedText API (or scrapes the transcript panel as fallback), builds "speech blocks," and slows playback around dialogue directly addresses the issue's suggestion to factor in spoken content, not just view density.
  • SponsorBlock integration: sponsor segments play at a fixed fast speed; intro/outro segments use a dedicated speed; if no SponsorBlock outro data exists, a heuristic treats the last ~3% of the video as outro.
  • Signals are blended by confidence-weighted average (computeCompositeSpeed) rather than heatmap alone, so future signals (e.g. word-difficulty/WPM from the issue) can be added to Signals without touching the engine.

Reliability fixes

  • Heatmap acquisition now checks belongsToVideo() before trusting cached/global data, preventing stale data from a previous video being applied to a new one.
  • Manual speed changes by the user (or another extension) are detected (expectedRate drift check) and pause Smart Speed instead of fighting the user.
  • Ad detection via MutationObserver on the player's ad-showing class, forcing base rate during ads.
  • Requests use AbortController so in-flight heatmap/caption/SponsorBlock fetches are cancelled when the user navigates away.
  • Segment boundary crossfade (_blendAcrossBoundary) smooths speed transitions over 2.5s, same idea as before but generalized to work with the composite signal.

Profiles

  • Profile resolution now supports a priority order across channel -> category -> language, configurable via smart_speed_rule_priority (previously: channel-name lookup only, keyed by fragile display-name strings).
  • Channel profiles are now keyed by canonical channel ID (channel:<id>) instead of display name, with automatic migration of legacy name-keyed profiles once an ID is available.

Target duration

  • Replaced the old greedy knapsack allocation (sort segments by intensity, eat into the least-intense first) with a binary search over a global speed bias that predicts total wall-clock time across all segments more accurate and works with the new multi-signal composite speed instead of the heatmap alone.

UI

  • Skip Intro / Skip Sponsor buttons added (driven by SponsorBlock data).
  • Live stream and Shorts whitelist toggle buttons added, shown only when relevant.
  • Indicator, whitelist shield, and skip buttons all refresh on a 1s interval via UI.refreshContextual instead of being scattered across handlers.

Will be addressed next in future work

  • Word-difficulty / WPM-based content-density signal

@ImprovedTube
ImprovedTube merged commit 9912a58 into code-charity:master Sep 24, 2026
1 check passed
@ImprovedTube2

Copy link
Copy Markdown
Member

Amazing!!

@ImprovedTube2

ImprovedTube2 commented Sep 24, 2026 •

Copy link
Copy Markdown
Member

Fully Tested? Ready to merge in Beta or everywhere?

predicts total wall-clock time.

Complete heatmap and transcript processing might run patiently? Per video or per segment? Yet a user could still target 1.5x average speed per video to compared improvements over static speed?

a heuristic treats the last ~3% of the video as outro.

depends video length i suppose.
many popular channels might apply something close to a 'wadsworth-constant' too ('first 30% are the intro')

smooths speed transitions over 2.5s, same idea

then #4265 remains open i guess

transcript ... dialogue

Structure, Chapters, Speakers might be labeled too 😮💡

Live stream and Shorts whitelist toggle buttons added, shown only when relevant.

Shown for every Short and live Live? Sticky toggle?

Indicator, whitelist shield, and skip buttons all refresh on a 1s interval via UI.refreshContextual instead of being scattered across handlers.

Should(n't) this depend the type? (Whitelist only changes on click? Skip only per segment?)

k-anonymity

currently a prefix of 4 chars results in average 1.38 videos among the 11.5 Million videos.
3 chars shows and average 44 videos

The DB is 2.5 GB gzipped. A few MB can store a reference to all Channel IDs appearing in it repeatedly.
(Potentially minimizing required API requests for most users)

a Channels (each have b Videos) b Videos (in the Sponsorblock DB) % of videos above % of channels above Number of channels above
1 160152 0.0% 0.0% 0
1 54303 1.4% 0.0% 1
1 38136 1.9% 0.0% 2
1 31961 2.2% 0.0% 3
1 29496 2.5% 0.0% 4
1 27883 2.7% 0.0% 5
1 27746 3.0% 0.0% 6
1 25135 3.2% 0.0% 7
1 21639 3.4% 0.0% 8
1 21218 3.6% 0.0% 9
1 21001 3.8% 0.0% 10
1 19074 4.0% 0.0% 11
1 17532 4.2% 0.0% 12
1 17294 4.3% 0.0% 13
1 15969 4.5% 0.0% 14
1 15704 4.6% 0.0% 15
1 14590 4.7% 0.0% 16
1 14018 4.9% 0.0% 17
1 13271 5.0% 0.0% 18
1 13263 5.1% 0.0% 19
1 13120 5.2% 0.0% 20
1 12785 5.3% 0.0% 21
1 12328 5.4% 0.0% 22
1 12309 5.6% 0.0% 23
1 11724 5.7% 0.0% 24
1 11641 5.8% 0.0% 25
1 11084 5.9% 0.0% 26
... ... ... ... ...
1 4477 9.7% 0.0% 90
1 4443 9.8% 0.0% 91
1 4386 9.8% 0.0% 92
1 4366 9.8% 0.0% 93
1 4349 9.9% 0.0% 94
1 4326 9.9% 0.0% 95
1 4304 10.0% 0.0% 96
1 4284 10.0% 0.0% 97
1 4272 10.0% 0.0% 98
1 4257 10.1% 0.0% 99
1 4246 10.1% 0.0% 100
1 4243 10.1% 0.0% 101
1 4240 10.2% 0.0% 102
1 4222 10.2% 0.0% 103
1 4206 10.3% 0.0% 104
1 4154 10.3% 0.0% 105
1 4142 10.3% 0.0% 106
1 4083 10.4% 0.0% 107
1 4012 10.4% 0.0% 108
1 3974 10.4% 0.0% 109
1 3931 10.5% 0.0% 110
1 3930 10.5% 0.0% 111
1 3923 10.5% 0.0% 112
1 3908 10.6% 0.0% 113
1 3902 10.6% 0.0% 114
1 3891 10.6% 0.0% 115
1 3876 10.7% 0.0% 116
1 3862 10.7% 0.0% 117
1 3843 10.7% 0.0% 118
1 3720 10.8% 0.0% 119
1 3712 10.8% 0.0% 120
1 3684 10.8% 0.0% 121
1 3679 10.9% 0.0% 122
1 3634 10.9% 0.0% 123
1 3631 10.9% 0.0% 124
1 3599 11.0% 0.0% 125
1 3583 11.0% 0.0% 126
2 3582 11.0% 0.0% 127
... ... ... ... ...
1 1126 18.6% 0.1% 592
1 1123 18.6% 0.1% 593
1 1121 18.6% 0.1% 594
2 1120 18.6% 0.1% 595
2 1119 18.6% 0.1% 597
3 1118 18.7% 0.1% 599
1 1117 18.7% 0.1% 602
1 1115 18.7% 0.1% 603
2 1114 18.7% 0.1% 604
3 1112 18.7% 0.1% 606
1 1110 18.8% 0.1% 609
1 1105 18.8% 0.1% 610
2 1104 18.8% 0.1% 611
4 1100 18.8% 0.1% 613
1 1097 18.8% 0.1% 617
1 1096 18.8% 0.1% 618
2 1095 18.8% 0.1% 619
1 1094 18.9% 0.1% 621
3 1093 18.9% 0.1% 622
2 1092 18.9% 0.1% 625
2 1090 18.9% 0.1% 627
... ... ... ... ...
4 686 23.2% 0.1% 1201
2 685 23.2% 0.1% 1205
4 684 23.2% 0.1% 1207
3 683 23.2% 0.1% 1211
4 682 23.2% 0.1% 1214
4 681 23.3% 0.1% 1218
2 679 23.3% 0.1% 1222
4 678 23.3% 0.1% 1224
2 677 23.3% 0.1% 1228
3 675 23.3% 0.1% 1230
3 674 23.3% 0.1% 1233
2 672 23.4% 0.1% 1236
... ... ... ... ...
8 367 31.0% 0.3% 3044
8 366 31.0% 0.3% 3052
12 365 31.0% 0.3% 3060
9 364 31.0% 0.3% 3072
13 363 31.1% 0.3% 3081
8 362 31.1% 0.3% 3094
10 361 31.1% 0.3% 3102
17 360 31.2% 0.3% 3112
12 359 31.2% 0.3% 3129
14 358 31.3% 0.3% 3141
... ... ... ... ...
130 108 48.5% 1.2% 13813
132 107 48.7% 1.2% 13943
149 106 48.8% 1.2% 14075
128 105 48.9% 1.2% 14224
120 104 49.0% 1.3% 14352
165 103 49.1% 1.3% 14472
158 102 49.3% 1.3% 14637
179 101 49.4% 1.3% 14795
150 100 49.6% 1.3% 14974
144 99 49.7% 1.3% 15124
177 98 49.8% 1.3% 15268
172 97 50.0% 1.4% 15445
154 96 50.1% 1.4% 15617
175 95 50.3% 1.4% 15771
... ... ... ... ...
394 58 56.4% 2.2% 25310
429 57 56.6% 2.3% 25704
434 56 56.9% 2.3% 26133
387 55 57.1% 2.3% 26567
465 54 57.3% 2.4% 26954
485 53 57.5% 2.4% 27419
462 52 57.7% 2.4% 27904
524 51 57.9% 2.5% 28366
499 50 58.1% 2.5% 28890
566 49 58.4% 2.6% 29389
560 48 58.6% 2.6% 29955
589 47 58.8% 2.7% 30515
621 46 59.1% 2.7% 31104
650 45 59.3% 2.8% 31725
618 44 59.6% 2.8% 32375
708 43 59.8% 2.9% 32993
690 42 60.1% 3.0% 33701
723 41 60.3% 3.0% 34391
737 40 60.6% 3.1% 35114
782 39 60.8% 3.1% 35851
811 38 61.1% 3.2% 36633
903 37 61.4% 3.3% 37444
896 36 61.7% 3.4% 38347
1011 35 62.0% 3.4% 39243
1025 34 62.3% 3.5% 40254
1092 33 62.6% 3.6% 41279
1136 32 62.9% 3.7% 42371
1194 31 63.2% 3.8% 43507
1231 30 63.5% 3.9% 44701
1306 29 63.8% 4.0% 45932
1330 28 64.2% 4.1% 47238
1446 27 64.5% 4.3% 48568
1617 26 64.8% 4.4% 50014
1703 25 65.2% 4.5% 51631
1765 24 65.6% 4.7% 53334
1979 23 65.9% 4.8% 55099
2043 22 66.3% 5.0% 57078
2230 21 66.7% 5.2% 59121
2552 20 67.1% 5.4% 61351
2626 19 67.6% 5.6% 63903
2871 18 68.0% 5.8% 66529
3223 17 68.5% 6.1% 69400
3557 16 68.9% 6.4% 72623
3973 15 69.4% 6.7% 76180
4426 14 70.0% 7.0% 80153
5020 13 70.5% 7.4% 84579
5818 12 71.1% 7.9% 89599
6621 11 71.7% 8.4% 95417
7578 10 72.3% 9.0% 102038
9196 9 73.0% 9.6% 109616
11490 8 73.7% 10.4% 118812
14364 7 74.5% 11.4% 130302
19212 6 75.4% 12.7% 144666
26002 5 76.4% 14.4% 163878
39002 4 77.5% 16.7% 189880
66386 3 78.9% 20.1% 228882
145184 2 80.6% 25.9% 295268
699381 1 83.1% 38.6% 440452

@AryaXDG

AryaXDG commented Sep 29, 2026 •

Copy link
Copy Markdown
Member Author

@ImprovedTube2 Thanks for the review!

Testing: I've tested it by hand, but there are no automated tests yet. I plan to add unit tests later. I'd call it a stable beta: the structure is modular, but I know of a few rough edges (e.g. the Shorts toggle also affects the short-video threshold) that I'll fix. Getting users to try it in Beta is the best way to find the remaining small bugs.

Processing: Heatmap and captions are processed once per video. Each tick (every 100ms) only does a cheap lookup. The target-duration solver runs only when the user clicks Apply. A target average speed (like 1.5x) would be easy to add, and it would make comparing against a static speed simple.

Outro / intro: The outro is the last 3%, capped at 15s, and only used when SponsorBlock has no outro data. I haven't added an intro rule yet. A Wadsworth-style rule is a good idea, and I'd make it optional.

#4265 / smoothing: It was built for the old version, so it isn't merged. Your comments there apply to the current 2.5s blend too, and I agree with most of them:

  • Timing: the change could happen anywhere in a segment, so I'd center the transition on the boundary instead of only ramping before it. The fixed 2.5s is also only ~1% of the video for a ~4 min video, so I'd scale the window with segment length (with a cap).
  • Shape: I'd use one smooth in/out curve, applied in log space so 1x to 2x feels like 2x to 4x, and I'll check by ear.
  • Threshold: the JS cost of setting the rate is negligible. I'll measure the audio side rather than guess.
  • Pauses: changing speed at pauses in speech (or at most every X seconds) should distort less than ramping mid-sentence. Captions often have no full stops, so I'd use gaps between caption cues.

I'd also add a single 0 to 1 smoothness setting that controls transition length (0 = instant), with a default close to today's behavior, instead of exposing several curve options.

Chapters and speakers: Good idea. Each can be added as a new signal without changing the engine.

Shorts / Live toggles: Right now they show on every Short or live stream, and the setting is global and sticky. I agree it could be per-type or per-session, and I'll look at it.

1s UI refresh: Agreed. I'll switch to events: the whitelist button updates on click, the Live/Shorts buttons on navigation, and the skip buttons when the segment changes.

k-anonymity: Your numbers made me check, so I ran the whole SponsorBlock dump (10.5M unique videos) through the same hashing our code uses. Your figures match a plain video-ID prefix (1.35 for 4 chars, 41 for 3). We send a hex SHA-256 prefix, which has fewer symbols per character:

Prefix Videos sharing it (avg)
4 chars (current) about 161 (min 111, max 219)
3 chars about 2,563

So 4 chars already hides each request among about 160 videos, and I kept user privacy in mind while writing this. Going to 3 would only make responses about 16x bigger. The channel-ID list is still a good idea to cut requests, and I'd do it as a follow-up.

Next: I'm planning a second major PR around the end of October that adds more signals so the speed adapts better to each video's context. One idea is to embed the transcript, split the video into sections, score each section's relevance to the video's main topic, and then score individual sentences within their section.

@hkzo

hkzo commented Oct 2, 2026

Copy link
Copy Markdown
Collaborator

@AryaXDG

While working on combining playback speed buttons A and B in connection with #4343, I noticed that the button's custom-speed setting, player_custom_playback_speed, is also used as the base rate for Smart Speed.

I'm planning to allow custom speeds up to 16× for the combined button. Since _baseRate() already accepts values up to 16, I think this change is within the range it expects.

With the current settings menu, both the custom speed and Smart Speed's target multiplier are limited to 4, so their product stays within 16×. Increasing the custom-speed limit to 16× would allow the final speed to exceed 16×, depending on targetSpeed. I confirmed in Chrome that setting a playback rate above 16× causes an error, so an upper-limit check when tick() calculates finalSpeed and passes it to _setRate() might be needed.

@ImprovedTube2

Copy link
Copy Markdown
Member

hi! @hkzo first we had "Speed watching: permanent Speed" (player_forced_playback_speed) Many users have this. (did _baseRate mean this? @AryaXDG )

Permanent Speed remains limited to 3.17x because i didn't want to be responsible for anything above.
Similarly Smart Speed couldn't help much higher.

16x 'fast forward' might be fun just usually not optimal, but of course shortcuts and Buttons may overwrite everything.
Advanced player button with memory as envisioned years ago: #4146 (comment)

@hkzo

hkzo commented Oct 3, 2026

Copy link
Copy Markdown
Collaborator

"Speed watching: permanent Speed" (player_forced_playback_speed) is separate because it uses the value from storage.player_playback_speed.

In this PR, _baseRate() is implemented as follows, clamping the rate between 0.0625 and 16:

https://github.com/code-charity/youtube/pull/4355/changes#diff-fe348b00e097a372c7e9f1fed3dfd74c51aea19b178bd016735c7f579aa68bd2R4225-R4232

 
_baseRate() { return toNum(ImprovedTube.storage.player_custom_playback_speed, 1, 0.0625, 16); },

_setRate(rate) {
    const v = this._getVideo();
    if (!v || Math.abs(v.playbackRate - rate) < 0.01) return;
    this.expectedRate = rate;
    v.playbackRate = rate;
},

In this PR, finalSpeed is implemented as follows,

https://github.com/code-charity/youtube/pull/4355/changes#diff-fe348b00e097a372c7e9f1fed3dfd74c51aea19b178bd016735c7f579aa68bd2R4521-R4522

const finalSpeed = isOverrideSpeed ? targetSpeed : this._baseRate() * targetSpeed;
this._setRate(finalSpeed);

Chromium(Chrome,Edge,...) throws a DOMException (NOT_SUPPORTED_ERR) when playbackRate exceeds 16.

@AryaXDG

AryaXDG commented Oct 3, 2026

Copy link
Copy Markdown
Member Author

Hey @hkzo @ImprovedTube2
Thanks for bringing this up.

Just to clarify on @ImprovedTube2 question, _baseRate() uses player_custom_playback_speed (the custom speed button's setting), not Permanent Speed (player_forced_playback_speed). Permanent Speed is completely separate and stays capped at 3.17× as intended.

To fix the DOMException error when speeds exceed 16×, I just opened and merged #4371.

Instead of checking tick(), I updated _setRate() so it automatically clamps all incoming rates to [0.0625, 16] using toNum(rate, 1, 0.0625, 16). This acts as a global safety guard for Smart Speed so Chrome will never throw an error, while keeping internal state (this.expectedRate) in sync!

@ImprovedTube2

ImprovedTube2 commented Oct 4, 2026 •

Copy link
Copy Markdown
Member

hi @hkzo @AryaXDG! it is great if we connect features!
tried to say,
as long as the sum of smart, permanent and manual is below 3.5x or so,
then all three could meaningfully be interpreted as an addition.
(Or for, Smart and Permanent, they could be mutually exclusive toggles, disabling each other, as done for other features).
Then, beyond a certain maximum speed, manual speed is rather bypassing the others, since at very fast speed like 4x it may be too hard to understand videos for long.
Spending even less time could be through automated skipping silence sections and manual skipping,
hile regular buttons, sliders or shortcuts may be optimized for productivity and for fun / hackability, it might be enough, if we just start to allow unlimited amounts in the input in the text fields next to linear sliders, but let the sliders show the common range) (if manual speed settings are offered beyond a maximum shortly productive level (a bit above 4x?), that appears to be for fun, for the effect and the faster, the shorter it could be applied with a timer, not requiring a second click to undo)

@ImprovedTube2

ImprovedTube2 commented Oct 4, 2026 •

Copy link
Copy Markdown
Member

We send a hex SHA-256 prefix

💡 sorry. (Then 5 digits might happen😀)

Getting users to try it in Beta is the best way to find the remaining small bugs.

Unfortuantely we dont see an amount of users per feature yet, but since Smart Speed is young and without many revisions, we may upload for everybody soon unless there is a clear disadvantage or missing legacy option of current users.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants