Speed up startup: parallelize streamer loading + cache Client-Version refresh (request feedback) #779
Description
Is your feature request related to a problem?
I noticed startup can take ~1 to 2 minutes or sometimes way longer than that, even for ~10–20 streamers because initialization does a lot of sequential network work (and also refreshes Client-Version on every GraphQL request). I made changes to reduce startup time and wanted to ask some with more knowledge if this is a good idea.
Proposed solution
1: Parallelized initial streamer loading
In TwitchChannelPointsMiner/TwitchChannelPointsMiner.py, startup was doing two sequential loops:
- Loop A: for each streamer, resolve channel id + set defaults + maybe set up IRC chat
- Loop B: for each streamer, load channel points context + check online status
Each iteration also had time.sleep(random.uniform(0.3, 0.7)), which adds up quickly (e.g. 14 streamers * 2 loops * ~0.5s avg ≈ 14s of sleeping alone, plus network latency).
Old (simplified)
for username in streamers_name:
time.sleep(random.uniform(0.3, 0.7))
streamer.channel_id = twitch.get_channel_id(username)
# settings + chat setup
streamers.append(streamer)
for streamer in streamers:
time.sleep(random.uniform(0.3, 0.7))
twitch.load_channel_points_context(streamer)
twitch.check_streamer_online(streamer)I replaced these loops with ThreadPoolExecutor so the network-bound calls run concurrently (still with a small jitter per task to avoid blasting Twitch all at once). I also kept ordering stable by collecting futures and consuming results in the original order.
New (simplified)
with ThreadPoolExecutor(max_workers=min(8, len(streamers_name))) as ex:
futures = {u: ex.submit(build_streamer, u) for u in streamers_name}
for u in streamers_name:
s = futures[u].result()
if s: streamers.append(s)
with ThreadPoolExecutor(max_workers=min(8, len(streamers))) as ex:
futures = {s.username: ex.submit(hydrate_streamer, s) for s in streamers}
hydrated = []
for s in streamers:
if futures[s.username].result():
hydrated.append(s)
streamers = hydrated2: Cached Client-Version so it isn’t refreshed on every GQL call
In TwitchChannelPointsMiner/classes/Twitch.py, post_gql_request() calls update_client_version() every time.
But update_client_version() does a requests.get(URL) and regex parse to find __twilightBuildID. That means every GQL request may also do an extra HTTP GET to Twitch’s main page, which is expensive during startup where many GQL calls happen back-to-back.
Before (simplified)
def post_gql_request(...):
headers = {"Client-Version": self.update_client_version(), ...}
return requests.post(..., headers=headers)
def update_client_version(...):
response = requests.get(URL)
self.client_version = regex_extract(response.text)
return self.client_versionI added a simple TTL cache:
Store client_version_last_fetch
Reuse self.client_version for max_age_seconds (default 600 seconds / 10 minutes)
After a fetch attempt (success or failure), update client_version_last_fetch so repeated failures don’t cause a tight retry loop
After (simplified)
def update_client_version(self, max_age_seconds=600):
now = time.time()
if self.client_version and (now - self.client_version_last_fetch) < max_age_seconds:
return self.client_version
try:
response = requests.get(URL)
self.client_version = regex_extract(response.text) or self.client_version
finally:
self.client_version_last_fetch = now
return self.client_versionWhy I think this helps
The Client-Version value doesn’t change often enough to justify fetching it for every request, especially during startup.
Caching reduces the number of HTTP requests significantly (roughly 1 per 10 minutes instead of 1 per GQL call).
Alternatives you've considered
No response
Additional context
No response