[Twitter / X] Stops downloading after a certain point, even though there are more images

[Ikkoru]

Used version 2025.12.01.

Stopped on 1857568929792798741 (2024.11.15) after which I got the infinite "rate limit" message.
When scrolling through this user's Media tab on X all the way to the bottom, the above post does appear to be the last one (it stops loading more images at that point).

However, I managed to manually find several posts from before this supposed last post:
2024.08.18;
2024.08.23;
2024.09.11;
2024.09.12.

[mikf]

There are lots of posts after the /medialimit (1857568929792798741), at least several hundreds or even thousands, but you need to use Twitter's search to access/find them, which seems to be "protected" by an infinite rate limit for you. I was able to get to 1835676545500995883 before getting hit with a rate limit for the first time.

Try your luck with https://x.com/search?q="from:Mt_ONIGIRI max_id:1857568929792798741"
Maybe Twitter will let you access more posts without immediate rate limit at some point.

edit:
The last ID seems to be 1736173390187417924

[Ikkoru]

Thanks for the suggestion. So it's just Twitter being Twitter...

I tried it but got hit with a rate limit without DLing a single image. Did you try the above with the default config settings? I wonder if I'm doing something wrong.

[mikf]

Did you try the above with the default config settings?

Yep

I wonder if I'm doing something wrong.

You are living in and connecting from a place that's heavily limited by Twitter.
Just move somewhere else /s.

I don't have many connection/ratelimit problems with my residential IP over here in Germany.

[Ikkoru]

You are living in and connecting from a place that's heavily limited by Twitter.

It's surprising that it wouldn't let me DL even a single image. It's not like the rate limit pops up instantly, so I assume it's doing something, yet nothing's happening...

No workarounds at the moment?

When increasing sleep-request to around 20, it doesn't rate limit me, but no images are being downloaded either. Is this an issue on Twitter's side of things?

[Ikkoru]

I switched on verbose, and it appears that using the link you gave me makes gallery-dl enter a loop of looking through the same tweets over and over.

Full logs (note, I replaced the space between ONIGIRI and max_id with a + as a test):

with plus.txt

without plus.txt

[mikf]

No workarounds at the moment?

None that i am aware of.

Is this an issue on Twitter's side of things?

Yep, pretty sure it is.

it appears that using the link you gave me makes gallery-dl enter a loop of looking through the same tweets over and over.

That's kind of on Twitter for not providing functioning cursor values and/or respecting them. Also: Works on my machine ™️

Try "search-pagination": "max_id"
Maybe you get better results with that.

I replaced the space between ONIGIRI and max_id with a + as a test):

The + makes no difference, as it is converted to a space before being used as search query anyways:

gallery-dl/gallery_dl/extractor/twitter.py

Line 682 in c6b2041

query = text.unquote(self.user.replace("+", " "))

In any case, your without_plus attempt ran without max_id:… parameter

[gallery-dl][debug] Starting DownloadJob for 'https://x.com/search?q=from:Mt_ONIGIRI'

[Ikkoru]

In any case, your without_plus attempt ran without max_id:… parameter

Yeah, I thought that was what happened, and that's why I did the plus thing. I assume it's CMD's fault. On that note, do you run it on Linux, or do you use a different shell?

I tried with "search-pagination": "max_id", and it's not in a loop anymore, but it still started from 1994627711969759504 (not the user's tweet).

Log: pagination max_id.txt

Config, just in case:

		"twitter":
		{
			"cookies"	: "./[Cookies]/x.com_cookies.txt",
			"base-directory": "Z:\\Visual Media\\[Pics]\\[Artists]\\",
			"filename"	: "{date:%Y.%m.%d} {tweet_id} {num:>02}.{extension}",
			"directory"	: ["Twitter", "{author['nick']} ({author['name']})"],
			"avatar"	: {
				"filename"	: "avatar {date:%Y.%m.%d}.{extension}"
			},
			"background": {
				"filename"	: "bg {date:%Y.%m.%d}.{extension}"
			},
			"unavailable": true,
			"include"	: ["avatar", "background", "timeline"],
			"locked"	: "wait",
			"expand"	: true,
			"replies"	: "self",
			"#": "Following setting is due to https://github.com/mikf/gallery-dl/issues/8617",
			"image-filter": "author is user",
			"search-pagination": "max_id",
			"sleep-request": "20.0-21.0",
			"sleep"		: "1.0-1.5",
			"archive"	: "$HOME/SQL_{category}.sqlite3",
			"skip"		: "abort:100000"
		},

[mikf]

"https://x.com/search?q="from:Mt_ONIGIRI+max_id:1857568929792798741""

Don't "double quote", use one quote at the start and one at the end

"https://x.com/search?q=from:Mt_ONIGIRI+max_id:1857568929792798741"

although it did use the max_id:… thingy this time.

			"expand"	: true,

This might be the problem and the reason you are getting duplicate Tweets en masse.

do you run it on Linux, or do you use a different shell?

I'm on Linux, using a standard VTE terminal and a Bash shell.

[Ikkoru]

although it did use the max_id:… thingy this time.

cmd is silly to the extreme, and I'm not sure why I'm still using it...

Just for more info, it started looping again.

Log: more pagination max.txt (search for 1995472461576409100)

I'll try removing expand, but from what I understood of the config, it won't download images of the author that they posted as replies to their own posts?

[Ikkoru]

Removed expand, still loops. It seems it's failing to use max_id?
Failed to update 'max_id' search query (AttributeError: 'TwitterSearchExtractor' object has no attribute '_cursor_prefix'). Falling back to 'cursor' pagination

without expand.txt

Currently trying without double quotes and with space instead of plus, but that shouldn't change anything, right?

[mikf]

The AttributeError was fixed yesterday in commit c6b2041

[Ikkoru]

Wow, blink and you'll miss it. (๑•́ㅿ•̀๑)ᔆᵒʳʳᵞ

[Ikkoru]

Updated to 1.31.0-dev:2025.12.02. Still looping.

expand true.txt

expand false.txt

Other than expand, the config is as above.