[Twitter] Tweet/media can be found through search on the web, but gallery-dl returns "No Tweet results"

[EPQLSA]

Initially, when I was using gallery-dl to download media from a certain user, I kept encountering errors like
Unable to retrieve Tweets from this timeline and Use '-o cursor=xxx...' to continue downloading from the current position.
Later, when I used -o cursor to continue downloading during my last attempt, gallery-dl finished without downloading any files, seemingly as if everything had already been downloaded. Due to the repeated errors from gallery-dl, I was concerned. So, I manually conducted web search to verify and found that the target user still has a significant amount of media that hasn't been downloaded.
Later, I made multiple attempts to download by using "search", but the results were the same. By using the -v option, I found out that gallery-dl returned "No Tweet results" while processing the tweets that hadn't been downloaded. However, I was able to find these media tweets on the web, and I could download them one by one by copying the URLs.
Is this common? Here are the URLs I used for the download or tested:
https://x.com/4Headboiii
https://x.com/search?q=(from:4Headboiii+since:2023-11-1+until:2023-12-22+filter:links)
I expected that gallery-dl should be able to download them.

[EPQLSA]

After several more tests, it seems there are reasons related to both Twitter and the handling strategy of gallery-dl.
On the web search results page under "Latest", the search results are more consistent with the results fetched by gallery-dl. Sometimes there are more results, sometimes fewer. I guess this is due to Twitter's limitations, as I noticed that when there were fewer search results, gallery-dl had at least once shown a "rate limit" message during my tests.
However, when switching to "Top", the results are not like those in "latest". The display shows more results, almost as if there were no limitations. Sometimes, there are tweets that only appear in "Top", and even when there are no limitations, it seems certain tweets just won't be included in "Latest". But I can't be sure if some tweets only appear in "Latest" and not in "Top". Perhaps this situation does exist.
I'm not sure if these issues have been previously discussed, as I haven't looked through past related issues in detail. Anyway, when gallery-dl handles "search", does it truly retrieve results from "Latest" like the web interface? If so, could there be an option to switch between fetching results from "Top" or "Latest" to accommodate various situations?

[mikf]

does it truly retrieve results from "Latest" like the web interface?

It does

Could there be an option to switch between fetching results from "Top"

Sure

[nisehime]

Interestingly, a few moths ago (when I checked) the Top result used to ignore max_id or date-specific modifiers. Seems to be not the case anymore?

Man I hate twitter search. So I tried to do this search again and now it downloads the older tweets that it couldn't download previously. And it does so with 'Latest' results page. It didn't have anything in Top or Latest last time. And to make it even more STUPID I get some error in the browser when using Latest, aahhhhhhhhh.

It's just too incosistent. You might try again like a month later or whatever and it could work. Garbage site.

[nisehime]

Also as for Unable to retrieve Tweets from this timeline I'm having the same issue #8317 (comment). And there seems to be other reports too. I'm tired manually restarting with cursor so once again asking to make it automatically.

However, you would need to make it to ignore Abort option because sometimes starting from cursor would start download from before the last downloaded file, leading to a skip and therefore potential false abort trigger.

[nisehime]

So I tried it. With Top timeline cursor pagination doesn't work the same as with Latest. At some point it stops as if it's the end, but changing max_id gives new results. search-pagination=id suits Top better, but it has other issue. Because Top is not timedate-ordered, the last downloaded ID may not be having lowest timedate which makes the next max_id result batch potentially miss something. For example:

First batch:

1. tweet 25-07-05
2. tweet 25-08-01
3. tweet 25-07-16

Second (cursor):
4. tweet 25-08-10
5. tweet 25-08-05
6. tweet 25-07-01

With search-pagination=id second batch will use max_id of 3rd tweet which means tweets 4 and 5 are not going to be returned by twitter API.

So the workaround algo is cursor -> no res -> max_id switch (using lowest ID downloaded, ie tweet_id < max_id_lowest -> max_id_lowest = tweet_id condition) -> cursor again -> repeat.

But even so it's still going to have the same issue, just at longer distance (at every max_id switch). Just minimizes missed tweets. Ideally it should always be cursor.

That said, sometimes it's Top having resutls and Latest not having, sometimes it's vice versa.

[EPQLSA]

Regarding "top", it seems to always stop at certain points, whether it's the cursor or max_id. I'm curious how Twitter handles "top" searches. If the number of search results isn't very large, and it's just enough to retrieve everything before the "stop point", does this help minimize the omission of some tweets?
If I use search-pagination=cursor and set a smaller date range for the search, for example since:2022-08-01 until:2023-02-01, by configuring the time span to one month and converting the URL of one search into multiple URLs:
since:2022-08-01 until:2022-09-01,
since:2022-09-01 until:2022-10-01,
...
since:2023-01-01 until:2023-02-01
and saving them in a txt file, then using -i to let gallery-dl through this txt file for several downloads, would this be feasible to some extent?

[nisehime]

Regarding "top", it seems to always stop at certain points, whether it's the cursor or max_id

Maybe that's precisely because of your date range? If you set since:2022-08-01 it won't go older than that of course.

and saving them in a txt file, then using -i to let gallery-dl through this txt file for several downloads, would this be feasible to some extent?

To some extent it seems feasible, although there simply might not be any tweets in those ranges, which would make it impractical to go through. And again, twitter search is just quite random in what results it returns, it may not return all tweets even if they exist.

[nisehime]

Actually it seems you're right. Even without since: maxid just stops too. But search-stop=3 seems to fix that.

Ahhh I see. If you run with -v it will explain everything

[twitter][debug] Failed to update 'max_id' search query (AttributeError: 'TwitterSearchExtractor' object has no attribute '_cursor_prefix'). Falling back to 'cursor' pagination

So it's not even using max_id with Top page in the first place. It doesn't explain however why I was able to scroll through the whole timeline this time. Right, because twitter search.

[mikf]

[twitter][debug] Failed to update 'max_id' search query (AttributeError: 'TwitterSearchExtractor' object has no attribute '_cursor_prefix'). Falling back to 'cursor' pagination

Fixed in c6b2041