Rate limiting

[durack1]

Hi folks, I've got a query regarding rate limiting. I've been hitting https://esgf-node.ornl.gov/esgf-1-5-bridge? with queries, to get Dataset/File information out of the index, and have been hitting intermittent timeout errors, e.g.,

urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='[esgf-node.ornl.gov](http://esgf-node.ornl.gov/)', port=443): Read timed out. (read timeout=5)

If I change the default timeout from 5 to 25 this seems to work, but sometimes I get the same error, and if I drop this to a smaller timeout, then I can reproduce the error above almost 100%.

Is there some rate limiting going on, and if yes, what is the current limit, can we get this reset for certain IPs/ranges etc?

@sashakames ping

[durack1]

Ok and even with timeout=25 I have hit the same error, so incrementing back up to 30.. This is going to get REALLY slow

[durack1]

Ok even with timeout=30 I am getting timeouts, this is becoming very painful.

And just for notes, this is 222 serial queries targeting a source_id/Dataset level query

[sashakames]

@durack1 222 consecutive queries might indeed be rate limited. @bstrdsmkr any advice here?

[nocollier]

If you are using Python, I have used the backoff package to help here. https://pypi.org/project/backoff/ These give you decorators that will keep trying to execute your function with an increasingly long time delay until success. Basically gives you a dynamic timeout option that starts small and grows and resets for the next request.

…

On Fri, Jun 27, 2025 at 3:18 PM Sasha Ames ***@***.***> wrote: *sashakames* left a comment (esgf2-us/esg_fastapi#76) <#76 (comment)> @durack1 <https://github.com/durack1> 222 consecutive queries might indeed be rate limited. @bstrdsmkr <https://github.com/bstrdsmkr> any advice here? — Reply to this email directly, view it on GitHub <#76 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKFCB5Q542GGIID4THBQ4D3FWKKBAVCNFSM6AAAAACAJSQKPGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTAMJUGEZTCOJXGM> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[durack1]

If you are using Python, I have used the backoff package to help here.

@nocollier thanks for the workarounds, always good to have these in my pocket.

I am more keen to ascertain what the conditions are that are imposed on users of the system, so am definitely keen to understand what a general user/query can do, and what will be stopped. As my experience was so intermittent, this is something other services/folks will encounter, so it's best to get a handle on how this is all working before the CMIP7 hoard descends in a couple of months.

We also have experience on the LLNL SOLR node where huge numbers of concurrent requests from server farms and clouds (Amazon Singapore, are you listening..?), so getting a handle on all this early would be better than having to deal with this real-time with the production CMIP7 system.

[bstrdsmkr]

@durack1 if you were to hit the rate limits, you should get a response code 429. In contrast, these are timeouts on the client side. Our upstream timeout is set to 60s and we've not seen any queries go above that yet

That said, I'd imagine you're doing something like stepping through results with a limit and an offset, is that the case? That sort of "deep indexing" is pretty expensive on Globus Search and can take a while to respond sometimes.

If you can share more about your queries and what you're trying to accomplish, we might be able to find some tricks or tweaks to speed things up

[sashakames]

I am more keen to ascertain what the conditions are that are imposed on users of the system, so am definitely keen to understand what a general user/query can do, and what will be stopped. As my experience was so intermittent, this is something other services/folks will encounter, so it's best to get a handle on how this is all working before the CMIP7 hoard descends in a couple of months.

On this point, agreed it will be important to learn how controls will impact potential users. However, I should point out that this particular API and its deployment are not the target system for CMIP7. That will be the STAC discovery API, which will be hosted by Globus production services (in "the cloud"). Moveover, CMIP6 and the other active projects will be migrated to this (part of ESGF-NG). Absolutely we need to communicate service-level requirements to ensure users can accomplish reasonable tasks. several hundred queries should be in scope for that, 1000s may warrant use of a lower-level cursor-API.

[durack1]

If you can share more about your queries and what you're trying to accomplish, we might be able to find some tricks or tweaks to speed things up

@bstrdsmkr here is the example that I've been using - I am guessing it'll run for you if you wanted to pull it down - https://github.com/PCMDI/input4MIPs_CVs/blob/main/scripts/pollESGF.py

[durack1]

Has a change happened to the esgf-1-5-bridge.

Just to clarify, I am querying "https://esgf-node.ornl.gov/esgf-1-5-bridge?.." and weirdly, now getting errors from the SOLR index "esgf-node.llnl.gov".

It seems that a cronjob that was working until 3 pm yesterday is now hitting esgf-node.llnl.gov

requests.exceptions.ConnectionError: HTTPSConnectionPool(host='esgf-node.llnl.gov', port=443): Max retries exceeded with url: /esg-search/search/?limit=1000&format=application%2Fsolr%2Bjson&source_id=DRES-CMIP-BB4CMIP7-2-0&project=input4mips&project=input4MIPs&distrib=fals
e&fields=* (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x149a98f1e850>: Failed to establish a new connection: [Errno 113] No route to host'))

[bstrdsmkr]

@durack1 I see 3 queries in there, the first two are targeting Solr directly with only the 3rd one going through the API. Think these should be equivalent queries:

1. https://github.com/PCMDI/input4MIPs_CVs/blob/main/scripts/pollESGF.py#L145
   curl -v -L 'https://esgf-node.ornl.gov/esgf-1-5-bridge/search?type=Dataset&replica=false&activity_id=input4MIPs&facets=source_id&limit=0'
2. https://github.com/PCMDI/input4MIPs_CVs/blob/main/scripts/pollESGF.py#L157
   curl -v -L 'https://esgf-node.ornl.gov/esgf-1-5-bridge/search?type=File&replica=false&activity_id=input4MIPs&facets=source_id&limit=0'
3. https://github.com/PCMDI/input4MIPs_CVs/blob/main/scripts/pollESGF.py#L218
   curl -v -L 'https://esgf-node.ornl.gov/esgf-1-5-bridge/search?type=Dataset&source_id=MRI-JRA55-do-1-5-0&project=input4mips,input4MIPs&limit=1000'

[bstrdsmkr]

Has a change happened to the esgf-1-5-bridge.

Just to clarify, I am querying "https://esgf-node.ornl.gov/esgf-1-5-bridge?.." and weirdly, now getting errors from the SOLR index "esgf-node.llnl.gov".

It seems that a cronjob that was working until 3 pm yesterday is now hitting esgf-node.llnl.gov

requests.exceptions.ConnectionError: HTTPSConnectionPool(host='esgf-node.llnl.gov', port=443): Max retries exceeded with url: /esg-search/search/?limit=1000&format=application%2Fsolr%2Bjson&source_id=DRES-CMIP-BB4CMIP7-2-0&project=input4mips&project=input4MIPs&distrib=fals
e&fields=* (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x149a98f1e850>: Failed to establish a new connection: [Errno 113] No route to host'))

looks like our replies crossed in the middle :)

If you're still running that script you linked above, the first two queries in it are still pointed to LLNL's Solr. Could that be the problem?

[durack1]

Think these should be equivalent queries

Yeah sorry, the code is on a branch, so what I have is here which works out to be

"https://esgf-node.ornl.gov/esgf-1-5-bridge?type=Dataset&replica=false&activity_id=input4MIPs" +
"&facets=source_id&data_node=esgf-data1.llnl.gov,esgf-data2.llnl.gov,aims3.llnl.gov,esgf-data.nersc.gov"

[bstrdsmkr]

Looks like that 3rd one is still pointing to Solr: https://github.com/PCMDI/input4MIPs_CVs/blob/bd47a23227b3512585265def371a551ad80b5ad5/scripts/pollESGF.py#L237

[durack1]

@bstrdsmkr good point, sorry for not being thorough.

So what do I map this to?

https://esgf-node.llnl.gov/esg-search/search/?limit=1000&
format=application%2Fsolr%2Bjson&
source_id=CR-CMIP-1-0-0&project=input4mips&project=input4MIPs&distrib=false&fields=*