Usability: increase PAGE_SIZE and DEFAULT_PAGE_SIZE

[Rdataflow]

Fixes Usability Issues in current versions of ArcGIS Pro and QGIS

1000 is a reasonable default page size

If @geoadmin wants to promote 🚀 the usage of STAC API such usability fixes are no brainer low hanging 😉
and consider M2M is only useful if client facing interfaces work

[ltshb]

@Rdataflow thanks for your PR, unfortunately we have to limit the page size to 100 due to performance reason for the moment. We will need to have a look if we can improve serailization performance in order to increase the limit before changing that.

[Rdataflow]

@ltshb maybe the following context can help you for your performance consideration:

• STAC plugins typically will query for all collections and thus if necessary query all the next pages automatically in one go - so there will arise such queryload anyway
• WMTSCapabilities.xml is a similar use case... it may be pregenerated and cached on CDN in case performance becomes an issue 👍

[ltshb]

@Rdataflow the issue is unfortunately not in the SQL query which is very fast but in the serialization done by https://www.django-rest-framework.org/ which is very slow.

[Rdataflow]

@ltshb so maybe it's useful to apply some kind of caching? i.e. whitenoise or wichever CDN cache you can apply for such serialized json, as currently there seems no cache from the response header X-Cache: Miss from cloudfront

[ltshb]

@Rdataflow we already have caching in Cloudfront but caching only works for GET request and would not solve the issue for the search endpoint that uses POST request. We have an internal ticket that will check deeper on what are the actual performance when increasing this value, as the performance that I'm aware off date from the STAC launch a few years ago. We will keep you posted as soon as we have more insight on the performance. Internal ticket is PB-1591 (only accessible by geoadmin team)

[Rdataflow]

@ltshb perfect caching already works for GET
nb: the request I actually observe in QGIS is this GET request

curl 'https://data.geo.admin.ch/api/stac/v1/collections' -H 'User-Agent: Mozilla/5.0 QGIS/34201/Windows Server 2022 Version 2009'  --compressed

IIUC it qualifies for caching - but it doesn't (AFAICT from the response header)
ps: ... and perfect you are on this issue already 😄

[ltshb]

@Rdataflow we cannot increase the limit because of performance reason. However we implemented the item-search conformance and also changed the behavior of the limit query parameter. Now the query parameter returns 200 OK even if the max limit is reached, the service automatically correct the limit to 100. This should help with QGIS. Can you try again and let me know if it helped ? Note this changes is only on the STAC v1 implemented, not in STAC v0.9.

[Rdataflow]

@ltshb nothing changed. as QGIS doesnt pass no params still only the first page is shown.