Feature/metadata offset pagination - #814
Conversation
Adds an `offset` query parameter (default 0) to the GET metadata endpoint in both JAX-RS and Spring MVC adapters. A new `getMetadataList` overload in `MetadataApiImpl` delegates to the new `Storage.getMetadata(limit, offset, ...)` default interface method, keeping all existing call sites unchanged. The `Storage` default method falls back to fetch-and-slice for non-database implementations. `DatabaseStorage` will benefit from a follow-up that appends `OFFSET ?` to the generated SQL for true server-side skipping. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
All three DB dialects now push offset to the database engine: - PostgreSQL / MySQL / H2: LIMIT ? OFFSET ? - Oracle: WHERE rn > ? AND rn <= offset + limit - SQL Server: OFFSET ? ROWS FETCH NEXT ? ROWS ONLY (replaces TOP N) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
| /** | ||
| * Get a paged list of metadata records, skipping the first {@code offset} matching records. | ||
| * Implementations that do not override this method fall back to fetching | ||
| * {@code maxNumberOfRecords + offset} records and slicing client-side. |
There was a problem hiding this comment.
I updated terminology used in the comment to make it more readable.
| * | ||
| * @param storageName Name of the storage to search. | ||
| * @param metadataNames The metadata names to return. | ||
| * @param limit Maximum number of results to return. |
There was a problem hiding this comment.
Parameter offset moet hier ook, met korte uitleg.
There was a problem hiding this comment.
I added this:
* @param offset Number of matching records to skip before returning results.
| } | ||
| int fetchCount = maxNumberOfRecords < 0 ? -1 : maxNumberOfRecords + offset; | ||
| List<List<Object>> all = getMetadata(fetchCount, metadataNames, searchValues, metadataValueType); | ||
| return offset < all.size() ? all.subList(offset, all.size()) : new java.util.ArrayList<>(); |
There was a problem hiding this comment.
Zie ik hier een default implementatie die gewoon alle records vanaf het eerste ophaalt en daarna de records die vóór de offset horen weglaat uit de return waarde?
There was a problem hiding this comment.
Ja. Dit zou, voor zover ik kan overzien, alleen in kunnen kicken voor niet-database-backed storage.
|
Ik had hem willen testen in combinatie met het FF!, maar ik zie dat je je branch op een fork hebt gemaakt. Dat ondersteunen de GitHub Actions alleen als het FF! is geforked. We moeten hem maar mergen en achteraf testen denk ik. |
|
Zojuist master erin gemerged en door middel van de daarin beschikbare ladybug-director.html kunnen testen dat file-backed storage goed werkt met chunking. Dit nadat ik eerder postgres en h2 backed ladybug storage ook al heb testen. |
|
Wat is de status van deze PR? Het ontbreken van deze functionaliteit in de LB backend is de enige reden dat ik vooralsnog veroordeeld ben tot het gebruiken van een fork. |
|
There is no progress. Before this PR can be merged it needs to be tested with an Oracle database. The person to do that is Jaco. Jaco is on holiday and will be back on August 10. After that he will probably be too busy to do this test. If you really need this PR we need to discuss when Jaco is back. |
3 participants
Add
offsetparameter to metadata list API for pagination supportThe
GET /metadata/{storage}endpoint only supportedlimit, making itimpossible to page through large result sets without re-fetching from the
beginning each time. This adds an
offsetquery parameter (default0,so all existing callers are unaffected) across both backends.
Changes
Storage— newdefaultmethodgetMetadata(maxNumberOfRecords, offset, ...).Existing
Storageimplementations require no changes; the default falls back tothe existing abstract method and slices client-side. Implementations that override
it get true server-side offset support automatically (see
DatabaseStoragebelow).DatabaseStorage— overrides the new default with a proper SQL-levelimplementation. Offset is pushed to the database engine; no rows are fetched and
discarded in Java. All three supported dialects are handled in
DbmsSupport:LIMIT ? OFFSET ?WHERE rn > ? AND rn <= offset + limit(row-number subquery)OFFSET ? ROWS FETCH NEXT ? ROWS ONLY(replacesTOP N)MetadataApiImpl—offsetthreaded through to the storage call.The previous overload without
offsetis removed; only one method now.MetadataApi(JAX-RS and Spring MVC) —@QueryParam("offset")/@RequestParam("offset")added with default0.No existing behaviour changes. Callers that omit
offsetget exactly whatthey got before.