Skip to content

Commit 4b7beea

Browse files
alistair3149claude
alistair3149
and
claude
committed
Replace totalRows pagination with cursors on the list endpoints
GET /schemas, /layouts, and /mappings filtered unreadable rows out of their pages but reported an unfiltered namespace count, so totalRows - visible leaked the number of read-restricted Schemas, Layouts, and Mappings. The endpoints now paginate with an opaque cursor over readable rows only: limit plus an optional cursor in, items plus nextCursor out. Restricted rows are skipped exactly like nonexistent ones, take no page space, and never surface in the cursor, so nothing about them is inferable from the pagination. The persistence lookups page the namespace by keyset (page_id) in batches and re-check read permission per title, so a request costs the rows it serves rather than the whole namespace. Offset pagination over a read-filtered list would re-scan from the start on every page, which stops scaling once Schema counts grow Wikidata-shaped. The admin tables map CdxTable's offset-based server pagination onto cursors with a per-row-offset cursor history, run the indeterminate "X-Y of many" pagination until the end is known, then report the exact count (the indeterminate next-button heuristic misses a listing that ends exactly on a page boundary). The schema picker's full enumeration follows nextCursor instead of counting rows. Fixes #1062 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 30087af commit 4b7beea

31 files changed

Lines changed: 1156 additions & 325 deletions

‎docs/api/rest-api.md‎

Lines changed: 27 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -15,11 +15,16 @@ Every endpoint is also published as a complete [OpenAPI 3.0 description](#full-s
1515

1616
## Permissions
1717

18-
The Subject, page-subjects, subject-labels, Schema, Layout, RDF export, and entity-dereference read endpoints enforce
19-
the caller's per-page `read` permission; page protection and `$wgNamespaceProtection` do not restrict them, because
20-
MediaWiki's `read` action ignores both. When you may not read a page they respond as if the data were absent — a `null`
21-
value, an empty list, or a `404` — never a `403`. `GET /subject-labels` omits the labels of Subjects whose page you
22-
cannot read; because that filter runs per result, it caps `limit` at 50.
18+
The Subject, page-subjects, subject-labels, Schema, Layout, Mapping, RDF export, and entity-dereference read endpoints
19+
enforce the caller's per-page `read` permission; page protection and `$wgNamespaceProtection` do not restrict them,
20+
because MediaWiki's `read` action ignores both. When you may not read a page they respond as if the data were absent — a
21+
`null` value, an empty list, or a `404` — never a `403`. `GET /subject-labels` omits the labels of Subjects whose page
22+
you cannot read; because that filter runs per result, it caps `limit` at 50.
23+
24+
The `GET /schemas`, `GET /layouts`, and `GET /mappings` list endpoints paginate with an opaque cursor over the rows you
25+
may read (see [Cursor pagination](#cursor-pagination)): a restricted Schema, Layout, or Mapping is skipped exactly like
26+
one that does not exist, and no total count is reported, so nothing about restricted rows can be inferred from the
27+
pagination.
2328

2429
Subject write endpoints require per-page `edit` permission and answer `403` on denial.
2530

@@ -70,7 +75,7 @@ A Schema defines a Subject type and its properties. For the body shape, see
7075

7176
| Endpoint | Description |
7277
|---|---|
73-
| `GET /neowiki/v0/schemas` | List Schemas. Paginated with `limit` and `offset`. |
78+
| `GET /neowiki/v0/schemas` | List Schemas. [Cursor-paginated](#cursor-pagination) with `limit` and `cursor`. |
7479
| `GET /neowiki/v0/schema/{schemaName}` | Fetch a Schema by name. |
7580
| `GET /neowiki/v0/schema-names/{search}` | Find Schema names by prefix. |
7681

@@ -80,7 +85,7 @@ A Layout defines how a Subject is displayed.
8085

8186
| Endpoint | Description |
8287
|---|---|
83-
| `GET /neowiki/v0/layouts` | List Layouts. Paginated with `limit` and `offset`. |
88+
| `GET /neowiki/v0/layouts` | List Layouts. [Cursor-paginated](#cursor-pagination) with `limit` and `cursor`. |
8489
| `GET /neowiki/v0/layout/{layoutName}` | Fetch a Layout by name. |
8590

8691
### Mappings
@@ -90,7 +95,7 @@ An ontology Mapping projects native Schemas to a target ontology. For the format
9095

9196
| Endpoint | Description |
9297
|---|---|
93-
| `GET /neowiki/v0/mappings` | List ontology Mappings, each with the names of its mapped Schemas. Paginated with `limit` and `offset`. |
98+
| `GET /neowiki/v0/mappings` | List ontology Mappings, each with the names of its mapped Schemas. [Cursor-paginated](#cursor-pagination) with `limit` and `cursor`. |
9499

95100
### Query
96101

@@ -100,6 +105,20 @@ An ontology Mapping projects native Schemas to a target ontology. For the format
100105

101106
<!-- REST-ENDPOINTS:END -->
102107

108+
## Cursor pagination
109+
110+
The Schema, Layout, and Mapping list endpoints paginate with an opaque cursor. Request up to `limit` items (1–50,
111+
default 10); the response carries the items and a `nextCursor`:
112+
113+
```json
114+
{ "schemas": [ ... ], "nextCursor": "1462" }
115+
```
116+
117+
Pass that value back as `cursor` to fetch the next page. `nextCursor` is `null` on the last page; a non-null cursor
118+
means another page follows, though in rare cases (trailing items that fail to load) that page may come back empty.
119+
Treat the cursor as opaque and do not construct one yourself; a malformed cursor is rejected with a `400`. Cursors stay
120+
valid across item creation and deletion, and no total count is reported.
121+
103122
## The `expand` parameter
104123

105124
The Subject read and page-subjects read endpoints take an optional multi-valued `expand` query parameter (pipe-separated,

‎phpstan-baseline.neon‎

Lines changed: 45 additions & 60 deletions
Original file line numberDiff line numberDiff line change
@@ -10,36 +10,6 @@ parameters:
1010
count: 1
1111
path: src/NeoWikiExtension.php
1212

13-
-
14-
message: "#^Cannot access property \\$page_title on array\\|stdClass\\|false\\.$#"
15-
count: 1
16-
path: src/Persistence/MediaWiki/DatabaseSchemaNameLookup.php
17-
18-
-
19-
message: "#^Instantiated class TitleValue not found\\.$#"
20-
count: 2
21-
path: src/Persistence/MediaWiki/DatabaseSchemaNameLookup.php
22-
23-
-
24-
message: "#^Method ProfessionalWiki\\\\NeoWiki\\\\Persistence\\\\MediaWiki\\\\DatabaseSchemaNameLookup\\:\\:dbResultToTitleValueArray\\(\\) has invalid return type TitleValue\\.$#"
25-
count: 1
26-
path: src/Persistence/MediaWiki/DatabaseSchemaNameLookup.php
27-
28-
-
29-
message: "#^Method ProfessionalWiki\\\\NeoWiki\\\\Persistence\\\\MediaWiki\\\\DatabaseSchemaNameLookup\\:\\:getFirstSchemaNames\\(\\) has invalid return type TitleValue\\.$#"
30-
count: 1
31-
path: src/Persistence/MediaWiki/DatabaseSchemaNameLookup.php
32-
33-
-
34-
message: "#^Method ProfessionalWiki\\\\NeoWiki\\\\Persistence\\\\MediaWiki\\\\DatabaseSchemaNameLookup\\:\\:getSchemaNamesMatching\\(\\) has invalid return type TitleValue\\.$#"
35-
count: 1
36-
path: src/Persistence/MediaWiki/DatabaseSchemaNameLookup.php
37-
38-
-
39-
message: "#^Method ProfessionalWiki\\\\NeoWiki\\\\Persistence\\\\MediaWiki\\\\DatabaseSchemaNameLookup\\:\\:searchSuggestionsToTitleArray\\(\\) has invalid return type TitleValue\\.$#"
40-
count: 1
41-
path: src/Persistence/MediaWiki/DatabaseSchemaNameLookup.php
42-
4313
-
4414
message: "#^Call to method getPrefixedText\\(\\) on an unknown class Title\\.$#"
4515
count: 1
@@ -195,36 +165,6 @@ parameters:
195165
count: 1
196166
path: src/GraphDatabasePlugins/Neo4j/Persistence/Neo4jQueryStore.php
197167

198-
-
199-
message: "#^Cannot access property \\$page_title on array\\|stdClass\\|false\\.$#"
200-
count: 1
201-
path: src/Persistence/MediaWiki/DatabaseLayoutNameLookup.php
202-
203-
-
204-
message: "#^Instantiated class TitleValue not found\\.$#"
205-
count: 1
206-
path: src/Persistence/MediaWiki/DatabaseLayoutNameLookup.php
207-
208-
-
209-
message: "#^Method ProfessionalWiki\\\\NeoWiki\\\\Persistence\\\\MediaWiki\\\\DatabaseLayoutNameLookup\\:\\:dbResultToTitleValueArray\\(\\) has invalid return type TitleValue\\.$#"
210-
count: 1
211-
path: src/Persistence/MediaWiki/DatabaseLayoutNameLookup.php
212-
213-
-
214-
message: "#^Method ProfessionalWiki\\\\NeoWiki\\\\Persistence\\\\MediaWiki\\\\DatabaseLayoutNameLookup\\:\\:getLayoutNames\\(\\) has invalid return type TitleValue\\.$#"
215-
count: 1
216-
path: src/Persistence/MediaWiki/DatabaseLayoutNameLookup.php
217-
218-
-
219-
message: "#^Method ProfessionalWiki\\\\NeoWiki\\\\Persistence\\\\LayoutNameLookup\\:\\:getLayoutNames\\(\\) has invalid return type TitleValue\\.$#"
220-
count: 1
221-
path: src/Persistence/LayoutNameLookup.php
222-
223-
-
224-
message: "#^Method ProfessionalWiki\\\\NeoWiki\\\\Persistence\\\\SchemaNameLookup\\:\\:getSchemaNamesMatching\\(\\) has invalid return type TitleValue\\.$#"
225-
count: 1
226-
path: src/Persistence/SchemaNameLookup.php
227-
228168
-
229169
message: "#^Call to method getHeader\\(\\) on an unknown class WebRequest\\.$#"
230170
count: 1
@@ -299,3 +239,48 @@ parameters:
299239
message: "#^Parameter \\#3 \\$subject of function preg_replace expects array\\|string, string\\|null given\\.$#"
300240
count: 1
301241
path: src/GraphDatabasePlugins/Neo4j/Application/KeywordCypherQueryValidator.php
242+
243+
-
244+
message: "#^Cannot access property \\$page_id on array\\|stdClass\\|false\\.$#"
245+
count: 1
246+
path: src/Persistence/MediaWiki/DatabaseLayoutNameLookup.php
247+
248+
-
249+
message: "#^Cannot access property \\$page_title on array\\|stdClass\\|false\\.$#"
250+
count: 1
251+
path: src/Persistence/MediaWiki/DatabaseLayoutNameLookup.php
252+
253+
-
254+
message: "#^Parameter \\#2 \\$op of method Wikimedia\\\\Rdbms\\\\IReadableDatabase\\:\\:expr\\(\\) expects '\\!\\='\\|'\\='\\|'\\\\\\\\x3C'\\|'\\\\\\\\x3C\\='\\|'\\\\\\\\x3E'\\|'\\\\\\\\x3E\\='\\|'LIKE'\\|'NOT LIKE', '\\>' given\\.$#"
255+
count: 1
256+
path: src/Persistence/MediaWiki/DatabaseLayoutNameLookup.php
257+
258+
-
259+
message: "#^Cannot access property \\$page_id on array\\|stdClass\\|false\\.$#"
260+
count: 1
261+
path: src/Persistence/MediaWiki/DatabaseMappingNameLookup.php
262+
263+
-
264+
message: "#^Cannot access property \\$page_title on array\\|stdClass\\|false\\.$#"
265+
count: 1
266+
path: src/Persistence/MediaWiki/DatabaseMappingNameLookup.php
267+
268+
-
269+
message: "#^Parameter \\#2 \\$op of method Wikimedia\\\\Rdbms\\\\IReadableDatabase\\:\\:expr\\(\\) expects '\\!\\='\\|'\\='\\|'\\\\\\\\x3C'\\|'\\\\\\\\x3C\\='\\|'\\\\\\\\x3E'\\|'\\\\\\\\x3E\\='\\|'LIKE'\\|'NOT LIKE', '\\>' given\\.$#"
270+
count: 1
271+
path: src/Persistence/MediaWiki/DatabaseMappingNameLookup.php
272+
273+
-
274+
message: "#^Cannot access property \\$page_id on array\\|stdClass\\|false\\.$#"
275+
count: 1
276+
path: src/Persistence/MediaWiki/DatabaseSchemaNameLookup.php
277+
278+
-
279+
message: "#^Cannot access property \\$page_title on array\\|stdClass\\|false\\.$#"
280+
count: 2
281+
path: src/Persistence/MediaWiki/DatabaseSchemaNameLookup.php
282+
283+
-
284+
message: "#^Parameter \\#2 \\$op of method Wikimedia\\\\Rdbms\\\\IReadableDatabase\\:\\:expr\\(\\) expects '\\!\\='\\|'\\='\\|'\\\\\\\\x3C'\\|'\\\\\\\\x3C\\='\\|'\\\\\\\\x3E'\\|'\\\\\\\\x3E\\='\\|'LIKE'\\|'NOT LIKE', '\\>' given\\.$#"
285+
count: 1
286+
path: src/Persistence/MediaWiki/DatabaseSchemaNameLookup.php

‎resources/ext.neowiki/src/application/SchemaLookup.ts‎

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -8,14 +8,14 @@ export interface SchemaSummary {
88

99
export interface SchemaSummaryPage {
1010
schemas: SchemaSummary[];
11-
totalRows: number;
11+
nextCursor: string | null;
1212
}
1313

1414
export interface SchemaLookup {
1515

1616
getSchema( schemaName: SchemaName ): Promise<Schema>;
1717
getSchemaNames( search: string ): Promise<string[]>;
18-
getSchemaSummaries( offset: number, limit: number ): Promise<SchemaSummaryPage>;
18+
getSchemaSummaries( cursor: string | null, limit: number ): Promise<SchemaSummaryPage>;
1919

2020
}
2121

@@ -43,16 +43,20 @@ export class InMemorySchemaLookup implements SchemaLookup {
4343
return [ ...this.schemaNames ];
4444
}
4545

46-
public async getSchemaSummaries( offset: number, limit: number ): Promise<SchemaSummaryPage> {
46+
public async getSchemaSummaries( cursor: string | null, limit: number ): Promise<SchemaSummaryPage> {
4747
const summaries = [ ...this.schemas.values() ].map( ( schema ) => ( {
4848
name: schema.getName(),
4949
description: schema.getDescription(),
5050
propertyCount: [ ...schema.getPropertyDefinitions() ].length,
5151
} ) );
5252

53+
// The cursor is opaque to callers; this fake encodes the next start index in it.
54+
const start = cursor === null ? 0 : parseInt( cursor, 10 );
55+
const end = start + limit;
56+
5357
return {
54-
schemas: summaries.slice( offset, offset + limit ),
55-
totalRows: summaries.length,
58+
schemas: summaries.slice( start, end ),
59+
nextCursor: end < summaries.length ? String( end ) : null,
5660
};
5761
}
5862

‎resources/ext.neowiki/src/components/LayoutsPage/LayoutsPage.vue‎

Lines changed: 14 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -113,6 +113,7 @@ import { CdxButton, CdxDialog, CdxIcon, CdxTable } from '@wikimedia/codex';
113113
import type { TableColumn } from '@wikimedia/codex';
114114
import { cdxIconAdd, cdxIconEdit, cdxIconTrash } from '@wikimedia/codex-icons';
115115
import { NeoWikiExtension } from '@/NeoWikiExtension.ts';
116+
import { useCursorPagination } from '@/composables/useCursorPagination.ts';
116117
import { useLayoutPermissions } from '@/composables/useLayoutPermissions.ts';
117118
import { useLayoutStore } from '@/stores/LayoutStore.ts';
118119
import { Layout } from '@/domain/Layout.ts';
@@ -128,10 +129,16 @@ const paginationSizeOptions: { value: number }[] = [
128129
];
129130
130131
const loading = ref( true );
131-
const totalRows = ref( 0 );
132132
const isCreatorOpen = ref( false );
133133
const pageSize = ref( paginationSizeOptions[ 0 ].value );
134134
const lastOffset = ref( 0 );
135+
// Undefined while the end of the listing is unknown, which keeps CdxTable in its indeterminate
136+
// pagination. Once a response carries a null cursor the exact count is known, and the table needs
137+
// it: its indeterminate next-button heuristic (a short page) misses a listing that ends exactly on
138+
// a page boundary. The count covers only rows this client has itself paged through, so it reveals
139+
// nothing the row listing did not already.
140+
const totalRows = ref<number | undefined>( undefined );
141+
const { cursorFor, recordNextCursor } = useCursorPagination();
135142
const { canCreateLayouts, canEditLayout: canEditLayouts, checkCreatePermission, checkEditPermission } = useLayoutPermissions();
136143
const layoutStore = useLayoutStore();
137144
@@ -194,19 +201,21 @@ async function fetchLayouts( offset: number, limit: number ): Promise<void> {
194201
pageSize.value = limit;
195202
lastOffset.value = offset;
196203
204+
const cursor = cursorFor( offset );
205+
const cursorParam = cursor === null ? '' : `&cursor=${ encodeURIComponent( cursor ) }`;
197206
const restApiUrl = NeoWikiExtension.getInstance().getMediaWiki().util.wikiScript( 'rest' );
198207
const httpClient = NeoWikiExtension.getInstance().newHttpClient();
199208
200209
const response = await httpClient.get(
201-
`${ restApiUrl }/neowiki/v0/layouts?limit=${ limit }&offset=${ offset }`
210+
`${ restApiUrl }/neowiki/v0/layouts?limit=${ limit }${ cursorParam }`
202211
);
203212
204213
if ( !response.ok ) {
205214
loading.value = false;
206215
return;
207216
}
208217
209-
const result: { layouts: LayoutSummary[]; totalRows: number } = await response.json();
218+
const result: { layouts: LayoutSummary[]; nextCursor: string | null } = await response.json();
210219
211220
rows.value = result.layouts.map( ( summary ) => ( {
212221
name: summary.name,
@@ -215,7 +224,8 @@ async function fetchLayouts( offset: number, limit: number ): Promise<void> {
215224
description: summary.description
216225
} ) );
217226
218-
totalRows.value = result.totalRows;
227+
recordNextCursor( offset, limit, result.nextCursor );
228+
totalRows.value = result.nextCursor === null ? offset + result.layouts.length : undefined;
219229
loading.value = false;
220230
}
221231

‎resources/ext.neowiki/src/components/MappingsPage/MappingsPage.vue‎

Lines changed: 14 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,7 @@ import { CdxButton, CdxDialog, CdxIcon, CdxTable } from '@wikimedia/codex';
106106
import type { TableColumn } from '@wikimedia/codex';
107107
import { cdxIconAdd, cdxIconEdit, cdxIconTrash } from '@wikimedia/codex-icons';
108108
import { NeoWikiExtension } from '@/NeoWikiExtension.ts';
109+
import { useCursorPagination } from '@/composables/useCursorPagination.ts';
109110
import { useMappingPermissions } from '@/composables/useMappingPermissions.ts';
110111
import MappingCreatorDialog from './MappingCreatorDialog.vue';
111112
import EditSummary from '@/components/common/EditSummary.vue';
@@ -118,10 +119,16 @@ const paginationSizeOptions: { value: number }[] = [
118119
];
119120
120121
const loading = ref( true );
121-
const totalRows = ref( 0 );
122122
const pageSize = ref( paginationSizeOptions[ 0 ].value );
123123
const lastOffset = ref( 0 );
124124
const isCreatorOpen = ref( false );
125+
// Undefined while the end of the listing is unknown, which keeps CdxTable in its indeterminate
126+
// pagination. Once a response carries a null cursor the exact count is known, and the table needs
127+
// it: its indeterminate next-button heuristic (a short page) misses a listing that ends exactly on
128+
// a page boundary. The count covers only rows this client has itself paged through, so it reveals
129+
// nothing the row listing did not already.
130+
const totalRows = ref<number | undefined>( undefined );
131+
const { cursorFor, recordNextCursor } = useCursorPagination();
125132
126133
const isDeleteConfirmOpen = ref( false );
127134
const deletingMappingName = ref( '' );
@@ -173,26 +180,29 @@ async function fetchMappings( offset: number, limit: number ): Promise<void> {
173180
pageSize.value = limit;
174181
lastOffset.value = offset;
175182
183+
const cursor = cursorFor( offset );
184+
const cursorParam = cursor === null ? '' : `&cursor=${ encodeURIComponent( cursor ) }`;
176185
const restApiUrl = NeoWikiExtension.getInstance().getMediaWiki().util.wikiScript( 'rest' );
177186
const httpClient = NeoWikiExtension.getInstance().newHttpClient();
178187
179188
const response = await httpClient.get(
180-
`${ restApiUrl }/neowiki/v0/mappings?limit=${ limit }&offset=${ offset }`
189+
`${ restApiUrl }/neowiki/v0/mappings?limit=${ limit }${ cursorParam }`
181190
);
182191
183192
if ( !response.ok ) {
184193
loading.value = false;
185194
return;
186195
}
187196
188-
const result: { mappings: MappingSummary[]; totalRows: number } = await response.json();
197+
const result: { mappings: MappingSummary[]; nextCursor: string | null } = await response.json();
189198
190199
rows.value = result.mappings.map( ( summary ) => ( {
191200
name: summary.name,
192201
schemas: summary.schemas
193202
} ) );
194203
195-
totalRows.value = result.totalRows;
204+
recordNextCursor( offset, limit, result.nextCursor );
205+
totalRows.value = result.nextCursor === null ? offset + result.mappings.length : undefined;
196206
loading.value = false;
197207
}
198208

0 commit comments

Comments
 (0)