Portal loosing consumers

[maksimlikharev]

when # of consumers exceeds 1000, it seems that Wicked cannot detect outliers that would be retrieved with consecutive pagination call, and assumes that those consumers are new and will try to create them.
This will result in DB primary key constraint violation on kong, resulting in failure of kong-adapter.

all sync ops, especially for potentially large # of entities, like consumers, should implement proper pagination, Kong does not allow > 1000 rows to be retrieved with a single call.

[shreyas-segu]

The same is also applicable to routes, plugins and apis. It is highly likely that the plugins and routes can also cause the same issue as number of plugins grow related to the number of subscriptions / plans

[DonMartin76]

Yup, this has already been fixed in the kong-1xx branch, if you check the code there. All object retrievals from Kong are done via the paging mechanisms there. Would it be possible for you to verify the functionality there?

[DonMartin76]

See here: https://github.com/Haufe-Lexware/wicked.haufe.io/blob/kong_1xx/src/kong-adapter/src/kong/utils.ts#L304

FWIW, it was not documented in previous versions that getting more than 1000 consumers/APIs/services/... would not work. As of Kong 1.0+ the API would complain with a size parameter larger than 1000.

[maksimlikharev]

we are not at 1x yet in production, and yes, > 1000 does not work.
we can probably move 1x into our fork at least for the consumers