ServerVersion being returned in descending order

[HusnaHariz]

Results are being returned with descending serverVersions from the practitioner and organization endpoints. Work to support serverVersions was being tracked here. The beam-etl requires that these come in an ascending order as is the case for events.

Facing the following error at the moment because newer events are being fetched first

2022-04-26 12:49:31,004 [main] ERROR com.onaio.beam.etl.pipes.RunPipeline  - Pipeline error:??? java.lang.RuntimeException: Batch was not monotonic - found an entity at an earlier server version than a previous entity: prevId=e8daa671-be96-4e1b-bd2d-4122afb88b03 prevServerVersion=48 id=e2eca7a5-5942-45ff-8add-6e558aec1608 serverVersion=47 reqServerVersion=0 batchServerVersions=48,47,46,45,44,43,42,41,40,39,38,37,36,35,34,33,32,31,30,29,28,27,26,25,24,23,22,21,20,19,18,17,16,15,14,13,12,11,10,9,8,7,6,3,5,4,2,1??? at com.onaio.beam.etl.OpenSrpRestSource$OpenSrpRestReader.checkMonotonic(OpenSrpRestSource.java:671)??? at com.onaio.beam.etl.OpenSrpRestSource$OpenSrpRestReader.fetchNextBatch(OpenSrpRestSource.java:614)??? 
...

[ageryck]

@HusnaHariz we following this up with OpenSRP devs
cc @hilpitome

[HusnaHariz]

Tested the fetching with the orderByType=ASC filter. This does reorder the responses, except it seems to be sorting using the id field and not the serverVersion field. If you try fetch organizations using the following url
https://crvs.labs.smartregister.org/opensrp/rest/organization?serverVersion=0&limit=500&orderByType=ASC

We get results that are ordered by id. It so happens that object with id = 1 has a serverVersion=52.

[
    {
        "id": 1,
        "identifier": "f71f1d0c-89b7-4ccd-b078-1a0cddea6e31",
        "active": true,
        "name": "crvs_chw_village1",
        "type": {
            "coding": [
                {
                    "system": "http://terminology.hl7.org/CodeSystem/organization-type",
                    "code": "team",
                    "display": "Team"
                }
            ]
        },
        "dateCreated": "2022-02-07T07:41:49.607Z",
        "dateEdited": "2022-02-07T07:41:49.607Z",
        "serverVersion": 52
    },
    {
        "id": 2,
        "identifier": "75addbc2-e47d-4a65-9517-683a61e36465",
        "active": true,
        "name": "crvs_agent_village1",
        "type": {
            "coding": [
                {
                    "system": "http://terminology.hl7.org/CodeSystem/organization-type",
                    "code": "team",
                    "display": "Team"
                }
            ]
        },
        "dateCreated": "2022-02-07T07:41:49.607Z",
        "dateEdited": "2022-02-07T07:41:49.607Z",
        "serverVersion": 2
    },
    {
        "id": 3,
        "identifier": "05550d67-24e0-4348-bd76-8573e6457a28",
        "active": true,
        "name": "Team West CHA",
        "type": {
            "coding": [
                {
                    "system": "http://terminology.hl7.org/CodeSystem/organization-type",
                    "code": "team",
                    "display": "Team"
                }
            ]
        },
        "dateCreated": "2022-02-07T07:41:49.607Z",
        "dateEdited": "2022-02-07T07:41:49.607Z",
        "serverVersion": 4
    }

It could be an issue of dummyData, but it appears the actual field that is ordered is id. If we assume that the entry with serverVersion=52 is an issue of dummy data, are we certain that the id value will always be equal to serverVersion and incremental. If yes we can work with this. But kindly confirm that this is the case.

Note
Cancelled the above because we strictly need it to order by serverVersion because serverVersion takes into consideration edits made to existing entries.

We will still need the particular entry with id=1 to be have its serverVersion updated. The pipeline can't be run otherwise. Getting the same non monotonic error against the organization endpoint.

2022-04-28 14:23:23,779 [main] ERROR com.onaio.beam.etl.pipes.RunPipeline - Pipeline error:??? java.lang.RuntimeException: Batch was not monotonic - found an entity at an earlier server version than a previous entity: prevId=f71f1d0c-89b7-4ccd-b078-1a0cddea6e31 prevServerVersion=52 id=75addbc2-e47d-4a65-9517-683a61e36465 serverVersion=2 reqServerVersion=0 batchServerVersions=52,2,4,5,3,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48???

[ageryck]

@HusnaHariz as long the the endpoint is gonna return incremental server version you do not have any concerns on the id column right ?

[HusnaHariz]

@ageryck Yes that is correct

[HusnaHariz]

@ageryck A second issue identified is that the endpoint doesn't apply the serverVersion filter. It always pulls all records from OpenSRP which results in the same non-monotonic error.

This can be tested out in postman. When trying to run the following with serverVersion=10, ie we want to fetch all records from serverVersion 10, we get all records returned from serverVersion=1

https://crvs.labs.smartregister.org/opensrp/rest/practitioner?serverVersion=10&limit=500&orderByType=ASC

[ekigamba]

@HusnaHariz Can we close this?