Tumbling Window affects sorting

[Baazad]

Hey, we are using eKuiper with EdgeX to sort (asc) incoming values with a simple rule.

RULE:

{
"triggered": true,
"id": "RuleItemNr",
"sql": "SELECT ItemNr from SomePlace GROUP BY ItemNr, TUMBLINGWINDOW(ss, 60) ORDER BY meta(ItemNr.origin) ASC",
"actions": [
    {
        "edgex": {
            "contentType": "application/json",
            "host": "edgex-redis",
            "messageType": "event",
            "port": 1234,
            "protocol": "redis",
            "topic": "edgex.events.processed.SomePlace.ItemNr"
        },
        "log": {}
    }
],
"options": {
    "isEventTime": false,
    "lateTolerance": 1000,
    "concurrency": 1,
    "bufferLength": 1024,
    "sendMetaToSink": false,
    "sendError": true,
    "qos": 0,
    "checkpointInterval": 300000
}

It sorts correctly if the tumbling window is set to 30 (seconds), but we realized that if we change the tumbling window to 60 (seconds) the sorted results differ. So, the Kuiper rule doesn’t sort (asc) entries according to the rule set and the observed order is non-deterministic.

Result with Tumbling Window set to 30 seconds (RESULT OKAY):

time="2023-06-16 06:42:00" level=info msg="sink result for rule RuleItemNr: [{"ItemNr":57061},{},{"ItemNr":57062},{"ItemNr":57063},{"ItemNr":57064},{"ItemNr":57065},{"ItemNr":57066},{"ItemNr":57067},{"ItemNr":57068},{"ItemNr":57069},{"ItemNr":57070}]" file="sink/log_sink.go:31" rule=RuleItemNr

time="2023-06-16 06:42:30" level=info msg="sink result for rule RuleItemNr: [{"ItemNr":57071},{},{"ItemNr":57072},{"ItemNr":57073},{"ItemNr":57074},{"ItemNr":57075},{"ItemNr":57076},{"ItemNr":57077},{"ItemNr":57078},{"ItemNr":57079},{"ItemNr":57080}]" file="sink/log_sink.go:31" rule=RuleItemNr

Result with Tumbling Window set to 60 seconds (RESULT NOT OKAY):

time="2023-06-16 06:41:00" level=info msg="sink result for rule RuleItemNr: [{"ItemNr":57031},{"ItemNr":57041},{"ItemNr":57042},{},{"ItemNr":57037},{"ItemNr":57036},{"ItemNr":57032},{"ItemNr":57033},{"ItemNr":57034},{"ItemNr":57035},{"ItemNr":57048},{"ItemNr":57044},{"ItemNr":57046},{"ItemNr":57045},{"ItemNr":57038},{"ItemNr":57047},{"ItemNr":57043},{"ItemNr":57050},{"ItemNr":57039},{"ItemNr":57049},{"ItemNr":57040}]" file="sink/log_sink.go:31" rule=RuleItemNr

The event of the format which we are receiving and working with is coming fron an opcua source:

        {
        "apiVersion": "x2",
        "id": "3814453f-2b43-4f10-b819-df4ecc1c1b7d",
        "deviceName": "SomeMachine",
        "profileName": "SomeMachineProfile",
        "sourceName": "ItemNr.",
        "origin": 1686906231721960083,
        "readings": [
            {
                "id": "93e5d5d3-90c7-47dc-af16-9b7f36dbd5c0",
                "origin": 1686906231721,
                "deviceName": "SomeMachine",
                "resourceName": "ItemNr",
                "profileName": "SomeMachineProfile",
                "valueType": "Int32",
                "value": "59899"
            }
        ],
        "tags": {
            "source timestamp": "2023-06-16 09:03:51.721942899 +0000 UTC m=+2706.551476272"
        }
    }

How come the tumbling windows affects the sorting of the values as seen in the example given above?
Do you guys have any idea about this issue or have seen any documentation? I haven’t found anything yet …
Appreciate your time and your effort in advance. Thanks !!

[ngjaying]

In your SQL, order by meta(itemNr.origin), but you expect to order by itemNr value, is it correct? Are these two order expects to be the same?

[Baazad]

We actually want to sort by the timestamp which is the origin. In best case we want to sort by the origin of the readings object, which is possible by sorting as following : itemNr -> origin, in this case we just sorted by the timestamp of the event itemNr.origin. So we do not intend to sort by value, rather by the time we receive them. Therefore it doens't matter if we wair for 30 seconds or 60 seconds. The values should be sorted asc by the timestamp. Hope this answers the question.

[ngjaying]

Currently, the meta function seems not working in a window. Could you try sort by itemNr itself?

[Baazad]

Yes, we have just tried that as well, but it also does not work. Following Rule was used:
SELECT itemNr from SomePlace GROUP BY itemNr, TUMBLINGWINDOW(ss, 60) ORDER BY itemNr ASC
Output:

time="2023-06-16 14:15:00" level=info msg="sink result for rule RuleItemNr: [{"ItemNr":66092},{},{"ItemNr":66093},{"ItemNr":66094},{"ItemNr":66095},{"ItemNr":66096},{"ItemNr":66097},{"ItemNr":66098},{"ItemNr":66099},{"ItemNr":66100},{"ItemNr":66101}]" file="sink/log_sink.go:31" rule=RuleItemNr

time="2023-06-16 14:16:00" level=info msg="sink result for rule RuleItemNr: [{"ItemNr":66114},{"ItemNr":66115},{"ItemNr":66116},{"ItemNr":66117},{"ItemNr":66118},{"ItemNr":66119},{"ItemNr":66120},{"ItemNr":66121},{}]" file="sink/log_sink.go:31" rule=RuleItemNr

time="2023-06-16 14:17:00" level=info msg="sink result for rule RuleItemNr: [{"ItemNr":66123},{"ItemNr":66135},{"ItemNr":66129},{"ItemNr":66134},{"ItemNr":66126},{"ItemNr":66124},{"ItemNr":66125},{"ItemNr":66132},{"ItemNr":66127},{"ItemNr":66130},{"ItemNr":66137},{"ItemNr":66136},{"ItemNr":66133},{"ItemNr":66122},{"ItemNr":66128},{"ItemNr":66141},{"ItemNr":66140},{"ItemNr":66139},{"ItemNr":66138},{"ItemNr":66131},{}]" file="sink/log_sink.go:31" rule=RuleItemNr

eKuiper version used is: version 1.7.6-alpine

Same output goes with 1.10.1

As you can also see in the examples given above the meta function does work with a tumbling window of 30 seconds, but as soon as we change it to 60 seconds both sorting by orgin and by itemNr does not seem to work.

[ngjaying]

I can't think about a reason that the window size would affect the sorting. Actually, they are two separate operators. We'll try to reproduce the problem. Thanks.