Which is faster? select.. where @rid > #... vs select .. where _processed <> true order by @rid asc

[jymcheong]

Trying to implementing a FIFO approach to process each record. Thanks.

[gramian]

Certainly the filter by RID (`SELECT ... WHERE @Rid > ...) given that this is the only filter used.

However, I needed to build something like that a while ago and ended up with the second variant. Because my type uses multiple buckets, the RIDs' ordering does not correspond to the entry time (only the second RID component does), thus I could not use the comparison operation. When using a single bucket this could work though.

[jymcheong]

@gramian
Currently using select from queue LIMIT 100. But noting @lvca's point that RIDs can be recycled, I have to reconsider...

[jymcheong]

Using a _sortID field since RID can be reused n be "smaller" value #1391 , potentially leading to a non-FIFO order.

CREATE DOCUMENT TYPE queue IF NOT EXISTS;
CREATE PROPERTY queue._sortID IF NOT EXISTS LONG (readonly, default sysdate().toLong());
CREATE INDEX ON queue (_sortID) NOTUNIQUE;

I have a single writer (enqueue) & reader (dequeue > process > delete) scenario...

Reader does SELECT from queue order by _sortid limit xxx

[lvca]

It depends on the use case. Do you have one FIFO for all the users? Or a FIFO per user or other entity? The problem with RIDs is that now are recycled. So if you delete the entry from the queue, the new assigned RID could be recycled, that means a lower number than the previous inserted.

The easiest and safe way is using edges. Edges are always added in sequence, so the last edge inserted is always the first one in the list (more a LIFO approach). We didn't set a link to the first element, it's something we should have done since the beginning in EdgeLinkedList classes: 2 pointers to browse in both directions.

How many elements in the queue you plan to have?

You could have a "Queue" vertex type representing the queue and a "QueueEntry" vertex type containing the element. Then you could have a "QueueHasEntry" edge type to connect both.

At this point to get the 1st element, you should traverse the whole list and get the last that is the first. Linked lists are very efficient, but it all depends on how many entries you have.

Having the last entry first speeds up the insertion in the queue, at least, but the "pop" operation has higher complexity. I wish I added a double linked pointers in the linked list, but there are not, and not so fast to change now.

[jymcheong]

@ivca

The problem with RIDs is that now are recycled.

Noted.

How many elements in the queue you plan to have?

In the orders of thousands. Ideally don't want the queue to accumulate. One writer , one queue. One reader n8n workflow doing dequeue -> process item -> delete pattern.

The queue is currently a Document type. I have ported partially a system that was (OrientDB + nodeJS), to (ArcadeDB + n8n workflows), using simple select from queue LIMIT 100... Currently FIFO order is there, but now that you mentioned that RID can be recycled, a smaller number reused could break the FIFO order.

I see that you are describing a Linked List approach, I have to evaluate the effort on my end to refactor :P

[jymcheong]

I will go read #1391. Thanks guys for the support!

[jymcheong]

@lvca from #1391 - I'd like to make this configurable, so the user can decide per bucket/type if the RIDs are reusable can't really find in the documentation.

Or rather, with RID recycling, would select from queue LIMIT 100 still main the FIFO behavior?