Skip to content

optimize in-memory history implementation #790

@slingamn

Description

@slingamn

In response to the latest CHATHISTORY draft, all ordering operations (on msgids or timestamps) are now implemented as intervals over nanosecond-resolution timestamps.

Right now, all in-memory history queries are basically linear scans. It would be interesting to optimize this so that in-memory buffers can be substantially larger than they are now (maybe 10,000 entries?), making always-on clients practical even without a MySQL backend.

Steps:

  1. Add should ensure that the buffer is always sorted by nanotime, if necessary by reordering messages within the ring buffer. In the typical case, no reordering should be necessary. We may want to put our thumb on the scale here by reassigning timestamps so that they are roughly sequential (even under many concurrent JOINs, such as after server restart or in ircstress's chanflood benchmark).
  2. Time-based queries can be implemented using binary search in the ring buffer
  3. Msgid-based queries will still require a linear scan to get the corresponding timestamp (although we could potentially generate msgids that include the timestamp in their value, e.g., 64 bits of nanotime plus 64 bits of randomness --- need to consider the security/privacy implications here)

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions