Re-index the events table with a combined topic/contract and ID

[Shaptic]

What

Replaces the existing indices on the events table with one built on the first topic AND id.

Why

Because the query always contains an ORDER BY id ASC clause, the index on topic1 can't be utilized properly: it results in a full scan because sorting is necessary after the lookup. You can see this in the query planner:

> explain query plan select hex(topic1) from events where topic1 = x'0000000f000000087472616e73666572' order by id asc limit 100;^C
QUERY PLAN
|--SEARCH events USING INDEX idx_topic1 (topic1=?)
`--USE TEMP B-TREE FOR ORDER BY

vs. a string-based search which can leverage the id index:

> explain query plan select hex(topic1) from events where hex(topic1) = upper('0000000f000000087472616e73666572') order by id asc limit 100;
QUERY PLAN
`--SCAN events USING INDEX sqlite_autoindex_events_1

Known limitations

There are ~67mm events in a week-long pubnet deployment, so we need to time this migration on the development cluster to add accurate release notes and operator expectations.

Maybe there's a way to change the query itself to use the indices properly?

[mollykarcher]

There are ~67mm events in a week-long pubnet deployment, so we need to time this migration on the development cluster to add accurate release notes and operator expectations.

In thinking about this more I'm not incredibly concerned about this. In general, I think we may (debatably) want to limit any long-running/blocking migrations like index creations to major releases boundaries. But people that deploy RPC (or anything captive-core) should already have deployment architectures that are tolerant of downtime on upgrade, because captive-core's restart time is not instant. Given that, I think anything on the order of minutes is probably tolerable, so 🤞

[tamirms]

nice catch!