Create api for syncing with database

[jenny-codes]

Add an api to remove the deleted + stale uris and their associated entities from the database, and insert the updated and new entries.

See #210 for more context.

For the deletion we use the delete from ... where id in ...​ batch delete which is a bit more complicated than the single delete but it proves to be much faster than the single deletion queries. Calling this out because we have discovered that the batch mutation queries do not always improve performance in other cases.

Benchmark

Benchmarking on Shopify core. Method: run the index once, and then index an empty collection of documents.

Before (no deletion, no re-indexing unchanged documents)

Timing breakdown
  Initialization      0.001s (  0.0%)
  Listing             3.331s ( 57.1%)
  Indexing            2.505s ( 42.9%)
  Querying            0.000s (  0.0%)
  Database            0.000s (  0.0%)
  Cleanup             0.000s (  0.0%)
  Total:              5.838s

After (enabling deleting, without batch delete)

The first attempt was to delete the uris one by one, as is the pattern so far, but it proves to be extremely slow, taking 37 minutes to delete the whole shopify core index data. Even in the medium corpus with 1000 files, it takes 9 seconds for the deletion to complete.

Timing breakdown
  Initialization      0.001s (  0.0%)
  Listing             1.435s (  0.1%)
  Indexing         2259.725s ( 99.9%)
  Querying            0.000s (  0.0%)
  Database            0.000s (  0.0%)
  Cleanup             0.000s (  0.0%)
  Total:           2261.162s

After (enabling deleting, with batch delete)

The second attempt (the current implementation) is using bulk deletion, which is much faster. It now takes 9 seconds to delete all data from Shopify core.

Timing breakdown
  Initialization      0.002s (  0.0%)
  Indexing            9.992s (100.0%)
  Querying            0.000s (  0.0%)
  Database            0.000s (  0.0%)
  Cleanup             0.000s (  0.0%)
  Total:              9.994s

[jenny-codes]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• Sync indexing result with database #283
• Create api for syncing with database #273 👈 (View in Graphite)
• Skip indexing unchanged files #210
• Create Errors::DatabaseError #284
• Create IndexResult for individual file indexing result #285
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.