Skip indexing unchanged files

[jenny-codes]

For #88

See the full work at main.../jennyshih/save-updated-graph

This is a different implementation than #173. In the previous version, here is the flow:

indexing starts 
-> 1. collect documents from workspace
      1-1 read content from file system
-> 2. *calculate diff against db state*
-> 3. *delete stale entries from db* 
-> 4. index individual files
-> 5. build global graph in memory 
-> 6. *save graph to db*

which has a few issues

1. To do the staleness check we need to read the document source from the filesystem before going into the parallel indexing (step 4), which can be a bottleneck for us down the road.
2. It attempts to save all entries into the database, including the ones that remain unchanged which is wasteful and a bit awkward.

This new attempt here is as follows:

-> 1. collect documents from workspace 
-> 2. index individual files
         2-1 read content from file system
         2-2 calculate diff
         2-3 exit early if same entry exists in db
-> 3. build global graph in memory (merging local index into global)
         3-1 when clearing global index duplicate uri data, clear the db too (not yet implemented)
         3-2 before adding the new data to memory, write to db (not yet implemented)
-> 4. delete the deleted entries from the db (not yet implemeneted) 

So for our four kinds of uris:

1. new uris: they will be saved individually during parallel indexing
2. updated uris: their data will be deleted from the db and re-inserted during parallel indexing
3. existing uris: we will not attempt to re-index them nor save them to db again. We will do nothing about them.
4. deleted uris: they will be cleaned up when we finish the parallel indexing of the present uris.

This pr implements the part where it looks up the db document entries and use it to inform the indexing worker to skip a file when it has not been changed.

Prs to come

1. Save the new/updated entries to db after indexing
2. Delete the entries in db that are no longer in the filesystem

Benchmark

Skipping the unchanged files reduced the db save time to 0 when we run the index twice on the same repo, but does not really affect the indexing time.

Before

Query statistics

Total declarations:         684127
With documentation:         71130 (10.4%)
Without documentation:      612997 (89.6%)
Total documentation size:   211400 bytes
Multi-definition names:     14691 (2.1%)

Definition breakdown:
  Method               315306
  InstanceVariable     178723
  Module               175442
  Class                100527
  Constant              60735
  AttrReader            31333
  AttrAccessor          11458
  AttrWriter              370
  GlobalVariable           36
  ClassVariable            11

Timing breakdown
  Initialization      0.001s (  0.0%)
  Listing             1.440s ( 11.6%)
  Indexing            2.439s ( 19.7%)
  Querying            0.223s (  1.8%)
  Database            8.276s ( 66.9%)
  Cleanup             0.000s (  0.0%)
  Total:             12.380s

Maximum RSS: 657997824 bytes (627.52 MB)

Indexed 85386 files
Found 684127 names
Found 873941 definitions
Found 85386 URIs

After

Query statistics

Total declarations:         1
With documentation:         0 (0.0%)
Without documentation:      1 (100.0%)
Total documentation size:   0 bytes
Multi-definition names:     0 (0.0%)

Definition breakdown:

Timing breakdown
  Initialization      0.002s (  0.0%)
  Listing             3.318s ( 58.9%)
  Indexing            2.300s ( 40.9%) 
  Querying            0.000s (  0.0%)
  Database            0.009s (  0.2%) <- this is for the querying
  Cleanup             0.000s (  0.0%)
  Total:              5.630s

Maximum RSS: 65519616 bytes (62.48 MB)

Indexed 0 files
Found 1 names
Found 0 definitions
Found 0 URIs

[jenny-codes]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• Skip indexing unchanged files #210 👈 (View in Graphite)
• Add content_hash to document struct #209
• Add content_hash to load_uri query #215
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.