TabNine suggestions block Sublime for multiple seconds (on popup and selection)

[kakra]

please complete the following information:

• OS version: Gentoo Linux
• Editor version: Sublime Text 3 Build 4113
• Programming language: Ruby on Rails
• Tabnine extension version: v3.2.0

Issue Details:

During auto-complete (just when the popup is supposed to show up), Sublime often blocks short amounts of time, sometimes several seconds. Looking at the process monitor, you can see that TabNine hangs in disk state and spikes IO usage. This blocking makes working and typing in code unusable. Selecting a suggested TabNine entry also blocks the editor for up to multiple seconds sometimes. This blocking happens like every 3-4 keypresses.

Background: metrics.db is stored on a btrfs partition which uses copy-on-write. If I recreate .config/TabNine with chattr +C and copy all contents from a backup, this file inherits the nocow-attribute. With this, Sublime feels a little bit smoother but it's still not very good.

TabNine should probably improve its IO patterns, and be copy-on-write friendly.

The system has 24 GB of RAM, and 11+ GB of that a available memory (around 10 GB of that is page cache). So this is not a low memory condition.

gz#6848

[boaz-codota]

Hey @kakra

Thank you for the awesome breakdown of the issue. We indeed not friendly to copy-on-write situations, and we will look into improving that.

Do you have any suggestions as to how to improve this? Or must we sacrifice consistency and batch write the state?

[kakra]

I think SQlite has a solution for this by using write-ahead logging (that's the WAL files), here's some starting point on the topic: https://wiki.tnonline.net/w/Blog/SQLite_Performance_on_Btrfs

Maybe the database files of TabNine should use something similar (I think you already use SQlite).

I've already done that for other SQlite databases and it can be done without knowledge for that application using the steps described in the link, section "How to enable SQLite WAL" (bottom of the page).

[kakra]

If possible, try to avoid fsync patterns for synchronization - fsync is quite horrible on btrfs performance-wise. Btrfs doesn't need it being a cow file system. But I'm not sure how to handle that, maybe through some env var like TABNINE_NO_FSYNC=1 or something that advanced users could set manually in their profile.

[kakra]

Also, avoid in-place updates of blocks in files and prefer appending data to some sort of log file which you could merge at process start or regular intervals back into the data file (similar to WAL).

[kakra]

I'm not sure if SQlite has file access modes which use fsync() - if they do, I'd research if that is really needed with WAL, and if it isn't, you may want to explicitly disable that access mode. Btrfs really doesn't like fsync because it flushes internal trees and transactions.