Solve Feed lookup delay

[asabya]

Everything In fairOS depends on the epoch based feeds. But we don't keep track of the feed updates. For this reason we have "noclue" of the feed updates and the latest feed lookup take a while to get the latest update.

With the above case downloading a 30kb file can take upto 8-10 seconds to download (4-5 seconds for the feed lookup alone). Note: We do the feed lookup for getting the file metadata and references of the data blocks of the file.

We need a way to either lower the lookup time or store the latest epoch to make fairOS ecosystem faster.

[asabya]

Solution 1: We use any other kv store to map "topic:epoch" and keep the kv store files inside or outside user account.

Case 1: we store the feed-epoch map in user account.

we can actually store these store related files outside any pod space with the root feed api that is initialised at user login in fairOS. We load these files after login and initialise the kv store and flush the files into swarm after logout (or at some interval or after every write).

This will increase the login and logout time but a using kv store that has support for go and js will make things even for fairOS and fdp-storage.

With some initial testing of this above said implementation the same 30kb file can be downloaded in 4-5 seconds to download (1-2 seconds for the feed lookup) making the total download speed 2x faster.

Case 2: We use a kv store outside user account that will be local.

This will be even faster than case 1 theoretically.

But on the other hand we compromise with interoperability. If we are using a local fairOS server with cli and FDA, we need to keep the store files in sync manually. Thats just only fairOS ecosystem, we have fdp-storage aswell. Syncing the files manually will be a nightmare.

On the down side, accessing a outdated kv store might create confusion returning old data.