Improve speed by caching the last few dataview queries

[MaxMoldmann]

Is your feature request related to a problem? Please describe.
My fault with 1500+ notes is loading Dataview queries slower and slower.

Describe the solution you'd like
One potential solution would be #1221 or to cache the results of the last few query runs and then serve them from cache if they are only a few minutes old. Add an option to manually run a query again - in case I added a new note and want it to show up despite the cache.

Describe alternatives you've considered
A more difficult solution would be a cache with an invalidation strategy, where Dataview keeps track of the last few changes in Links or Tags and then only serves a query from the Cache if the entry is not affected by recent changes.

[holroy]

Would it be possible to see some of the queries which are running slowly in your setup? Do you think they could be rewritten/reorder, so that you eliminate more of the file set earlier on in the query?

From personal experience careful ordering of the clauses can do magic when it comes to execution speed.

[MaxMoldmann]

Every Map of Content / Category page dynamically lists: Sub-Categories, Sub-Notes and other Notes that Link to this Note:

Sub-Categories

LIST WITHOUT ID link(file.name, title)
FROM #Category
WHERE contains(up, [[]])
SORT file.name ASC

Sub-Notes

LIST WITHOUT ID link(file.name, title) 
FROM !#Category
WHERE contains(up, [[]])
SORT file.name ASC

Linked Notes

LIST WITHOUT ID link(file.name, title) 
FROM [[]] AND !outgoing([[]]) AND !#Category
WHERE !contains(up, [[]])
SORT file.name ASC

[MaxMoldmann]

On central topics with many notes it takes roughly 3 seconds to display the dynamic DataView content on an Intel i7 @ 2.5 GHz. Smaller topics render almost instantly. I have to note though that on an M3 MacBook loading time is negligible everywhere.

[holroy]

In your query, you're doing FROM #category which limits which file set to use in the test of the query, and then afterwards you're doing a test for links to the current file. Have you considered including that link test into the FROM as well?

My reasoning is that given a vault of, say 10 000 files, where half of them have that category, but only a few of those again have the proper link, your query will have to scan those 5000 files in order to verify the WHERE clause. This is under the qualified assumption that it's easier/faster/ more efficient to process the FROM clauses since they're know to be cached metadata lists, instead of scanning metadata (or lookups) on each and every file.

Where as if you added something like FROM [[]] and #category, you would only need to scan a file set that already have a link to your current file. So your per file tests is now only done on the limited set of files actually linked to n the current file. It could still be a free link, or in another property, or... , but it is still relatively cheap as compared to checking several thousands of files.

---

In short, when dealing with increasingly larger vaults, I strongly suggest limiting as much as possible in the FROM clause, even though you might need to further filter it with a similar filter in a WHERE clause.

[holroy]

Regarding the actual request, it would be near impossible for dataview to know when a query should be cached or not. In most cases, when I work with queries, I also work with the underlying data set show cased in the query, so I would definitely not want the cached result. In some cases, I'm already hindered by the query not re-executing because it doesn't detect changes in my data set. So the request is hard to implement.

In addition, you can tweak the auto refresh timing of a query, or even turn it off, and then only rely on doing the refresh at your given choice of time through triggering the Dataview: Rebuild current view command.

Taking it even a step further, you could look into using the Dataview Publisher plugin, which persists the query entirely allowing for links to be included in graph view and/or be published as static content not just dynamic content.

---

Still I've been considering whether Dataview should have internal options to persist queries and only refresh those queries when explicitly told so. That would also address your use case here. I'll try to remember this issue if/when I get around to implementing persisting queries.