How to disable Helm caching for files based operations OR make it less aggressive

[MattMicheletti]

Suggestion

Hey Thierry,

I wanted to ask about a semi-frequent issue I have been running into while using Helm. Essentially, Helm has internal caching when it is doing almost any file-based operation (e.g. finding files, listing directories, etc.) and that can often have me run into issues where Helm gets out-of-sync with the real, actual underlying file system due to a stale cache. Often this happens when doing things like running build commands for projects that clean, create, or modify various files and/or directories. Until I found out that Helm had that internal caching, I mistakenly thought certain things weren't working in my projects and spent hours wasting time debugging things, thinking they weren't working when they really were and really did make file system changes.

Description

I recently found out about the helm-refresh command that can be manually invoked during a Helm session on a directory, as needed. However, I wonder if there is a more sophisticated automated solution could be employed here to make the caching either less aggressive or a setting that completely bypasses caching altogether. I was unable to find a clear, easy way to completely turn off caching/disable it entirely for all of Helm.

Solutions

Absent not being able to turn off caching at all, my initial idea is to have a couple of callbacks in some hooks (possibly before/after-init hook for find files)/advice around some methods (:after helm-find-files, :after helm-find-files-up-one-level, :after helm-execute-persistent-action) whereby at least a single refresh is done per each directory visited for the first visit via Helm during a Helm session/command invocation (i.e. doing C-x C-f) but only once for that directory and only for that session, after that the cache takes back over and is in-use for the remainder of that session until the next session/command invocation (whereby the cycle starts all over). In essence, allowing caching to still be used during a session to make traversal still fast after an initial visit to a directory.

What are your thoughts? Is there a better way to solve this issue overall or is this on the right track?

Thanks,
Matt

[thierryvolpiatto]

Hello Matt, Matt Micheletti ***@***.***> writes:

● MattMicheletti created an issue (emacs-helm/helm#2758) Suggestion Hey Thierry, I wanted to ask about a semi-frequent issue I have been running into while using Helm. Essentially, Helm has internal caching when it is doing almost any file-based operation (e.g. finding files, listing directories, etc.) and that can often have me run into issues where Helm gets out-of-sync with the real, actual underlying file system. Often this happens when doing things like running build commands for projects that clean, create, or modify various files and/or directories. Until I found out that Helm had that internal caching, I mistakenly thought certain things weren't working in my projects and spent hours wasting time debugging thinking things weren't working when they really were and did make file system changes.

It is working fine here, here a small recipe: 1) Visit with helm-find-files (C-x C-f here) the directory "~/tmp/" which contains the files "foo", "bar" and "baz". 2) From a shell outside of emacs, run rm "foo". 3) Visit again "~/tmp/" in emacs (always with helm-find-files), the content of the directory is "bar" and "baz" i.e. "foo" is not appearing.

Description I recently found out about the helm-refresh command that can be manually invoked during a Helm session on a directory, as needed. However, I wonder if there is a more sophisticated automated solution could be employed here to make the caching either less aggressive or a setting that completely bypasses caching altogether. I was unable to find a clear, easy way to completely turn off caching/disable it entirely for all of Helm.

The automated solution is to use Inotify, it is what helm uses. It is working like this (basically): Each time you visit a new directory, helm is caching the content of this directory and at the same time tell inotify to watch this directory. when this directory is modified outside of helm, inotify delete the cache for this directory so that when you visit it again a fresh cache is created again. It seems that for some reasons Inotify is not working on your system, this is what you have to fix probably, unfortunately I know nothing about your system and your installation (please fill in the bug template when you submit a bug).

What are your thoughts? Is there a better way to solve this issue overall or is this on the right track?

Inotify is the best thing to use, it works fine since years in Helm, may be your system is not providing this?

…

-- Thierry

[MattMicheletti]

Thanks for the fast reply. Apologies for not including system information, I am usually pretty good about that. Here is my system information:

Standard environment disclosure:

• OS: MacOS
• OS Version: 15.4.1 (Sequoia)
• Chip: Apple Silicon
• Helm version: 1b41178 (version 3.9.1) (haven't upgraded just yet)
• Emacs Version: 27.2.1 (yes, I know it's out of support)
• Port: Mitsuharu Yamamoto / Railwaycat
• Other Notes: - No special distributions / etc.
• Relevant Common Packages Used:
  • Helm
  • Standard internal Emacs packages

According to my search on my system, Google, and AI I don't believe macOS supports inotify (AFAIK, that's a Linux kernel only mechanism). I am not sure how Helm's file caching works on macOS then if it's not using inotify. As well as why it sometimes it works great and then other times, it stops picking up changes in some directories.

Regarding the example recipe for re-production, I did it and it did indeed work. However, I have ran into this issue semi-frequently after using external commands such as build tools (e.g. Apache Maven, Gradle, etc.) and shell scripts. Via both external terminal and via Emacs built in compile and start-process-shell-command methods. I have a few video snippets of random times where it has happened and I have needed to do a helm-refresh when browsing the directory via helm-find-files. I could try to provide them somehow, however I am not sure it's very helpful.

It's intermittent for sure and often a lot of reads, writes, and deletes are happening to directories when it occurs (as one would expect when compiling large projects), I imagine at least since those are the only times I can think of when it has occurred.

[thierryvolpiatto]

Matt Micheletti ***@***.***> writes:

● MattMicheletti left a comment (emacs-helm/helm#2758) Thanks for the fast reply. Apologies for not including system information, I am usually pretty good about that. Here is my system information: Standard environment disclosure: • OS: MacOS • OS Version: 15.4.1 (Sequoia) • Chip: Apple Silicon • Helm version: 1b41178 (version 3.9.1) (haven't upgraded just yet) • Emacs Version: 27.2.1 (yes, I know it's out of support) • Port: Mitsuharu Yamamoto / Railwaycat • Other Notes: - No special distributions / etc. • Relevant Common Packages Used: □ Helm □ Standard internal Emacs packages According to my search on my system, Google, and AI I don't believe macOS supports inotify (AFAIK, that's a Linux kernel only mechanism). I am not sure how Helm's file caching works on macOS then if it's not using inotify.

Helm uses filenotify.el which is a unified interface to all file notifications systems, see: https://www.gnu.org/software/emacs/manual/html_node/elisp/File-Notifications.html AFAIK on MacOS filenotify is using kqueue, see: https://en.wikipedia.org/wiki/Kqueue So it is supported on MacOs. If the variable `file-notify--library` is non nil your Emacs should work with kqueue: ,---- | Non-nil when Emacs has been compiled with file notification support. | The value is the name of the low-level file notification package | to be used for local file systems. Remote file notifications | could use another implementation. `---- So on your system (featurep 'kqueue) should return `kqueue`. Apparently Emacs is compiled by default with notify support, I never had to specify something on Emacs compilation, I use now emacs-30 but on emacs-27 helm was working properly as well, I see no reasons why it would not still working nowaday. Can you check if filenotify is working properly on your system? If not perhaps we should report a bug on Emacs.

As well as why it sometimes it works great and then other times, it stops picking up changes in some directories. Regarding the example recipe for re-production, I did it and it did indeed work. However, I have ran into this issue semi-frequently after using external commands such as build tools (e.g. Apache Maven, Gradle, etc.) and shell scripts. Via both external terminal and via Emacs built in compile and start-process-shell-command methods. I have a few video snippets of random times where it has happened and I have needed to do a helm-refresh when browsing the directory via helm-find-files. I could try to provide them somehow, however I am not sure it's very helpful.

You lost me here, is file notifications working sometimes or not working at all? Thanks.

…

-- Thierry

[MattMicheletti]

Again, thank you so much for the super fast reply. My apologies for my delayed response, I was spending some time investigating some of the items you mentioned.

Helm uses filenotify.el which is a unified interface to all file
notifications systems, see:

https://www.gnu.org/software/emacs/manual/html_node/elisp/File-Notifications.html

AFAIK on MacOS filenotify is using kqueue, see:

https://en.wikipedia.org/wiki/Kqueue

So it is supported on MacOs.

If the variable file-notify--library is non nil your Emacs should work
with kqueue:

,----
| Non-nil when Emacs has been compiled with file notification support.
| The value is the name of the low-level file notification package
| to be used for local file systems. Remote file notifications
| could use another implementation.
`----

So on your system (featurep 'kqueue) should return kqueue.

Apparently Emacs is compiled by default with notify support, I never had
to specify something on Emacs compilation, I use now emacs-30 but on
emacs-27 helm was working properly as well, I see no reasons why it
would not still working nowaday.

Can you check if filenotify is working properly on your system?
If not perhaps we should report a bug on Emacs.

Ah, ok, that clarifies things. Yes, I checked my system and it is indeed using kqueue. Emacs is loading it and it is working. This is new information for me and quite interesting to learn, thank you!

You lost me here, is file notifications working sometimes or not working
at all?

Apologies, I can see how it was confusing. Put simply, Helm's helm-find-files does work and does show a given directory's contents correctly ... for a while. At some point though, in a session, when I go to that same directory in helm-find-files it reports seeing X files. However, when I then go to visit that same directory in macOS Finder, for example, I see Y files. Helm/Emacs got out of sync with the real file system and is not showing what is really there. I have to do a manual refresh with helm-refresh or completely restart Emacs.

As I was doing more research trying to find a way to show this does exist, I stumbled onto the steps to reproduce the issue (which is good news hopefully). I even have video recordings of me doing this both initially by accident and via these exact steps.

Steps to Reproduce

Heads-up, my steps to reproduce are always exceptionally detailed and these are key press-by-key press essentially. I am not being condescending or disrespectful, I am merely providing every single action I took in case any one of them might help shed light on the issue or provide necessary context.

Pre-Requisite Resources

example_script.sh

The following script simulates some common I/O operations that are similar to, as a facsimile at best, to those found while invoking common build tools. The issue of helm-find-files not displaying the right contents doesn't always happen, but usually it is more likely to occur when there are a lot of I/O operations for the directory, ergo build tools.

#!/bin/sh
# Temporary directory
TEMP_DIRECTORY=/tmp/fiz;

for no_of_times in {1..10}; do
	# Clean any previous output files
	rm -rf $TEMP_DIRECTORY;
	
	# Create temporary directories for output files
	mkdir -p $TEMP_DIRECTORY;
	
	echo "Starting ${no_of_times}/10 times...";
	for index in {1..100}; do
		echo "Hello ${index}" &> "$TEMP_DIRECTORY/output_${index}.txt";
	done
	
	for ((index = 1; index <= no_of_times; index++)); do
		rm "$TEMP_DIRECTORY/output_${index}.txt";
	done
	echo "Completed ${no_of_times}/10 times!";
done

echo "Creating final directory contents: foo.txt, bar.txt, baz.txt";
# Clean any previous output files
rm -rf $TEMP_DIRECTORY;

# Create temporary directories for output files
mkdir -p $TEMP_DIRECTORY;

echo "Foo" &> "$TEMP_DIRECTORY/foo.txt";
echo "Bar" &> "$TEMP_DIRECTORY/bar.txt";
echo "Baz" &> "$TEMP_DIRECTORY/baz.txt";

echo "Completed!";

Pre-Requisite Steps/Assumptions:

0. Ensure you don't have a directory named fiz in your /tmp directory

Steps:

1. Open a new external shell (iTerm2 is what I am using)
2. Navigate to cd /tmp
3. Open another new external shell where the shell script is located to prepare for running it (it should NOT be in the temp directory, just to ensure absolutely nothing else is interfering with the test ... mine was in my ~/.emacs.d/temp/example_script.sh)
4. Open a new Emacs instance
5. Open the shell script in a buffer
6. Split the window below
7. Staying in the top window
8. Open the scratch buffer
9. Type helm-ff-use-notify and evaluate it via C-j to verify it is t
10. Delete the line
11. Delete the window displaying the scratch buffer
12. Focus on your external shell prepared to run the shell script
13. Focus back to Emacs
14. Focus on your external shell prepared to run the shell script
15. Focus on your external shell with the temp directory open
16. Focus back to Emacs
17. Split the window below
18. Focus on the bottom window
19. Navigate to /tmp/fiz/buzz.txt via helm-find-files
20. Accept to create the directory when Helm prompts you
21. Type "Buzz" in the buzz.txt buffer
22. Save the buzz.txt buffer to the buzz.txt file
23. Kill the buzz.txt buffer via kill-buffer invoked via C-x k
24. Delete the bottom window open in Emacs via delete-window
25. Focus on your external shell with the temp directory open
26. Run tree for the temp directory to confirm fiz is created
27. Run tree fiz for the temp directory to confirm fiz contains buzz.txt only
28. Focus on your external shell prepared to run the shell script
29. Execute the shell script via sh ./example_script.sh
30. Focus on your external shell with the temp directory open
31. Run tree fiz for the temp directory to confirm fiz contains foo.txt, bar.txt, baz.txt only
32. Focus back to Emacs
33. Split the window below
34. Focus on the bottom window
35. Navigate to /tmp/fiz via helm-find-files and open baz.txt
36. Focus on your external shell prepared to run the shell script
37. Execute the shell script via sh ./example_script.sh again
38. Execute the shell script via sh ./example_script.sh again (yes, a second time)
39. Execute the shell script via sh ./example_script.sh again (yes, a total of three times)
40. Focus back to Emacs bottom window (i.e. baz.txt)
41. Invoke helm-find-files via C-x C-f and cancel via C-g
42. Focus on the top window in Emacs (i.e. example_script.sh)
43. Comment out lines 23 - 32 (inclusive) (i.e. the statement starting echo "Creating final ... until and including the statement starting echo "Baz" &> ...)
44. Save the example_script.sh buffer to the example_script.sh file via save-buffer invoked via s-s (mine is bound to Cmd + S as a customization)
45. Focus on your external shell prepared to run the shell script
46. Execute the shell script via sh ./example_script.sh again
47. Focus back to Emacs bottom window (i.e. baz.txt)
48. Invoke helm-find-files via C-x C-f (on baz.txt as the active buffer and file)
49. NOTICE: You are in /tmp/fiz and you should see Output_11.txt - Output.100.txt (approximately 90 files) but you don't ... this is where the problem is
50. Navigate up a directory via deleting the fiz directory name in the minibuffer path of Helm (or use C-l, it should not matter but I am just giving the exact steps I did)
51. Cancel helm-find-files via C-g
52. Invoke helm-find-files again and cancel via C-g
53. Focus on your external shell with the temp directory open
54. Run tree fiz for the temp directory to confirm fiz contains Output_11.txt - Output.100.txt (approximately 90 files)
55. Focus back to Emacs bottom window (i.e. baz.txt)
56. Invoke helm-find-files via C-x C-f, again on baz.txt confirming you DON'T see Output_11.txt - Output.100.txt (approximately 90 files) and cancel via C-g
57. Invoke helm-find-files via C-x C-f, again on baz.txt ... still confirming you DON'T see Output_11.txt - Output.100.txt (approximately 90 files) and cancel via C-g
58. Navigate up a directory via deleting the fiz directory name in the minibuffer path of Helm (or use C-l, again, it should not matter but I am just giving the exact steps I did)
59. Navigate into any other directory in tmp (which I had some and most people will) to verify helm-find-files is doing something (sanity check) and cancel via C-g
60. Invoke helm-find-files via C-x C-f, again on baz.txt and invoke helm-refresh via C-c C-u
61. NOTICE: You are in /tmp/fiz and you will see Output_11.txt - Output.100.txt (approximately 90 files) now after doing a refresh

Musings / Speculative Thoughts

NOTE: - WITHOUT doing helm-refresh, even killing the *helm-find-files* buffer does NOT fix the issue (which might make sense due to the caching and so forth behind the scenes). Also, navigating to /temp/fiz via helm-find-files through any other location (i.e. buffer, window, etc.) in Emacs/Helm also does NOT fix the issue either.

I hope this helps and allows you to see the issue as well and reproduce it on your end.

For now, I am using a bit of a sledgehammer to the whole issue. The solution I have come up with is to turn off helm-ff-use-notify and then wrap helm-ff-directory-files with :around advice that invokes the original method with the force-update parameter to always be true (e.g. (funcall helm-ff-directory-files-orig-method directory t)). It's ideal or even optimal and I need to do more testing but this should fix the problem visa-vie brute-force.

Please let me know if I could provide you any more information or assist in debugging this issue.

Thanks,
Matt

[thierryvolpiatto]

Matt Micheletti ***@***.***> writes:

Apologies, I can see how it was confusing. Put simply, Helm's helm-find-files does work and does show a given directory's contents correctly ... for a while. At some point though, in a session, when I go to that same directory in helm-find-files it reports seeing X files. However, when I then go to visit that same directory in macOS Finder, for example, I see Y files. Helm/Emacs got out of sync with the real file system and is not showing what is really there. I have to do a manual refresh with helm-refresh or completely restart Emacs.

I think I guess what may be wrong, at some point when the changes in a directory are too frequent filenotify (kqueue for you) gets confused, in this case (didn't checked yet) I hope `file-notify-valid-p` returns nil on the watcher handling directory. If this is right I may fix this on the helm side, I will commit a patch ASAP. Currently I am testing the watcher but I refresh only it when invalid instead of refreshing as well the cache, so hopefully doing this may fix the issue. That said I think even if we fix this on the helm side, it should be reported as an emacs bug to have more infos on this. I didn't yet reproduce with your recipe, but I will today if I have some time. Thanks for your detailed report and recipe, really appreciate.

…

-- Thierry

[thierryvolpiatto]

I commited a change (4621d85) that may fix this, I will try to reproduce your bug using your recipe in next days, let me know if you can reproduce on your side. Thanks. PS: I have no MacOs machine, so I can only try to reproduce on my (Arch)Linux box.

…

-- Thierry

[thierryvolpiatto]

Matt Micheletti ***@***.***> writes:

Steps:

[...]

48. Invoke helm-find-files via C-x C-f (on baz.txt as the active buffer and file) 49. NOTICE: You are in /tmp/fiz and you should see Output_11.txt - Output.100.txt (approximately 90 files) but you don't ... this is where the problem is

Ok with the recentest helm I can't reproduce this (didn't try yet without my patch applied), I do see all the Output_<n>.txt files. Let me know if you if you can reproduce on your side. Thanks.

…

-- Thierry

[MattMicheletti]

Hey Thierry,

Thanks for the quick patch! I backported the patch and tested it on my end and it appears to be working; I am no longer seeing the issue when I run the steps to reproduce. Seems to be working great, as expected. I think this does indeed fix it, however, I will keep an eye out all the same.

I am still curious to see if you can reproduce the issue on your end, without the aforementioned patch applied. It would be a good double check and to get confirmation of the underlying issue, just to be safe.

Thanks again,
Matt

[thierryvolpiatto]

Matt Micheletti ***@***.***> writes:

● MattMicheletti left a comment (emacs-helm/helm#2758) Hey Thierry, Thanks for the quick patch! I backported the patch and tested it on my end and it appears to be working; I am no longer seeing the issue when I run the steps to reproduce. Seems to be working great, as expected. I think this does indeed fix it, however, I will keep an eye out all the same.

That's great news! Yes let me know if something happen, we keep the issue open for now.

I am still curious to see if you can reproduce the issue on your end, without the aforementioned patch applied. It would be a good double check and to get confirmation of the underlying issue, just to be safe.

No, I can't reproduce the issue, so I thing it is specific to Kqueue. Thanks again for your report and very good recipe!

…

-- Thierry

[chuvke]

This commit 4621d85 seems to break the listing of files when using tramp for me. After entering the remote name e.g /ssh:root@ip:
I dont get a listing of the files anymore. I reverted to prev rev of this file, then it works as expected

[thierryvolpiatto]

chuvke ***@***.***> writes:

● chuvke left a comment (emacs-helm/helm#2758) This commit seems to break the listing of files when using tramp for me. After entering the remote name e.g ***@***.***: I dont get a listing of the files anymore. I reverted to prev rev of this file, then it works as expected

I can't reproduce here (arch linux), tried with ssh and sudo methods. Can you give more details on how to reproduce? Thanks.

…

— Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today! You are receiving this because you commented.*Message ID: ***@***.***>

-- Thierry

[chuvke]

I've some additional info

• I see it with many tramp methods, scp, ssh, shhx, rpc etc
• Not on all remotes, i've ubuntu 24.02 (OK), embedded device (NOK)
• I can create a new file when I enter a valid path
• I can't open an existing file when i enter a valid path (no files shown)
• When i try to open an existing directory (no files shown) I get error message [Trying to exit with no candidates]

[thierryvolpiatto]

chuvke ***@***.***> writes:

● chuvke left a comment (emacs-helm/helm#2758) I've some additional info • I see it with many tramp methods, scp, ssh, shhx, rpc etc • Not on all remotes, i've ubuntu 24.02 (OK), embedded device (NOK)

What are embedded devices for you?

• I can create a new file when I enter a valid path

Yes, this is unrelated, it is another source.

• I can't open an existing file when i enter a valid path (no files shown) • When i try to open an existing directory (no files shown) I get error message [Trying to exit with no candidates]

This is normal, you have no candidates. Can you try to reproduce from a minimal Emacs (./emacs-helm.sh)? What is your system (Linux, MacOs...)? Also try to reinstall Helm and restart Emacs, I hardly see how this last change would lead to a listing with no candidates as it affects only how the watchers are created/deleted. I tried on several machines and it is working everywhere. Try also C-c C-u from helm-find-files, maybe you saved inadvertently one of the vars handling cache (desktop, psession or whatever)? Thanks.

…

— Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today! You are receiving this because you commented.*Message ID: ***@***.***>

-- Thierry

[chuvke]

What are embedded devices for you?

In this case a custom board (Yocto based linux) with regular openssh and sh as shell

Can you try to reproduce from a minimal Emacs (./emacs-helm.sh)?
What is your system (Linux, MacOs...)?

Yes I tried with a fresh install of emacs (empty config, add melpa, install helm 20260712, run emacs-helm.sh). I'm using Linux (cachyOS, arch)

Try also C-c C-u from helm-find-files, maybe you saved inadvertently one
of the vars handling cache (desktop, psession or whatever)?

I tried to hit C-c C-u to refresh the listing but no difference, still empty list.

One observation is that even with the prev rev, 0aa8991, when i enter /ssh:root@<ip>: then at first i get also an empty list. I then have to delete the last : and re-enter : to see the list. I don't have to do that on the Ubuntu remote, there i get the dir listing directly. Maybe that is an additional clue. With the newer rev 4621d85, this delete and re-enter : doesn't work.

Is there something I can do to extract more info, like adding debug statements or so ? I checked the tramp log and that one seems to be ok, no error.