Does not work/tests fail on OpenBSD

[marvinthepa]

On OpenBSD, with both node v16.14.2 and v12.22.6, webpack does not recompile when files are changed.
It used to work when chokidar was used for file watching, i.e. before watchpack 2.

watchpack version: 2.4.0
webpack version: 5.73.0

The tests fail as well (on main branch, dc690bb/2.4.0):

  1) Assumption                                            
       should have a file system with correct mtime behavior (fs.watch):
     Error: Timeout of 20000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/data/git/msander/watchpack/test/Assumption.js)
      at Test.Runnable._timeoutError (/data/git/msander/watchpack/node_modules/mocha/lib/runnable.js:440:10)
      at Timeout.<anonymous> (/data/git/msander/watchpack/node_modules/mocha/lib/runnable.js:251:24)
      at listOnTimeout (node:internal/timers:559:17)
      at processTimers (node:internal/timers:502:7)

  2) Assumption                                            
       should detect removed directory:
     Error: Didn't receive a event about removed directory
      at /data/git/msander/watchpack/test/Assumption.js:334:13
      at Timeout._onTimeout (/data/git/msander/watchpack/test/helpers/TestHelper.js:129:3)
      at listOnTimeout (node:internal/timers:559:17)
      at processTimers (node:internal/timers:502:7)

  3) DirectoryWatcher                                      
       should detect a file change:
     Error: Timeout of 10000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/data/git/msander/watchpack/test/DirectoryWatcher.js)
      at Test.Runnable._timeoutError (/data/git/msander/watchpack/node_modules/mocha/lib/runnable.js:440:10)
      at Timeout.<anonymous> (/data/git/msander/watchpack/node_modules/mocha/lib/runnable.js:251:24)
      at listOnTimeout (node:internal/timers:559:17)
      at processTimers (node:internal/timers:502:7)

  4) DirectoryWatcher                                      
       "after each" hook:                                  

      Uncaught AssertionError: expected 1 to equal 0
      + expected - actual                                  

      -1                                                   
      +0                                                   
                                                           
      at Assertion.fail (/data/git/msander/watchpack/node_modules/should/lib/assertion.js:92:17)
      at Assertion.value (/data/git/msander/watchpack/node_modules/should/lib/assertion.js:164:19)
      at checkAllWatcherClosed (/data/git/msander/watchpack/test/helpers/TestHelper.js:25:51)
      at TestHelper.del (/data/git/msander/watchpack/test/helpers/TestHelper.js:64:4)
      at Timeout._onTimeout (/data/git/msander/watchpack/test/helpers/TestHelper.js:129:3)
      at listOnTimeout (node:internal/timers:559:17)
      at processTimers (node:internal/timers:502:7)

  5) ManyWatchers                                          
       "before each" hook for "should watch more than 4096 directories":

      AssertionError: expected 1 to equal 0
      + expected - actual                                  

      -1                                                   
      +0                                                   
                                                           
      at Assertion.fail (/data/git/msander/watchpack/node_modules/should/lib/assertion.js:92:17)
      at Assertion.value (/data/git/msander/watchpack/node_modules/should/lib/assertion.js:164:19)
      at checkAllWatcherClosed (/data/git/msander/watchpack/test/helpers/TestHelper.js:25:51)
      at TestHelper.before [as _before] (/data/git/msander/watchpack/test/helpers/TestHelper.js:41:2)
      at Context.TestHelper.before (/data/git/msander/watchpack/test/helpers/TestHelper.js:32:8)
      at callFnAsync (/data/git/msander/watchpack/node_modules/mocha/lib/runnable.js:400:21)
      at Hook.Runnable.run (/data/git/msander/watchpack/node_modules/mocha/lib/runnable.js:342:7)
      at next (/data/git/msander/watchpack/node_modules/mocha/lib/runner.js:317:10)
      at Immediate._onImmediate (/data/git/msander/watchpack/node_modules/mocha/lib/runner.js:347:5)
      at processImmediate (node:internal/timers:466:21)

  6) ManyWatchers                                                                                                      
       "after each" hook:                                                                                              
                                                                                                                       
      Uncaught AssertionError: expected 1 to equal 0   
      + expected - actual                                  
                                                           
      -1                                                   
      +0                                                   
                                                           
      at Assertion.fail (/data/git/msander/watchpack/node_modules/should/lib/assertion.js:92:17)
      at Assertion.value (/data/git/msander/watchpack/node_modules/should/lib/assertion.js:164:19)
      at checkAllWatcherClosed (/data/git/msander/watchpack/test/helpers/TestHelper.js:25:51)
      at TestHelper.del (/data/git/msander/watchpack/test/helpers/TestHelper.js:64:4)
      at Timeout._onTimeout (/data/git/msander/watchpack/test/helpers/TestHelper.js:129:3)
      at listOnTimeout (node:internal/timers:559:17)                                                                   
      at processTimers (node:internal/timers:502:7)                                                                    
                                                                                                                       
  7) Watchpack                                                                                                         
       "before each" hook for "should watch a single file":                               
                                                           
      AssertionError: expected 1 to equal 0        
      + expected - actual                                  

      -1                                                   
      +0                                                   
                                                           
      at Assertion.fail (/data/git/msander/watchpack/node_modules/should/lib/assertion.js:92:17)
      at Assertion.value (/data/git/msander/watchpack/node_modules/should/lib/assertion.js:164:19)
      at checkAllWatcherClosed (/data/git/msander/watchpack/test/helpers/TestHelper.js:25:51)
      at TestHelper.before [as _before] (/data/git/msander/watchpack/test/helpers/TestHelper.js:41:2)
      at Context.TestHelper.before (/data/git/msander/watchpack/test/helpers/TestHelper.js:32:8)
      at callFnAsync (/data/git/msander/watchpack/node_modules/mocha/lib/runnable.js:400:21)
      at Hook.Runnable.run (/data/git/msander/watchpack/node_modules/mocha/lib/runnable.js:342:7)
      at next (/data/git/msander/watchpack/node_modules/mocha/lib/runner.js:317:10)
      at Immediate._onImmediate (/data/git/msander/watchpack/node_modules/mocha/lib/runner.js:347:5)
      at processImmediate (node:internal/timers:466:21)

  8) Watchpack                                             
       "after each" hook:                                  

      Uncaught AssertionError: expected 1 to equal 0
      + expected - actual                                  

      -1                                                   
      +0                                                   
                                                           
      at Assertion.fail (/data/git/msander/watchpack/node_modules/should/lib/assertion.js:92:17)
      at Assertion.value (/data/git/msander/watchpack/node_modules/should/lib/assertion.js:164:19)
      at checkAllWatcherClosed (/data/git/msander/watchpack/test/helpers/TestHelper.js:25:51)
      at TestHelper.del (/data/git/msander/watchpack/test/helpers/TestHelper.js:64:4)
      at Timeout._onTimeout (/data/git/msander/watchpack/test/helpers/TestHelper.js:129:3)
      at listOnTimeout (node:internal/timers:559:17)
      at processTimers (node:internal/timers:502:7)

The tests pass for the last watchpack 1 release..

I realize that this is probably an issue with the underlying fs.watch implementation, but do you have any hints on how I could get webpack working properly? Relying on polling is not an option, as it utilizes 100% CPU when used with any reasonable polling interval.
Other solutions that rely on kqueue (e.g. https://github.com/guard/guard) do work, so it cannot be impossible. I would like to use one of those, is it possible to notify webpack of changed files e.g. via a notification on a port or domain socket? Could this be achieved in a plugin?

[alexander-akait]

@marvinthepa Is watchpack version: 2.3.6 working fine?

[marvinthepa]

@alexander-akait I did not find a 2.3.6 Version, did you mean 2.3.1?. Tests in Version 2.3.1 fail as well.

I also tried v2.0.0 and v2.0.0-beta.0 and 805b665 (which removed the chokidar dependency). Tests work with the previous commit a1f8f3f. I do not assume that there was a working commit since then 😿

[marvinthepa]

I checked the first test more closely.

It seems that when watching a directory, the change events

• are only ever fired when files appear or disappear
• are always rename
• never contain a file name

I guess this worked with chokidar because it does the normalization as described in https://github.com/paulmillr/chokidar#how (to get the file name from the rename event of the directory) and registers new watchers for files that appear, so they generate events, too.

You would not consider re-introducing chokidar version 3 as a dependency, would you? webpack-dev-server still has a dependency on chokidar anyway, so it would not even introduce new dependencies for people who use it..

[alexander-akait]

@marvinthepa In long term dev server will not use chokidar too, maybe you want to backport the solution from chokidar?

[marvinthepa]

maybe you want to backport the solution from chokidar?

I could give it a try if I find the time, but it seems to me the problem is a complicated one in which I lack expertise. So I probably would end up copying and pasting a lot of code from chokidar into watchpack, and I do not quite see how this would be an improvement over using chokidar directly..

[alexander-akait]

@marvinthepa chokidar has many perf problems and required a compilation after instalattion (this is not a problem, but happens for wsl and other exotic systems/subsystems)

[marvinthepa]

I had a look through the solutions, and if my (very limited) understanding is correct, fixing this in watchpack would require significant changes. For example, using a watcher per directory is not sufficient, as changes to files do not trigger a change event on the containing directory - you need an extra DirectoryWatcher per file (making it a misnomer), and you have to manage them as well.
I don't think I will find the time to fix this in the near future.

I will not close this issue though, I still find some hope that somebody else wants to tackle this. Would be interesting if this is an issue on other BSDs as well..

It seems to me the whole "watching files" thing is very frustrating on node. Mainly because node itself does not really provide a platform-agnostic API..