Separate configuration for ignored files and watching

[cmahnke]

Currently the configuration for ignoring files doesn't distinguish between files for deployment and for watching. This can lead to problems with huge trees.

My use case are sites including large image pyramids, generated by vips. These are subdirectories including fragments of images which are used to provide zoomable high resolution images to the user.

The result are several dozen image fragment for each provided image. This blows up the content tree quite a lot, but since these are generated externally, they doesn't need to be watched. Adding these folders to the ignoreFiles in the module mount section, also exclude them from access for the templates, which leads to the problem, that they can't get required metadata, like the image size.

Since hugo serve actually breaks down on huge trees, including them have the side effect, that hugo takes ages to start up until it fails:

goroutine 1 [running]:
runtime.throw({0x105831102?, 0x1400aaa8480?})
	runtime/panic.go:1047 +0x40 fp=0x1400b7017a0 sp=0x1400b701770 pc=0x1040fd8a0
runtime.sigNoteSetup(0x90)
	runtime/os_darwin.go:101 +0xc4 fp=0x1400b7017d0 sp=0x1400b7017a0 pc=0x1040fa854
os/signal.signal_enable(0x2)
	runtime/sigqueue.go:205 +0x40 fp=0x1400b7017f0 sp=0x1400b7017d0 pc=0x10412d510
os/signal.enableSignal(...)
	os/signal/signal_unix.go:49
os/signal.Notify.func1(0x2)
	os/signal/signal.go:145 +0x7c fp=0x1400b701810 sp=0x1400b7017f0 pc=0x1055df99c
os/signal.Notify(0x1400aa26cc0, {0x1400b701ad0, 0x2, 0x1040d36f4?})
	os/signal/signal.go:165 +0x258 fp=0x1400b701890 sp=0x1400b701810 pc=0x1055df878
github.com/gohugoio/hugo/commands.(*commandeer).serve(0x14000a0cb00, 0x140003def00)
	github.com/gohugoio/hugo/commands/server.go:593 +0x454 fp=0x1400b701b00 sp=0x1400b701890 pc=0x10560e674
github.com/gohugoio/hugo/commands.(*serverCmd).server(0x140003def00, 0x14000433800, {0x0?, 0x0?, 0x0?})
	github.com/gohugoio/hugo/commands/server.go:297 +0x2c8 fp=0x1400b701c50 sp=0x1400b701b00 pc=0x10560b3b8
github.com/gohugoio/hugo/commands.(*commandsBuilder).newServerCmdSignaled.func1(0x14000433800?, {0x140006c1ae0?, 0x2?, 0x2?})
	github.com/gohugoio/hugo/commands/server.go:96 +0x3c fp=0x1400b701ca0 sp=0x1400b701c50 pc=0x10560af9c
github.com/spf13/cobra.(*Command).execute(0x14000433800, {0x140006c1ac0, 0x2, 0x2})
	github.com/spf13/cobra@v1.6.1/command.go:916 +0x5c8 fp=0x1400b701de0 sp=0x1400b701ca0 pc=0x1045f2048
github.com/spf13/cobra.(*Command).ExecuteC(0x14000a02900)
	github.com/spf13/cobra@v1.6.1/command.go:1044 +0x35c fp=0x1400b701ea0 sp=0x1400b701de0 pc=0x1045f279c
github.com/gohugoio/hugo/commands.Execute({0x140001a2010, 0x3, 0x3})
	github.com/gohugoio/hugo/commands/hugo.go:92 +0x94 fp=0x1400b701f20 sp=0x1400b701ea0 pc=0x1055fa0c4
main.main()
	github.com/gohugoio/hugo/main.go:23 +0x50 fp=0x1400b701f70 sp=0x1400b701f20 pc=0x105613bb0
runtime.main()
	runtime/proc.go:250 +0x24c fp=0x1400b701fd0 sp=0x1400b701f70 pc=0x1040ffe1c
runtime.goexit()
	runtime/asm_arm64.s:1172 +0x4 fp=0x1400b701fd0 sp=0x1400b701fd0 pc=0x104131654

goroutine 2 [force gc (idle), 3 minutes]:
runtime.gopark(0xb9792e7d4377?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:363 +0xe4 fp=0x1400006efa0 sp=0x1400006ef80 pc=0x1041001e4
runtime.goparkunlock(...)
	runtime/proc.go:369
runtime.forcegchelper()
	runtime/proc.go:302 +0xac fp=0x1400006efd0 sp=0x1400006efa0 pc=0x10410007c
runtime.goexit()
	runtime/asm_arm64.s:1172 +0x4 fp=0x1400006efd0 sp=0x1400006efd0 pc=0x104131654
created by runtime.init.6
	runtime/proc.go:290 +0x24

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:363 +0xe4 fp=0x1400006f770 sp=0x1400006f750 pc=0x1041001e4
runtime.goparkunlock(...)
	runtime/proc.go:369
runtime.bgsweep(0x0?)
	runtime/mgcsweep.go:297 +0x10c fp=0x1400006f7b0 sp=0x1400006f770 pc=0x1040eb75c
runtime.gcenable.func1()
	runtime/mgc.go:178 +0x28 fp=0x1400006f7d0 sp=0x1400006f7b0 pc=0x1040dfcb8
runtime.goexit()
	runtime/asm_arm64.s:1172 +0x4 fp=0x1400006f7d0 sp=0x1400006f7d0 pc=0x104131654
created by runtime.gcenable
	runtime/mgc.go:178 +0x74

goroutine 4 [GC scavenge wait]:
runtime.gopark(0x1e636c8e?, 0x1e537517?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:363 +0xe4 fp=0x1400006ff50 sp=0x1400006ff30 pc=0x1041001e4
runtime.goparkunlock(...)
	runtime/proc.go:369
runtime.(*scavengerState).park(0x10730ada0)
	runtime/mgcscavenge.go:389 +0x5c fp=0x1400006ff80 sp=0x1400006ff50 pc=0x1040e96ac
runtime.bgscavenge(0x0?)
	runtime/mgcscavenge.go:622 +0xac fp=0x1400006ffb0 sp=0x1400006ff80 pc=0x1040e9cbc
runtime.gcenable.func2()
	runtime/mgc.go:179 +0x28 fp=0x1400006ffd0 sp=0x1400006ffb0 pc=0x1040dfc58
runtime.goexit()
	runtime/asm_arm64.s:1172 +0x4 fp=0x1400006ffd0 sp=0x1400006ffd0 pc=0x104131654
created by runtime.gcenable
	runtime/mgc.go:179 +0xb8

goroutine 18 [finalizer wait, 5 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:363 +0xe4 fp=0x1400006a580 sp=0x1400006a560 pc=0x1041001e4
runtime.goparkunlock(...)
	runtime/proc.go:369
runtime.runfinq()
	runtime/mfinal.go:180 +0x120 fp=0x1400006a7d0 sp=0x1400006a580 pc=0x1040deee0
runtime.goexit()
	runtime/asm_arm64.s:1172 +0x4 fp=0x1400006a7d0 sp=0x1400006a7d0 pc=0x104131654
created by runtime.createfing
	runtime/mfinal.go:157 +0x84

goroutine 19 [GC worker (idle), 5 minutes]:
runtime.gopark(0xb940386f2151?, 0x3?, 0x77?, 0x8c?, 0x0?)
	runtime/proc.go:363 +0xe4 fp=0x1400006af40 sp=0x1400006af20 pc=0x1041001e4
runtime.gcBgMarkWorker()
	runtime/mgc.go:1235 +0xec fp=0x1400006afd0 sp=0x1400006af40 pc=0x1040e209c
runtime.goexit()
	runtime/asm_arm64.s:1172 +0x4 fp=0x1400006afd0 sp=0x1400006afd0 pc=0x104131654
created by runtime.gcBgMarkStartWorkers
	runtime/mgc.go:1159 +0x28

goroutine 20 [GC worker (idle)]:
runtime.gopark(0xb98c14cf8c36?, 0x3?, 0x48?, 0x58?, 0x0?)
	runtime/proc.go:363 +0xe4 fp=0x14000087f40 sp=0x14000087f20 pc=0x1041001e4
runtime.gcBgMarkWorker()
	runtime/mgc.go:1235 +0xec fp=0x14000087fd0 sp=0x14000087f40 pc=0x1040e209c
runtime.goexit()
	runtime/asm_arm64.s:1172 +0x4 fp=0x14000087fd0 sp=0x14000087fd0 pc=0x104131654
created by runtime.gcBgMarkStartWorkers
	runtime/mgc.go:1159 +0x28

goroutine 34 [GC worker (idle)]:
runtime.gopark(0xb98c14d02247?, 0x3?, 0xc0?, 0xc6?, 0x0?)
	runtime/proc.go:363 +0xe4 fp=0x14000085f40 sp=0x14000085f20 pc=0x1041001e4
runtime.gcBgMarkWorker()
	runtime/mgc.go:1235 +0xec fp=0x14000085fd0 sp=0x14000085f40 pc=0x1040e209c
runtime.goexit()
	runtime/asm_arm64.s:1172 +0x4 fp=0x14000085fd0 sp=0x14000085fd0 pc=0x104131654
created by runtime.gcBgMarkStartWorkers
	runtime/mgc.go:1159 +0x28

goroutine 21 [GC worker (idle)]:
runtime.gopark(0xb98c14cf9118?, 0x1?, 0x45?, 0xfe?, 0x0?)
	runtime/proc.go:363 +0xe4 fp=0x1400006bf40 sp=0x1400006bf20 pc=0x1041001e4
runtime.gcBgMarkWorker()
	runtime/mgc.go:1235 +0xec fp=0x1400006bfd0 sp=0x1400006bf40 pc=0x1040e209c
runtime.goexit()
	runtime/asm_arm64.s:1172 +0x4 fp=0x1400006bfd0 sp=0x1400006bfd0 pc=0x104131654
created by runtime.gcBgMarkStartWorkers
	runtime/mgc.go:1159 +0x28

goroutine 35 [GC worker (idle)]:
runtime.gopark(0xb98c14cf8b66?, 0x1?, 0xf1?, 0x86?, 0x0?)
	runtime/proc.go:363 +0xe4 fp=0x140004a7f40 sp=0x140004a7f20 pc=0x1041001e4
runtime.gcBgMarkWorker()
	runtime/mgc.go:1235 +0xec fp=0x140004a7fd0 sp=0x140004a7f40 pc=0x1040e209c
runtime.goexit()
	runtime/asm_arm64.s:1172 +0x4 fp=0x140004a7fd0 sp=0x140004a7fd0 pc=0x104131654
created by runtime.gcBgMarkStartWorkers
	runtime/mgc.go:1159 +0x28

goroutine 36 [GC worker (idle)]:
runtime.gopark(0x10734bfe0?, 0x3?, 0xe8?, 0xc6?, 0x0?)
	runtime/proc.go:363 +0xe4 fp=0x14000080f40 sp=0x14000080f20 pc=0x1041001e4
runtime.gcBgMarkWorker()
	runtime/mgc.go:1235 +0xec fp=0x14000080fd0 sp=0x14000080f40 pc=0x1040e209c
runtime.goexit()
	runtime/asm_arm64.s:1172 +0x4 fp=0x14000080fd0 sp=0x14000080fd0 pc=0x104131654
created by runtime.gcBgMarkStartWorkers
	runtime/mgc.go:1159 +0x28

goroutine 37 [GC worker (idle)]:
runtime.gopark(0xb98c14cf8d84?, 0x3?, 0xe3?, 0x63?, 0x0?)
	runtime/proc.go:363 +0xe4 fp=0x14000587f40 sp=0x14000587f20 pc=0x1041001e4
runtime.gcBgMarkWorker()
	runtime/mgc.go:1235 +0xec fp=0x14000587fd0 sp=0x14000587f40 pc=0x1040e209c
runtime.goexit()
	runtime/asm_arm64.s:1172 +0x4 fp=0x14000587fd0 sp=0x14000587fd0 pc=0x104131654
created by runtime.gcBgMarkStartWorkers
	runtime/mgc.go:1159 +0x28

goroutine 5 [GC worker (idle)]:
runtime.gopark(0xb98c14cf829c?, 0x1?, 0x30?, 0x2f?, 0x14000098000?)
	runtime/proc.go:363 +0xe4 fp=0x1400006e740 sp=0x1400006e720 pc=0x1041001e4
runtime.gcBgMarkWorker()
	runtime/mgc.go:1235 +0xec fp=0x1400006e7d0 sp=0x1400006e740 pc=0x1040e209c
runtime.goexit()
	runtime/asm_arm64.s:1172 +0x4 fp=0x1400006e7d0 sp=0x1400006e7d0 pc=0x104131654
created by runtime.gcBgMarkStartWorkers
	runtime/mgc.go:1159 +0x28

goroutine 6 [GC worker (idle)]:
runtime.gopark(0xb98c14cf9c7d?, 0x3?, 0xcf?, 0x10?, 0x0?)
	runtime/proc.go:363 +0xe4 fp=0x14000086f40 sp=0x14000086f20 pc=0x1041001e4
runtime.gcBgMarkWorker()
	runtime/mgc.go:1235 +0xec fp=0x14000086fd0 sp=0x14000086f40 pc=0x1040e209c
runtime.goexit()
	runtime/asm_arm64.s:1172 +0x4 fp=0x14000086fd0 sp=0x14000086fd0 pc=0x104131654
created by runtime.gcBgMarkStartWorkers
	runtime/mgc.go:1159 +0x28

goroutine 7 [GC worker (idle)]:
runtime.gopark(0xb98c14cff0a3?, 0x1400058c080?, 0x18?, 0x14?, 0x0?)
	runtime/proc.go:363 +0xe4 fp=0x140004a6f40 sp=0x140004a6f20 pc=0x1041001e4
runtime.gcBgMarkWorker()
	runtime/mgc.go:1235 +0xec fp=0x140004a6fd0 sp=0x140004a6f40 pc=0x1040e209c
runtime.goexit()
	runtime/asm_arm64.s:1172 +0x4 fp=0x140004a6fd0 sp=0x140004a6fd0 pc=0x104131654
created by runtime.gcBgMarkStartWorkers
	runtime/mgc.go:1159 +0x28

goroutine 50 [select, 1 minutes]:
runtime.gopark(0x14000589f78?, 0x3?, 0xe0?, 0x64?, 0x14000589f62?)
	runtime/proc.go:363 +0xe4 fp=0x14000589e00 sp=0x14000589de0 pc=0x1041001e4
runtime.selectgo(0x14000589f78, 0x14000589f5c, 0x140000b0080?, 0x0, 0x0?, 0x1)
	runtime/select.go:328 +0x688 fp=0x14000589f20 sp=0x14000589e00 pc=0x104110af8
go.opencensus.io/stats/view.(*worker).start(0x140000b0080)
	go.opencensus.io@v0.23.0/stats/view/worker.go:276 +0x88 fp=0x14000589fb0 sp=0x14000589f20 pc=0x105252828
go.opencensus.io/stats/view.init.0.func1()
	go.opencensus.io@v0.23.0/stats/view/worker.go:34 +0x28 fp=0x14000589fd0 sp=0x14000589fb0 pc=0x105251ba8
runtime.goexit()
	runtime/asm_arm64.s:1172 +0x4 fp=0x14000589fd0 sp=0x14000589fd0 pc=0x104131654
created by go.opencensus.io/stats/view.init.0
	go.opencensus.io@v0.23.0/stats/view/worker.go:34 +0xa4

goroutine 1310 [select]:
runtime.gopark(0x140091d97a8?, 0x2?, 0xb0?, 0x8d?, 0x140091d9734?)
	runtime/proc.go:363 +0xe4 fp=0x140091d95b0 sp=0x140091d9590 pc=0x1041001e4
runtime.selectgo(0x140091d97a8, 0x140091d9730, 0x0?, 0x0, 0x0?, 0x1)
	runtime/select.go:328 +0x688 fp=0x140091d96d0 sp=0x140091d95b0 pc=0x104110af8
github.com/gohugoio/hugo/commands.(*commandeer).newWatcher.func1()
	github.com/gohugoio/hugo/commands/hugo.go:888 +0xac fp=0x140091d97d0 sp=0x140091d96d0 pc=0x1055fed1c
runtime.goexit()
	runtime/asm_arm64.s:1172 +0x4 fp=0x140091d97d0 sp=0x140091d97d0 pc=0x104131654
created by github.com/gohugoio/hugo/commands.(*commandeer).newWatcher
	github.com/gohugoio/hugo/commands/hugo.go:886 +0x350

goroutine 1309 [select]:
runtime.gopark(0x140091d8f78?, 0x3?, 0x1?, 0x0?, 0x140091d8f2a?)
	runtime/proc.go:363 +0xe4 fp=0x140091d8d90 sp=0x140091d8d70 pc=0x1041001e4
runtime.selectgo(0x140091d8f78, 0x140091d8f24, 0x0?, 0x0, 0x0?, 0x1)
	runtime/select.go:328 +0x688 fp=0x140091d8eb0 sp=0x140091d8d90 pc=0x104110af8
github.com/gohugoio/hugo/watcher.(*Batcher).run(0x1400b13a570)
	github.com/gohugoio/hugo/watcher/batcher.go:67 +0xe4 fp=0x140091d8fb0 sp=0x140091d8eb0 pc=0x1055ec8c4
github.com/gohugoio/hugo/watcher.New.func1()
	github.com/gohugoio/hugo/watcher/batcher.go:56 +0x28 fp=0x140091d8fd0 sp=0x140091d8fb0 pc=0x1055ec7a8
runtime.goexit()
	runtime/asm_arm64.s:1172 +0x4 fp=0x140091d8fd0 sp=0x140091d8fd0 pc=0x104131654
created by github.com/gohugoio/hugo/watcher.New
	github.com/gohugoio/hugo/watcher/batcher.go:56 +0x218

goroutine 51 [IO wait, 5 minutes]:
runtime.gopark(0xffffffffffffffff?, 0xffffffffffffffff?, 0x23?, 0x0?, 0x104145b30?)
	runtime/proc.go:363 +0xe4 fp=0x1400006cca0 sp=0x1400006cc80 pc=0x1041001e4
runtime.netpollblock(0x0?, 0x0?, 0x0?)
	runtime/netpoll.go:526 +0x158 fp=0x1400006cce0 sp=0x1400006cca0 pc=0x1040f96c8
internal/poll.runtime_pollWait(0x12f28def8, 0x72)
	runtime/netpoll.go:305 +0xa0 fp=0x1400006cd10 sp=0x1400006cce0 pc=0x10412b150
internal/poll.(*pollDesc).wait(0x14005b74660?, 0x14007a18200?, 0x1)
	internal/poll/fd_poll_runtime.go:84 +0x28 fp=0x1400006cd40 sp=0x1400006cd10 pc=0x1041676c8
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x14005b74660, {0x14007a18200, 0x200, 0x200})
	internal/poll/fd_unix.go:167 +0x1e0 fp=0x1400006cdc0 sp=0x1400006cd40 pc=0x1041689f0
os.(*File).read(...)
	os/file_posix.go:31
os.(*File).Read(0x14006c3a6e8, {0x14007a18200?, 0x0?, 0x12f3fdc60?})
	os/file.go:118 +0x5c fp=0x1400006ce20 sp=0x1400006cdc0 pc=0x1041716dc
bytes.(*Buffer).ReadFrom(0x14005b7ce18, {0x10632d180, 0x14006c3a6e8})
	bytes/buffer.go:202 +0x90 fp=0x1400006ce80 sp=0x1400006ce20 pc=0x10421a7b0
io.copyBuffer({0x106327c40, 0x14005b7ce10}, {0x10632d180, 0x14006c3a6e8}, {0x0, 0x0, 0x0})
	io/io.go:413 +0x108 fp=0x1400006cf10 sp=0x1400006ce80 pc=0x104161788
io.Copy(...)
	io/io.go:386
os/exec.(*Cmd).writerDescriptor.func1()
	os/exec/exec.go:407 +0x44 fp=0x1400006cf70 sp=0x1400006cf10 pc=0x1042af884
os/exec.(*Cmd).Start.func1(0x0?)
	os/exec/exec.go:544 +0x2c fp=0x1400006cfb0 sp=0x1400006cf70 pc=0x1042b03fc
os/exec.(*Cmd).Start.func2()
	os/exec/exec.go:545 +0x30 fp=0x1400006cfd0 sp=0x1400006cfb0 pc=0x1042b0390
runtime.goexit()
	runtime/asm_arm64.s:1172 +0x4 fp=0x1400006cfd0 sp=0x1400006cfd0 pc=0x104131654
created by os/exec.(*Cmd).Start
	os/exec/exec.go:543 +0x5d0

goroutine 52 [IO wait, 5 minutes]:
runtime.gopark(0xffffffffffffffff?, 0xffffffffffffffff?, 0x23?, 0x0?, 0x104145b30?)
	runtime/proc.go:363 +0xe4 fp=0x14004725b20 sp=0x14004725b00 pc=0x1041001e4
runtime.netpollblock(0x0?, 0x0?, 0x0?)
	runtime/netpoll.go:526 +0x158 fp=0x14004725b60 sp=0x14004725b20 pc=0x1040f96c8
internal/poll.runtime_pollWait(0x12f28e0d8, 0x72)
	runtime/netpoll.go:305 +0xa0 fp=0x14004725b90 sp=0x14004725b60 pc=0x10412b150
internal/poll.(*pollDesc).wait(0x14005b74540?, 0x14007ac8000?, 0x1)
	internal/poll/fd_poll_runtime.go:84 +0x28 fp=0x14004725bc0 sp=0x14004725b90 pc=0x1041676c8
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x14005b74540, {0x14007ac8000, 0x1000, 0x1000})
	internal/poll/fd_unix.go:167 +0x1e0 fp=0x14004725c40 sp=0x14004725bc0 pc=0x1041689f0
os.(*File).read(...)
	os/file_posix.go:31
os.(*File).Read(0x14006c3a6d8, {0x14007ac8000?, 0x3?, 0x45ba6?})
	os/file.go:118 +0x5c fp=0x14004725ca0 sp=0x14004725c40 pc=0x1041716dc
bufio.(*Reader).fill(0x14005b74600)
	bufio/bufio.go:106 +0xfc fp=0x14004725ce0 sp=0x14004725ca0 pc=0x1042aa2fc
bufio.(*Reader).ReadByte(0x14005b74600)
	bufio/bufio.go:265 +0x2c fp=0x14004725d00 sp=0x14004725ce0 pc=0x1042aab2c
github.com/bep/godartsass.(*conn).ReadByte(0x1060f25a0?)
	<autogenerated>:1 +0x30 fp=0x14004725d20 sp=0x14004725d00 pc=0x104364a60
encoding/binary.ReadUvarint({0x12f3fdc80, 0x14005b7e780})
	encoding/binary/varint.go:133 +0x80 fp=0x14004725d50 sp=0x14004725d20 pc=0x1042305d0
github.com/bep/godartsass.(*Transpiler).input(0x14007a88200)
	github.com/bep/godartsass@v0.16.0/transpiler.go:264 +0x6c fp=0x14004725fb0 sp=0x14004725d50 pc=0x104362dbc
github.com/bep/godartsass.Start.func1()
	github.com/bep/godartsass@v0.16.0/transpiler.go:70 +0x28 fp=0x14004725fd0 sp=0x14004725fb0 pc=0x104361e68
runtime.goexit()
	runtime/asm_arm64.s:1172 +0x4 fp=0x14004725fd0 sp=0x14004725fd0 pc=0x104131654
created by github.com/bep/godartsass.Start
	github.com/bep/godartsass@v0.16.0/transpiler.go:70 +0x2e0

goroutine 1311 [runnable]:
github.com/gohugoio/hugo/livereload.Initialize.func1()
	github.com/gohugoio/hugo/livereload/livereload.go:108 fp=0x140091d47d0 sp=0x140091d47d0 pc=0x1055ea2c0
runtime.goexit()
	runtime/asm_arm64.s:1172 +0x4 fp=0x140091d47d0 sp=0x140091d47d0 pc=0x104131654
created by github.com/gohugoio/hugo/livereload.Initialize
	github.com/gohugoio/hugo/livereload/livereload.go:108 +0x40

goroutine 1308 [syscall, 1 minutes]:
syscall.syscall6(0x14008f26400?, 0x106328a60?, 0x140091d5478?, 0x1040d9be4?, 0x140091d5498?, 0x104e515f4?, 0x14007077dd8?)
	runtime/sys_darwin.go:45 +0x64 fp=0x140091d53f0 sp=0x140091d5330 pc=0x10412db84
golang.org/x/sys/unix.kevent(0x14008abc480?, 0x14009297110?, 0x140091d54e8?, 0x1041162b4?, 0x140091d54b8?, 0x1040d4ea4?)
	golang.org/x/sys@v0.2.0/unix/zsyscall_darwin_arm64.go:276 +0x58 fp=0x140091d5460 sp=0x140091d53f0 pc=0x104296f88
golang.org/x/sys/unix.Kevent(0x140091d55d8?, {0x0?, 0x10795cf18?, 0x10?}, {0x140091d5660?, 0x2909468e70?, 0x5548?}, 0x12f7a0ca0?)
	golang.org/x/sys@v0.2.0/unix/syscall_bsd.go:399 +0x40 fp=0x140091d54a0 sp=0x140091d5460 pc=0x104296db0
github.com/fsnotify/fsnotify.(*Watcher).read(0x140091d5548?, {0x140091d5660?, 0x14008d01b00?, 0xa})
	github.com/fsnotify/fsnotify@v1.6.0/backend_kqueue.go:702 +0x48 fp=0x140091d5500 sp=0x140091d54a0 pc=0x104d83d88
github.com/fsnotify/fsnotify.(*Watcher).readEvents(0x14006f1b730)
	github.com/fsnotify/fsnotify@v1.6.0/backend_kqueue.go:487 +0x90 fp=0x140091d57b0 sp=0x140091d5500 pc=0x104d82b20
github.com/fsnotify/fsnotify.NewWatcher.func1()
	github.com/fsnotify/fsnotify@v1.6.0/backend_kqueue.go:155 +0x28 fp=0x140091d57d0 sp=0x140091d57b0 pc=0x104d81228
runtime.goexit()
	runtime/asm_arm64.s:1172 +0x4 fp=0x140091d57d0 sp=0x140091d57d0 pc=0x104131654
created by github.com/fsnotify/fsnotify.NewWatcher
	github.com/fsnotify/fsnotify@v1.6.0/backend_kqueue.go:155 +0x1f8

[cmahnke]

Just to give you an idea:

$ find content -type f |wc -l
   32454

and

$ hugo version
hugo v0.108.0+extended darwin/arm64 BuildDate=unknown

[bep]

Why do you think this problem is related to the file watcher?

[cmahnke]

Actually I don't have a valid clue. But as you might remember I was quite busy developing Hugo sites beginning 2021 and back then there have been such problems quite early (in terms of tree growth).

But maybe it's just a hugo serve problem...

Can you give me a hint how to narrow it down?

The error itself seems to be a timeout:

goroutine 52 [IO wait, 5 minutes]

There isn't a building problem since (output croped):

$ hugo
[...]

                   |  DE    
-------------------+--------
  Pages            |   782  
  Paginator pages  |     2  
  Non-page files   | 31976  
  Static files     |   881  
  Processed images |   365  
  Aliases          |   355  
  Sitemaps         |     1  
  Cleaned          |     0  

Total in 11680 ms

[bep]

What you do experience if you do:

hugo server --watch=false

Or maybe also:

hugo server --poll

[cmahnke]

Thanks, the first ( which is hugo server --watch=false) already did it (well, at least starting hugo):

                   |  DE    
-------------------+--------
  Pages            |   782  
  Paginator pages  |     2  
  Non-page files   | 31976  
  Static files     |   881  
  Processed images |   365  
  Aliases          |   355  
  Sitemaps         |     1  
  Cleaned          |     0  

Built in 7396 ms
Environment: "development"
Serving pages from memory
Running in Fast Render Mode. For full rebuilds on change: hugo server --disableFastRender
Web Server is available at http://localhost:1313/ (bind address 127.0.0.1)
Press Ctrl+C to stop

But now the watcher is gone. But since building ist quite fast, this will do for further development.

Do you need any more information, in case you want to fix this instead of make the watcher configurable?

[bep]

What does

hugo server --poll

Do?

There isn't a building problem since (output croped):

There are more differences between hugo and hugo server than the watching, one major thing is that we (by default) writes everything to memory in server mode. There are flags to disable that.

Do you need any more information, in case you want to fix this instead of make the watcher configurable?

We have millions of options already. We're not adding more without understanding that it's needed.

[cmahnke]

What does

hugo server --poll

Do?

$ hugo server --poll 1000ms

goroutine 95410 [select, 2 minutes]:
runtime.gopark(0x1402af9ff18?, 0x3?, 0x78?, 0xfe?, 0x1402af9feba?)
	runtime/proc.go:363 +0xe4 fp=0x1402af9fd40 sp=0x1402af9fd20 pc=0x1024541e4
runtime.selectgo(0x1402af9ff18, 0x1402af9feb4, 0x1402af9ffa8?, 0x0, 0x140b6a79440?, 0x1)
	runtime/select.go:328 +0x688 fp=0x1402af9fe60 sp=0x1402af9fd40 pc=0x102464af8
github.com/gohugoio/hugo/livereload.(*hub).run(0x10565c280)
	github.com/gohugoio/hugo/livereload/hub.go:39 +0x94 fp=0x1402af9ffb0 sp=0x1402af9fe60 pc=0x10393dd64
github.com/gohugoio/hugo/livereload.Initialize.func1()
	github.com/gohugoio/hugo/livereload/livereload.go:108 +0x28 fp=0x1402af9ffd0 sp=0x1402af9ffb0 pc=0x10393e2e8
runtime.goexit()
	runtime/asm_arm64.s:1172 +0x4 fp=0x1402af9ffd0 sp=0x1402af9ffd0 pc=0x102485654
created by github.com/gohugoio/hugo/livereload.Initialize
	github.com/gohugoio/hugo/livereload/livereload.go:108 +0x40

goroutine 95411 [select, 2 minutes, locked to thread]:
runtime.gopark(0x1402adb3fa0?, 0x2?, 0x88?, 0x3e?, 0x1402adb3f9c?)
	runtime/proc.go:363 +0xe4 fp=0x1402adb3e30 sp=0x1402adb3e10 pc=0x1024541e4
runtime.selectgo(0x1402adb3fa0, 0x1402adb3f98, 0x0?, 0x0, 0x0?, 0x1)
	runtime/select.go:328 +0x688 fp=0x1402adb3f50 sp=0x1402adb3e30 pc=0x102464af8
runtime.ensureSigM.func1()
	runtime/signal_unix.go:991 +0x190 fp=0x1402adb3fd0 sp=0x1402adb3f50 pc=0x102468e10
runtime.goexit()
	runtime/asm_arm64.s:1172 +0x4 fp=0x1402adb3fd0 sp=0x1402adb3fd0 pc=0x102485654
created by runtime.ensureSigM
	runtime/signal_unix.go:974 +0xf4

goroutine 95412 [syscall, 2 minutes]:
runtime.sigNoteSleep(0x0)
	runtime/os_darwin.go:123 +0x20 fp=0x1402b1d5790 sp=0x1402b1d5750 pc=0x10244e8a0
os/signal.signal_recv()
	runtime/sigqueue.go:149 +0x2c fp=0x1402b1d57b0 sp=0x1402b1d5790 pc=0x10248131c
os/signal.loop()
	os/signal/signal_unix.go:23 +0x1c fp=0x1402b1d57d0 sp=0x1402b1d57b0 pc=0x103933dcc
runtime.goexit()
	runtime/asm_arm64.s:1172 +0x4 fp=0x1402b1d57d0 sp=0x1402b1d57d0 pc=0x102485654
created by os/signal.Notify.func1.1
	os/signal/signal.go:151 +0x2c

goroutine 95413 [IO wait, 2 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:363 +0xe4 fp=0x140d451bb60 sp=0x140d451bb40 pc=0x1024541e4
runtime.netpollblock(0x140d451bbf8?, 0x24bfcc4?, 0x1?)
	runtime/netpoll.go:526 +0x158 fp=0x140d451bba0 sp=0x140d451bb60 pc=0x10244d6c8
internal/poll.runtime_pollWait(0x12d69aea8, 0x72)
	runtime/netpoll.go:305 +0xa0 fp=0x140d451bbd0 sp=0x140d451bba0 pc=0x10247f150
internal/poll.(*pollDesc).wait(0x14000119c80?, 0x104525360?, 0x0)
	internal/poll/fd_poll_runtime.go:84 +0x28 fp=0x140d451bc00 sp=0x140d451bbd0 pc=0x1024bb6c8
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0x14000119c80)
	internal/poll/fd_unix.go:614 +0x1d0 fp=0x140d451bca0 sp=0x140d451bc00 pc=0x1024bfd30
net.(*netFD).accept(0x14000119c80)
	net/fd_unix.go:172 +0x28 fp=0x140d451bd60 sp=0x140d451bca0 pc=0x102530a78
net.(*TCPListener).accept(0x14000343a70)
	net/tcpsock_posix.go:142 +0x28 fp=0x140d451bda0 sp=0x140d451bd60 pc=0x102546c78
net.(*TCPListener).Accept(0x14000343a70)
	net/tcpsock.go:288 +0x2c fp=0x140d451bde0 sp=0x140d451bda0 pc=0x102545e8c
net/http.(*onceCloseListener).Accept(0x10468e6c0?)
	<autogenerated>:1 +0x30 fp=0x140d451be00 sp=0x140d451bde0 pc=0x10282a040
net/http.(*Server).Serve(0x1400e136f00, {0x10468d030, 0x14000343a70})
	net/http/server.go:3070 +0x30c fp=0x140d451bf30 sp=0x140d451be00 pc=0x10280696c
github.com/gohugoio/hugo/commands.(*commandeer).serve.func3()
	github.com/gohugoio/hugo/commands/server.go:617 +0x30 fp=0x140d451bf60 sp=0x140d451bf30 pc=0x1039631e0
golang.org/x/sync/errgroup.(*Group).Go.func1()
	golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 +0x5c fp=0x140d451bfd0 sp=0x140d451bf60 pc=0x1030d44ac
runtime.goexit()
	runtime/asm_arm64.s:1172 +0x4 fp=0x140d451bfd0 sp=0x140d451bfd0 pc=0x102485654
created by golang.org/x/sync/errgroup.(*Group).Go
	golang.org/x/sync@v0.1.0/errgroup/errgroup.go:72 +0xa4

r0      0x0
r1      0x0
r2      0x0
r3      0x0
r4      0x104241970
r5      0x16da6ec70
r6      0xa
r7      0x0
r8      0x3a7e26f536541563
r9      0x3a7e26f45bf2e563
r10     0x2
r11     0xfffffffd
r12     0x10000000000
r13     0x0
r14     0x0
r15     0x0
r16     0x148
r17     0x200b309a8
r18     0x0
r19     0x6
r20     0x16da6f000
r21     0x1807
r22     0x16da6f0e0
r23     0x0
r24     0xffffffffffffffff
r25     0x103e1dfa0
r26     0x16da6ec80
r27     0x28
r28     0x14000002b60
r29     0x16da6ec20
lr      0x1a061dcec
sp      0x16da6ec00
pc      0x1a05e7224
fault   0x1a05e7224

Fails, with quite some long massage, to long to attach. And it took a lot longer then the five minutes above.

$ hugo server --poll 1000ms &> poll-error.txt
$ du -chs poll-error.txt 
 96M	poll-error.txt
 96M	total

I can also try with longer intervals, like 2000ms, 5000ms. More doesn't really make sense to me since, that't getting in the time span of a regular rebuild.

There isn't a building problem since (output croped):

There are more differences between hugo and hugo server than the watching, one major thing is that we (by default) writes everything to memory in server mode. There are flags to disable that.

Yes, I know. That used to work about 1,5 years ago, but with smaller trees:

hugo server --disableFastRender --renderToDisk

But now it doesn't.

Do you need any more information, in case you want to fix this instead of make the watcher configurable?

We have millions of options already. We're not adding more without understanding that it's needed.

Certainly it's better to fix the root cause for this. But I'm not sure how well a fix would scale with let's say 100.000 files (just a made up but not unreasonable number).

---

Update

$ hugo server --poll 5000ms

Doesn't work either, but even after about 20 min there isn't an error message...
But it gave me a hint: Runtime and log length relate to the interval.

I've attached a compressed log of

hugo server --poll 100ms &> poll-error.txt

poll-error.txt.gz