watchmedo shell-command: drop events during execution of command

[jenisys]

(original title): watchmedo shell-command --wait seems to fire more often than necessary

Version: watchdog 0.6.0

When I execute the watchmedo script with "--wait --command=..." seems to be triggered to often. The command seems to be executed for each detected event according to the generated command-output. This seems to unnecessary and inefficient. Normally, I would expect the following behavior

# -- PSEUDO-CODE:
loop forever:
    Wait until event occurs.
    Remove all pending events (from the scheduler queue).
    Execute the command (once for all removed/pending events)

REASON:
When the command is executed it should rebuild whatever and be responsible for all accumulated events.

The efficient behaviour could also be enabled by a new, additional command-line option.

NOTE:
If you do not use the "--wait" option you can still use the nice debug/diag feature:

watchmedo shell-command ... --command='echo "${watch_src_path}"'

to detect which filesystem events occurred.

[ryandesign]

Please don't change it; --wait works as I expect and need it to. Consider watching a directory for tarballs and untarring them as they arrive, and not wanting to hit the disk with too many simultaneous operations:

watchmedo shell-command --wait --pattern '*.tar' --command '[ "${watch_event_type}" == "created" ] && tar xf "${watch_src_path}"'

[jenisys]

@ryandesign
In my opinion you do not need the --wait option for your use case (but I didn't try it out).

The command-line help states which behavior the --wait option provides
(or should provide but the implemented behavior does not match the description).

$ watchmedo shell-command -h
...
  -w, --wait            wait for process to finish to avoid multiple
                        simultaneous instances

[tamland]

Can you explain again what you wanted or what the bug is? --wait means it will wait for the shell command aka. "the process" to finish before continuing. Otherwise it will be executed asynchronous and you may get multiple simultaneous instances of it.

[jenisys]

As described in the first message.
The the command is executed too often in my opinion.

EXAMPLE:
watchmedo waits for filesystem events with a timeout (before the command is triggered).
3 filesystem events occur.
Then the command is triggered (and basically serves all 3 events).
When the command is done, it will be triggered twice again.

[tamland]

Still don't get it. It's execute exactly once. Just tested by triggering a batch of events:

./watchmedo.py shell-command --command='sleep 5; echo "${watch_src_path}"' test
echoes all of them at once.

./watchmedo.py shell-command --wait --command='sleep 5; echo "${watch_src_path}"' test
echoes one by one in a 5 sec interval

As expected.

[chrisconley]

This is an issue for us as well. We're using watchdog to watch our source directory to trigger an automatic build of the software. When changing git branches, events are triggered for each file and directory change. Ideally, all of those events would trigger just one build.

What we'd like is to have the functionality of the --wait option, but ensure there can only be a maximum of one event on the queue.

[jonaldinger]

To accommodate both @ryandesign and @chrisconley perhaps --wait could remain unmodified, and one of the following additional options could be implemented:

1. --max-wait-queue : max number of items to add to the queue (would be 1 in @chrisconley 's case)
2. --event-delay : number of ms to delay before dispatching the event. Assuming a delay value of 100ms, Event A occurs. It waits 100ms. If Event B occurs within 100ms, Event A is killed and Event B waits 100ms. If no events occur within 100ms, Event B is dispatched. This would solve the issue of changing git branches (which currently adds an event to the queue for every file and directory modified... this can easily be hundreds of queued executions), or adding/removing files (which triggers both a 'file changed' and 'directory changed' event), while ensuring that the last event in a rapid series is the one that gets dispatched (i.e. when the last file in a git checkout is modified).

[CharString]

To workaround this I wrapped my testrunner command with a lockfile. From my tricks.yaml:

 shell_command: if [ -e 'trick-lock' ]; then exit 0; else touch trick-lock; django-admin.py test --noinput people; rm trick-lock; fi

[brainwarped]

Using --command 'fab test', always runs tests twice for me... using -w doesn't make a difference. Is this a problem with fabfiles?

[ferrouswheel]

Reading this issue's comments as a newcomer it seems people are talking about different functionality.

@jonaldinger's suggestions seem the most sensible and make watchmedo a lot more useful.

Even copying a file into a directory will trigger multiple events:

• 1 created event
• (file size divided by buffer size) modified events

My specific use case is that I want to uncompress an archive when it's been copied, but watchdog will spam the command for every event. I only would like to uncompress the archive when it's finished copying.

While I can work around it, @jonaldinger's --event-delay would make this use case trivial.

[timbertson]

+1, --event-delay would be great! Currently watchdog with --wait kicks off 6 builds in a row when I save one file in vim, which is particularly overwhelming when there's a build error.

I would have thought --event-delay implies --max-wait-queue=1 though, so I won't know that both flags would be necessary. If you're going to be dropping events at all, it doesn't seem like there's a use case for keeping multiple events that accumulate during a command's execution.

[nuket]

@gfxmonk I've added a --smoothing=<seconds> parameter to run after the triggering event is detected and ignore all of the events that gather in those <seconds>.

It does cause the command to run after <seconds> delay, but that's somewhat unavoidable.

The modifications are in Pull #231 or in nuket/watchdog@11c64d6.