More comprehensive docs about `--restartsOnError` option

[Mr0grog]

I’ve got some crawls I run regularly and that sometimes crash in the middle, so I’ve got them wrapped in code to restart using the latest state file after looking at the log output to determine whether restarting makes sense (mainly I just look for the log line that says “crawl status: interrupted”).

The --help docs mention a --restartsOnError option that I’m not currently using:

if set, assume will be restarted if interrupted, don't run post-crawl processes on interrupt [boolean] [default: false]

But that description seemed overly concise and left me with some questions (what doesn’t happen? What do I need to account for?). A brief search of the code also makes it clear this option has some non-obvious implications for general behavior: it changes the 17 exit code to 0 and causes some things to throw instead of exit more gracefully (which might also affect exit codes?).

It would be great if there was some more detail about what this option does and how to use it effectively, maybe as part of the docs on interrupting/restarting crawls. I’d be happy to take a stab at that if someone could fill me in on the details a bit more here.

[ikreymer]

Yeah, we should definitely expand the docs for this option. It means that any non-zero exit code will result in crawler getting restarted automatically.

The goal of this flag is to be able to support running as standalone Docker container and running in k8s with slightly different semantics.

It is used together with the Browsertrix app, which runs the crawler in K8s, where a non-zero response means the pod will be restarted. This has some implications for semantics, as the crawler assumes if it exits with non-zero code, it'll be restarted again, while exit code zero means it will not. This results in slightly different semantics than running on the command-line, for example:

• The fatal error code is set to 0, instead of 17. Fatal means the crawler should not restart, but in 'restart on exit mode', the only way not to restart is setting code to 0
• When gracefully interrupting, don't always run final post crawl actions, such as generating & uploading WACZ. The crawler expects to be restarted again and can continue.
• If browser is stuck/crashes, don't attempt to kill/restart browser, just exit with an error code as the crawler will be restarted.

[ikreymer]

You should set this flag if you intend to always restart the crawler if it exits with non-zero error code.