Validate config file and provide user-friendly error messages

[jack-mil]

Currently, if the configuration file contains unknown keys, is invalid json,
or the data-type of a value is incorrect, a variety of runtime errors can occur throughout
the code.

The config file should be validated for correctness before attempted parsing and loading
of the values, and the user should be notified of any errors.

This effort should also help with bringing the config file and cli options closer together,
and providing a more centralized source of truth for the runtime configuration.

Maybe I need to look into how other projects handle lots of CLI args that need to override any
static config file at runtime (fallback).

[jack-mil]

@znarfm I don't have a set solution on how this can be done. I'll let you figure out what is best if you are still interested. I am open to introducing a dependecy if it makes the code cleaner or safer.
Should I hold off the v3.3 release for this feature? If it will likely be a big rework, or backwards incompatible, I'll just add it to v4.0.

Also no pressure if you don't want to. Just giving the option since you mentioned working on this.

[znarfm]

I have viewed how other Python CLI tools like yt-dlp handles their config files. Their config files include just the command line flags (one option per line). I think this would be good as we won't need to have code for verifying if the entries in the config file are valid or not, since they will be immediately passed into the parser argparse, then the parser will throw messages about which option is wrong or not just like running it in the terminal and manually inputting the custom flags/options.

However, this would obviously change how we write config files as we would shift from .json to just plain text file. So maybe a v4 release would be needed for this change.

Then, maybe we could add code to generate a template config file with comments or just put it in the project's readme, like this?

What are your thoughts on this?

[jack-mil]

Hi! That sounds like a good idea! I like relying on argparse (or potentially other cli framework, if you think it would be beneficial) to avoid duplicated logic. You will probably have to add some new flags that currently only exist as a config option (user-agent, at least).

I think a generated template config file with commented default flags will be important for discoverability.

Obviously this will be a major breaking change, but I am fine with that. It would be nice to have a warning message of some sort if old config is detected, to let users know they need to check GitHub releases for update notes.