Designing for layered configs

[epage]

This is inspired by #2759 in which there is a proposal for one approach for layering. Thought the actual layering best practices would best be split out in a discussion

[epage]

#2759's approach

Use case: I would like to merge ArgMatches into Config settings using the following rules:

1. ArgMatches have higher precedence over Config if they are explicitly set.
2. If config setting is present but ArgMatches has an argument with default value then config value is taken.
3. If config does not set the setting, then ArgMatches default value it taken.

[gibfahn]

From my experience with layered configs, you have some Args not in the Config and some Config not exposed as Args with each requiring defaulting of some sort.

You're probably right, but it feels like using serde to deserialise a config file into args, and then updating from std::env::args() would be quite neat. However it's worth noting this doesn't work with subcommands (at least with the derive API), as you end up with a SubCommand enum rather than a SubCommand struct, e.g. something like:

#[derive(Parser)]
struct Args {
    #[clap(subcommand)]
    cmd: Option<SubCommand>,
}

#[derive(Parser)]
pub enum SubCommand {
  // ...
}

but for a config file parse, you'd want something like:

#[derive(Deserialize)]
pub struct SubCommand {
  // ...
}

[epage]

Could you expand on why an enum vs a struct makes a difference?

[gibfahn]

Basically when you run a command like:

my-app subcommand-one --flag
my-app subcommand-two --other_flag

Each invocation only passes one subcommand, so you want enum SubCommand not struct SubCommand.

However in a config file, you would want to be able to set defaults for all subcommands, not just one, so the config file might look like this:

global:
  color: always

subcommand_one:
  flag: some-value

subcommand_two:
  other_flag: other-value

Which might deserialise to:

#[derive(Deserialize)]
struct Config {
  global: GlobalOpts,
  #[serde(flatten)]
  subcommand: SubCommand,
}

#[derive(Deserialize)]
struct SubCommand {
  // ...
}

So two bits that I ran into trying to make a config file work were:

1. The global options will contain a SubCommand enum (which can be hidden/ignored so not a big issue).
2. I had to manually create a subcommand struct from the subcommand enum I already used in clap.

Will see if I can push up a full worked example.

[epage]

Good point.

Seems like this could be handled by having all arguments for your subcommand in a separate struct. For the CLI, you then integrate them with an enum. For your config, you integrate them with a struct. This does require manually connecting the pieces though.

[gibfahn]

Yep, that's what I ended up doing, but duplicating the fields and docstrings seems painful, and like something a macro of some kind could do for you.

[epage]

At one point, I would use a

• trait ConfigSource with a method per field that returned an Option<RefType>.
• The Config and Args would both implement this.
• Config would have an update(other: &dyn ConfigSource) for merging
• Config would also have methods per field that return RefType, defaulting the value if it is None.

Example from cargo-release

[epage]

As I ended up with child-structs and things, the boiler plate for this became too much and I wasn't seeing value out of using the trait. I moved onto a concrete Config. See below.

[epage]

Recently, I've been using a struct Config

• Every field is Option<Owned>
• Config has a update(other: &Config)
• Config has a method per field that returns RefType, defaulting the value if it is None
• Args has a to_config

Example from cargo-release

[romange]

Agree with me that while this approach has most control over the layering logic, it's pretty verbose. Currently I also use clap-derived configs with Option<> only fields. The problem with this approach is that I have to fill in default values myself via 3rd layer (after arguments and config files), but then clap help message does not provide information about default values - which I think is suboptimal.

[epage]

Mentioned in another thread but for --help output with clap, #1695 could help.

[epage]

There is also the popular config crate which is designed around layered config

[epage]

There isn't clap integration atm

[epage]

The big reason why I've avoided this is it looks like it delays all error reporting until after merge, not letting you tell users which config source was the problem.

[romange]

Exactly, I used config crate for merging. I think using Option<> for everything is a hack that should be fixed.

struct ServerOpts {
    #[clap(short, long)]
    port: Option<u16>,

    #[clap(long)]
    bind: Option<IpAddr>,
    // Configuration toml file.
    #[clap(long, value_hint = ValueHint::FilePath)]
    config: Option<PathBuf>,
}

impl ServerOpts {
    fn default_config() -> Result<Config, ConfigError> {
        let mut settings = Config::new();

        settings
            .set_default("port", 8080)?
            .set_default("bind", "127.0.0.1")?;

        Ok(settings)
    }

    pub fn new() -> Result<Self, Report> {
        // Just to get a custom config path
        let app = Self::into_app();
        let matches = app.get_matches();

        let mut settings = Self::default_config()?;

        // Priority is: config -> environment -> runtime args.
        if let Ok(config) = matches.value_of_t::<PathBuf>("config") {
            settings.merge(File::from(config))?;
            debug!("file args {:?}", settings);
        }

        settings.merge(Environment::with_prefix("THANKFORCLAP"))?;

        // Merge runtime arguments into settings by converting them to toml first and then merging toml string.
        let runtime_args = <Self as FromArgMatches>::from_arg_matches(&matches)
            .expect("could not parse arguments");

        let toml = toml::to_string(&runtime_args).expect("could not convert to toml");
        settings.merge(File::from_str(&toml, FileFormat::Toml))?;

        let final_args = settings.try_into::<Self>()?;

        Ok(final_args)
    }
}

[epage]

Could you clarify what you find is hacky about Option, rather than me guessing (probably incorrectly) which part of that code sample you don't like?

[epage]

For --help output with clap, #1695 could help.

[pksunkara]

Maintainer's notes:

• Allow adding "external argv"s to be parsed alongside/before command line #748 alludes to a couple of different pieces:
  • A File parser driven by clap::App
  • Turn that config file into argv and merge it with existing argv
  • Or turn that config into an ArgMatches and allow merging ArgMatches

---

Note: There's a lot of previous discussion on #748. Let's continue this discussion there?

[epage]

I looked through that and it seemed to mostly be focused on one approach (external argv) while this is discussing the the broader problem.

[epage]

As with #2759, my concern with an config parser being driven by clap::App is there are three types of config

• Those sources from all sources
• Those sourced from CLI only
• Those sourced from command line only

Having an "easy route" that focuses on this will help encourage people to put everything in both the config and CLI when it shouldn't all be there. This also limits the type of config you can store.

I will say someone can implement this on top of clap and give it a try to see how well it works.

[epage]

The problem with merging argv is order can matter, whether with subcommands or with other less common cases like #2419. This becomes a solution that is severely limited for who it can help and could encourage people to not design their CLI like they should to get an easy path for config layering.

[epage]

clap3's Parser trait has something like ArgMatches merging. It only supports overwriting but not any kind of merging or conflict. It'd be interesting to see people give it a try to see how well it works.

From my own experience, whether the overwrite, merge, etc is usually more field dependency. Its hard to have a one-size-fits-all policy at the top level.

Also, until we have #2683, we'll be severely limited in what config can be stored inside of an ArgMatches.

[epage]

For completeness, I'm going to mention argfile support. The formats currently supported by the argfile crate do not support comments and you can only specify it on the command line, you can't load it yourself to extend argv. This does have the benefit of allowing the user to control where the args are inserted.

[epage]

Instead of a solution, this thread will be exploring requirements and their prioritization

Various requirements include:

• Merge from args, env, and undefined set of files
  • Some files will be on the command line, some will be implicitly discovered, sometimes a user will pass a --isolated flag to disable implicit discovery
  • The merge strategy is field specific (replace, append, conflict, etc)
  • Tell users what layer an error came from
  • Handling schema migration with layering is complicated
• Some config is CLI, env, or file specific while some is shared
• We want defaults to show up in --help
• Users tend to want a --dump-config for both debugging and as a jumping off point
• Config file format is application-specific
  • TOML is frequently used in Rust. Some will want YAML or a json variant. For git users, you will want to integrate in with .gitconfig and git's env variables.

What additional requirements are there? Is there a viable subset that covers enough people for a shared solution? What building blocks can be made to help with all of this?

[epage]

Hidden: yes, not present in help.

What is the use case for hidden? Are you thinking in terms of renaming flags and keeping compatibility?

[nyurik]

Hidden: yes, not present in help.

What is the use case for hidden? Are you thinking in terms of renaming flags and keeping compatibility?

@epage I would think a case where config file / env var has something not settable via CLI args would be hidden, but would still make it into the primary args struct.

[epage]

That sounds like a specific solution to this requirement:

Some config is CLI, env, or file specific while some is shared

[nyurik]

correct. Anything hidden is basically something unused/unsettable by the CLI. I guess there could be some undocumented/unpublished params too that should work but shouldn't show up in the help, but they wouldn't be hidden I think (i.e. for backwards compat support?). Still hacky though.

[nyurik]

Another aspect to consider is extendability/modularity: some applications may have git-like structure, where depending on the top-level command, there are different sub-options. I see two general use cases:

• global config files work the same regardless of the top level command, and the app chooses which part of the config to use
• top-level commands can have different config files, possibly required for the command to proceed, e.g. app copy <src.config> <dst.config>

[jaskij]

Problems with early error reporting:

1. If an option is required, the program often doesn't care where it comes from - so it is actually impossible to check for missing options until after the merge.
2. If the merge strategy is replace, does it matter that an invalid option was overwritten?

[epage]

If an option is required, the program often doesn't care where it comes from - so it is actually impossible to check for missing options until after the merge.

Yes, if you have required config, you can't check that until the end. However, in my layering so far, the only required data is taken on the command line and is not layered with anything else. It'd be interesting to explore examples otherwise.

If the merge strategy is replace, does it matter that an invalid option was overwritten?

It represents a latent bug, unless there is an intentional reason you are overlaying on top of that bug. I can invent situations for that but unsure how likely they are to occur.

[nyurik]

Apparently there are a few more options, but neither support Derive:

• viperus - a bit dated (last release 2years ago, last modified half a year ago). Supports Clap. Lots of file formats - JSON, TOML, YAML, dotenv file ,java properties config files. Supports auto-reloading (config watching) (2k downloads)
  • cannot use Clap3 until Iterating over ArgMatches #1206 is fixed.
• clap_conf - popular (16k downloads). Supports ENV, Clap, Toml.
• twelf supports layering and Clap Derive, a bit buggy, under active development.

[virtualritz]

Also the relatively recent twelf crate has layering & clap support (incl. Derive – but it needs work).

[nyurik]

thx @virtualritz, added to the list. BTW, I suspect the Args::command().ignore_errors(true).get_matches() trick is what is missing in the other ones to support Derive -- i'll hack on it a bit to see if i can make clap_conf or viperus work with it.

[epage]

Looking for feedback on the upcoming clap API design, in particular from a layered config perspective but also if people have thoughts in general

See #3792

[jaskij]

I basically don't use the builder API, so no help from me here, sadly.

One thing I've been looking forward to is validators which are run after parsing - so I can, for example, check if a number is in the valid range. Preferably without parsing the string twice. Right now I'm working around this by using a separate function which is called after clap.

[epage]

Depending on what you mean by "after parsing", the new API allows

    #[clap(value_parser = clap::value_parser!(u16).range(1..))]
    port: u16

• The string will only be parsed once
• For something outside of 1..=u16::MAX, the user will be given an error saying it needs to be within that range.

Granted, if the range is dependent on runtime factors, like a config file, this won't work. At least we will only parse the string once.

[jaskij]

By "after parsing" I mean that the validator gets passed a typed value, not a string.

Range on numbers is nice, works for most if not all of my needs along the other existing validators. But would be nice to be able to pass a function too.

[luketpeterson]

After a 4-day diversion into a config system that involved becoming very familiar with the code for the twelf crate (some might call it a yak-shave), I have arrived at a fairly clear vision for the design of a a config layering feature that I personally would like to use.

First a few observations:

• Errors from the parsing of the ArgMatches (get_matches) ideally wouldn’t be thrown all the way to the user unless those fields can’t be assigned values from one of the other layers.

Optional arguments aren’t enough to accomplish this however, because the user still needs to see the error if no layer was ultimately able to supply the argument value.

• I don’t recommend the way twelf has a dependency on a large number of format crates (e.g. toml, json, etc. etc.) because it leads to a cargo features mess, not to mention the lack of clean support for formats not included. I think a better option is to just allow a layer to invoke a generic closure that returns an object that implements Deserialize (or DeserializeOwned).

Most serde format crates have one or two line samples that provide an interface like this.

• I don’t know whether it’s better to force the user to implement Deserialize on the same struct they implement the Clap traits on, or whether it’s better to auto-generate a parallel struct that is the Deserialize cousin of the Clap struct from which the Command is created.

There are pros and cons to each. On the one hand, having to re-specify so many field attributes for Clap and Serde is a pain, and there is no struct if the Command builder API was used instead. But on the other hand, allowing the user to derive Deserialize independently gives a lot more control / flexibility. It sounds like pain either way.

A third solution may be to side-step this concern altogether by providing a way to Deserialize directly to ArgMatches.

So, for an API, I think an elegant design might be:

1. A way to deserialize directly to an ArgMatches.
2. A way to merge multiple ArgMatches together.
3. A way to parse the args (get_matches) without throwing an error to the user right away
4. A way to validate a final merged ArgMatches against a command, and throw errors at that point.

Finally, it could all be wrapped up in a convenience API that presents a nice “Layer” API.

I am offering to prototype this if it sounds like a good design / direction to everyone. Although It will probably take me a few weeks / months to get back to this as I already lost too much time to config management and I’m behind on other work.

[epage]

I've got some further questions

• How are arg-only or config-only fields handled?
• Shouldn't validation errors from a layer be reported more eagerly? It seems like required and conflicts would be the errors that should be delayed
• Would the ArgMatches deserializer just use the type hints for storing of types?
• When merging, how do we know when we should overwrite vs merge?
  • Note that ArgMatches doesn't really have knowledge of whats inside the AnyValue and so we can't do any processing on it once its stored (which also means we can't serialize)
• How would we handle defaults?

[luketpeterson]

How are arg-only or config-only fields handled?

I was imagining ArgMatches could be extended to include a source field for each arg, indicating which layer the value came from. So the merge and validation logic could ensure that only values from supported layers are merged, and appropriate errors are thrown if the values came from the wrong layers.

Shouldn't validation errors from a layer be reported more eagerly? It seems like required and conflicts would be the errors that should be delayed

Other errors certainly could be thrown more eagerly. As you say, the required and conflict errors are the main ones to postpone, but if we have a mechanism to postpone error throwing to a later call then we could move all validation there. It could be implemented either way.

Would the ArgMatches deserializer just use the type hints for storing of types?

Yes. That is what I was thinking. There may be some devils in the details that I'll discover when actually implementing it.

When merging, how do we know when we should overwrite vs merge?

I don't have a clear idea for a "deep-merge". For atomic (irreducible) args / configs, which I believe is most of them, it's just a matter of specifying which item takes precedence. I could see merging lists, etc. being a useful enhancement, but I need to understand the use case better to propose how to control the behavior. So IMO deep-merge could be punted, pending more user feedback.

How would we handle defaults?

I was imagining defaults would be treated like another layer by the logic. Maybe an implicit layer since Default would probably be the layer of last resort, so it doesn't dominate any other arg / cfg sources.

[gibfahn]

I think that if update_from_arg_matches() used .contains_id() && .value_source() != ValueSource::DefaultValue instead of just .contains_id() to check whether the value is present, you might be able to do something like this:

use clap::{CommandFactory, FromArgMatches, Parser};
use serde::Deserialize;

#[derive(Debug, Parser, Deserialize)]
struct Opts {
    #[clap(long)]
    a: Option<String>,

    #[serde(default)]
    #[clap(long, default_value = "def")]
    b: String,

    #[clap(long)]
    c: bool,
}

const CONFIG: &str = "
a: one
b: two
c: false
";

fn main() {
    let matches = <Opts as CommandFactory>::command().get_matches();
    let mut opts: Opts = serde_yaml::from_str(CONFIG).unwrap();
    opts.update_from_arg_matches(&matches).unwrap();
    dbg!(&opts);
}

Basically creating an ArgMatches from the command-line args, parsing the config file into the Opts struct, and then updating it with the values from the command line if set.

You'd probably have to keep the serde default value and the clap default value in sync though.

[sofiedotcafe]

Thank you, this was userful!

[gibfahn]

There is also the clap_serde_derive crate, which seems to offer a nice solution by creating a struct for your config that has everything wrapped in an Option<>.

I haven't used it or looked at the source due to the AGPL license, but if it handles subcommands properly it seems like a neat solution.

[cavivie]

I want to express such a use case:

use std::path::PathBuf;

use clap::Parser;
use figment::{
    providers::{Env, Format, Serialized, Json},
    Figment,
};
use serde::{Deserialize, Serialize};

fn conf_default_name() -> String {
    "world".to_string()
}

#[derive(Parser, Debug, Serialize, Deserialize)]
struct Config {
    /// A file to override default config.
    #[clap(short, long, value_parser)]
    config: Option<PathBuf>,

    /// Name of the person to greet.
    #[clap(short, long, value_parser, default_value_t = conf_default_name())]
    #[serde(default = "conf_default_name")]
    name: String,
}

fn main() {
    // Create config builder.
    let mut config: Figment = Figment::new();

    // Parse CLI arguments.
    let cli_args = Config::parse();

    // Override with config file.
    if let Some(config_path) = cli_args.config.clone() {
        config = config.merge(Json::file(config_path));
    }

    // Override with `APP_`-prefixed environment variables and CLI args.
    let config: Config = config
        .merge(Env::prefixed("APP_"))
        .merge(Serialized::defaults(cli_args))
        .extract()
        .unwrap();

    println!("{:?}", config);
}

In this case, I expect the override priority to be: cli arg value > config file > default value, and the application can be started from ffi and other non-cli methods.
Then I set default values ​​for both clap and serde. The problem here is that the default value of serde is always overwritten by clap, and the args specified in the config file are still affected by the default value of clap.
If name = "hello" is configured in config.json, it is expected to display hello, but it actually displays world.

cargo run -- --config config.json

Some people may say that setting the name field to Option and not providing a default value for clap can solve this problem. But I would say that this does not fundamentally solve the problem and is not ergonomic. Imagine that you expect clap to output the default value explanation for you when there is no configuration file (that is, the help information of clap should display the default value of name instead of the stupid None).
The special feature of this use case is that the name field is always not None, and the value is either obtained from the cli arg, from the configuration file, or the default value, but the cli help information can display the default value of the argument. I can't think of how to do this well at present.

[0xForerunner]

@luketpeterson did you ever start a prototype for this? If so I'd love to take a look.

[luketpeterson]

@luketpeterson did you ever start a prototype for this? If so I'd love to take a look.

Started, yes. But barely. Sadly I just didn't have the free days to spend getting it into any kind of shape I can share (yet). I noticed you have been working in the config crate; do you have any thoughts about how interop with it could work?

Somehow I didn't find the config crate last year when I was working heavily on this. I tried to use twelf but ultimately didn't like the design so I rolled my own. But it seems config and clap are the "blessed" (or at least dominant) solutions for their respective problem areas so it makes sense they should leverage the same description structures where it makes sense.

[benaryorg]

For the first time in forever (literally) I write something that needs both command line input as well as config and environment and I end up here.

A few thoughts on potential design;

• clap already has multiple "input sources" internally, since it is able to fall back to environment variables when something is not present on the CLI
• clap needs to integrate into this, otherwise we run into the same issues as virtually all solutions proposed in this thread, for example something replaces fields with Option versions behind the scenes and causes e.g. --help to print everything as optional
• we all choose clap for our CLI parsing because it has nice error reporting, sophisticated parsing, etc., anything working between clap and the data structure necessarily affects those aspects, so the solution needs to be either between the CLI and clap (like generating "fake" arguments based on a config file) which sounds very error prone given varargs and friends, or needs to be actively queried by clap
• there is (or used to be?, been a while) a means of handing clap a custom argument parser for custom data types; this uses the same "interface" as passing the environment variables for clap to read from however it conflicts with the intended use of those validators

Adding everything together I think the solution would look something like this (using owned datatypes and leaving out a lot of necessary generic-related stuff):

#[derive(Parser, Debug)]
#[command(loader_t = foo)] // this is inherited by all values
struct Config {
	#[arg(short, long, default_value = "::1", loader_key = "listen.address")] // specifying an explicit config key
	address: IpAddr,
	#[arg(short, long, default_value_t = 8080)] // uses its internal Id as key
	port: u16,
	#[arg(short, long, loader = secret_loader)] // 
	secret: String,
}

// this should be allowed to be associated with a struct of sorts to allow for caching keys instead of loading the file each time
// T here is the type, meaning `foo::<IpAddr>("listen.address".to_string(), None)` will be called if I don't provide an address myself
// The default value for clap is therefore invisible to the loader in this bit of design, which seems reasonable if the loader is instructed to just hand out a value if it has one (as with config files), but I can imagine there to be use-cases that need the loader to provide a value only when no default is provided (but I can't think of any)
// Then again the default value could also be a loader of sorts.
fn foo<T>(key: String, provided: Option<T>) -> Option<T> {
	// the provided value is what clap parsed from the CLI, this allows precedence higher than CLI for some loaders
	// others, like this one can short circuit
	if provided.is_some() { return provided; }
	let config = serde::from_file("my_config.toml");
	if let Ok(Some(value)) = config.get(key) {
		// this will make clap use this value instead of the (optional) provided one
		return Some(value);
		// note that we used `loader_t` (similar to `default_value_t`) above, which means this uses T directly
		// there could be a `loader` one which returns string-like types which are then still parsed by clap, with error reporting from clap
		// for that there should be some sort of additional context which the loader can provide, such that users get errors like:
		//  value "foo" from loader "TOML parser, file $f, line $n" is not valid for "port" of type u16
		// (just more user friendly)
		// (it could also integrate with Result to allow the loader to return something like a syntax error in a file via clap)
	}

	// when returning None, this will make clap panic if the value is required, same as with the CLI
	None
}

// fn secret_loader() could query additional configuration files (such as encrypted secrets or a secret store a la Vault)

Note that something about hierarchical structures is missing here, maybe something like loader_namespace on the command level which then moves all the keys into that section by default (potentially recursively) such that loading a key from the JSON {"foo": {"bar": "baz"}} works.
In absence of that the loader can still interpret nested keys by means of dot separation or such, but getting a &[String] as the key would probably be better.

With this kind of thing we can also have composite loaders which implement precedence like the twelf layers system.

This would pull all of the complexity of parsing files (or making network connections for something like cloud-init) out of clap.
On the other hand the environment variable loading could be refactored to match this kind of thing, basicall a loader which falls back to using the env key if clap did not find a value, which could serve as a template/scaffolding/example for other loaders.

This is just me, someone who writes Rust every other year or so for a project (or AoC), suggesting an API that may or may not even work when generics and lifetimes get involved.
However given that this discussion has been stale for a while (unless I, as usual, am too scatterbrained to notice some obvious activity).
Cunningham's Law being what it is, the responses to this may come up with a good design though, or correct me on something I missed.

FWIW, right now clap sort of has a loader that goes like this:

fn loader(...) {
	if let Some(v) = provided { return v; }
	if let Some(v) = getenv(format!("{}{}", env_prefix, key.upcase()) { return v; }
	if let Some(v) = default_value { return v; }
	None
}

And I think factoring that into a separate bit of code which can be replaced (or augmented, since one loader could just chain another like warp filters do) by another crate (on user request) would be ideal.
Some configuration context may need to be given (such as requiring the loader to not be a function rather than a struct with trait impl (with blanket impl for matching functions)), but I wouldn't know how to accomplish that in the context of clap's derive system, so it may need to be provided as part of the Config::parse(); which could then instead be a HashMap to allow multiple named loaders to be provided, such as the different secret loader in the example.

Aight, that's all I have on my mind right now, hopefully this pushes anything or anyone in a good direction ^^

[aleferna12]

My use case for this feature is that I have a very large Parameters struct that takes many many parameters to configure a model. Because the struct is so large (and there are no obvious defaults), I definitely need support for a configuration file. However only having a configuration file without clap would mean that there is no obvious place to put documentation for each parameter (I currently have it in the conf file itself). Integrating both would make for a much more ergonomic user experience.

[aschey]

Some config libraries that use a partial/layered approach can support this with a single struct that derives both the config library's macros and clap's macros without introducing any coupling at the library level. I like schematic and confique, so I've been working on getting some necessary changes upstreamed for this in both.

The basic strategy is to derive your configuration plus clap::Args on a dedicated struct, and then merge it into the main clap::Parser using #[command(flatten)]. It ends up being a bit attribute-heavy, but it does solve the problem of code/documentation duplication.

This works with schematic on the latest published version.

use clap::{Args, Parser};
use schematic::Config;

#[derive(Config, Debug)]
#[config(partial(derive(Args)))]
pub struct AppConfig {
    /// app port
    #[setting(default = 3000, partial(arg(long, env = "APP_PORT")))]
    port: usize,
    /// allowed hosts
    #[setting(
        default = vec!["localhost".to_string()],
        partial(arg(long, env = "APP_HOSTS")))]
    allowed_hosts: Vec<String>,
}

#[derive(Parser, Debug)]
struct Cli {
    // Note: PartialAppConfig is a struct generated by schematic that has all types wrapped in Option
    // The attributes wrapped in partial() are applied to this struct
    #[command(flatten)]
    app_config: PartialAppConfig,
    ...
}

The values in PartialAppConfig are all optional and can be used to override the values in the config file, using the same schema.

Here's the same approach using confique (won't work on the published version until this commit is released).

use clap::{Args, Parser};
use confique::Config;

#[derive(Config, Debug)]
#[config(partial_attr(derive(Args, Debug)))]
pub struct AppConfig {
    /// app port
    #[config(default = 3000, partial_attr(arg(long)))]
    port: usize,
    /// allowed hosts
    #[config(default = ["localhost"], partial_attr(arg(long)))]
    allowed_hosts: Vec<String>,
}

#[derive(Parser, Debug)]
struct Cli {
    #[command(flatten)]
    config: PartialAppConfig,
    ...
}

[aschey]

Both libraries would support that, I think. You can load from multiple files and they will get merged together at the end:

// confique
Conf::builder()
    .file("./file1.toml")
    .file("./file2.toml");

// schematic
ConfigLoader::new()
    .file("./file1.toml")
    .file("./file2.toml");

[epage]

If I'm understanding your point, then thats just layering and independent of the schema/definition.

[aschey]

Sorry, I'm probably missing something. If you have two different schemas, wouldn't you need to serialize them to separate structs? Schematic supports nested configs where you can load two different schemas independently and merge them together at the end.

[epage]

That requires pulling the parts of the schema together into one schema to then do a single read. What if the entire schema isn't known? You could create a bunch of one-off parent structs to then nest your one part of your config into or you can do deserialize a subset of the data.

Take cargo's [term] table . Its defined in a struct. Instead of having that struct be term: TermConfig into some parent Config struct, we instead can do gctx.get::<TermConfig>("term")

[aschey]

I see, so it's lazy loading parts of the config where it needs to do some extra steps like ignoring deserialization errors or some more complex post-processing. Makes sense.

[laduke]

Thanks for doing those PRs @aschey. Kind of surprised there isn't one blessed unified thing for this yet.

I'm testing out the schematic one and I noticed one thing

The default value doesn't get passed to clap. I think? I didn't look at the code, just testing experimentially.

The reason I'd want that is clap adds it to the automatic help. I had to add "3000" twice to get it to print.

    #[setting(default = 3000, partial(arg(long, default_value = "3000")), env = "SOME_PORT")]
    port: usize

Options:
      --port <PORT>  Some Port [default: 3000]

    #[setting(default = 3000, partial(arg(long)), env = "SOME_PORT")]
    port: usize

Options:
      --port <PORT>  Some Port

I imagine this being hard to figure out.

[aschey]

If you set a default for the CLI argument, then won't that always override the value coming from your config file? That value will get passed to clap if it's set in the arg() section.

[laduke]

haha yeah sorry. Too many factors to think about at once. I got too focused on the help output for a minute

[rejuvenile]

I wanted an approach which:

• supported all native clap features, including defaults
• allowed a layered config with figment, where the command line is the final override
• supported a distributed configuration, where each module can have it's own struct CONFIG without requiring a "grand unified struct". This is convenient in a monorepo, where there can be a lot of libraries crossed with a lot binaries (including special unit test runners).

I hacked something up which appears to work. It uses linkme to allow creation of a config struct in every module. A macro helps with registration. The macro also defines a trait which uses the concrete type of the struct for easy deserialization. main() then calls parse_config(), which:

• creates a clap Command
• adds the arguments from all the linked config structs (this will also detect duplicate argument names, but only at runtime)
• gets all the matches (thus, allowing clap default error handling to be used)
• removes all the default values from a clone of the matches, thus leaving only the overrides
• then, for every config struct, we:
  • call figment.merge() with <config_struct>::from_arg_matches() using the original set of matches. This populates the defaults (well, really, everything) we got from clap.
  • add any other figment layers (TOML, environment variables, etc.)
  • deserialize this into a local copy of the config struct
  • update that config struct with the CLI overrides/non-defaults (via update_from_arg_matches())
  • initialize the local module's OnceLock with a copy of the final struct

/// A registrar for a module's configuration.
pub trait ConfigRegistrar {
    /// The name of the module, used as a key in config files and for namespacing.
    fn name(&self) -> &'static str;
    /// A function that adds this module's command-line arguments to the main `clap` command.
    fn add_args(&self, cmd: Command) -> Command;
    /// A function to extract this module's config from the final Figment
    /// and initialize the module's static config.
    fn initalize_args(
        &self, matches_without_defaults: &ArgMatches, matches_without_defaults: &ArgMatches,
    );
}

#[distributed_slice]
pub static CONFIG_REGISTRARS: [&'static (dyn ConfigRegistrar + Sync)];

/// Parses the configuration for the application based on the command-line arguments.
///
/// # Panics
///
/// - Panics if any operation related to clearing default values from `matches_without_defaults`
///   fails unexpectedly, such as attempting to clear a non-existent argument.
pub fn parse_config() {
    let mut command = Command::new(constants::BINARY_NAME)
        .ignore_errors(false) // TODO: parameterize this
        .about(constants::ABOUT)
        .version(constants::BINARY_VERSION_CORE)
        .long_version(constants::VERSION_STRING_LONG.as_str());
    for registrar in CONFIG_REGISTRARS {
        command = registrar.add_args(command);
    }
    let matches = command.get_matches();
    let mut matches_without_defaults = matches.clone();

    for arg_id in matches.ids() {
        let key = arg_id.as_str();
        if matches.value_source(key).unwrap() == ValueSource::DefaultValue {
            matches_without_defaults.try_clear_id(key).unwrap();
        }
    }

    for registrar in CONFIG_REGISTRARS {
        registrar.initalize_args(&matches, &matches_without_defaults);
    }
}

pub mod prelude {
    pub use clap::{ArgMatches, Args, Command, FromArgMatches};
    pub use figment::{
        Figment,
        providers::{Env, Format, Serialized, Toml},
    };
    pub use serde::{Deserialize, Serialize};
}

/// Implements `ConfigRegistrar` for a given struct and registers it
/// in the `CONFIG_REGISTRARS` distributed slice.
#[macro_export]
macro_rules! register_config {
    ($struct_type:ty, $config_name:literal, $config_static_name:ident) => {
        struct ConfigRegistrarImpl;

        impl $crate::config::ConfigRegistrar for ConfigRegistrarImpl {
            fn name(&self) -> &'static str {
                $config_name
            }

            fn add_args(&self, app: $crate::prelude::Command) -> $crate::prelude::Command {
                <$struct_type as $crate::prelude::Args>::augment_args(app)
            }

            fn initalize_args(
                &self, matches: &$crate::prelude::ArgMatches,
                matches_without_defaults: &$crate::prelude::ArgMatches,
            ) {
                use $crate::prelude::FromArgMatches;

                let figment = $crate::prelude::Figment::new()
                    .merge($crate::prelude::Serialized::defaults(
                        <$struct_type>::from_arg_matches(matches).unwrap(),
                    ))
                    .merge($crate::config::config_toml())
                    .merge($crate::config::config_env());
                let mut config: $struct_type = figment.extract().unwrap();
                config.update_from_arg_matches(matches_without_defaults).unwrap();
                $config_static_name.set(config).unwrap();
            }
        }

        static $config_static_name: std::sync::OnceLock<$struct_type> = std::sync::OnceLock::new();

        #[linkme::distributed_slice($crate::config::CONFIG_REGISTRARS)]
        static CONFIG_REGISTRAR: &(dyn $crate::config::ConfigRegistrar + Sync) =
            &ConfigRegistrarImpl;
    };
}

#[cfg(test)]
mod tests {
    use clap::Args;
    use serde::{Deserialize, Serialize};
    use tracing_test::traced_test;

    use super::*;
    use crate::register_config;

    #[derive(Serialize, Deserialize, Debug, Args)]
    pub struct Config {
        #[arg(
            long = "message",
            env = "APP_MODULE_A_MESSAGE",
            default_value = "foo",
            env = "FOO_ENV"
        )]
        pub message: String,

        #[arg(long = "retries", default_value_t = 3)]
        pub retries: u32,
    }

    register_config!(Config, "unit_test", CONFIG);

    #[test]
    #[traced_test(level = "info")]
    fn test_run_logic() {
        parse_config();
        println!("{}", CONFIG.get().unwrap().message.as_str());
    }
}

One downside to the above code is that the business logic of merging is defined in the macro. This isn't convenient, and requires all modules to be rebuilt on any change to that logic. This is lazyness on my part due to the <$struct_type>::from_arg_matches(). The macro could call out into common logic for most of this. Another disadvantage is that the figment merge work for all flags is re-performed for every module's struct. At first glance, I didn't see any way to merge a cached layer with figment, but I didn't try very hard. Perhaps it's quite easy. This is the price you pay for per-module structs with decentralized registration, but this approach should work and be more efficient with a centralized struct, since there will only be one. I also haven't thought about subcommands much, as I don't use them, but I don't know why they wouldn't work.

[Hecatron]

After coming across this thread, I managed to come up with an example that uses the derive method in combination with figment to load defaults -> toml file -> clap cli arguments in that order

• docs: Added example for figment toml config #6162

[rejuvenile]

After coming across this thread, I managed to come up with an example that uses the derive method in combination with figment to load defaults -> toml file -> clap cli arguments in that order

• docs: Added example for figment toml config #6162

This is great! Have a look at #2763 (comment), though, it doesn't require use of Option or anything special with serde (you can use standard Clap for your struct) and accomplishes the same. If you remove the macro and the distributed options, the code gets much simpler; basically what's in initalize_args().