Think about interaction with clap to allow loading/overriding config values from CLI args

[LukasKalbertodt]

Often it's useful to allow users to override some (or all) configuration values via command line. Confique should work nicely for that use case. The currently best way to do it is probably to convert the CLI values to a partial type (manually) and then add it via Builder::preloaded. I would like to investigate whether this can be made more convenient and with less duplicate code.

If this improvement has to be CLI-library specific, I am pretty sure I only want to support clap. I only ever use clap with the derive feature and I think it's the most mature library.

Just to throw some random ideas into this issue, maybe one can annotate config fields with #[config(clap(...))] and if any are annotated this way, we will generate an additional type containing the fields annotated that way that has the derive(clap::*) on it. And has a method to convert it to a partial config type. This extra type can then be flattened into the main clap type. But again, haven't thought about this too deeply yet.

[ssokolow]

For the record, I also use Gumdrop in my actix-web projects, where I don't need the path arguments to support accepting non-UTF8-able POSIX paths (i.e. where they're just going to be a path to a server root folder or config file) and I want to benefit from the reduced compile times and smaller output binaries.

[LukasKalbertodt]

Fair point. It's certainly worth considering how one could support multiple cli libraries. Or even be completely library-agnostic.

[hmuendel]

I would also love to have an integration with clap. I followed your workaround described above and I find it quite ok. However I was not able to derive Config directly on my clap cli flag struct, because of defaults. I totally agree with your view, that the finally parsed config should not contain options any longer, but this lead me to have options in my clap struct to see if the user provided the flag. I expect the flag to take precedence over all other config options (since it is the most direct user input). So I had another struct with options removed and config derived on it, which I could then construct from the partials.
But this misses out on the documentation benefits from clap.

I didn't see a better way of doing it for now, but I still find your crate awesome and clean and can live with the solution. Just wanted to raise the point of defaults in clap and confique.

[Nemo157]

A different kind of approach to support overriding via the command line is to have a general flag that accepts a fragment of config and applies it, this is the approach taken by cargo --config and jj --config-toml. This seems relatively simple to do now with confique, and is maybe just worth adding an example of, I used something like:

#[derive(confique::Config, Debug)]
#[config(partial_attr(derive(Clone, Debug)))]
#[config(partial_attr(serde(deny_unknown_fields)))]
struct Config {
    ...
}

type Partial = <Config as confique::Config>::Partial;

impl Config {
    fn load(fragments: Vec<Partial>) -> Result<Self> {
        let dirs = ProjectDirs::from(...).ok_or_eyre("cannot get config directory")?;
        let mut builder = Config::builder();
        // reverse so that later fragments take precedence
        for fragment in fragments.into_iter().rev() {
            builder = builder.preloaded(fragment)
        }
        Ok(builder.env().file(dirs.config_dir().join("config.toml")).load()?)
    }
}

#[derive(Debug, Parser)]
struct App {
    /// Config overrides to apply, these should be fragments of the config file.
    #[arg(long = "config-toml", value_name = "TOML", value_parser = toml::from_str::<Partial>)]
    config_fragments: Vec<config::Partial>,

    ...
}

fn main() -> Result<()> {
    let app = App::parse();
    let config = Config::load(app.config_fragments)?;
    tracing::trace!(?config, "loaded config");

    ...
}

[aschey]

I think this can be accomplished without having to add a dependency on clap or have any implementations specific to it. We need to be able to use partial_attr on struct fields (PR created for this: #44):

use clap::Args;

#[derive(confique::Config, Debug)]
#[config(partial_attr(derive(Args, Debug)))]
pub struct AppConfig {
    #[config(partial_attr(arg(long)))]
    port: usize,
    #[config(partial_attr(arg(long)))]
    allowed_hosts: Vec<String>,
}

which allows you to embed the partial struct inside a clap Parser:

use clap::Parser;

#[derive(Parser, Debug)]
struct Cli {
    #[command(flatten)]
    app_config: PartialAppConfig,
    // other fields
}

and use it like this:

let cli = Cli::parse();
let config = AppConfig::builder()
    .preloaded(cli.app_config)
    .env()
    .file(&config_path)
    .load()?;

The only remaining issue is that clap requires structs to use Option without the fully-qualified name, but confique's Config macro generates structs that use std::option::Option, causing clap's parsing to fail. My initial reaction was that this should be fixed in clap, but apparently this is a feature to work around the lack of true compile-time reflection.

To get around this, we could either

• provide another attribute to the derive macro to allow users to override the type name used when generating optional fields, something like #[option_type_name="Option"]
• remove the fully-qualified path and generate structs that use Option instead of std::option::Option

I think the second option is probably fine since Option is in the global prelude, but I could be convinced otherwise.

[LukasKalbertodt]

Regarding the fully qualified Option: as far as I can see it, either we use the fully qualified path or we fully disallow a symbol Option to refer to anything else than std::option::Option in the derive(Config) scope. Which is prooobably fine, in particular since confique also attributes special behavior to Option fields. There are some theoretical counter examples, where this would be a breaking change. But mh, I'm not entirely opposed to changing this.

Details

Just changing confique to emit Option instead of std::option::Option would make this break:

#[derive(Config)]
struct MealConfig {
    option: Option,
}

enum Option { Vegan, Vegetarian, Anything }

Even if no type called Option is used in the config struct, but is just in scope, it is complicated. The inner module (with the partial type) has use super::*; because the user might use types for struct fields that are defined next to the main Config struct. These need to be imported in the partial module. And the macro cannot know where a field type is defined, so cannot know if it needs to be imported.

But isn't it possible to use std::option::Option with clap and still get the behavior by specifying the appropriate attributes (.action(ArgAction::Set).required(false))?

[aschey]

Yeah, there's really not a robust solution unfortunately. I can be sympathetic to use cases for aliasing common types (Result being the most common one), but I don't see a compelling reason for having a custom type called Option considering how many naming conflicts that would cause.

But isn't it possible to use std::option::Option with clap and still get the behavior by specifying the appropriate attributes (.action(ArgAction::Set).required(false))?

Good thought! That's for the builder API, but I did some testing and it does actually work if you override the value parser using the derive API like this: #[arg(value_parser=value_parser!(usize))], replacing usize with whatever type matches the struct field. That's a decent workaround, but it's pretty unintuitive considering the opaque errors that clap spits out when it can't figure out the correct types.

[LukasKalbertodt]

I don't see a compelling reason for having a custom type called Option considering how many naming conflicts that would cause.

Yup, I thought about it a bit more and I agree. In the next major version, confique will disallow having Option to refer to anything other than std::option::Option. Then confique can emit simple, not-fully-qualified Option field types for the partial type, which makes this clap use case work (together with #44).

That's a decent workaround, but it's pretty unintuitive

Thanks for finding the workaround. And yes, that's not great DX at all. With #44 being released as 0.3.1, this means you can technically already do it with a released version of confique, but yes, still not great. As I said above, in the next major version this will be fixed.

---

I've thought about this general issue a bit more. I think there are a few ways/use cases/options on how to integrate a CLI parser with confique.

1. A separate & dedicated CLI parser (e.g. a type Cli deriving clap::Parser) from which a Config::Partial is created and added to builer.preloaded(...). This requires some manual work and potentially some duplication (of docs, types). However, I still think that in production applications this will be the most common approach: you might only want some of the config values in the CLI, parsing values from CLI might be different, or you might specify different docs for the argument.
2. Use partial_attr to derive clap::Parser on the partial types. That's what @aschey came up with. This results in a very attribute-heavy struct definition but avoids any duplication. You can even ignore some fields in the CLI by using the clap skip attribute.
3. Specify config snippets via CLI to override, like @Nemo157 suggested. Or build your own --override foo.bar=23. It's a bit of manual code, but both absolutely doable.
4. Built-in support: confique knows directly about clap (or other libraries) to improve DX.

I think with 1, 2 and 3, many most cases would already catered to. My next plans would be:

• Make (2) work well: the Option thing mentioned above, forwarding doc comments to partial fields, ... and potentially more
• Potentially make (1) and (3) easier. Will see if there are some easy ways to improve DX.
• Add examples and docs for all of these. It's important to show users how to do it, so that they can get started quickly.

After all of that, we can reconsider if a more built-in and tightly-coupled support is needed.

[lu-zero]

Would be possible to have a new release since at least the code for 2 seems to be all landed?

[LukasKalbertodt]

I just published v0.4.0 which includes support for layer_attr on struct fields, uses plain Option in generated code (meaning that clap understands the type), and forward docs to the layer type. With this, I think there is decent support for CLI stuff via clap for example. You don't need to duplicate docs or flag definitions. I still think many apps would opt for a separate type for CLI args.

I'd be interested in getting some feedback on this from people who actually use confique and use CLI as one layer to load the config. Is the solution as is fine? What are pain points?

Once I gathered a bit more experience, I will again update the docs to give nicer guidance on possible ways to integrate CLI with confique.

[LukasKalbertodt]

Oh and take a look at the clap example I added: https://github.com/LukasKalbertodt/confique/blob/main/examples/clap.rs