Env settings are not loaded if property is not defined as field (using extra="allow" in model_config)

[alessandro-gandolfi-ors]

I'm trying to load settings without defining a class with every property listed (there is one but not accessible by the code I'm writing)

The code for loading the settings manages env vars and json sources, with env vars being the priority and no properties defined, is the following:

from pydantic_settings import BaseSettings as PydanticBaseSettings, PydanticBaseSettingsSource, SettingsConfigDict, JsonConfigSettingsSource

class BaseConfiguration(PydanticBaseSettings):
    model_config = SettingsConfigDict(
        env_prefix="PREFIX_",
        env_nested_delimiter='__',
        json_file=["/path/to/file.json"],
        extra="allow"
    )
    
    @classmethod
    def settings_customise_sources(
        cls,
        settings_cls: type[PydanticBaseSettings],
        init_settings: PydanticBaseSettingsSource,
        env_settings: PydanticBaseSettingsSource,
        dotenv_settings: PydanticBaseSettingsSource,
        file_secret_settings: PydanticBaseSettingsSource,
    ) -> tuple[PydanticBaseSettingsSource, ...]:
        return env_settings, JsonConfigSettingsSource(settings_cls)

Having the following json at /path/to/file.json:

{
  "random_prop_json": "aaaaa"
}

And having the following env var with a value saved correctly in the os (using devcontainer, saved using launch.json in VS Code but could be saved with any other method):

"PREFIX_random_prop_env": "bbbbb"

Loads only the properties of the json when calling the ctor BaseConfiguration(), resulting only in:

{
  "random_prop_json": "aaaaa"
}

When I would expect the instance of BaseConfiguration() to contain something more like:

{
  "random_prop_json": "aaaaa",
  "random_prop_env": "bbbbb"
}

---

Tested with the following packages versions:

• pydantic==2.12.5
• pydantic-settings==2.13.1
• pydantic_core==2.41.5

[hramezani]

Thanks @alessandro-gandolfi-ors for reporting this.

Env settings source doesn't care about extra config. It loops overt defined field and tries to collect values for them. This is by design otherwise, we need to load all OS env variables.

[alessandro-gandolfi-ors]

@hramezani Thanks for the reply and info. I understand the logic behind ignoring extra properties from env vars not to bloat the loaded settings with useless values from the os.

It would be reasonable at least to try to load env vars with the specified prefix in the name. In my opinion it would be even better to load all of the os env vars if the developer doesn't specify a prefix and "force" them to use one if they need to filter the loaded env vars, moreover with the explicit extra="allow" configuration, it would be technically right even if inconvenient. The designed behaviour should be in the docs, the only hint at it is not even in the env var section but in the .env file section ("Pydantic settings consider extra config in case of dotenv file").

I know my case is particular but the behaviour change from one settings source to another is not intuitive at all. I'll use the .env files since those accept extra settings properties.

[hramezani]

@hramezani Thanks for the reply and info. I understand the logic behind ignoring extra properties from env vars not to bloat the loaded settings with useless values from the os.

It would be reasonable at least to try to load env vars with the specified prefix in the name. In my opinion it would be even better to load all of the os env vars if the developer doesn't specify a prefix and "force" them to use one if they need to filter the loaded env vars, moreover with the explicit extra="allow" configuration, it would be technically right even if inconvenient. The designed behaviour should be in the docs, the only hint at it is not even in the env var section but in the .env file section ("Pydantic settings consider extra config in case of dotenv file").

I know my case is particular but the behaviour change from one settings source to another is not intuitive at all. I'll use the .env files since those accept extra settings properties.

Agree, this is inconsistent between different setting sources. Unfortunately, there are some other inconsistencies between sources as well.

I am open to improve it but we shouldn't break the existing behavior. probably the best way is to introduce a config default to current behaviour. we can't change the current behaviour unit V3