How can Pydantic settings configuration be used with multiple YAML files?

edit

There are two approaches to use Pydantic settings configuration with multiple YAML files:

1. Merge YAML files using a Python script:

In this approach, you can write a Python script that reads multiple YAML files, merges them into a single YAML object, and passes it to Pydantic's BaseSettings subclass.

For example,

from typing import List
from pydantic import BaseModel

class Settings(BaseModel):
    databases: List[str]
    # other settings ...

def load_settings():
    import yaml
    merged_dict = {}
    with open('config1.yaml') as file1, open('config2.yaml') as file2:
        yaml1, yaml2 = yaml.safe_load(file1), yaml.safe_load(file2)
        merged_dict.update(yaml1)
        merged_dict.update(yaml2)
    return Settings(**merged_dict)

Here, load_settings function reads two YAML files (config1.yaml and config2.yaml) and merges them into a single dictionary merged_dict. After that merged_dict is passed to Settings class to create a Pydantic Settings object.

1. Use Pydantic's built-in support for multiple config files:

Pydantic provides support for combining configuration files of different formats (e.g, YAML, JSON, and environment variables) using an approach called "complex settings".

For example,

from typing import List
from pydantic import BaseSettings, Field

class Settings(BaseSettings):
    databases: List[str] = Field([], env='DBS')

    class Config:
        env_prefix = 'MYAPP_'
        env_file = '.env'

        @classmethod
        def _yaml(cls, path: str):
            with open(path) as f:
                return f.read()

        @classmethod
        def customise_sources(
            cls, init_settings, env_settings, file_secret_settings
        ):
            return env_settings or file_secret_settings or init_settings

Here, Config class defines a custom behaviour to load configuration settings. It specifies env_prefix to identify environment variables with MYAPP_ prefix, env_file to include environment variables from .env file, _yaml class method to read YAML files and customise_sources method to combine the settings from environment variables, YAML files, and defaults.

In this approach, you need to keep the YAML files in a predefined location (e.g., config.yaml), and Pydantic automatically detects and loads it. To load additional YAML files, you can call Settings.Config._yaml method and pass the file path as an argument.

For example,

settings = Settings(_yaml('config1.yaml'))

Here, Settings(_yaml('config1.yaml')) creates a Settings object by loading settings from config.yaml and config1.yaml files.

Note: You can also use environment variables and command-line arguments to override the settings defined in YAML files.