April 14, 2022
Python configuration management
hydra-core package
Hydra is a Python library that allows you to access parameters from a configuration file inside a Python script.
Create exemplary main.yaml
in directory
config
raw:
path: data/raw/sample.csv
processed:
path: data/processed/processed.csv
final:
path: data/final/final.csv
then we can access the value inside the configuration file by adding
the decorator @hydra.main
on a specific function.
Inside this function, we can access the value under
processed
and path
by using a dot
notation: config.processed.path
.
"""
This is the demo code that uses hydra to access the parameters in under the directory config.
Author: Khuyen Tran
"""
import hydra
from omegaconf import DictConfig
from hydra.utils import to_absolute_path as abspath
@hydra.main(config_path="../config", config_name='main')
def process_data(config: DictConfig):
"""Function to process the data"""
raw_path = abspath(config.raw.path)
print(f"Process data using {raw_path}")
print(f"Columns used: {config.process.use_columns}")
if __name__ == '__main__':
process_data()
From: https://towardsdatascience.com/how-to-structure-a-data-science-project-for-readability-and-transparency-360c6716800
decouple package
Python Decouple: Strict separation of settings from code
Decouple helps you to organize your settings so that you can change parameters without having to redeploy your app.
It also makes it easy for you to:
- store parameters in ini or .env files; define comprehensive default values;
- properly convert values to the correct data type;
- have only one configuration module to rule all your instances. -It was originally designed for Django, but became an independent generic tool for separating settings from code.
Envvars works, but since os.environ
only returns
strings, it’s tricky.
Let’s say you have an envvar DEBUG=False
. If
you run:
if os.environ['DEBUG']:
print True
else:
print False
It will print True
, because
os.environ['DEBUG']
returns the string
“False”. Since it’s a non-empty string, it will be
evaluated as True.
Decouple provides a solution that doesn’t look like a workaround:
config('DEBUG', cast=bool)
.
From: package description on pypi
omegaconf
OmegaConf is a hierarchical configuration system, with support for merging configurations from multiple sources (YAML config files, dataclasses/objects and CLI arguments) providing a consistent API regardless of how the configuration was created.
Tags:
python
configuration
hydra
decouple