Skip to main content

load_meta_data

The load_meta_data module provides the foundational logic for reading and parsing the YAML configuration files that drive the automated platform. It translates your project definitions into the technical objects used by the Bronze, Silver, and Gold loaders.

get_objects_by_folder​

def get_objects_by_folder(
        tablefolder: str,
        config_manager: ConfigManager=None) -> list[TableConfig] | None

Loads metadata configurations from a specified folder or file and returns a list of TableConfig objects. If a single file is specified, metadata will be loaded only from that file. Otherwise, the function traverses the specified folder and loads metadata from all YAML files.

Arguments:

  • tablefolder str - The path to the folder or file containing YAML metadata files. If it starts with "Files/", it will be resolved to the default lakehouse path.
  • config_manager ConfigManager - An instance of ConfigManager used to initialize and validate configurations.

Returns:

  • List[TableConfig] - A list of TableConfig dataclass instances created from the loaded YAML metadata.

Raises:

  • Exception - If config_manager is not initialized or if an error occurs during metadata loading and config_manager.stop_at_error is set to True.

get_model_by_file​

def get_model_by_file(modelfile: str, config_manager: ConfigManager) -> Model

Loads a Model object by reading a given YAML file. Ensures that the configuration manager is initialized and sets up logging for the operation.

If the input file path starts with a specific prefix, it is converted to an absolute path. The function validates the file type to ensure it is a YAML file and parses its content to create a Model instance.

Arguments:

  • modelfile - The path to the metadata YAML file. It can be relative or absolute. If it starts with 'Files/', it is converted to the absolute path in the lakehouse directory.
  • config_manager - An instance of ConfigManager that is responsible for initialization and managing configurations.

Returns:

  • Model - The Model object created from the provided YAML file.

Raises:

  • Exception - If the configuration manager is not initialized.
  • Exception - If the provided file is not a YAML file.
  • Exception - If the stop-at-error policy is enabled in the configuration manager and an error occurs during the process.