Skip to main content

load_data_gold

The load_data_gold module is the final stage of the data pipeline. It consumes Silver tables and applies the business logic defined in the Tabular Model to create high-performance semantic models and reporting views.

modelloader​

def modelloader(data_frame: DataFrame, load_config: LoadConfig,
                model_config: Model, config_manager: ConfigManager=None)

Loads data into the gold layer of a data platform using the specified configurations and model.

This function handles loading a DataFrame into the data platform's "gold" layer. It configures logging, saves metadata, and uses the provided model configuration to apply necessary transformations and loading procedures. Proper initialization of LoadConfig, Model, and ConfigManager is required before calling this function.

Arguments:

  • data_frame DataFrame - The data to be loaded into the platform.
  • load_config LoadConfig - Configuration settings for the loading process.
  • model_config Model - Model configuration required for transformations and validations.
  • config_manager ConfigManager - Manages configuration details and ensures proper initialization.

Returns:

  • str - A string indicating the outcome of the loading process, or None if successful.

Raises:

  • Exception - If load_config.model_object_name is missing.
  • Exception - If config_manager is not properly initialized.
  • Exception - If model_config is not provided.
  • Exception - If any other error occurs during the loading process and Model5 is set to True.