Skip to main content

loaders.history

load_dataframe_bronze_history​

def load_dataframe_bronze_history(df_bronze: DataFrame,
                                  table_config: TableConfig,
                                  config_manager: ConfigManager) -> None

Loads the given DataFrame into the bronze history table while applying upsert and delete logic based on the configured history settings and schema. This function processes the incoming DataFrame by comparing it against the existing records in the history table to update, delete, or insert records accordingly.

The behavior and mechanism of operations, such as identifying keys for join conditions and filtering data, depend heavily on the configuration provided through table_config and config_manager.

Arguments:

  • df_bronze DataFrame - The input DataFrame containing the source data to be processed and loaded into the bronze history table.
  • table_config TableConfig - Configuration object for the target table, containing metadata such as key columns, history settings, table name, and other relevant configurations.
  • config_manager ConfigManager - Configuration manager providing access to global configurations, such as lakehouse settings, paths, schemas, and other metadata for processing.

Raises:

  • Exception - Raised if the bronze lakehouse cannot be retrieved, history settings are invalid, history functionality is not enabled for the table, or other critical configurations are missing that halt processing.