Skip to main content

loaders.tableloader

dataframe_to_table​

def dataframe_to_table(df_source: DataFrame, load_config: LoadConfig,
                       table_config: TableConfig,
                       config_manager: ConfigManager)

Loads a DataFrame into a Delta table with various load types (FULL, APPEND, MERGE) and handles schema validation, column updates, and logging.

Arguments:

  • df_source DataFrame - Source Spark DataFrame to load
  • load_config LoadConfig - Configuration for the load process
  • table_config TableConfig - Configuration for the target table
  • config_manager ConfigManager - Global configuration manager

Raises:

  • ValueError - If the source DataFrame does not match the target table schema or if duplicate keys are found
  • Exception - If an error occurs during the load process

Returns:

None

dataframe_to_gold​

def dataframe_to_gold(df_source: DataFrame, load_config: LoadConfig,
                      model: Model, config_manager: ConfigManager)

Loads a DataFrame into a Gold layer Delta table with surrogate key handling and validation.

Arguments:

  • df_source DataFrame - Source Spark DataFrame to load
  • load_config LoadConfig - Configuration for the load process
  • model Model - Data model containing table definitions
  • config_manager ConfigManager - Global configuration manager

Raises:

  • ValueError - If the table is not found in the model or if the source DataFrame does not match the target table schema
  • Exception - If an error occurs during the load process

Returns:

None

dataframe_to_bronze​

def dataframe_to_bronze(df_source: DataFrame, table_config: TableConfig,
                        object_info: ObjectInfo,
                        config_manager: ConfigManager) -> None

Loads a DataFrame into a Bronze layer Delta table with system columns and validation.

Arguments:

  • df_source DataFrame - Source Spark DataFrame to load
  • table_config TableConfig - Configuration for the target table
  • object_info ObjectInfo - Source object information
  • config_manager ConfigManager - Global configuration manager

Raises:

  • ValueError - If source DataFrame is empty or invalid
  • Exception - If bronze lakehouse configuration is missing

Returns:

None