Skip to main content

loaders.tableloader

The tableloader utility provides the core writing logic for the entire platform. Whether you are landing data in Bronze, merging in Silver, or publishing to Gold, these functions handle the physical storage interactions, schema enforcement, and Delta table operations.

View Functions

dataframe_to_table​

def dataframe_to_table(df_source: DataFrame, load_config: LoadConfig,
                       table_config: TableConfig,
                       config_manager: ConfigManager)

Loads a DataFrame into a Delta table with various load types (FULL, APPEND, MERGE) and handles schema validation, column updates, and logging.

Arguments:

  • df_source DataFrame - Source Spark DataFrame to load
  • load_config LoadConfig - Configuration for the load process
  • table_config TableConfig - Configuration for the target table
  • config_manager ConfigManager - Global configuration manager

Raises:

  • ValueError - If the source DataFrame does not match the target table schema or if duplicate keys are found
  • Exception - If an error occurs during the load process

Returns:

None

dataframe_to_gold​

def dataframe_to_gold(df_source: DataFrame, load_config: LoadConfig,
                      model: Model, config_manager: ConfigManager)

Loads a DataFrame into a Gold layer Delta table with surrogate key handling and validation.

Arguments:

  • df_source DataFrame - Source Spark DataFrame to load
  • load_config LoadConfig - Configuration for the load process
  • model Model - Data model containing table definitions
  • config_manager ConfigManager - Global configuration manager

Raises:

  • ValueError - If the table is not found in the model or if the source DataFrame does not match the target table schema
  • Exception - If an error occurs during the load process

Returns:

None

dataframe_to_bronze​

def dataframe_to_bronze(df_source: DataFrame, table_config: TableConfig,
                        object_info: ObjectInfo,
                        config_manager: ConfigManager) -> None

Loads a DataFrame into a Bronze layer Delta table with system columns and validation.

Arguments:

  • df_source DataFrame - Source Spark DataFrame to load
  • table_config TableConfig - Configuration for the target table
  • object_info ObjectInfo - Source object information
  • config_manager ConfigManager - Global configuration manager

Raises:

  • ValueError - If source DataFrame is empty or invalid
  • Exception - If bronze lakehouse configuration is missing

Returns:

None