Skip to main content

easyfabric.loaders.tableloader

json​

logging​

DeltaTable​

DataFrame​

Window​

coalesce​

col​

concat_ws​

expr​

lit​

row_number​

sha2​

when​

IntegerType​

LongType​

StringType​

ConfigManager​

LoadConfig​

LoadType​

Model​

ObjectInfo​

TableConfig​

align_dataframe_to_target_schema​

check_for_duplicate_keys_dataframe​

current_timestamp​

get_spark​

load_dataframe_silver​

merge_loadconfig_into_target​

spark_operation_with_retries​

spark​

dataframe_to_silver​

def dataframe_to_silver(df_source: DataFrame, load_config: LoadConfig,
                        table_config: TableConfig,
                        config_manager: ConfigManager)

dataframe_to_bronze_old​

def dataframe_to_bronze_old(df_source: DataFrame, load_config: LoadConfig,
                            table_config: TableConfig,
                            config_manager: ConfigManager)

dataframe_to_gold​

def dataframe_to_gold(df_source: DataFrame, load_config: LoadConfig,
                      model: Model, config_manager: ConfigManager)

Loads a DataFrame into a Gold layer Delta table with surrogate key handling and validation.

Arguments:

  • df_source DataFrame - Source Spark DataFrame to load
  • load_config LoadConfig - Configuration for the load process
  • model Model - Data model containing table definitions
  • config_manager ConfigManager - Global configuration manager

Raises:

  • ValueError - If the table is not found in the model or if the source DataFrame does not match the target table schema
  • Exception - If an error occurs during the load process

Returns:

None

handle_full_load​

def handle_full_load(load_config: LoadConfig, df_source, config_manager)

handle_append_load​

def handle_append_load(load_config: LoadConfig, df_source, config_manager)

handle_append_with_delete_load​

def handle_append_with_delete_load(load_config: LoadConfig, df_source,
                                   config_manager)

load_handlers​

dataframe_to_bronze​

def dataframe_to_bronze(df_source: DataFrame, table_config: TableConfig,
                        object_info: ObjectInfo,
                        config_manager: ConfigManager)

Loads a DataFrame into a Bronze layer Delta table with system columns and validation.

Arguments:

  • df_source DataFrame - Source Spark DataFrame to load
  • table_config TableConfig - Configuration for the target table
  • object_info ObjectInfo - Source object information
  • config_manager ConfigManager - Global configuration manager

Raises:

  • ValueError - If source DataFrame is empty or invalid
  • Exception - If bronze lakehouse configuration is missing

Returns:

  • Dataframe - The loaded DataFrame after processing and validation