Skip to main content

easyfabric.loaders.table_utils

logging​

time​

DataFrame​

ConfigManager​

TableConfig​

get_spark​

spark_operation_with_retries​

def spark_operation_with_retries(operation_func,
                                 config_manager: ConfigManager,
                                 operation_name: str = "Spark Operation")

Executes a Spark/Delta operation with retry logic for concurrency conflicts.

Arguments:

  • operation_func - A callable that performs the Spark/Delta operation.
  • config_manager ConfigManager - Configuration manager for retry settings.
  • operation_name str - Name of the operation for logging.

refresh_table​

def refresh_table(table_config: TableConfig,
                  config_manager: ConfigManager,
                  layer: str,
                  return_table: bool = False,
                  history: bool = False) -> DataFrame | None

Refresh the metadata of a Spark table and optionally return it as a DataFrame, with retry logic on failure.

Arguments:

  • table_config TableConfig - Table configuration object.
  • config_manager ConfigManager - Configuration manager for environment/layer.
  • layer str - Data layer name.
  • return_table bool, optional - If True, return the refreshed Spark DataFrame. Defaults to False.
  • history bool, optional - If True, get historical version of table. Defaults to False.

Returns:

DataFrame | None: Spark DataFrame if return_table is True, else None.

truncate_bronze_table​

def truncate_bronze_table(table_config: TableConfig,
                          config_manager: ConfigManager)

Deletes all data from a specified bronze lakehouse table by using Truncate and refreshes the table cache.

Arguments:

  • table_config TableConfig - Configuration object containing the table details.
  • config_manager ConfigManager - Configuration manager to retrieve lakehouse and schema details.

Raises:

  • Exception - If no bronze lakehouse is found in the configuration.

delete_silver​

def delete_silver(table_config: TableConfig, config_manager: ConfigManager)

Deletes all data from the specified silver table and refreshes the cache to ensure the latest state of the table is visible for subsequent operations. This function retrieves the silver lakehouse configuration, constructs the necessary table name and performs deletion using Spark.

Arguments:

  • table_config TableConfig - Configuration object that provides table-related information such as table names. It is used to determine the silver table name linked to the provided configuration.
  • config_manager ConfigManager - Configuration manager instance that provides access to various lakehouse configurations, including the silver lakehouse details required for this operation.

Raises:

  • Exception - If no silver lakehouse is found in the configuration. An error is logged before raising this exception to provide insight into the root cause.