Skip to main content

easyfabric.data.loadconfig

re​

dataclass​

Enum​

Optional​

ConfigManager​

DataClassFromDictMixin​

LoadType Objects​

class LoadType(Enum)

Enum of supported data load types (Full, Append, Merge, etc.).

FULL​

APPEND​

APPENDWITHDELETE​

MERGE​

to_snake_case​

def to_snake_case(string: str) -> str

LoadConfig Objects​

@dataclass
class LoadConfig(DataClassFromDictMixin)

Configuration class for managing data load settings.

This class encapsulates the configuration settings for managing data load operations, enabling fine-grained control over various aspects such as load type, error handling, logging, and destination specifications. It is designed to adapt and configure data loading workflows seamlessly based on user-defined settings.

Attributes:

  • _layer str - Operational layer associated with the configuration. Defaults to "not set".
  • dry_run bool - Indicates if the process should be executed in dry-run mode. Defaults to True.
  • auto_null_column bool - Determines if null values should be automatically managed for columns. Defaults to True.
  • load_type LoadType - Specifies the type of load operation. Defaults to LoadType.FULL.
  • stop_at_error bool - Specifies whether the process should stop when an error occurs. Defaults to True.
  • business_key_check bool - Indicates if business keys should be validated during the load. Defaults to True.
  • log_row_count bool - Determines if row counts should be logged during the process. Defaults to False.
  • key_violation_action str - Action to be taken when key violations occur. Defaults to "raise".
  • destination_schema str - Schema of the destination table. Defaults to "dbo".
  • destination_table Optional[str] - Name of the destination table. Defaults to None.
  • dry_run0 Optional[str] - Specifies the model object name used in operations. Defaults to None.
  • dry_run1 Optional[str] - The query for handling delete scenarios during merge operations. Defaults to None.

dry_run​

auto_null_column​

load_type​

stop_at_error​

business_key_check​

log_row_count​

key_violation_action​

load_history_table​

model_object_name​

merge_delete_query​

destination_full_tablename​

get_destination_lakehouse​

@staticmethod
def get_destination_lakehouse(config_manager: ConfigManager)

Resolve the destination lakehouse name for the bronze layer.

Arguments:

  • config_manager - The active ConfigManager instance.

Returns:

The lakehouse name string for the bronze layer.

get_destination_abfs​

@staticmethod
def get_destination_abfs(config_manager: ConfigManager)

Get the ABFS path for the bronze-layer destination.

Arguments:

  • config_manager - The active ConfigManager instance.

Returns:

The ABFS URI string for the bronze lakehouse.

layer​

@property
def layer()