Silver

Overview

Materialize a type-converted Delta table from bronze. With keephistory: false, silver is a flat typed copy of the latest bronze. With keephistory: true, silver maintains its own Silver.his history table populated from Bronze.his; silverloadtype selects either a full reload or an incremental merge into the existing silver table. Per-column type conversions and expressions live on the same per-table YAML as bronze.

What gets generated

Stage	Component	Output
Build	EasyFabric Generator (EFG)	GenerateFabricObjects → Fabric lakehouse tables (silver + history)
Runtime	EasyFabric Runtime (EFR)	easyfabric.load_data_silver.run, easyfabric.load_data_silver.dataframeloader

Example YAML

Dataplatform/DP/Objects/AdvWorks/orders.yaml

  Table:
  Connection: adv-advworks
  SourceTable: orders.csv
  DataPlatformObjectname: orders
  PreBronzeNotebook: 
    Notebook: Pull_Github
    Param001: orders.csv
    Param002: orders.csv
  KeepHistory: true
  Columns:
  - SourceColumn: SalesOrderID
    SourceDataType: varchar(255)
    IsPrimaryKey: true
  - SourceColumn: LineItem
    SourceDataType: int
    IsPrimaryKey: true
  - SourceColumn: OrderDate
    SourceDataType: datetime
  - SourceColumn: CustomerID
    SourceDataType: int
  - SourceColumn: ProductID
    SourceDataType: int  
  - SourceColumn: OrderQty
    SourceDataType: decimal(28,5)    
  - SourceColumn: LineItemTotal
    SourceDataType: decimal(28,5)

Schema reference

Required fields marked *. Linked types are collapsible — click to expand inline.

Fabric Object

Name	Type	Description
Connection *	String	Connection to use for this object (as defined in connection)
Fields *	List<FabricAttribute>	Fields from the source object
SourceTable *	String	Name of the object in the source

Show optional fields (6)

Name	Type	Description
DataPlatformObjectName	String	To override the source name that is used in the dataplatform
Description	String	Description of the source
IsActive	true/false	Set to active for generating this object (default=true)
KeepHistory	true/false	Set to true if history is required in the silver layer (default=true). Primary key required.
Prefix	String	Set a prefix (default = '')
SourceSchema	String	Source schema in the source (if applicable)

Fabric Attribute

Name	Type	Description
SourceColumn *	String	Name of the attribute in the source system
SourceDataType *	String	Datatype of the source (use source datatype, will be converted to dataplatform types automatically)

Show optional fields (5)

Name	Type	Description
Classification	ClassificationType	Set the classification of a field (sensitive, restricted, internal, public (default = public)
IsActive	true/false	Attribute is active (default=true)
IsNullable	true/false	Attribute can be null (default = true)
IsPrimaryKey	true/false	Attribute is part of the primary key (default = false)
Name	String	To override the name used in the dataplatform (default is the sourcecolumn name used)

Runtime-only fields

These fields are read by the EFR at runtime and have no EFG counterpart.

• keephistory
• silverloadtype
• silverdatatype
• silverexpression
• issilveronly
• isdeletedidentifier

EasyFabric Runtime

load_data_silver.run

def run(tablefile: str, config_manager: ConfigManager=None)

Executes the silver extraction and transformation process for a table, utilizing the configuration information provided in the ConfigManager. This involves loading, processing, and merging table data within the silver lakehouse layer.

The function ensures the proper configuration of logging, verifies layer activity and table activation status, and invokes pre-silver and post-silver workflows if defined. It uses corresponding notebooks or direct data operations for the silver loading process.

Arguments:

• tablefile str - Path to the YAML file containing table configuration details.
• config_manager ConfigManager - An instance of ConfigManager, pre-initialized with application configuration, connection, and lakehouse details.

Returns:

• Optional[str] - An error message containing the name of the failed table and the exception details, or None if the operation finishes successfully.

Raises:

• Exception - If ConfigManager is not initialized before invoking this function.
• Exception - If the bronze lakehouse configuration is not found.
• Exception - If the stop_at_error setting is enabled and an exception occurs in processing.

load_data_silver.dataframeloader

def dataframeloader(data_frame: DataFrame, table_config: TableConfig,
                    load_config: LoadConfig, config_manager: ConfigManager=None)

Loads data into a silver layer table in a lakehouse environment.

This function facilitates loading data from a given DataFrame into a table specified by a table configuration within a silver layer of the lakehouse architecture. It uses the provided configuration details to establish connections, manage runtime settings, and log relevant information during the operation. It validates critical configurations and raises appropriate exceptions in case of missing or invalid details.

Arguments:

• data_frame DataFrame - The input pandas DataFrame containing data to load.
• table_config TableConfig - Configuration object specifying table details and related connection configurations.
• load_config LoadConfig - Configuration object holding load-specific settings, including the layer and runtime options.
• config_manager ConfigManager - Centralized configuration management object used to retrieve connection settings and maintain runtime parameters.

LoadConfig fields

Runtime parameter bag — construct in code and pass to the loader. All fields are optional unless flagged below.

Field	Type	Description
_layer	str	Operational layer associated with the configuration. Defaults to "not set".
dry_run	bool	Indicates if the process should be executed in dry-run mode. Defaults to True.
auto_null_column	bool	Determines if null values should be automatically managed for columns. Defaults to True.
load_type	LoadType	Specifies the type of load operation. Defaults to LoadType.FULL.
stop_at_error	bool	Specifies whether the process should stop when an error occurs. Defaults to True.
business_key_check	bool	Indicates if business keys should be validated during the load. Defaults to True.
log_row_count	bool	Determines if row counts should be logged during the process. Defaults to False.
key_violation_action	str	Action to be taken when key violations occur. Defaults to "raise".
destination_schema	str	Schema of the destination table. Defaults to "dbo".
destination_table	Optional[str]	Name of the destination table. Defaults to None.

Returns:

• str - A string indicating the result of the load operation, either as success or an error message.

Raises:

• Exception - If load_config is None, if ConfigManager is not properly initialized, or if no silver lakehouse configuration is found.
• Exception - If there is an issue while performing the load operation and config_manager.stop_at_error is set to True.

Related data classes

• easyfabric.data.TableConfig
• easyfabric.data.Column