Skip to main content

YAML Properties Reference

This page provides a comprehensive overview of all YAML properties used across the EasyFabricCore system. It specifically identifies which properties are utilized by the major system components.

Component Overview​

  • API: Web API that receives JSON requests (converted from YAML) to generate Fabric objects.
  • Dataplatform: Desktop application (Generator.exe) that reads YAML files to generate notebooks, tables, and configurations.
  • Wheel Package (easyfabric-py): Python library used at runtime in Fabric notebooks to load and process data.

Required vs. optional properties​

Unless a property's description is marked (required), the property is optional and may be omitted.

  • A (default=…) annotation shows the value the system uses when you leave the property out.
  • A property with no default annotation is still optional unless marked (required); when omitted it is simply left unset, or inherited from a higher config level (see note 6).

Configuration Properties​

Defined in the main environment configuration file (e.g., dp.dev.yaml).

PropertyAPIDataplatformWheel PackageDescription
BronzeConnectionPrefix✅✅✅Use prefix from ConnectionPrefix for bronze objects (default=true)
BronzeKeepSourceColumnNames✅✅—Use source names even when overridden on bronze (default=false)
BronzeObjectPrefix✅✅✅Prefix for bronze objects (default='')
BronzeTableSettings✅✅—Table settings for Bronze layer (Delta properties)
ColumnNamesToLower✅✅—Generate all column names to lower (default=true)
ColumnNamesAutoFix✅✅—Auto fix columns with only numbers by adding 'z' prefix (default=true)
DefaultSeparator✅✅✅Separator used for object names (default='_')
DimBusinesskeyPrefix✅✅—Prefix of businesskey column for dimensions (default='BK_')
DimObjectPrefix✅✅—Prefix used for dimensions (default='D')
DimSurrogatekeyPrefix✅✅—Prefix used for dimension surrogate keys (default='SK_')
FactObjectPrefix✅✅—Prefix used for fact objects (default='F')
HistoryTablePostfix✅✅✅Postfix used for objects with history (default='')
IsBronzeStringTypes✅✅✅Create fields in bronze layer with datatype string (default=true)
Lakehouses✅✅✅List of lakehouses used in medallion layer
ObjectPartitionColumn✅✅—Name of partition column (default='SYSTEMPARTITION')
ObjectSourceTagColumn✅✅—Name of system source tag column (default='SYSTEMSOURCETAG')
RemoveNewLineFromScripts✅✅—Remove newline from generated scripts (default=true)
SchemaHistory✅✅✅Schema name for history tables (default='his')
SilverConnectionPrefix✅✅✅Use connection prefix for object names on silver (default=true)
SilverConvertedColumn✅✅—Create Silver column with converted value (default=true)
SilverConvertedPostfix✅✅✅Postfix for converted column in silver (default='_convert')
SilverInitialColumn✅✅—Create Silver column with original value (default=true)
SilverObjectPrefix✅✅—Prefix for silver objects (default='')
SilverPrimarykeySeparator✅✅—Separator for SYSTEMPRIMARYKEY field (default='
SilverTableSettings✅✅—Table settings for Silver layer (Delta properties)
SurrogateKeyUnknownValue✅✅—Default value for surrogate key when no match (default='-1')
TransformationBusinessKeysNotNull✅✅—Business keys in transformation objects mandatory (default=false)
TransformationObjectPrefix✅✅—Prefix used for transformation objects (default='T')
DefaultEncodingHintKeyColumns✅✅—Default encoding hint for key columns (default='Value')
DefaultEncodingHintRegularColumns✅✅—Default encoding hint for regular columns (default='Default')

Connection Properties​

Defined in connection YAML files (e.g., adv-advworks.yaml).

PropertyAPIDataplatformWheel PackageDescription
ConnectionName✅✅✅Name of the connection (required)
ConnectionPrefix✅✅✅Prefix used when connection prefix is enabled (required)
ConnectionType——✅Type of connection (e.g., fabricfiles, sqlserver)
BronzeCleanupActive——✅Enable automatic cleanup of expired bronze history rows (default: unset — cleanup off)
BronzeFolder——✅Folder path for bronze files
BronzeKeyViolationAction——✅Action on key violations (raise/keepone/remove)
BronzeLoadOrder——✅Load order for bronze notebooks (default=0)
BronzeLoadSkip——✅Skip bronze loading (default=false)
BronzeMaxRetentionHours——✅Retention window in hours for bronze history cleanup when BronzeCleanupActive is on (default: unset)
BronzeNotebookConcurrency——✅Concurrency for bronze notebooks (default=0)
BronzeTableSettings✅✅—Table settings for Bronze at connection level
Charset——✅Character set for file reading
Container——✅Storage container name
Dateformat——✅Default date format for parsing
Delimiter——✅CSV delimiter (default=',')
Escape——✅Escape character for CSV
FileExtension——✅File extension filter
FileType——✅Type of file (csv, json, xml, parquet, etc.)
Header——✅CSV has header row (default=true)
IgnoreNullKey——✅Ignore records with null keys (default=true)
InferSchema——✅Infer schema from data (default=false)
IsBronzeStringTypes✅✅✅Override config setting for this connection
IsTriggerable——✅Connection can be triggered (default=true)
JsonPath——✅Path to the nested array of records, e.g. $.rows. Omit for files that are themselves a JSON array (top-level records load directly)
KeyvaultSecretConnectionString——✅Keyvault secret name for connection string
MaxFileAgeHours——✅Max age in hours of a source file; older files are reported per BronzeLoadViolationAction (and caught even when the source is unchanged) (default=2)
Mode——✅Spark read mode (default='PERMISSIVE')
Multiline——✅Support multiline records (default=false)
NotebookTimeout——✅Default timeout for notebooks (default=1800)
ProcessType——✅Type of processing
Quote——✅Quote character for CSV
QuoteMode——✅Quote mode for CSV
SilverLoadOrder——✅Load order for silver notebooks (default=0)
SilverNotebookConcurrency——✅Concurrency for silver notebooks (default=10)
SilverTableSettings✅✅—Table settings for Silver at connection level
SkipIfSourceUnchanged——✅Skip the load when source files are unchanged since the last run (default: unset — inherited; see Skip-if-Unchanged)
SourceFolder——✅Source folder path

Object/Table Properties​

Defined in object YAML files (e.g., customers.yaml).

PropertyAPIDataplatformWheel PackageDescription
Connection✅✅✅Connection to use for this object (required)
SourceTable✅✅✅Name of the object in the source
SourceSchema✅✅—Source schema (if applicable)
DataPlatformObjectName✅✅✅Override source name in dataplatform
Description✅✅—Description of the source
IsActive✅✅✅Set to active for generating (default=true)
KeepHistory✅✅✅Enable history tracking (default=true; the runtime default differs — see note 9)
KeepOneDayRecords——✅Keep only one day of records (default=false)
Layers✅✅✅Which layers to generate (ALL/Bronze/Silver/Gold, default=ALL)
Prefix✅✅—Set a prefix for this object
Columns / Fields✅✅✅List of columns/fields from source
BronzeTableSettings✅✅—Table settings for Bronze at object level
SilverTableSettings✅✅—Table settings for Silver at object level
SourceFilter——✅Filter query for source data
SourceFolder——✅Override source folder
SourceOrder——✅Order by clause for source
BronzeFolder——✅Override bronze folder
BronzeKeyViolationAction——✅Override connection setting
BronzeLoadOrder——✅Override connection load order
BronzeLoadSkip——✅Skip bronze load for this table (default=false)
BronzeLoadViolationAction——✅Response to load violations including stale files (maxfileagehours): continue warns, stop raises (default=stop)
BronzeMaxExpectedDuration——✅Max expected duration for bronze (default=1800)
BronzeCleanupActive——✅Override connection: enable cleanup of expired bronze history rows
BronzeMaxRetentionHours——✅Override connection: retention window in hours for bronze history cleanup
SkipIfSourceUnchanged——✅Skip the load when source files are unchanged since the last run
DateFormat——✅Override date format for this table
FileExtension——✅Override file extension
FileType——✅Override file type
Delimiter——✅Override connection CSV delimiter
Header——✅Override connection: CSV has header row
Multiline——✅Override connection: support multiline records
Escape——✅Override connection CSV escape character
Quote——✅Override connection CSV quote character
QuoteMode——✅Override connection CSV quote mode
Charset——✅Override connection character set for file reading
JsonPath——✅Override connection JSON path for extracting data
HistorySettings——✅History-specific settings
IgnoreNullKey——✅Override ignore null key setting (default=true)
PreBronzeNotebook——✅Notebook to run before bronze
MidBronzeNotebook——✅Notebook to run during bronze
PostBronzeNotebook——✅Notebook to run after bronze
PreSilverNotebook——✅Notebook to run before silver
MidSilverNotebook——✅Notebook to run during silver
PostSilverNotebook——✅Notebook to run after silver
BronzeNotebook——✅Custom bronze notebook
SilverNotebook——✅Custom silver notebook
SheetName——✅Excel sheet name
SilverLoadOrder——✅Override silver load order
SilverLoadType——✅Load type (full/incremental, default='full')
SilverMaxExpectedDuration——✅Max expected duration for silver (default=1800)
Trigger——✅Trigger configuration
XmlRowPath——✅XML row path for parsing

Column/Field Properties​

Defined within the Columns or Fields array in object files.

PropertyAPIDataplatformWheel PackageDescription
SourceColumn✅✅✅Name of the attribute in source system
SourceDataType✅✅✅Datatype of the source
DataPlatformName✅✅✅Override name used in dataplatform
IsPrimaryKey✅✅✅Attribute is part of primary key (default=false)
IsNullable✅✅✅Attribute can be null (default=true for non-PK)
IsActive✅✅✅Attribute is active (default=true)
IsSilverOnly✅✅✅Attribute only generated for Silver (default=false)
IsPartition✅✅—Attribute is part of partition column (default=false)
SilverDataType✅✅✅Datatype used in Silver layer
SilverExpression✅✅✅Expression to fill Silver column
SilverConvertedColumn✅✅✅Create converted column (default=true)
SourceExpression✅✅✅Expression for Bronze layer conversion
Classification✅✅—Field classification (sensitive/restricted/internal/public)
HstgType✅✅—Type used in HSTG (historization)
DefaultValue——✅Default value for column
DateFormat——✅Date format for this column
DestinationColumn——✅Destination column name (legacy)
ConversionExpression——✅Conversion expression (legacy)
IsZOrder——✅Use column for Z-ordering (default=false)
IsDeletedIdentifier——✅Column identifies deleted records (default=false)

History Settings Properties​

Nested within HistorySettings in table configuration.

PropertyAPIDataplatformWheel PackageDescription
BronzeDeleteFilterQuery——✅Filter query for bronze delete operations
BronzeSkipDelete——✅Skip delete operations in bronze (default=false)
SourceFilterQuery——✅Filter query for source data
SilverFilterQuery——✅Filter query for silver data

Notebook Configuration Properties​

Nested within notebook configuration objects (e.g., PreBronzeNotebook).

PropertyAPIDataplatformWheel PackageDescription
Notebook——✅Notebook name to execute
Timeout——✅Notebook execution timeout in seconds (optional; default=3600 / 1 hour when omitted)
Param001 - Param003——✅Parameters to pass to notebook

Trigger Properties​

Nested within Trigger configuration.

PropertyAPIDataplatformWheel PackageDescription
Notebook——✅Notebook to trigger
NotifyWebhookUrl——✅Webhook URL for notifications
NotifyWebhookType——✅Type of webhook notification

Lakehouse Properties​

Defined within the Lakehouses array in environment configuration.

PropertyAPIDataplatformWheel PackageDescription
Lakehouse✅✅✅Name of the lakehouse
Layer✅✅✅Layer (Bronze/Silver/Gold/Meta)
Workspace✅✅✅Fabric workspace name
UseSchema✅✅—Use schema for this lakehouse (default=true)
DefaultSchema✅✅—Schema to use (default='dbo' or 'his' for history)

Table Settings Properties​

Nested within BronzeTableSettings or SilverTableSettings. These accept any valid Delta Lake table properties.

PropertyAPIDataplatformWheel PackageDescription
delta.enableChangeDataFeed✅✅—Enable change data feed
delta.autoOptimize.optimizeWrite✅✅—Enable optimize write
delta.autoOptimize.autoCompact✅✅—Enable auto compaction
delta.logRetentionDuration✅✅—Log retention duration
delta.deletedFileRetentionDuration✅✅—Deleted file retention
spark.sql.parquet.vorder.default✅✅—Enable V-Order
delta.stats.extended.collect✅✅—Collect extended statistics
delta.stats.extended.inject✅✅—Inject extended statistics
(Any Delta property)✅✅—FabricTableSettings accepts arbitrary Delta properties

Notes​

  1. API receives YAML converted to JSON in web requests and uses C# model classes to deserialize
  2. Dataplatform reads YAML files directly from disk and uses the same C# model classes
  3. Wheel Package reads YAML files at runtime in notebooks using Python dataclasses
  4. Properties marked with ✅ are actively used by that component
  5. Properties marked with — are not used by that component
  6. Table Settings can be defined at three levels (Configuration → Connection → Object) with object-level overriding connection-level, which overrides configuration-level
  7. The Dataplatform and API share the same model classes, so they support identical properties for object/field/configuration definitions
  8. The Wheel Package has additional runtime properties for orchestration, notebook execution, and data loading that are not needed during generation
  9. KeepHistory default differs between generation and runtime. The generator (API / Dataplatform) treats an omitted KeepHistory as true, so generated objects get history tables by default — this is the effective default for normal use. The runtime Wheel Package, however, defaults keephistory to false when the property is genuinely absent from the config it loads. In the normal flow this divergence is invisible because the generated YAML carries an explicit value; it only surfaces if you build a TableConfig by hand without setting KeepHistory. When in doubt, set KeepHistory explicitly.