Gold Partitioning

Gold fact tables can grow to hundreds of millions of rows, and reloading the whole table on every run quickly becomes the slowest step in the pipelineAn automated "conveyor belt" that moves data from one place to another or performs a task automatically.. Gold partitioning lets you refresh a fact table one slice at a time — replacing only the partitions present in the incoming batch and leaving the rest of the table untouched.

This is not Tabular partitioning

There are two independent "partition" concepts in EasyFabric, and it is easy to mix them up:

• Gold (lakehouseA place where you store both "raw" data (like files) and "organized" data (like tables). It combines the best of a File Cabinet and a Database.) partitioning — described on this page. A physical SYSTEMPARTITION column on the Gold Delta table, used by the appendwithdelete load type to overwrite a slice of a fact table. This governs how data is written to the lakehouse.
• Tabular (semantic-model) partitioning — how the Power BI / Fabric semantic model is sliced for refresh, using date-based partition schemes (yyyy, yyyyMM, yyyyMMdd), filters, and history annotations. See Tabular → Partitioning.

They are separate systems: a Gold table can carry a SYSTEMPARTITION column and the semantic modelThe "brain" of your data that tells Power BI how different pieces of information relate to each other. on top of it can additionally be split into import partitions. This page is only about the former.

Gold load types

The write mode is set with load_type on the LoadConfig you pass to modelloader. LoadConfig is a runtime parameter bag — you construct it in the Gold notebook, it is not read from model.yaml.

load_type	Behaviour
full (default)	Overwrites the entire target table with the incoming DataFrame.
append	Appends the incoming rows. Nothing is deleted — removing or replacing rows is the calling notebook's responsibility.
appendwithdelete	Partition-scoped overwrite: atomically replaces the partitions present in the batch in a single Delta transaction. This is the incremental-refresh primitive for fact tables.

merge exists in the LoadType enum but is not wired up for the Gold layer — requesting it raises Load type ... is not implemented. Use appendwithdelete for incremental fact loads.

The SYSTEMPARTITION column

Gold partitioning is column-based. Each row of a fact table carries a partition value in a column named SYSTEMPARTITION. The value is a discrete string that identifies the slice the row belongs to — for example a load date, a year-month, or a source batch id. Rows that share a SYSTEMPARTITION value form one partition.

Partition handling applies to fact tables only (TableType: Fact). Dimension tables are keyed on their surrogate/business key instead and have no partition column.

Producing the column

The SYSTEMPARTITION column should carry a real partition value on the fact source you hand to modelloader. It flows down from ingestion: mark the source attribute(s) that make up the partition with IsPartition in the object configuration (see the IsPartition field). If the column is absent from the source, the loader synthesises it with the placeholder value ~not set~ — which puts the entire batch into a single partition, so every appendwithdelete load replaces that same one slice instead of giving you real partition-scoped refreshes. For genuine incremental fact loads, make sure each row carries its own SYSTEMPARTITION value. The replace predicate keys on the SYSTEMPARTITION name specifically, so keep the default partition column name.

How appendwithdelete works

When you load a fact table with load_type="appendwithdelete", the loader replaces the affected partitions in a single atomicAn operation that either fully happens or not at all — never half-finished. If it fails partway through, it's as if it never started, so your data is never left in a broken, in-between state. Delta transaction:

1. It reads the distinct SYSTEMPARTITION values present in the incoming DataFrame.
2. It writes the DataFrame in a single overwrite commit, scoped to exactly the affected partitions. On a default (unpartitioned) table it uses Delta's replaceWhere with the collected partition values; on a table that is physically partitioned by SYSTEMPARTITION it uses Delta's dynamic partition overwrite (partitionOverwriteMode=dynamic) instead. The two mechanisms are never combined in one write — Fabric forbids that — and both remove the existing rows in the affected partitions and write the new rows as one commit.

Because it is one transaction, there is no window in which the partitions are missing: if the job fails partway through, the target is left exactly as it was, and re-running the same batch produces the same result. Two guardrails protect the write:

• NULL partition values are rejected. If any row's SYSTEMPARTITION is NULL the load fails fast rather than silently leaving orphaned rows behind.
• An empty batch is a no-op. If the source carries no partition values the loader logs that there is nothing to replace and skips the write.

The net effect is that every partition present in the batch is fully replaced, while partitions absent from the batch are left exactly as they were. There is no per-row matching — the unit of replacement is the whole partition.

from easyfabric import load_data_gold, Model, LoadConfig

mdl = Model.from_yaml_file("Files/Models/DM/model.yaml")

# df_sales holds only the partitions you want to refresh, e.g. the last 3 load dates.
# Each row carries a SYSTEMPARTITION value identifying its slice.
load_config = LoadConfig.from_dict({
    "model_object_name": "F_Sales",
    "load_type": "appendwithdelete",
})

load_data_gold.modelloader(data_frame=df_sales, load_config=load_config, model_config=mdl)

Worked example

Suppose F_Sales already holds partitions 2026-07-19, 2026-07-20, and 2026-07-21, and your source query returns rows for 2026-07-21 and 2026-07-22:

Partition	Before	Incoming batch	After
2026-07-19	1.2M rows	—	1.2M rows (untouched)
2026-07-20	1.1M rows	—	1.1M rows (untouched)
2026-07-21	0.9M rows	1.0M rows	1.0M rows (replaced)
2026-07-22	—	1.3M rows	1.3M rows (added)

The batch's two partitions are atomically replaced; the two older partitions are never read or rewritten.

Physical partitioning (opt-in)

By default the Gold transformation table is created as a plain Delta table: SYSTEMPARTITION is an ordinary column and the partition-replace is expressed as a replaceWhere predicate. You can opt in to physically partitioning the transformation table by SYSTEMPARTITION per fact table in model.yaml:

- Table:
  Name: Sales
  TableType: Fact
  PartitionTransformationTable: true

With PartitionTransformationTable: true the generated create script adds PARTITIONED BY (SYSTEMPARTITION), and the appendwithdelete loader detects the physical partitioning at runtime and switches to Delta's dynamic partition overwrite: the write atomically replaces exactly the partitions present in the batch, with no hand-built predicate. Reads that filter on SYSTEMPARTITION also benefit from partition pruning. Everything else is unchanged — the load stays a single atomic transaction, a mid-operation failure leaves the target untouched, NULL partition values are rejected, and an empty batch is a no-op.

The flag only applies to fact tables (only facts have a transformation table) and requires the partition field; combining it with HasPartitionField: false — or setting it on a dimension — fails validation. Dimension tables are never partitioned.

When to opt in

Physical partitioning pays off when partition-filtered reads matter and the partition values are low-cardinality — a year, a month, a region, a source batch tag. Each distinct SYSTEMPARTITION value becomes a physical directory: high-cardinality values (e.g. a per-day value over many years, or worse a per-run id) create many small files, which hurts read performance and V-Order effectiveness in Fabric. If your partition values are high-cardinality, keep the default unpartitioned layout — appendwithdelete is equally atomic there via replaceWhere. The generated table keeps delta.autoOptimize.optimizeWrite enabled either way.

Migrating an existing table

Delta cannot ALTER an existing table into a partitioned layout — the physical file layout differs, and the generated create scripts only apply to tables that do not exist yet. Enabling PartitionTransformationTable on an already-deployed fact therefore requires recreating its transformation table once:

1. Set PartitionTransformationTable: true in model.yaml and regenerate, so the new create script carries PARTITIONED BY (SYSTEMPARTITION).

2. In the Gold lakehouse, rewrite the existing table. Either drop and recreate it (run the generated create script, then reload from Silver), or keep the data with a CTAS-style rewrite:

   CREATE TABLE lh_gold.dbo.t_sales_new
   PARTITIONED BY (SYSTEMPARTITION)
   AS SELECT * FROM lh_gold.dbo.t_sales;

   then drop the old table and rename the new one into place.

3. Run the next appendwithdelete load as normal — the loader picks up the partitioned layout automatically.

The reverse migration (turning partitioning off again) is the same drop-and-recreate in the other direction. Plan the rewrite in a maintenance window: the table is unavailable between the drop and the reload.

Incremental fact loads

There is no "refresh the last N partitions" loop inside the loader — the refresh scope is simply whatever your source query returns. To run an incremental fact load you:

1. Restrict the source query in the Gold notebook to the partitions you want to refresh (for example, the recent load dates or the open period).
2. Ensure each row carries its SYSTEMPARTITION value.
3. Load with appendwithdelete.

Because the replace is scoped to exactly the partitions in the batch and runs as a single atomic transaction, this is safe to re-run: reloading the same partitions produces the same result, and widening the query to include older partitions re-swaps those too. A first load needs no special handling — into an empty table there is nothing to replace, so the batch is simply written; keep appendwithdelete. Reach for full only when you want to rebuild the entire table and drop partitions that are no longer in the source.

Relationship to the semantic model

The SYSTEMPARTITION column and the partition-replace behaviour described here are purely about how the Gold lakehouse Delta table is written. Slicing the Power BI / Fabric semantic model on top of that table into import partitions — by year, month, or filter, with history partitions that are skipped on incremental refreshes — is configured separately via the Partitions: block on the Tabular table. See Tabular → Partitioning for that.