Skip to main content

Row-count sweep

count_by_folder walks the table metadata in a folder and records the current row count of every table across the requested lakehouse layers and schemas. count_by_model does the same for the Gold tables defined by a model. Both are inventory and trending functions: a lightweight, read-only sweep that runs as the final stage of a pipeline and makes zero changes to the hot path.

Overview​

The sweep exists to answer "how many rows are in each table right now, per run" without touching the loading logic. It is the b job in a pipeline — the observational tail that runs after all loads (the a jobs) have completed:

  • It only ever reads — it counts rows, it never optimizes, vacuums, or writes data. Nothing about it can affect a load.
  • It runs last, so a slow or failing sweep cannot delay or break the tables it is measuring. Counting is off the critical path.
  • It emits one EVENT log row per table, in the RowCount category. The row's objectname is the physical table name (the connection-prefixed name as it appears in the lakehouse, e.g. int_customers) and its log context holds every count as {"objectname": ..., "row_counts": [{"layer": ..., "dbo": ..., "his": ...}]} — one array element per layer with its schema counts inlined — carrying the run identifiers so downstream analytics can join sweeps to loads and to each other.

Because it is metadata-driven, adding a table to a folder automatically brings it into the sweep — no per-table wiring.

Bronze and Silver: count_by_folder​

Bronze and Silver tables are folder-defined, so count_by_folder enumerates them from the object metadata. For every table resolved from the folder, the function:

  1. Resolves the full name of each requested layer/schema variant (e.g. Bronze.dbo.customers and Bronze.his.customers).
  2. Reads each row count — using Spark when a session is active, and delta-rs otherwise (see Backend).
  3. Emits one EVENT row for the table: objectname is the physical table name and the log context is {"objectname": ..., "row_counts": [{"layer": ..., "dbo": ..., "his": ...}]} for that table.

Counts run concurrently across tables (max_workers, default 10). A table that cannot be resolved or read is logged and skipped — it never aborts the sweep — so one bad table does not cost you the counts for all the others.

dbo and his schemas​

By default the sweep counts both the dbo schema and the history schema (his). But dbo does not mean the same thing in every layer: in Bronze it is the landing zone — the raw materialization of what was ingested from source this run — while in Silver it is the record, the type-converted current-state table downstream consumers read. The two dbo counts therefore measure different populations and are expected to differ; read each in its layer's terms rather than trending them as a single series.

The his count is usually the interesting one for change-tracked tables: it is the full versioned row set behind the current snapshot. History is only counted for tables with keephistory enabled; tables without it contribute a dbo count only.

Narrow the sweep with the schemas parameter — schemas=["his"] for history-only, schemas=["dbo"] for current-state-only. Each schema counted for a table appears under its layer in that table's log context.

The event row and the downstream join​

Each table's row lands in the RowCount log category with log type EVENT. Filter on log_category = 'RowCount' to isolate the sweep, read objectname for the physical table name, and pull the counts straight from the JSON context. The context is a flat, T-SQL-friendly shape: the table name under objectname and a row_counts array with one element per layer, each element carrying the layer plus its dbo / his counts as sibling keys:

{
  "objectname": "int_customers",
  "row_counts": [
    { "layer": "Bronze", "dbo": 97466, "his": 152387396 },
    { "layer": "Silver", "dbo": 322493, "his": 152387396 }
  ]
}

Here the two dbo values are not the same measure: Bronze.dbo (97 466) is what landed from source this run, while Silver.dbo (322 493) is the accumulated current record — the same table name in two layers, two different populations.

Because row_counts is an array of layer records, it reads cleanly from T-SQL with OPENJSON — one row per layer, columns for the schema counts:

SELECT
    JSON_VALUE(log_context, '$.objectname') AS objectname,
    rc.layer,
    rc.dbo,
    rc.his
FROM logs
CROSS APPLY OPENJSON(log_context, '$.row_counts')
    WITH (
        layer varchar(20) '$.layer',
        dbo   bigint       '$.dbo',
        his   bigint       '$.his'
    ) AS rc
WHERE log_category = 'RowCount';

The sweep reads the logging run identifiers (run_id, activity_id, parent_run_id) from the logging config globals that set_run_id / init_logging populate. If logging has not been initialized yet, the sweep bootstraps it (log_source="Maintenance"), so it is safe to call standalone.

Because every row carries the run_id, the sweep output is joinable on run_id downstream. That is what turns per-run snapshots into trends: join a run's sweep to that same run's load telemetry to see, per table, what was loaded versus what is now on disk.

Full-load reconciliation​

For a full load, the Bronze.dbo sweep count is a direct reconciliation check against the load: the landing zone should hold exactly the rows the load read from source, i.e.

bronze_dbo_count == source_in

Silver.dbo reconciles differently — it is the record after type conversion (and, for a merge load, after accumulation across runs), so it is checked against the materialized record, not against source_in. A mismatch on a full-load Bronze table is a signal — rows were dropped, duplicated, or filtered somewhere between source and the landing zone — and the join on run_id is what lets you spot it per run.

Usage​

from easyfabric.maintenance import count_by_folder

counts = count_by_folder(
    "Files/Objects/Sales",
    layers=["Bronze", "Silver"],
)

# counts == [
#   {"object": "int_customers", "layer": "Bronze", "schema": "dbo", "table": "Bronze.dbo.int_customers", "row_count": 1042},
#   {"object": "int_customers", "layer": "Bronze", "schema": "his", "table": "Bronze.his.int_customers", "row_count": 5310},
#   {"object": "int_customers", "layer": "Silver", "schema": "dbo", "table": "Silver.dbo.int_customers", "row_count": 1042},
#   ...
# ]

The returned list is one entry per counted (layer, schema) for programmatic use; the log collapses each table into a single event row with all its counts in context. Run it as the last cell of the orchestration notebook, after all loads have finished, so the counts reflect the fully materialized state of the run.

Gold: count_by_model​

Gold tables are model-based, not folder-based — they are defined in a model rather than object metadata — so count_by_folder rejects Gold and gold is swept with count_by_model instead. It enumerates the Gold tables of a model and counts each one with the same backend, event row, and record shape — one EVENT row per table, objectname the physical gold table name, and the same {"objectname": ..., "row_counts": [...]} context, here with a single Gold layer element:

from easyfabric.maintenance import count_by_model

counts = count_by_model("SalesModel")

# counts == [
#   {"object": "factsales", "layer": "Gold", "schema": "dbo", "table": "Gold.dbo.factsales", "row_count": 1042},
#   ...
# ]

The log context for a Gold table mirrors the Bronze/Silver shape:

{
  "objectname": "factsales",
  "row_counts": [
    { "layer": "Gold", "dbo": 1042 }
  ]
}

Backend​

Row counting selects its engine automatically, mirroring how the logging layer persists:

  • Spark, when a Spark session is active — spark.table(name).count().
  • delta-rs otherwise — DeltaTable(...).to_pyarrow_dataset().count_rows(), which sums the row counts from each Delta data file's Parquet footer without reading any column data.

Because the delta-rs path needs no Spark session, the sweep also runs on Spark-free (pure-Python) Fabric compute. It avoids a full data scan — the counts come from file footers, not the column data — but it does read each data file's footer, so its cost scales with the number of files rather than being a single transaction-log lookup.

Importing easyfabric also quiets delta-rs's benign delta_kernel "unknown file type" WARNING (a known effect of Fabric-written Delta logs) by defaulting RUST_LOG to warn,delta_kernel=error — set at import, before any delta-rs use, so it takes effect on the first count. A RUST_LOG you set yourself is left untouched.

Parameters​

count_by_folder​

ParameterTypeDefaultDescription
table_folderstrRequiredFolder containing the YAML table metadata to sweep.
config_managerConfigManagerNoneConfig manager instance; initialized automatically when omitted.
except_folderslist[str][]Sub-folders to exclude from the sweep.
except_fileslist[str][]Individual metadata files to exclude.
layerslist[str]RequiredLayers to count, any of ["Bronze", "Silver"] (case-insensitive). Gold is rejected — use count_by_model.
schemaslist[str]["dbo", "his"]Schemas to count: dbo, his, or both. his is skipped for tables without keephistory.
max_workersint10Maximum concurrent counts.
skip_missing_tablesboolTrueSkip (log) tables that do not exist instead of raising.

count_by_model​

ParameterTypeDefaultDescription
model_namestrRequiredName of the model whose Gold tables to sweep.
config_managerConfigManagerNoneConfig manager instance; initialized automatically when omitted.
max_workersint10Maximum concurrent counts.
skip_missing_tablesboolTrueSkip (log) tables that do not exist instead of raising.

Returns: list[dict], one entry per successfully counted (layer, schema) with object, layer, schema, table, and row_count.

Raises: count_by_folder raises ValueError if layers is empty, contains Gold, or contains a value other than Bronze or Silver, or if schemas contains anything other than dbo/his.

Cost​

The sweep is deliberately cheap and can be left on:

  • No full data scan. Neither backend reads column data to count: Spark uses its own count(), and the delta-rs path sums Parquet-footer row counts. Cost scales with the number of data files (one footer read each), not with row volume — and the delta-rs path needs no Spark session at all.
  • One log row per table — every count for a table folds into a single EVENT row, so the sweep writes N rows for N tables rather than one per layer/schema variant. The work scales linearly with the number of tables, runs concurrently, and sits off the critical path at the end of the pipeline, so it adds observability without taxing the loads.