Deployment options

In Azure DevOps on the Pipelines page, there is a pipelineAn automated "conveyor belt" that moves data from one place to another or performs a task automatically. to deploy an environment to Microsoft FabricAn all-in-one data and analytics platform from Microsoft. Think of it as a "digital warehouse" for all your company's information.. When 'Run Pipeline' is chosen, the following parameters can be set:

Name	Description
Branch/tag	The branch to be deployed (default=main)
environments	Add one or more environments. For example -tst or multiple environments at once
Bronze / Silver / Gold	Three separate booleans selecting which layers to deploy (at least one must be selected)
Deploy Notebooks	Deploy the notebooks to Fabric (Changed/All/No)
Deploy SQL Objects	Deploy objects to the lakehouseA place where you store both "raw" data (like files) and "organized" data (like tables). It combines the best of a File Cabinet and a Database. (Changed/All/No)
Deploy Tabular Models	Deploy tabular models to PowerBI.com (Changed/All/No)
Deploy Reports	Deploy Power BI reports to PowerBI.com (Changed/All/No)
Cleanup obsolete notebooks and tables	Report or delete notebooks and lakehouse tables that no longer exist in the repository (No/Report/Delete, default No). See Cleanup of obsolete objects

Deploy options (Changed/All/No)

The four deploy parameters share the same values:

• Changed: Only artifacts touched since the previous commit (detected with a git diff) are deployed. When a shared file changes (a notebook template, reports.yaml, or model.yaml for the Gold objects), the pipeline widens the selection automatically.
• All: Everything in the category is deployed, regardless of changes.
• No: This step is skipped completely.

The metadata files (YAMLA simple way to write configurations. It's basically a list that computers can read easily.) are uploaded to the Meta lakehouse unconditionally on every deploymentThe process of "pushing a button" to make your configuration actual, working software in the cloud. — there is no separate parameter for them.

note

In the current canonical pipeline the Power BI report deployment step (pbix) is disabled (commented out), so the "Deploy Reports" parameter has no effect until that step is re-enabled.

Cleanup of obsolete objects

By default the pipeline never deletes anything in Fabric. Notebooks and tables you remove from the repository stay in the workspace until you delete them by hand. The Cleanup obsolete notebooks and tables parameter changes that:

• No: nothing is deleted or reported (default).
• Report: a dry run. Every notebook and table that would be deleted is listed as a pipeline warning, nothing is deleted.
• Delete: the obsolete objects are deleted, and every deletion is listed as a pipeline warning.

Deleting is only carried out when the environment allows it. Each dp.<env>.yaml carries a cleanup block that acts as the second lock, so an acceptance environment can allow deletion while production stays on report-only:

 cleanup:
   notebooks:
     enabled: true
     exceptfolders: [Users, Development]
   tables:
     enabled: false
     exceptschemas: [his]
     excepttables: []

When the pipeline runs with Delete and enabled is false, that category is reported instead of deleted.

Notebooks. The cleanup compares the workspace with the complete set of generated notebooks (all layers, regardless of the layer and Changed/All selections). Only notebooks inside managed folders are candidates: the folders named in the notebook.folder setting of the generated notebooks (for example Bronze, Silver, Gold, DAG, Loaders and Users) and their subfolders. Notebooks in the workspace root or in any other folder are never touched. exceptfolders removes a managed folder, or a subfolder of it, from the cleanup; the template excludes Users (personal USR_ notebooks) and Development. When the generated set is empty the step stops without deleting anything.

Tables. The lakehouse deployment already compares the generated table scripts with the tables present in each schema it deploys. With cleanup enabled the tables that exist in Fabric but no longer in the generated scripts are dropped, per selected layer. Schemas that do not occur in the generated scripts are never scanned. exceptschemas keeps whole schemas (the history schema by default), excepttables keeps tables matching a schema.table wildcard pattern such as dbo.manual_*.

warning

Dropping a table deletes its data. Fabric has no recycle bin for lakehouse tables. Run with Report first and keep tables.enabled on false in environments where a mistake would be costly.

Semantic models and Power BI reports are outside the scope of the cleanup.

The deployment pipeline in Azure DevOps will do the following:

Build

• Check the yaml files for common errors before deploying
• Generate the scripts for the objects to use for deployment

Deployment

• Copy yaml files to the Meta Lakehouse
• Deploy notebooks to the Meta, Bronze, Silver and Gold lakehouse
• Run a notebook that will create* the objects in the different lakehouses