Custom extensions
Every data platform has its unique characteristics. While many situations are similar, each company has its own specific requirements. Sometimes, it is necessary to perform actions that do not fit the standard workflow. In these scenarios, it is helpful to know that custom actions can be added to the flow.
The most logical place to implement these custom actions is in a notebook.
Between every layer, there is an option to run a custom notebook.
Pre notebook and post notebook
To run a specific notebook before loading data into a layer, define a Pre notebook on the object yaml file. In this case before running bronze:
Table:
Connection: exa-example
SourceTable: example.json
DataPlatformObjectname: example
PreBronzeNotebook: Pull_example_file
PreBronzeNotebookParams:
- Param001: SomeValue
When the action to load bronze data is activated and if a prebronzenotebook is defined, this notebook will be run, automatically the following parameters are supplied to the notebook:
- sourcetable
- sourcefilter
- bronzefolder
- keyvault
Optionally it is possible to supply 3 additional parameters (Param001, Param002, Param003). In this example there is one parameter Param001 supplied with a value of 'SomeValue'.
The notebook that is called should have a parameter cell with the following:
sourcetable=""
sourcefilter=""
bronzefolder=""
keyvault=""
param001=""
param002=""
param003=""
Works equally as the pre notebook, but is run after the loading has finished.