Databricks Reference Guide │ Asset Bundles · Azure DevOps CI/CD · Lakeflow Declarative Pipelines
Databricks
Advanced Topics Reference Guide
In-Depth Explanations with Real-World Examples
Topic 10 — Databricks Asset Bundles | Configure Production-Grade DABs | CI/CD using
DABs (IaC)
Topic 11 — Setup Azure DevOps Pipeline with DABs | Complete CI/CD Process
Topic 12 — Lakeflow Spark Declarative Pipelines | Code Editor | AUTO CDC | External
Target Sinks
© 2025 Databricks Reference Guide
Page
Databricks Reference Guide │ Asset Bundles · Azure DevOps CI/CD · Lakeflow Declarative Pipelines
Topic 10: Databricks Asset Bundles | Configure
Production-Grade DABs | CI/CD using DABs
(IaC)
10.1 What Are Databricks Asset Bundles (DABs)?
Databricks Asset Bundles (DABs) are the official Infrastructure-as-Code (IaC) framework for
Databricks. They allow you to define your entire Databricks project — notebooks, jobs,
pipelines, clusters, permissions, secrets, SQL warehouses, and dashboards — as YAML
configuration files that are version-controlled in Git and deployed through automated CI/CD
pipelines.
DABs replace ad-hoc manual deployments (clicking through the Databricks UI to create jobs
and clusters) with a repeatable, auditable, environment-aware deployment system. The same
bundle can be deployed to dev, staging, and production environments by simply switching the
target — all configuration differences are handled automatically.
Traditional Deployment
Databricks Asset Bundles (DABs)
Manual UI clicks to create jobs and clusters
YAML-defined resources deployed via CLI
No version control for infrastructure
Git-versioned — every change tracked and
reviewable
'Works on my machine' environment drift
Environment targets (dev/staging/prod) with
overrides
Manual copy of notebooks between
environments
Notebooks deployed automatically as part of the
bundle
No deployment audit trail
Full CI/CD pipeline history in Azure DevOps /
GitHub
Difficult to reproduce in disaster recovery
bundle deploy rebuilds entire project from scratch
Permissions configured manually per
environment
Permissions declared in YAML, applied
consistently
10.2 DAB Project Structure
© 2025 Databricks Reference Guide
Page
Databricks Reference Guide │ Asset Bundles · Azure DevOps CI/CD · Lakeflow Declarative Pipelines
A Databricks Asset Bundle is a directory containing a databricks.yml root configuration file,
environment-specific override files, notebook and Python files, and optionally SQL files and
resource templates. The directory is a Git repository that CI/CD pipelines act on.
# Typical DAB project directory structure
my_databricks_project/
├── databricks.yml
# Root bundle config — required
├── resources/
│
├── jobs/
│
│
├── etl_job.yml
# Job definition
│
│
├── ml_training_job.yml
│
│
└── reporting_job.yml
│
├── pipelines/
│
│
└── lakeflow_pipeline.yml
│
├── clusters/
│
│
└── shared_cluster.yml
│
└── permissions/
│
└── rbac.yml
├── src/
│
├── notebooks/
│
│
├── bronze_ingestion.py
│
│
├── silver_transform.py
│
│
└── gold_aggregation.py
│
├── pipelines/
│
│
└── declarative_pipeline.py
│
└── utils/
│
└── helpers.py
├── tests/
│
├── unit/
│
└── integration/
├── .github/workflows/
# OR azure-pipelines/
│
└── deploy.yml
└── README.md
10.3 The databricks.yml Root Configuration
The databricks.yml file is the heart of every DAB project. It defines the bundle name, the
Databricks workspace to deploy to, environment targets with their specific overrides, and
references to resource definition files.
Real-World Example: RetailAnalytics Data Platform
# databricks.yml — Root bundle configuration
bundle:
name: retail-analytics-platform
© 2025 Databricks Reference Guide
Page
Databricks Reference Guide │ Asset Bundles · Azure DevOps CI/CD · Lakeflow Declarative Pipelines
git:
origin_url: https://dev.azure.com/retailco/DataPlatform/_git/retail-analytics
branch: ${bundle.git.branch}
# ─── Workspace connection (resolved per target) ───────────────────────
workspace:
host: ${var.workspace_host}
root_path: /Shared/.bundle/${bundle.name}/${bundle.target}
# ─── Variables with defaults (overridden per target) ──────────────────
variables:
workspace_host:
description: Databricks workspace URL
default: https://adb-dev-1234.azuredatabricks.net
catalog:
description: Unity Catalog catalog name
default: dev_catalog
schema:
description: Default schema
default: retail_dev
cluster_node_type:
description: Worker node type
default: Standard_DS3_v2
num_workers:
description: Number of worker nodes
default: 2
notification_email:
description: Alert email for job failures
default: dev-alerts@retailco.com
# ─── Include resource definition files ────────────────────────────────
include:
- resources/jobs/*.yml
- resources/pipelines/*.yml
- resources/permissions/*.yml
# ─── Deployment targets (environments) ────────────────────────────────
targets:
dev:
mode: development
# dev mode: adds [dev username] prefix to
resources
default: true
workspace:
host: https://adb-dev-1234.azuredatabricks.net
variables:
catalog: dev_catalog
schema: retail_dev
num_workers: 1
# smaller cluster in dev = cost savings
notification_email: dev-alerts@retailco.com
staging:
mode: development
workspace:
host: https://adb-staging-5678.azuredatabricks.net
© 2025 Databricks Reference Guide
Page
Databricks Reference Guide │ Asset Bundles · Azure DevOps CI/CD · Lakeflow Declarative Pipelines
variables:
catalog: staging_catalog
schema: retail_staging
num_workers: 2
notification_email: staging-alerts@retailco.com
production:
mode: production
# prod mode: no prefix, strict permissions
workspace:
host: https://adb-prod-9999.azuredatabricks.net
variables:
catalog: prod_catalog
schema: retail_prod
num_workers: 8
# full-size production cluster
notification_email: oncall@retailco.com
permissions:
- level: CAN_MANAGE
group_name: data-platform-admins
- level: CAN_VIEW
group_name: data-consumers
10.4 Defining Jobs in YAML
Jobs are defined as YAML files in the resources/jobs/ directory. Each job file describes the full
job specification — schedule, tasks, cluster configuration, notifications, and retry behaviour.
# resources/jobs/daily_etl_job.yml
resources:
jobs:
daily_etl_pipeline:
name: retail-daily-etl-${bundle.target}
description: 'Daily ETL pipeline: Bronze ingestion → Silver transform →
Gold aggregation'
# ─── Job-level cluster configuration ───────────────────────────
job_clusters:
- job_cluster_key: etl_cluster
new_cluster:
spark_version: 15.4.x-scala2.12
node_type_id: ${var.cluster_node_type}
num_workers: ${var.num_workers}
spark_conf:
spark.databricks.delta.preview.enabled: 'true'
spark_env_vars:
CATALOG: ${var.catalog}
SCHEMA: ${var.schema}
data_security_mode: SINGLE_USER
autoscale:
min_workers: 2
© 2025 Databricks Reference Guide
Page
Databricks Reference Guide │ Asset Bundles · Azure DevOps CI/CD · Lakeflow Declarative Pipelines
max_workers: 8
# ─── Task definitions ──────────────────────────────────────────
tasks:
- task_key: bronze_ingestion
description: 'Ingest raw data from source systems into Bronze layer'
job_cluster_key: etl_cluster
notebook_task:
notebook_path: ./src/notebooks/bronze_ingestion.py
base_parameters:
catalog: ${var.catalog}
schema: ${var.schema}
env: ${bundle.target}
timeout_seconds: 3600
retry_on_timeout: true
max_retries: 2
- task_key: silver_transform
description: 'Transform and clean Bronze data into Silver layer'
depends_on:
- task_key: bronze_ingestion
job_cluster_key: etl_cluster
notebook_task:
notebook_path: ./src/notebooks/silver_transform.py
base_parameters:
catalog: ${var.catalog}
schema: ${var.schema}
- task_key: gold_aggregation
description: 'Build Gold layer fact and dimension tables'
depends_on:
- task_key: silver_transform
job_cluster_key: etl_cluster
notebook_task:
notebook_path: ./src/notebooks/gold_aggregation.py
base_parameters:
catalog: ${var.catalog}
schema: ${var.schema}
- task_key: data_quality_checks
description: 'Run Great Expectations / dbt tests on Gold tables'
depends_on:
- task_key: gold_aggregation
job_cluster_key: etl_cluster
python_wheel_task:
package_name: retailco_dq
entry_point: run_checks
parameters:
- --catalog=${var.catalog}
- --env=${bundle.target}
# ─── Job schedule ───────────────────────────────────────────────
schedule:
quartz_cron_expression: '0 0 2 * * ?'
# 2 AM daily
© 2025 Databricks Reference Guide
Page
Databricks Reference Guide │ Asset Bundles · Azure DevOps CI/CD · Lakeflow Declarative Pipelines
timezone_id: UTC
pause_status: UNPAUSED
# PAUSED in dev, UNPAUSED in
prod
# ─── Notifications ──────────────────────────────────────────────
email_notifications:
on_failure:
- ${var.notification_email}
on_success:
- ${var.notification_email}
# ─── Job-level permissions ──────────────────────────────────────
permissions:
- level: CAN_MANAGE_RUN
group_name: data-engineers
- level: CAN_VIEW
group_name: data-analysts
10.5 DABs CLI — Core Commands
The Databricks CLI (v0.200+) is the primary tool for working with DABs. Install it, authenticate,
and use the bundle subcommand for all DAB operations.
# ─── Install Databricks CLI ─────────────────────────────────────────────
# macOS / Linux:
curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh
| sh
# Windows (PowerShell):
winget install Databricks.DatabricksCLI
# Verify installation:
databricks --version
# Output: Databricks CLI v0.220.0
# ─── Authenticate ───────────────────────────────────────────────────────
# Option 1: OAuth U2M (interactive login — for local development)
databricks auth login --host https://adb-dev-1234.azuredatabricks.net
# Opens browser → complete Azure AD OAuth flow
# Option 2: Service Principal (M2M — for CI/CD pipelines)
export DATABRICKS_HOST=https://adb-prod-9999.azuredatabricks.net
export DATABRICKS_CLIENT_ID=<service-principal-client-id>
export DATABRICKS_CLIENT_SECRET=<service-principal-secret>
# ─── Core DAB Commands ──────────────────────────────────────────────────
# Validate the bundle configuration (syntax check, schema validation):
databricks bundle validate
© 2025 Databricks Reference Guide
Page
Databricks Reference Guide │ Asset Bundles · Azure DevOps CI/CD · Lakeflow Declarative Pipelines
# Deploy to the default target (dev):
databricks bundle deploy
# Deploy to a specific target:
databricks bundle deploy --target staging
databricks bundle deploy --target production
# Run a specific job after deployment:
databricks bundle run daily_etl_pipeline
databricks bundle run daily_etl_pipeline --target production
# Run a specific task within a job:
databricks bundle run daily_etl_pipeline --task bronze_ingestion
# View current deployment state:
databricks bundle summary
databricks bundle summary --target production
# Destroy all resources in a target (useful for dev cleanup):
databricks bundle destroy --target dev
databricks bundle destroy --target dev --auto-approve
10.6 Production-Grade DAB Configuration
Production deployments require additional configuration for security, reliability, and cost
management. Here are the critical additions for a production-ready bundle.
Dynamic Job Pausing by Environment
# In dev: pause scheduled jobs to avoid accidental runs on dev data
# In production: unpause to enable real schedules
# resources/jobs/daily_etl_job.yml
resources:
jobs:
daily_etl_pipeline:
schedule:
quartz_cron_expression: '0 0 2 * * ?'
timezone_id: UTC
# Use variable that resolves per target:
pause_status: ${var.schedule_pause_status}
# In databricks.yml variables section:
variables:
schedule_pause_status:
default: PAUSED
# Default: paused (safe for dev)
© 2025 Databricks Reference Guide
Page
Databricks Reference Guide │ Asset Bundles · Azure DevOps CI/CD · Lakeflow Declarative Pipelines
# In the production target override:
targets:
production:
variables:
schedule_pause_status: UNPAUSED
# Only prod runs on schedule
Secrets and Secure Variable Handling
# Store sensitive config in Databricks Secrets, not YAML:
databricks secrets create-scope --scope retail-prod-secrets
databricks secrets put --scope retail-prod-secrets --key db_password
databricks secrets put --scope retail-prod-secrets --key api_key
# Reference secrets in notebook tasks via spark_env_vars:
tasks:
- task_key: bronze_ingestion
job_cluster_key: etl_cluster
notebook_task:
notebook_path: ./src/notebooks/bronze_ingestion.py
new_cluster:
spark_env_vars:
# Reference secret — never expose value in YAML
DB_PASSWORD: '{{secrets/retail-prod-secrets/db_password}}'
API_KEY:
'{{secrets/retail-prod-secrets/api_key}}'
# In the notebook, access via dbutils:
# db_password = dbutils.secrets.get('retail-prod-secrets', 'db_password')
Artifact Management — Wheel Files and Packages
# Build and deploy custom Python packages as part of the bundle
# databricks.yml — artifact definition:
artifacts:
retailco_utils:
type: whl
build: pip wheel ./src/packages/retailco_utils -w dist/
path: ./src/packages/retailco_utils
# Reference artifact in a task:
tasks:
- task_key: data_quality_checks
python_wheel_task:
package_name: retailco_utils
entry_point: run_dq_checks
libraries:
- whl: ./dist/retailco_utils-1.0.0-py3-none-any.whl
© 2025 Databricks Reference Guide
Page
Databricks Reference Guide │ Asset Bundles · Azure DevOps CI/CD · Lakeflow Declarative Pipelines
# When you run 'databricks bundle deploy':
# 1. DABs runs 'pip wheel' to build the package
# 2. Uploads the .whl to DBFS / workspace files
# 3. References the uploaded path in the job definition
# 4. Job installs the wheel at runtime before running the task
Production DAB Best Practices
1. Never hardcode workspace hosts, catalog names, or credentials in YAML — use variables.
2. Use mode: development for dev/staging (auto-prefixes resources, prevents collision).
3. Pause schedules in all non-production targets — prevent dev runs on production data.
4. Store all secrets in Databricks Secret Scopes — reference via {{secrets/scope/key}}.
5. Pin Spark versions explicitly — don't use 'latest' in production job clusters.
6. Set retry policies and timeouts on every task — production jobs must be resilient.
7. Define permissions in YAML — don't rely on manual UI permission grants.
8. Tag all jobs and clusters with environment and team metadata for cost allocation.
© 2025 Databricks Reference Guide
Page
Databricks Reference Guide │ Asset Bundles · Azure DevOps CI/CD · Lakeflow Declarative Pipelines
Topic 11: Setup Azure DevOps Pipeline with
Databricks Asset Bundles (DABs) | Complete
CI/CD Process
11.1 Architecture Overview — DABs + Azure DevOps
A complete CI/CD pipeline for Databricks uses Azure DevOps Pipelines to automate testing,
validation, and deployment of DABs across environments. The pipeline is triggered by Git
events (pull requests, merges to main), runs automated tests and bundle validation, and
deploys to the appropriate Databricks workspace.
CI/CD Stage
Trigger
Actions
Target
PR Validation
Every pull request to any
branch
Unit tests, bundle validate,
syntax checks
Dev
workspace
(ephemeral)
Dev Deploy
Merge to develop branch
bundle deploy, integration
tests, bundle run
Dev
workspace
Staging Deploy
Merge to staging branch
bundle deploy, smoke tests,
approval gate
Staging
workspace
Production Deploy
Merge to main + manual
approval
bundle deploy, bundle run (if
needed)
Production
workspace
11.2 Azure DevOps Setup — Prerequisites
Step 1: Create a Service Principal in Azure AD
# Create Service Principal for Databricks access
# Run in Azure Cloud Shell or local az CLI:
# Login to Azure:
az login
# Create the Service Principal:
az ad sp create-for-rbac \
--name 'databricks-cicd-sp' \
--role Contributor \
© 2025 Databricks Reference Guide
Page
Databricks Reference Guide │ Asset Bundles · Azure DevOps CI/CD · Lakeflow Declarative Pipelines
--scopes /subscriptions/<subscription-id>/resourceGroups/<resource-group>
# Output — save these values:
# {
#
'appId':
'<client-id>',
#
'password':
'<client-secret>',
#
'tenant':
'<tenant-id>'
# }
# Grant the Service Principal access to the Databricks workspace:
# Databricks workspace → Settings → Identity and Access → Service Principals
# → Add Service Principal → databricks-cicd-sp → Admin role
# Or via Databricks CLI:
databricks service-principals create \
--display-name 'databricks-cicd-sp' \
--application-id <client-id>
Step 2: Configure Azure DevOps Variable Groups
# In Azure DevOps: Pipelines → Library → Variable Groups
# Create separate variable groups per environment:
# Variable Group: databricks-dev
# Variables:
#
DATABRICKS_HOST:
https://adb-dev-1234.azuredatabricks.net
#
DATABRICKS_CLIENT_ID:
<dev-service-principal-client-id>
#
DATABRICKS_CLIENT_SECRET: <dev-sp-secret> [mark as secret ⚿]
#
DATABRICKS_TENANT_ID:
<azure-tenant-id>
#
BUNDLE_TARGET:
dev
# Variable Group: databricks-staging
# Variables:
#
DATABRICKS_HOST:
https://adb-staging-5678.azuredatabricks.net
#
DATABRICKS_CLIENT_ID:
<staging-sp-client-id>
#
DATABRICKS_CLIENT_SECRET: <staging-sp-secret> [secret ⚿]
#
BUNDLE_TARGET:
staging
# Variable Group: databricks-production
# Variables:
#
DATABRICKS_HOST:
https://adb-prod-9999.azuredatabricks.net
#
DATABRICKS_CLIENT_ID:
<prod-sp-client-id>
#
DATABRICKS_CLIENT_SECRET: <prod-sp-secret> [secret ⚿]
#
BUNDLE_TARGET:
production
# Link Variable Groups to the pipeline in azure-pipelines.yml
11.3 Complete Azure DevOps Pipeline YAML
© 2025 Databricks Reference Guide
Page
Databricks Reference Guide │ Asset Bundles · Azure DevOps CI/CD · Lakeflow Declarative Pipelines
The azure-pipelines.yml file defines the full CI/CD pipeline — from pull request validation
through production deployment. This file lives in the root of the DAB Git repository.
azure-pipelines.yml — Full Production Pipeline
# azure-pipelines.yml
# Full CI/CD pipeline: PR Validation → Dev → Staging → Production
trigger:
branches:
include:
- main
- develop
- staging
paths:
include:
- 'src/**'
- 'resources/**'
- 'databricks.yml'
pr:
branches:
include:
- main
- develop
# ─── Reusable variable templates ─────────────────────────────────────
variables:
PYTHON_VERSION: '3.11'
DATABRICKS_CLI_VERSION: '0.220.0'
# ═══════════════════════════════════════════════════════════════════════
# STAGE 1: PR Validation (runs on all Pull Requests)
# ═══════════════════════════════════════════════════════════════════════
stages:
- stage: Validate
displayName: 'Validate — Unit Tests & Bundle Check'
condition: eq(variables['Build.Reason'], 'PullRequest')
jobs:
- job: UnitTests
displayName: 'Run Unit Tests'
pool:
vmImage: ubuntu-latest
steps:
- task: UsePythonVersion@0
inputs:
versionSpec: $(PYTHON_VERSION)
- script: |
pip install pytest pytest-cov pyspark delta-spark
pip install -r requirements.txt
displayName: 'Install dependencies'
© 2025 Databricks Reference Guide
Page
Databricks Reference Guide │ Asset Bundles · Azure DevOps CI/CD · Lakeflow Declarative Pipelines
- script: |
pytest tests/unit/ \
--tb=short \
--cov=src \
--cov-report=xml:coverage.xml \
--junit-xml=test-results.xml
displayName: 'Run unit tests'
- task: PublishTestResults@2
inputs:
testResultsFormat: JUnit
testResultsFiles: test-results.xml
condition: always()
- task: PublishCodeCoverageResults@1
inputs:
codeCoverageTool: Cobertura
summaryFileLocation: coverage.xml
- job: BundleValidate
displayName: 'Validate DAB Bundle'
pool:
vmImage: ubuntu-latest
variables:
- group: databricks-dev
steps:
- script: |
curl -fsSL https://raw.githubusercontent.com/databricks/setupcli/main/install.sh | sh
databricks --version
displayName: 'Install Databricks CLI'
- script: |
databricks bundle validate --target dev
displayName: 'Validate bundle configuration'
env:
DATABRICKS_HOST:
$(DATABRICKS_HOST)
DATABRICKS_CLIENT_ID:
$(DATABRICKS_CLIENT_ID)
DATABRICKS_CLIENT_SECRET: $(DATABRICKS_CLIENT_SECRET)
DATABRICKS_TENANT_ID:
$(DATABRICKS_TENANT_ID)
# ═══════════════════════════════════════════════════════════════════════
# STAGE 2: Deploy to Dev
# ═══════════════════════════════════════════════════════════════════════
- stage: DeployDev
displayName: 'Deploy → Development'
dependsOn: []
condition: |
and(
ne(variables['Build.Reason'], 'PullRequest'),
eq(variables['Build.SourceBranchName'], 'develop')
)
variables:
© 2025 Databricks Reference Guide
Page
Databricks Reference Guide │ Asset Bundles · Azure DevOps CI/CD · Lakeflow Declarative Pipelines
- group: databricks-dev
jobs:
- deployment: DeployToDev
displayName: 'Deploy DAB to Dev workspace'
environment: databricks-dev
pool:
vmImage: ubuntu-latest
strategy:
runOnce:
deploy:
steps:
- checkout: self
- script: |
curl -fsSL
https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh
displayName: 'Install Databricks CLI'
- script: |
databricks bundle deploy --target dev
displayName: 'Bundle deploy to Dev'
env:
DATABRICKS_HOST:
$(DATABRICKS_HOST)
DATABRICKS_CLIENT_ID:
$(DATABRICKS_CLIENT_ID)
DATABRICKS_CLIENT_SECRET: $(DATABRICKS_CLIENT_SECRET)
DATABRICKS_TENANT_ID:
$(DATABRICKS_TENANT_ID)
- script: |
databricks bundle run daily_etl_pipeline --target dev
echo 'Dev integration test run triggered'
displayName: 'Run integration smoke test'
env:
DATABRICKS_HOST:
$(DATABRICKS_HOST)
DATABRICKS_CLIENT_ID:
$(DATABRICKS_CLIENT_ID)
DATABRICKS_CLIENT_SECRET: $(DATABRICKS_CLIENT_SECRET)
DATABRICKS_TENANT_ID:
$(DATABRICKS_TENANT_ID)
# ═══════════════════════════════════════════════════════════════════════
# STAGE 3: Deploy to Staging (with approval gate)
# ═══════════════════════════════════════════════════════════════════════
- stage: DeployStaging
displayName: 'Deploy → Staging'
dependsOn: DeployDev
condition: |
and(
succeeded('DeployDev'),
eq(variables['Build.SourceBranchName'], 'staging')
)
variables:
- group: databricks-staging
jobs:
- deployment: DeployToStaging
displayName: 'Deploy DAB to Staging workspace'
environment: databricks-staging
# Approval gate configured here in ADO
pool:
© 2025 Databricks Reference Guide
Page
Databricks Reference Guide │ Asset Bundles · Azure DevOps CI/CD · Lakeflow Declarative Pipelines
vmImage: ubuntu-latest
strategy:
runOnce:
deploy:
steps:
- checkout: self
- script: |
curl -fsSL
https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh
databricks bundle deploy --target staging
displayName: 'Bundle deploy to Staging'
env:
DATABRICKS_HOST:
$(DATABRICKS_HOST)
DATABRICKS_CLIENT_ID:
$(DATABRICKS_CLIENT_ID)
DATABRICKS_CLIENT_SECRET: $(DATABRICKS_CLIENT_SECRET)
DATABRICKS_TENANT_ID:
$(DATABRICKS_TENANT_ID)
# ═══════════════════════════════════════════════════════════════════════
# STAGE 4: Deploy to Production (manual approval required)
# ═══════════════════════════════════════════════════════════════════════
- stage: DeployProduction
displayName: 'Deploy → Production'
dependsOn: DeployStaging
condition: |
and(
succeeded('DeployStaging'),
eq(variables['Build.SourceBranchName'], 'main')
)
variables:
- group: databricks-production
jobs:
- deployment: DeployToProduction
displayName: 'Deploy DAB to Production workspace'
environment: databricks-production # Manual approval gate in ADO
Environments
pool:
vmImage: ubuntu-latest
strategy:
runOnce:
deploy:
steps:
- checkout: self
- script: |
curl -fsSL
https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh
displayName: 'Install Databricks CLI'
- script: |
echo 'Deploying bundle to Production workspace'
databricks bundle deploy --target production
echo 'Production deployment complete'
displayName: 'Bundle deploy to Production'
env:
© 2025 Databricks Reference Guide
Page
Databricks Reference Guide │ Asset Bundles · Azure DevOps CI/CD · Lakeflow Declarative Pipelines
DATABRICKS_HOST:
$(DATABRICKS_HOST)
DATABRICKS_CLIENT_ID:
$(DATABRICKS_CLIENT_ID)
DATABRICKS_CLIENT_SECRET: $(DATABRICKS_CLIENT_SECRET)
DATABRICKS_TENANT_ID:
$(DATABRICKS_TENANT_ID)
- script: |
echo 'Production deployment summary:'
databricks bundle summary --target production
displayName: 'Show deployment summary'
env:
DATABRICKS_HOST:
$(DATABRICKS_HOST)
DATABRICKS_CLIENT_ID:
$(DATABRICKS_CLIENT_ID)
DATABRICKS_CLIENT_SECRET: $(DATABRICKS_CLIENT_SECRET)
DATABRICKS_TENANT_ID:
$(DATABRICKS_TENANT_ID)
11.4 Configuring Azure DevOps Environments and Approval
Gates
Azure DevOps Environments provide the mechanism for manual approval gates before
production deployments. Configure them in Azure DevOps under Pipelines → Environments.
# Azure DevOps Environment Setup (done in ADO UI):
# 1. Navigate to: Pipelines → Environments → New environment
#
Name: databricks-dev
| Resource: None
#
Name: databricks-staging | Resource: None
#
Name: databricks-production | Resource: None
# 2. Configure approval on databricks-production:
#
databricks-production → Approvals and checks → Add check
#
→ Approvals → Approvers: [Lead Data Engineer, Platform Architect]
#
→ Timeout: 24 hours
#
→ Allow approvers to approve their own runs: No
# 3. Configure branch policy on databricks-production:
#
→ Branch control → Allowed branches: refs/heads/main
#
This prevents deploying to production from feature branches
# 4. Add business hours deployment window (optional):
#
→ Business hours check → Mon-Fri 9AM-4PM UTC
#
→ Prevents production deployments outside business hours
# When the pipeline reaches the Production stage:
# → Pipeline pauses and emails the approvers
# → Approvers review the change in ADO and approve or reject
# → On approval: production deployment proceeds
# → On rejection or timeout: pipeline fails safely
© 2025 Databricks Reference Guide
Page
Databricks Reference Guide │ Asset Bundles · Azure DevOps CI/CD · Lakeflow Declarative Pipelines
11.5 Branching Strategy for DABs CI/CD
A robust branching strategy prevents accidental production deployments and ensures every
change is reviewed and tested before reaching production.
Branch
Purpose
Deploys To
Protection
feature/*
Individual developer work on a
feature or fix
Nothing (runs PR
validation)
PR required to
merge to
develop
develop
Integration branch — all features
merged here
Dev workspace
(auto)
PR required, CI
must pass
staging
Pre-production validation branch
Staging workspace
(auto)
PR from develop
only, CI must
pass
main
Production branch — source of truth
Production (manual
approval)
PR from staging
only, 2
reviewers, CI
must pass
# Developer workflow:
# 1. Create feature branch from develop:
git checkout develop
git pull origin develop
git checkout -b feature/add-customer-segmentation
# 2. Make changes to notebooks, YAML, Python code
# 3. Test locally using dev target:
databricks bundle validate --target dev
databricks bundle deploy --target dev
databricks bundle run daily_etl_pipeline --target dev
# 4. Push and create PR → ADO runs PR validation pipeline
git push origin feature/add-customer-segmentation
# Create PR: feature/add-customer-segmentation → develop
# ADO automatically runs: unit tests + bundle validate
# 5. PR approved and merged to develop:
# ADO automatically deploys to dev workspace
# 6. When ready for staging — create PR: develop → staging
# ADO auto-deploys to staging after merge
# 7. When ready for production — create PR: staging → main
© 2025 Databricks Reference Guide
Page
Databricks Reference Guide │ Asset Bundles · Azure DevOps CI/CD · Lakeflow Declarative Pipelines
# ADO pauses at production stage → sends approval request
# Lead engineer approves → production deployment runs
Complete CI/CD Outcome — RetailCo Before and After DABs + Azure DevOps
Before DABs CI/CD:
• Deployments: manual Databricks UI clicks, 45-90 minutes per deployment
• Errors: 1-2 production incidents per month from missed configuration steps
• Environment drift: dev and prod jobs had different cluster sizes and libraries
• No version control for job/cluster definitions
After DABs + Azure DevOps Pipeline:
• PR to production: automated, fully tested, < 8 minutes pipeline runtime
• Zero manual UI clicks required for any deployment
• Environment differences handled entirely by variable groups
• Full Git history of every infrastructure change with author and PR link
• Production incidents from deployments: reduced from 2/month to 0 in 6 months
© 2025 Databricks Reference Guide
Page
Databricks Reference Guide │ Asset Bundles · Azure DevOps CI/CD · Lakeflow Declarative Pipelines
Topic 12: Lakeflow Spark Declarative Pipelines |
New Pipeline Code Editor | AUTO CDC | External
Target Sinks
12.1 What Are Lakeflow Spark Declarative Pipelines?
Lakeflow Spark Declarative Pipelines (formerly known as Delta Live Tables / DLT) is Databricks'
managed ETL framework for building reliable, scalable data pipelines using a declarative SQL
or Python API. Instead of writing imperative Spark code that explicitly manages execution order,
checkpoints, retries, and schema evolution, you declare WHAT your data should look like and
Lakeflow figures out HOW to build and maintain it.
In 2024-2025, Databricks significantly enhanced Lakeflow with a new Pipeline Code Editor
(replacing the legacy notebook interface), richer AUTO CDC capabilities, and External Target
Sinks for writing pipeline output to destinations outside the Lakehouse.
Traditional Spark ETL
Lakeflow Declarative Pipelines
Manually manage execution order of
transformations
Lakeflow builds and manages the DAG
automatically
Write checkpoint management for streaming
Built-in checkpointing — no manual management
Handle schema evolution with ALTER TABLE
scripts
Automatic schema evolution with configurable
policies
Retry logic written in application code
Configurable retry and error handling at pipeline
level
Data quality checks as separate scripts
Built-in expectations (data quality constraints)
CDC implemented with complex MERGE SQL
AUTO CDC with one APPLY CHANGES INTO
declaration
No lineage tracking
Full lineage visible in Pipeline Graph UI
12.2 The New Pipeline Code Editor
Databricks introduced a new Pipeline Code Editor in 2024 that replaces the older notebookbased pipeline authoring experience. The new editor provides a dedicated, purpose-built
interface for writing and testing Lakeflow pipeline code — with real-time syntax validation,
© 2025 Databricks Reference Guide
Page
Databricks Reference Guide │ Asset Bundles · Azure DevOps CI/CD · Lakeflow Declarative Pipelines
pipeline graph preview, integrated debugging, and table lineage visualisation without needing to
run the full pipeline.
Key Features of the New Code Editor
•
Split-pane view: Write pipeline code on the left, see the live pipeline graph on the right.
•
Inline table preview: Click any table in the pipeline graph to preview its schema and
sample data.
•
Real-time validation: Syntax errors and unresolved table references are highlighted as
you type.
•
Multi-file support: Pipeline code can be split across multiple Python or SQL files in the
same workspace folder.
•
Run mode selection: Run the full pipeline, a single table update, or a dry-run validation
from the editor.
•
Integrated expectations panel: View data quality metrics for each table without leaving
the editor.
Creating a Pipeline in the New Code Editor
# Navigate to: Databricks workspace → Lakeflow → Pipelines → Create Pipeline
# Pipeline Settings:
# Name:
retail-lakehouse-pipeline
# Pipeline mode:
Triggered (or Continuous for streaming)
# Storage location: /pipelines/retail-lakehouse
# Target catalog:
retailco_prod
# Target schema:
retail_pipeline
# Cluster:
Auto-select (or pin to specific cluster policy)
# In the new Code Editor, create files directly:
# File 1: bronze_layer.py
# File 2: silver_layer.py
# File 3: gold_layer.py
# File 4: expectations.py
# All files in the pipeline folder are automatically included
# No need to configure notebook paths separately
12.3 Declarative Pipeline Code — Bronze to Gold
Scenario: ShopStream — a streaming e-commerce company building a complete Lakeflow
pipeline from raw Kafka events to curated Gold analytics tables.
© 2025 Databricks Reference Guide
Page
Databricks Reference Guide │ Asset Bundles · Azure DevOps CI/CD · Lakeflow Declarative Pipelines
bronze_layer.py — Raw Ingestion with Expectations
# bronze_layer.py — Ingest raw data into Bronze layer
import dlt
from pyspark.sql import functions as F
from pyspark.sql.types import *
# ─── Bronze: Raw order events from Kafka ───────────────────────────────
@dlt.table(
name
= 'bronze_order_events',
comment
= 'Raw order events from Kafka. Append-only, no
transformations.',
table_properties = {
'quality': 'bronze',
'pipelines.autoOptimize.zOrderCols': 'event_timestamp'
}
)
@dlt.expect('valid_order_id',
'order_id IS NOT NULL')
@dlt.expect('valid_timestamp', 'event_timestamp IS NOT NULL')
@dlt.expect_or_drop('valid_amount', 'order_amount > 0')
def bronze_order_events():
return (
spark.readStream
.format('kafka')
.option('kafka.bootstrap.servers', 'pkc-xxx.confluent.cloud:9092')
.option('subscribe', 'shopstream-order-events')
.option('startingOffsets', 'latest')
.option('kafka.security.protocol', 'SASL_SSL')
.option('kafka.sasl.mechanism', 'PLAIN')
.load()
.select(
F.col('key').cast('string').alias('event_key'),
F.from_json(
F.col('value').cast('string'),
StructType([
StructField('order_id',
StringType()),
StructField('customer_id',
StringType()),
StructField('product_id',
StringType()),
StructField('order_amount',
DoubleType()),
StructField('event_type',
StringType()),
StructField('event_timestamp', TimestampType()),
])
).alias('data'),
F.col('timestamp').alias('kafka_timestamp')
)
.select('event_key', 'data.*', 'kafka_timestamp')
)
# ─── Bronze: Raw customer profiles from Auto Loader ────────────────────
@dlt.table(
name
= 'bronze_customer_profiles',
comment = 'Customer profile snapshots from CRM system via ADLS'
)
© 2025 Databricks Reference Guide
Page
Databricks Reference Guide │ Asset Bundles · Azure DevOps CI/CD · Lakeflow Declarative Pipelines
@dlt.expect_or_drop('valid_customer', 'customer_id IS NOT NULL AND email IS NOT
NULL')
def bronze_customer_profiles():
return (
spark.readStream
.format('cloudFiles')
.option('cloudFiles.format', 'json')
.option('cloudFiles.inferColumnTypes', 'true')
.option('cloudFiles.schemaHints',
'customer_id STRING, email STRING, tier STRING, region STRING')
.load('abfss://raw@shopstreamstorage.dfs.core.windows.net/crm-exports/')
)
silver_layer.py — Cleaned and Enriched Data
# silver_layer.py — Transform and clean Bronze data into Silver
import dlt
from pyspark.sql import functions as F
# ─── Silver: Cleaned and deduplicated order events ─────────────────────
@dlt.table(
name
= 'silver_orders',
comment = 'Cleaned, deduplicated order events with standardised fields'
)
@dlt.expect_all_or_drop({
'valid_order_id' : 'order_id IS NOT NULL',
'valid_customer' : 'customer_id IS NOT NULL',
'positive_amount' : 'order_amount > 0',
'valid_event_type': "event_type IN
('ORDER_PLACED','ORDER_UPDATED','ORDER_CANCELLED')"
})
def silver_orders():
return (
dlt.read_stream('bronze_order_events')
.withColumn('order_date', F.to_date('event_timestamp'))
.withColumn('order_amount_usd',
F.when(F.col('order_amount') < 0, 0)
.otherwise(F.col('order_amount')))
.withColumn('event_type', F.upper(F.trim(F.col('event_type'))))
.withColumn('customer_id', F.upper(F.trim(F.col('customer_id'))))
.withColumn('ingestion_timestamp', F.current_timestamp())
.dropDuplicates(['order_id', 'event_type', 'event_timestamp'])
.select(
'order_id', 'customer_id', 'product_id',
'order_amount_usd', 'event_type', 'event_timestamp',
'order_date', 'ingestion_timestamp'
)
)
# ─── Silver: Cleaned customer profiles ─────────────────────────────────
@dlt.table(name='silver_customers')
@dlt.expect_or_drop('valid_email', "email LIKE '%@%'")
© 2025 Databricks Reference Guide
Page
Databricks Reference Guide │ Asset Bundles · Azure DevOps CI/CD · Lakeflow Declarative Pipelines
def silver_customers():
return (
dlt.read_stream('bronze_customer_profiles')
.withColumn('email', F.lower(F.trim(F.col('email'))))
.withColumn('tier',
F.upper(F.trim(F.col('tier'))))
.withColumn('region', F.initcap(F.trim(F.col('region'))))
)
gold_layer.py — Business-Ready Analytics Tables
# gold_layer.py — Build Gold analytics tables from Silver
import dlt
from pyspark.sql import functions as F
# ─── Gold: Daily order summary (batch aggregate) ────────────────────────
@dlt.table(
name
= 'gold_daily_order_summary',
comment = 'Daily order metrics by customer segment and product'
)
def gold_daily_order_summary():
# dlt.read() for batch (non-streaming) reads of Silver tables
orders
= dlt.read('silver_orders')
customers = dlt.read('silver_customers')
return (
orders
.join(customers, on='customer_id', how='left')
.groupBy('order_date', 'tier', 'region')
.agg(
F.count(F.col('order_id')).alias('order_count'),
F.countDistinct('customer_id').alias('unique_customers'),
F.sum('order_amount_usd').alias('total_revenue_usd'),
F.avg('order_amount_usd').alias('avg_order_value')
)
)
# ─── Gold: Real-time running total (streaming) ──────────────────────────
@dlt.table(
name
= 'gold_running_revenue',
comment = 'Continuously updated running revenue total by date'
)
def gold_running_revenue():
return (
dlt.read_stream('silver_orders')
.withWatermark('event_timestamp', '10 minutes')
.groupBy(
F.window('event_timestamp', '1 hour').alias('hour_window'),
'order_date'
)
.agg(
F.sum('order_amount_usd').alias('hourly_revenue'),
F.count('order_id').alias('hourly_orders')
)
© 2025 Databricks Reference Guide
Page
Databricks Reference Guide │ Asset Bundles · Azure DevOps CI/CD · Lakeflow Declarative Pipelines
)
12.4 AUTO CDC — APPLY CHANGES INTO
AUTO CDC is one of Lakeflow's most powerful features. It automatically processes Change
Data Capture (CDC) streams — handling inserts, updates, and deletes from source systems —
and applies them to a target Delta Lake table as SCD Type 1 (upsert) or SCD Type 2 (historical)
records. One APPLY CHANGES INTO declaration replaces hundreds of lines of complex
MERGE SQL.
APPLY CHANGES INTO — SCD Type 1 (Upsert)
# AUTO CDC: Process CDC stream from SQL Server into Silver customer table
# Source: CDC events contain insert/update/delete operations
# Step 1: Ingest raw CDC events into Bronze
@dlt.table(
name
= 'bronze_cdc_customers',
comment = 'Raw CDC events from SQL Server customers table'
)
def bronze_cdc_customers():
return (
spark.readStream
.format('cloudFiles')
.option('cloudFiles.format', 'parquet')
.load('abfss://cdc@storage.dfs.core.windows.net/customers/')
)
# Step 2: APPLY CHANGES INTO — SCD Type 1 (latest value wins)
dlt.create_streaming_table(
name
= 'silver_customers_cdc',
comment = 'Current customer state — SCD Type 1 upsert from CDC stream'
)
dlt.apply_changes(
target
= 'silver_customers_cdc',
source
= 'bronze_cdc_customers',
# Column that uniquely identifies each customer record:
keys
= ['customer_id'],
# Column containing the CDC operation type:
# Expects values: 'INSERT', 'UPDATE', 'DELETE'
sequence_by
= 'cdc_timestamp',
# Use timestamp to order events
apply_as_deletes = F.expr("operation = 'DELETE'"),
apply_as_truncates = F.expr("operation = 'TRUNCATE'"),
© 2025 Databricks Reference Guide
Page
Databricks Reference Guide │ Asset Bundles · Azure DevOps CI/CD · Lakeflow Declarative Pipelines
# Columns to include in the target table:
except_column_list = ['operation', 'cdc_timestamp', '_rescued_data'],
# SCD Type 1 (default): just upsert — no history kept
stored_as_scd_type = 1
)
# Result: silver_customers_cdc always contains the CURRENT state
# of every customer. Deletes are physically removed.
# No MERGE SQL needed — Lakeflow handles it automatically.
APPLY CHANGES INTO — SCD Type 2 (Historical Tracking)
# APPLY CHANGES INTO with SCD Type 2:
# Preserves full history with __START_AT and __END_AT columns
dlt.create_streaming_table(
name
= 'silver_customers_history',
comment = 'Full customer history with SCD Type 2 — tracks all changes'
)
dlt.apply_changes(
target
= 'silver_customers_history',
source
= 'bronze_cdc_customers',
keys
= ['customer_id'],
sequence_by
= 'cdc_timestamp',
apply_as_deletes = F.expr("operation = 'DELETE'"),
except_column_list = ['operation', 'cdc_timestamp'],
# SCD Type 2: adds __START_AT and __END_AT columns
# Current row: __END_AT = NULL
# Historical row: __END_AT = timestamp when it was superseded
stored_as_scd_type = 2
)
# Resulting table structure:
# customer_id | tier
| region | __START_AT
| __END_AT
# CUST001
| Bronze | North | 2024-01-01 08:00:00 | 2024-03-15 14:22:00
# CUST001
| Silver | North | 2024-03-15 14:22:00 | 2024-08-01 09:11:00
# CUST001
| Gold
| West
| 2024-08-01 09:11:00 | NULL ← current
# Query current state:
# SELECT * FROM silver_customers_history WHERE __END_AT IS NULL
# Query state at a point in time:
# SELECT * FROM silver_customers_history
# WHERE __START_AT <= '2024-06-01' AND (__END_AT > '2024-06-01' OR __END_AT IS
NULL)
© 2025 Databricks Reference Guide
Page
Databricks Reference Guide │ Asset Bundles · Azure DevOps CI/CD · Lakeflow Declarative Pipelines
Data Quality Expectations Reference
# Expectation decorators and their behaviour when violated:
# @dlt.expect — Log violation but KEEP the row:
@dlt.expect('valid_email', "email LIKE '%@%'")
# Use for: soft warnings, non-critical validation
# @dlt.expect_or_drop — DROP violating rows, log metric:
@dlt.expect_or_drop('positive_amount', 'order_amount > 0')
# Use for: filtering invalid records from downstream tables
# @dlt.expect_or_fail — FAIL the pipeline if any row violates:
@dlt.expect_or_fail('non_null_id', 'order_id IS NOT NULL')
# Use for: critical constraints where bad data = stop everything
# @dlt.expect_all — Multiple expectations, all logged:
@dlt.expect_all({
'valid_order'
: 'order_id IS NOT NULL',
'valid_customer': 'customer_id IS NOT NULL',
'valid_date'
: 'order_date IS NOT NULL'
})
# @dlt.expect_all_or_drop — All must pass or row is dropped:
@dlt.expect_all_or_drop({
'valid_order_id': 'order_id IS NOT NULL',
'valid_amount' : 'order_amount > 0'
})
# @dlt.expect_all_or_fail — All must pass or pipeline fails:
@dlt.expect_all_or_fail({
'no_nulls'
: 'order_id IS NOT NULL',
'no_duplicates': 'ROW_NUMBER() OVER (PARTITION BY order_id ORDER BY ts) = 1'
})
12.5 External Target Sinks
External Target Sinks allow Lakeflow pipelines to write output not just to internal Unity Catalog
Delta Lake tables, but to external destinations — Azure Event Hubs, Apache Kafka, and other
streaming sinks. This enables reverse ETL patterns, real-time data sharing, and event-driven
architectures where the Lakehouse produces events for downstream consumers.
Writing Pipeline Output to Kafka (External Sink)
# External Target Sink: Write processed Gold table data to Kafka
# Use case: ShopStream publishes real-time order confirmations to
© 2025 Databricks Reference Guide
Page
Databricks Reference Guide │ Asset Bundles · Azure DevOps CI/CD · Lakeflow Declarative Pipelines
# an external Kafka topic consumed by customer notification services
import dlt
from pyspark.sql import functions as F
@dlt.table(
name
= 'gold_order_confirmations',
comment = 'Order confirmation events ready for external publishing'
)
def gold_order_confirmations():
return (
dlt.read_stream('silver_orders')
.filter(F.col('event_type') == 'ORDER_PLACED')
.select(
F.col('order_id').alias('key'),
F.to_json(F.struct(
'order_id', 'customer_id', 'order_amount_usd',
'order_date', 'event_timestamp'
)).alias('value')
)
)
# Configure the External Sink in the pipeline settings:
# (Done via Lakeflow Pipeline Configuration JSON or UI)
# {
#
'sink': {
#
'type': 'kafka',
#
'options': {
#
'kafka.bootstrap.servers': 'pkc-xxx.confluent.cloud:9092',
#
'topic': 'shopstream-order-confirmations',
#
'kafka.security.protocol': 'SASL_SSL',
#
'kafka.sasl.mechanism': 'PLAIN',
#
'kafka.sasl.jaas.config': '{{secrets/confluent/jaas_config}}'
#
}
#
}
# }
Writing to Azure Event Hubs (External Sink)
# External Sink: Write to Azure Event Hubs
# Event Hubs uses the Kafka protocol — same API, different endpoint
@dlt.table(
name
= 'gold_customer_events_sink',
comment = 'Customer tier change events for downstream CRM notification'
)
def gold_customer_events_sink():
return (
dlt.read_stream('silver_customers_cdc')
.select(
F.col('customer_id').alias('key'),
© 2025 Databricks Reference Guide
Page
Databricks Reference Guide │ Asset Bundles · Azure DevOps CI/CD · Lakeflow Declarative Pipelines
F.to_json(F.struct(
'customer_id', 'tier', 'region', 'email'
)).alias('value')
)
)
# Event Hubs Sink Configuration:
# {
#
'sink': {
#
'type': 'kafka',
-- Event Hubs uses Kafka protocol
#
'options': {
#
'kafka.bootstrap.servers':
#
'shopstream-eventhub.servicebus.windows.net:9093',
#
'topic': 'shopstream-customer-events',
#
'kafka.security.protocol': 'SASL_SSL',
#
'kafka.sasl.mechanism': 'PLAIN',
#
'kafka.sasl.jaas.config':
#
'org.apache.kafka.common.security.plain.PlainLoginModule required
#
username="$ConnectionString"
#
password="{{secrets/eventhubs/connection_string}}";'
#
}
#
}
# }
External Sink Type
Protocol
Configuration Key
Use Case
Apache Kafka
Native Kafka
Producer
kafka.* options
Stream results to external
Kafka consumers
Azure Event Hubs
Kafka-compatible
Same as Kafka, Event
Hubs endpoint
Azure-native event
streaming to Logic Apps,
Functions
AWS Kinesis
Kinesis Data
Streams
kinesis.* options
AWS-native streaming for
Lambda, Firehose
HTTP Webhook
REST API
url, headers, method
Push records to REST
APIs, webhooks
Complete Lakeflow Pipeline Outcome — ShopStream Real-World Results
Architecture deployed:
Bronze: Kafka (order events) + Auto Loader (CRM exports) → Streaming Tables
Silver: Cleaned + deduped events + SCD Type 2 customer history via AUTO CDC
Gold: Daily summaries (batch) + Running revenue (streaming) + Order confirmations (Kafka sink)
Before Lakeflow (legacy Spark ETL):
• 4,200 lines of Spark code across 18 notebooks
• CDC implemented as nightly MERGE jobs — 4-hour latency
• Pipeline failures required manual checkpoint repair
© 2025 Databricks Reference Guide
Page
Databricks Reference Guide │ Asset Bundles · Azure DevOps CI/CD · Lakeflow Declarative Pipelines
• Zero data quality metrics or violation tracking
After Lakeflow Declarative Pipelines:
• 680 lines of clean declarative Python (84% code reduction)
• CDC latency: 4 hours → < 2 minutes (AUTO CDC continuous mode)
• Pipeline failures: self-healing with automatic retry
• Data quality: 12 expectations tracking 99.97% pass rate
• External Kafka sink: order confirmations reach notification service in < 30 seconds
© 2025 Databricks Reference Guide
Page
0
You can add this document to your study collection(s)
Sign in Available only to authorized usersYou can add this document to your saved list
Sign in Available only to authorized users(For complaints, use another form )