Data Quality Gates: Lakehouse Monitoring and DLT Expectations

Table of Contents

Databricks Challenge - This article is part of a series.

Part : This Article

Part : Delta Sharing: Zero-Copy Data Exchange Across Workspaces and Beyond

Part : Multi-Tenant Data Platform: Row Filters, Column Masks, and ABAC on Unity Catalog

Part : Real-Time Anomaly Detection with Structured Streaming and Delta Lake

Part : SCD Type 2 Done Right: Declarative Pipelines and Change Data Feed

The dashboard is wrong. Someone notices on Tuesday. The root cause turns out to be a column that started arriving as NULL three weeks ago. The pipeline didn’t fail. The data landed. The table grew. Nothing broke – except every metric downstream.

Silent data quality regressions are the most expensive kind of bug. They don’t crash anything, so they don’t trigger alerts. They corrupt analytics gradually, and by the time someone notices, you’re explaining to leadership why last month’s board deck used bad numbers.

The fix is defense-in-depth: row-level validation that catches known-bad data at ingestion time, plus statistical monitoring that catches unknown-bad data by detecting distributional drift. Neither layer alone is sufficient. Expectations catch rows that violate explicit rules. Monitoring catches shifts that no rule anticipated.

Two Layers of Quality
#

flowchart TB
    subgraph Layer1["Layer 1: Row-Level (DLT Expectations)"]
        R["Incoming rows"]
        E1["expect: email is valid"]
        E2["expect_or_drop: amount > 0"]
        E3["expect_or_fail: customer_id NOT NULL"]
        R --> E1 & E2 & E3
    end

    subgraph Layer2["Layer 2: Table-Level (Lakehouse Monitoring)"]
        T["Materialized table"]
        P["Statistical profiling"]
        D["Drift detection
(vs. baseline)"]
        T --> P --> D
    end

    subgraph Alert["Alerting"]
        S["Slack / PagerDuty"]
        DB["Quality Dashboard"]
    end

    E1 & E2 & E3 --> T
    D --> S
    D --> DB

    style R fill:#3498db,color:white
    style T fill:#2ecc71,color:white
    style D fill:#e74c3c,color:white
    style S fill:#f39c12,color:white

Layer 1: DLT Expectations (Row-Level Validation)
#

Expectations are constraints defined inline with your Declarative Pipeline tables. They fire on every row, every micro-batch. Three modes:

Mode	Behavior	Use Case
`expect`	Log violation, keep the row	Soft validation, metrics tracking
`expect_or_drop`	Silently drop violating rows	Filtering bad data without failing
`expect_or_fail`	Fail the pipeline	Hard constraint, data must be correct

Setup: Orders Pipeline with Expectations
#

1
2
3


CREATE CATALOG IF NOT EXISTS ecommerce;
CREATE SCHEMA IF NOT EXISTS ecommerce.quality_demo;
CREATE VOLUME IF NOT EXISTS ecommerce.quality_demo.orders_landing;

Generate sample data with intentional quality issues:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47


import json
import random
from datetime import datetime, timedelta, timezone

def generate_orders(n=500, bad_pct=0.1):
    orders = []
    for i in range(n):
        order = {
            "order_id": f"ORD-{i:06d}",
            "customer_id": f"C{random.randint(1, 200):04d}",
            "email": f"user{random.randint(1, 200)}@example.com",
            "amount": round(random.uniform(10, 500), 2),
            "quantity": random.randint(1, 10),
            "postal_code": f"{random.randint(10000, 99999)}",
            "region": random.choice(["US-WEST", "US-EAST",
                                     "EU-WEST", "APAC"]),
            "order_date": (
                datetime(2026, 4, 1, tzinfo=timezone.utc)
                + timedelta(days=random.randint(0, 19))
            ).isoformat(),
        }

        # Inject quality issues
        if random.random() < bad_pct:
            issue = random.choice([
                "null_email", "negative_amount",
                "bad_postal", "null_customer"
            ])
            if issue == "null_email":
                order["email"] = None
            elif issue == "negative_amount":
                order["amount"] = -round(random.uniform(1, 100), 2)
            elif issue == "bad_postal":
                order["postal_code"] = "XXXXX"
            elif issue == "null_customer":
                order["customer_id"] = None

        orders.append(order)
    return orders

volume_path = "/Volumes/ecommerce/quality_demo/orders_landing"
orders = generate_orders(500, bad_pct=0.1)
with open(f"{volume_path}/orders_batch_001.json", "w") as f:
    for o in orders:
        f.write(json.dumps(o) + "\n")

print(f"Wrote {len(orders)} orders with ~10% quality issues")

Define the Pipeline with Expectations
#

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


import dlt
from pyspark.sql import functions as F

@dlt.table(
    comment="Raw orders ingested from landing zone"
)
def orders_raw():
    return (
        spark.readStream
        .format("cloudFiles")
        .option("cloudFiles.format", "json")
        .option("cloudFiles.inferColumnTypes", "true")
        .load("/Volumes/ecommerce/quality_demo/orders_landing/")
    )


@dlt.expect("valid_email",
            "email IS NOT NULL AND email RLIKE '^[^@]+@[^@]+\\\\.[^@]+$'")
@dlt.expect_or_drop("positive_amount", "amount > 0")
@dlt.expect_or_fail("has_customer_id", "customer_id IS NOT NULL")
@dlt.expect("valid_postal_code",
            "postal_code RLIKE '^[0-9]{5}$'")
@dlt.table(
    comment="Validated orders - bad amounts dropped, null customers fail pipeline"
)
def orders_validated():
    return dlt.read_stream("orders_raw")

What happens at runtime:

valid_email: Rows with invalid emails pass through but are logged as violations. You see the count in the pipeline event log.
positive_amount: Rows with negative or zero amounts are silently dropped. They never reach orders_validated.
has_customer_id: If any row has a NULL customer_id, the entire pipeline fails. This is your hard constraint – orders without customers are corrupted data.
valid_postal_code: Soft check, logged but not enforced. Use this to track data quality trends without disrupting the pipeline.

Querying Expectation Results
#

After the pipeline runs, the event log records every expectation evaluation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


-- Expectation violations per run
SELECT
    details:flow_definition.output_dataset AS table_name,
    details:flow_progress.data_quality.expectations[*].name
        AS expectation_name,
    details:flow_progress.data_quality.expectations[*].dataset
        AS dataset,
    details:flow_progress.data_quality.expectations[*]
        .passed_records AS passed,
    details:flow_progress.data_quality.expectations[*]
        .failed_records AS failed
FROM event_log(TABLE(ecommerce.quality_demo.orders_validated))
WHERE event_type = 'flow_progress'
ORDER BY timestamp DESC
LIMIT 10

Layer 2: Lakehouse Monitoring (Statistical Drift Detection)
#

Expectations catch rows that violate rules you wrote. But what about quality issues nobody anticipated? A region column that suddenly has 90% of values as “UNKNOWN” when it used to be evenly distributed. An amount column whose mean shifts from $200 to $50 because of a pricing bug upstream.

Lakehouse Monitoring profiles your table statistically and compares current distributions against a baseline. No rules to write – it detects anomalies in the data itself.

Enable a Monitor
#

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


from databricks.sdk import WorkspaceClient

w = WorkspaceClient()

# Create a monitor on the validated orders table
monitor = w.quality_monitors.create(
    table_name="ecommerce.quality_demo.orders_validated",
    output_schema_name="ecommerce.quality_demo._monitoring",
    assets_dir="/Workspace/Users/you/monitoring_assets",
    schedule={"quartz_cron_expression": "0 0 * * * ?",
              "timezone_id": "UTC"},
    slicing_exprs=["region"],
    baseline_table_name="ecommerce.quality_demo.orders_baseline",
    inference_log=None,
)

print(f"Monitor created: {monitor.monitor_name}")

Configuration breakdown:

output_schema_name: Where profiling results are stored (auto-created tables)
schedule: How often to re-profile. Hourly is aggressive for most tables – daily is more typical
slicing_exprs: Profile per-slice, not just globally. Drift in a single region gets detected even if the global distribution looks fine
baseline_table_name: The “known good” snapshot to compare against. If omitted, the monitor uses the first profile as the baseline

Create the Baseline
#

1
2
3
4
5
6


# Snapshot current data as the baseline
# Do this when you know the data quality is correct
spark.sql("""
    CREATE TABLE IF NOT EXISTS ecommerce.quality_demo.orders_baseline
    AS SELECT * FROM ecommerce.quality_demo.orders_validated
""")

Query Drift Results
#

After the monitor runs, it populates profiling and drift tables:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


-- Profile metrics: distribution stats per column
SELECT
    column_name,
    metric_name,
    baseline_value,
    current_value,
    drift_type,
    abs(current_value - baseline_value) AS absolute_drift
FROM ecommerce.quality_demo._monitoring.profile_metrics
WHERE drift_type IS NOT NULL
ORDER BY absolute_drift DESC

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


-- Which columns drifted the most?
SELECT
    column_name,
    COUNT(*) AS drift_count,
    COLLECT_SET(drift_type) AS drift_types,
    MAX(abs(current_value - baseline_value)) AS max_drift
FROM ecommerce.quality_demo._monitoring.profile_metrics
WHERE drift_type IS NOT NULL
GROUP BY column_name
ORDER BY max_drift DESC

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


-- Drift by region slice
SELECT
    slice_key,
    slice_value,
    column_name,
    metric_name,
    baseline_value,
    current_value,
    drift_type
FROM ecommerce.quality_demo._monitoring.profile_metrics
WHERE drift_type IS NOT NULL
    AND slice_key = 'region'
ORDER BY slice_value, column_name

Custom Metrics
#

Built-in profiling covers standard statistics (mean, stddev, null ratio, distinct count). For business-specific quality metrics, define custom metrics:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


from databricks.sdk.service.catalog import (
    MonitorMetric, MonitorMetricType
)

# Custom metric: percentage of orders with valid postal codes
w.quality_monitors.update(
    table_name="ecommerce.quality_demo.orders_validated",
    custom_metrics=[
        MonitorMetric(
            type=MonitorMetricType.CUSTOM_METRIC_TYPE_AGGREGATE,
            name="valid_postal_pct",
            input_columns=["postal_code"],
            definition="AVG(CASE WHEN postal_code RLIKE '^[0-9]{5}$'"
                       " THEN 1.0 ELSE 0.0 END)",
            output_data_type="DOUBLE",
        ),
        MonitorMetric(
            type=MonitorMetricType.CUSTOM_METRIC_TYPE_AGGREGATE,
            name="avg_order_value",
            input_columns=["amount"],
            definition="AVG(amount)",
            output_data_type="DOUBLE",
        ),
    ],
)

Integrating Alerts
#

Webhook Alerts to Slack
#

Set up a notification destination in the Databricks workspace, then link it to the monitor:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


# Create a Slack webhook notification destination
from databricks.sdk.service.settings import (
    CreateNotificationDestinationRequest
)

w.notification_destinations.create(
    display_name="data-quality-alerts",
    config={
        "slack": {
            "url": "https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
        }
    }
)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


# Link notifications to the monitor
w.quality_monitors.update(
    table_name="ecommerce.quality_demo.orders_validated",
    notifications={
        "on_failure": {
            "email_addresses": ["[email protected]"]
        },
        "on_new_classification_tag_detected": {
            "email_addresses": ["[email protected]"]
        }
    }
)

Quality Dashboard
#

Build a dashboard over the monitoring system tables to visualize quality trends:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


-- Quality trend over time (for a dashboard panel)
SELECT
    window.start AS profile_date,
    column_name,
    metric_name,
    current_value
FROM ecommerce.quality_demo._monitoring.profile_metrics
WHERE column_name IN ('amount', 'postal_code', 'email')
    AND metric_name IN ('percent_nulls', 'mean', 'distinct_count')
ORDER BY profile_date, column_name

Putting Both Layers Together
#

The two layers complement each other:

Aspect	DLT Expectations	Lakehouse Monitoring
Scope	Row-level	Table-level / column-level
When	Every micro-batch (streaming) or batch	Scheduled (cron)
What it catches	Known violations (rules you write)	Unknown drift (statistical)
Response	Drop/fail/log per row	Alert, drift report
False positives	None (rules are exact)	Possible (statistical thresholds)
Setup effort	Inline decorators	SDK call + baseline

A complete quality gate looks like this:

Expectations block obviously bad data at ingestion: null keys fail the pipeline, negative amounts get dropped, invalid formats get logged
Monitoring catches subtle regressions after the data lands: distribution shifts, unusual null spikes, column cardinality changes
Alerts notify the right people: Slack for the data team, email for governance, PagerDuty for critical pipelines
Dashboard gives the historical view: quality trends over weeks and months, per-slice breakdowns

Gotchas
#

Monitor refresh frequency vs. cost. Profiling wide tables (100+ columns) is not free. Each monitor run reads the entire table (or the incremental window) and computes statistics per column per slice. Running hourly on a billion-row table will cost real compute. Start with daily or weekly and increase frequency only for critical tables.

Expectation naming collisions. If two expectations have the same name on different tables, the event log becomes confusing. Name them descriptively: orders_valid_email, not just valid_email. There’s no enforcement of unique names across tables.

Baseline staleness. Your baseline represents “what good data looks like.” If the business genuinely changes (new regions, different pricing), the baseline becomes outdated and the monitor flags everything as drift. Refresh the baseline periodically after validating that the new distribution is correct.

expect_or_fail in streaming pipelines. If a single bad row triggers a pipeline failure, you need to fix the source data and restart. For high-volume streaming, this can mean hours of reprocessing. Use expect_or_fail only for constraints that truly indicate corrupt data (null keys), not for soft quality issues. Use expect_or_drop for fields where dropping bad rows is acceptable.

Slicing expressiveness. slicing_exprs accepts column expressions, not arbitrary SQL. You can slice by region or DATE(order_date), but complex slicing logic needs to be materialized as a column first.

Conclusion
#

Data quality is not a single check. It’s two systems working at different granularities.

DLT Expectations are your first line – they enforce what you know must be true about every row. Null keys, positive amounts, valid formats. These are deterministic, zero-false-positive validations baked into your pipeline.

Lakehouse Monitoring is your second line – it catches what you didn’t anticipate. Distribution shifts, cardinality changes, unusual null rates. These are statistical signals that something changed upstream, even if every individual row passes your explicit rules.

Neither layer replaces the other. A row can pass every expectation and still be part of a distributional anomaly. A statistical drift can be real and intentional (new market launch) rather than a bug. Both layers together give you confidence that bad data doesn’t silently corrupt your analytics.

Author

Mariusz Derela

Cyber Security Specialist | DevSecOps | AI/ML

Databricks Challenge - This article is part of a series.

Part : This Article

Part : Delta Sharing: Zero-Copy Data Exchange Across Workspaces and Beyond

Part : Multi-Tenant Data Platform: Row Filters, Column Masks, and ABAC on Unity Catalog

Part : Real-Time Anomaly Detection with Structured Streaming and Delta Lake

Part : SCD Type 2 Done Right: Declarative Pipelines and Change Data Feed

Two Layers of Quality #

Layer 1: DLT Expectations (Row-Level Validation) #

Setup: Orders Pipeline with Expectations #

Define the Pipeline with Expectations #

Querying Expectation Results #

Layer 2: Lakehouse Monitoring (Statistical Drift Detection) #

Enable a Monitor #

Create the Baseline #

Query Drift Results #

Custom Metrics #

Integrating Alerts #

Webhook Alerts to Slack #

Quality Dashboard #

Putting Both Layers Together #

Gotchas #

Conclusion #

Related