Zerobus Ingest: Building a NiFi Processor for Databricks' New Streaming Primitive

On February 23, 2026, Databricks quietly promoted Zerobus Ingest from preview to GA. No fanfare, no keynote. Just a docs page update and a SDK bump to 0.2.0.

Zerobus replaces the entire Kafka→Spark Structured Streaming→Delta pipeline with a single gRPC call. No brokers, no partitions, no consumer groups. You send JSON, you get an offset ACK, your data lands in a Delta table. Sub-200ms P50 latency. Fully serverless.

I wanted to see how fast I could integrate it with an existing dataflow platform. So I built an Apache NiFi processor for it, going from zero to data landing in Databricks in a weekend. This post covers the integration, the gotchas, and a monitoring gap that most teams will overlook.

What Zerobus replaces

#

The traditional Databricks ingestion story involves at least two systems you have to operate:

flowchart LR
    Source["Data Source"] --> Kafka["Kafka Broker"]
    Kafka --> SS["Spark Structured\nStreaming"]
    SS --> Delta["Delta Table"]

    style Kafka fill:#dc2626,color:#fff
    style SS fill:#dc2626,color:#fff

Kafka alone is a full-time job: topic management, partition rebalancing, consumer lag, retention policies, schema registry. Then Spark adds checkpoint management, micro-batch tuning, and cluster sizing. Each is a potential failure point that needs monitoring, alerting, and on-call rotation.

Zerobus collapses this to:

flowchart LR
    Source["Data Source"] --> ZB["Zerobus\n(Serverless)"]
    ZB --> Delta["Delta Table"]

    style ZB fill:#6366f1,color:#fff

One gRPC stream. No infrastructure. The data goes from your producer directly into a Delta table:

For high-volume telemetry (security events, metrics, logs), this is the primitive you want. No intermediaries, no tuning, just send and forget.

The cost side

#

Most Databricks shops today run some variant of Kafka → Delta Live Tables (DLT) → Delta table. DLT handles schema enforcement, data quality expectations, and incremental processing well. But it comes with a bill.

DLT pipelines need always-on compute to consume from Kafka. That means DBUs, VM hours, and network egress, 24/7, whether data is flowing or not. For a typical security telemetry pipeline doing 50 MB/s sustained, you’re looking at a DLT pipeline running on a cluster that never sleeps. Add multiple sources, multiple tables, and suddenly your “streaming” budget is dominated by the compute layer sitting between Kafka and Delta.

Zerobus eliminates that entire layer. If your data is already well-formed JSON matching the target schema (and in many telemetry use cases, it is), there’s no transformation step needed. No DLT pipeline, no always-on cluster, no DBU burn. The data goes directly from your producer into the Delta table. The only cost is Zerobus ingestion itself, which is pure serverless. You pay for bytes ingested, not for idle compute waiting for bytes to arrive.

flowchart LR
    subgraph Before["Traditional (always-on compute)"]
        S1["Source"] --> K["Kafka"] --> DLT["DLT Pipeline\n💰 DBUs + Compute"] --> D1["Delta Table"]
    end

    subgraph After["Zerobus (serverless)"]
        S2["Source"] --> ZB["Zerobus\n(pay per byte)"] --> D2["Delta Table"]
    end

    style DLT fill:#dc2626,color:#fff
    style ZB fill:#6366f1,color:#fff

For pipelines where the data doesn’t need transformation, just reliable delivery, Zerobus isn’t just simpler. It’s meaningfully cheaper.

Building the NiFi processor

#

Apache NiFi is the Swiss army knife of data routing. Visual, backpressure-aware, battle-tested in enterprises that move terabytes daily. The problem? No Zerobus connector exists. Day one of GA, zero integrations outside the official Python/Java SDKs.

So I built one. The source is on GitHub under Apache 2.0.

The SDK: Java wrapping Rust wrapping gRPC

#

The official Java SDK (com.databricks:zerobus-ingest-sdk:0.2.0) isn’t pure Java. It’s a thin JNI layer over a compiled Rust binary that manages the gRPC stream, TLS, authentication, token refresh, compression, and stream recovery. Good design choice. You get Rust’s performance and safety guarantees with Java’s ecosystem compatibility.

But it also means platform-specific native libraries (linux-x86_64 only as of GA), and some real challenges with NiFi’s classloader isolation. More on that below.

Architecture decisions

#

The processor, PutZerobusIngest, is a single Java class, ~270 lines. The key decisions:

1. One persistent gRPC stream per processor instance. Opened in @OnScheduled, reused across all onTrigger invocations, closed in @OnStopped. No per-FlowFile connection overhead.

2. Batch ingestion with offset-based ACK. NiFi pulls FlowFiles in configurable batches (default: 100). Each batch becomes a single ingestRecordsOffset() call. The processor blocks on waitForOffset() before routing to success, guaranteeing delivery.

3. Three-way routing. Success, failure (non-retriable: schema mismatch, auth error), retry (transient: network blip, stream reset). NiFi handles retry natively via backpressure and penalization.

4. Automatic stream recovery. If the gRPC stream drops mid-session, the processor calls sdk.recreateStream() before the next batch. The SDK handles reconnection with configurable exponential backoff.

flowchart TB
    subgraph NiFi["Apache NiFi"]
        FF["FlowFiles\n(JSON)"] --> PZ["PutZerobusIngest"]
        PZ -->|"success"| S["Success Queue"]
        PZ -->|"failure"| F["Failure Queue"]
        PZ -->|"retry"| R["Retry Queue"]
        R -.->|"penalize + retry"| PZ
    end

    subgraph Databricks["Databricks"]
        PZ -->|"gRPC stream"| ZB["Zerobus\nServerless"]
        ZB --> DT["Delta Table"]
    end

    style PZ fill:#6366f1,color:#fff
    style ZB fill:#6366f1,color:#fff

The core code

#

The lifecycle is deceptively simple. Stream opens once:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

@OnScheduled
public void onScheduled(final ProcessContext context) throws Exception {
    sdk = new ZerobusSdk(endpoint, workspace);

    StreamConfigurationOptions options = StreamConfigurationOptions.builder()
            .setMaxInflightRecords(maxInflight)
            .setRecovery(true)
            .setRecoveryRetries(5)
            .setRecoveryTimeoutMs(30000)
            .setAckCallback(new AckCallback() {
                public void onAck(long offsetId) {
                    getLogger().debug("ACK for offset {}", offsetId);
                }
                public void onError(long offsetId, String message) {
                    getLogger().warn("Error for offset {}: {}", offsetId, message);
                }
            })
            .build();

    stream = sdk.createJsonStream(table, clientId, clientSecret, options).join();
}

Every trigger reads a batch, sends it, waits for confirmation:

1
2
3
4
5

Optional<Long> lastOffset = stream.ingestRecordsOffset(records);
if (lastOffset.isPresent()) {
    stream.waitForOffset(lastOffset.get());
}
session.transfer(flowFiles, REL_SUCCESS);

The SDK handles everything else: TLS, OAuth token refresh, gRPC framing, compression.

Walkthrough: from zero to data in Databricks

#

Actual deployment on a running NiFi instance with an existing Kafka-based flow.

Starting point: NiFi with Kafka

#

My test environment already had a flow with four host generators (host1, host2, host3a, host3b) publishing security telemetry to Kafka via PublishKafka_2_6:

Step 1: Add the processor

#

After deploying the NAR, PutZerobusIngest appears in the processor palette. Searching for “zero” filters 365 processors down to one:

The processor description shows the key details: persistent gRPC stream, batch ingestion, offset-based acknowledgment. Version 0.1.0, bundled as la.dere.nifi - nifi-zerobus-nar.

Step 2: Configure settings

#

The processor settings show the bundle identity and standard NiFi configuration:

Step 3: Set properties

#

The properties tab exposes the Zerobus configuration. Seven properties, five required:

After filling in the endpoint, workspace URL, target table, and service principal credentials:

The batch size is set to 10 for this test (default is 100). The Service Principal Client Secret shows as “Sensitive value set”. NiFi encrypts it at rest.

Step 4: Wire it up and run

#

I connected PutZerobusIngest in parallel with the existing Kafka publisher, same data source, fan-out to both destinations simultaneously:

The processor immediately started consuming FlowFiles. The stats show 14 tasks completed with active read/write throughput.

Step 5: Verify in Databricks

#

Data from all four hosts landing in the Delta table demo.default.zerobus_ingestion:

1
2

SELECT host, cast(timestamp / 1000 as timestamp) as timestamp
FROM demo.default.zerobus_ingestion;

All four hosts (host1, host2, host3a, host3b) with timestamps showing real-time ingestion. From NiFi to queryable Delta table, no Kafka in between.

The gotchas (and there were a few)

#

These are the issues I hit, documented so you don’t have to rediscover them.

1. JNI classloader isolation

#

The hardest one. NiFi uses NAR (NiFi Archive) packaging with strict classloader isolation, so each NAR gets its own classloader. The Zerobus SDK’s Rust backend spawns native threads via JNI AttachCurrentThread. These threads inherit the system classloader, not the NAR classloader.

Result: NoClassDefFoundError: com/databricks/zerobus/NonRetriableException. The SDK class exists in the NAR but the native thread can’t see it.

The fix: extract the SDK JAR from the NAR and place it on NiFi’s system classpath (lib/) in addition to bundling it in the NAR. The Dockerfile handles this automatically.

2. ARM64: SDK is x86_64 only

#

The Zerobus SDK ships native libraries for linux-x86_64 only. On Apple Silicon (OrbStack, Rancher Desktop), you need --platform linux/amd64 when building the Docker image and Rosetta emulation enabled in your container runtime. Production x86_64 clusters are unaffected.

3. Endpoint format: https:// required

#

The SDK parses the workspace ID from the endpoint URL. Without the https:// prefix, it fails with Failed to extract workspace_id from zerobus_endpoint. The property now clearly documents the expected format.

4. Permissions: ALL_PRIVILEGES is not enough

#

This one cost me hours. The service principal had ALL_PRIVILEGES inherited from the parent catalog, which you’d think covers everything. It doesn’t.

Zerobus Ingest uses fine-grained OAuth authorization_details scoping. The OIDC token request includes the specific table and operation. Only explicit MODIFY and SELECT grants on the target table satisfy this check. Inherited privileges are ignored.

1
2
3
4
5

-- This works:
GRANT MODIFY, SELECT ON TABLE demo.default.zerobus_ingestion TO `sp-nifi`;

-- This doesn't (even though it should):
GRANT ALL_PRIVILEGES ON CATALOG demo TO `sp-nifi`;

5. Auth is service-principal-only

#

No PATs, no managed identity, no instance profiles. OAuth 2.0 M2M with client ID + client secret only. This is intentional for automated pipelines, but it means you need a secrets management story.

6. Schema must match exactly

#

No schema evolution on the fly. Your JSON must match the target Delta table schema at write time. Mismatches produce NonRetriableException routed to the failure relationship.

Monitoring Zerobus itself

#

Zerobus is serverless and largely invisible. There’s no cluster to SSH into, no Spark UI to check. But Databricks does expose two system tables in system.lakeflow that give you stream-level and ingestion-level telemetry.

Stream events: system.lakeflow.zerobus_stream

#

This table tracks stream lifecycle: opens, closes, and errors. Key columns:

How many streams are active right now?

1
2
3
4
5

SELECT COUNT(stream_id) AS active_streams
FROM system.lakeflow.zerobus_stream
WHERE table_name = 'demo.default.security_events'
  AND closed_time IS NULL
  AND opened_time > CURRENT_TIMESTAMP - INTERVAL 15 MINUTES;

Ingestion metrics: system.lakeflow.zerobus_ingest

#

This table tracks actual data landing in Delta, aggregated per commit, not per record:

Average ingestion rate over the last hour:

1
2
3
4
5
6

SELECT table_name,
       SUM(committed_bytes) / 3600 AS avg_bytes_per_sec,
       SUM(committed_records) AS total_records
FROM system.lakeflow.zerobus_ingest
WHERE commit_time >= CURRENT_TIMESTAMP - INTERVAL 1 HOUR
GROUP BY table_name;

Detect ingestion gaps (no commits for 5+ minutes):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

WITH commits AS (
    SELECT table_name, commit_time,
           LEAD(commit_time) OVER (PARTITION BY table_name ORDER BY commit_time) AS next_commit
    FROM system.lakeflow.zerobus_ingest
    WHERE commit_time >= CURRENT_TIMESTAMP - INTERVAL 1 HOUR
)
SELECT table_name, commit_time, next_commit,
       TIMESTAMPDIFF(SECOND, commit_time, next_commit) AS gap_seconds
FROM commits
WHERE TIMESTAMPDIFF(SECOND, commit_time, next_commit) > 300
ORDER BY gap_seconds DESC;

These tables are solid for pipeline-level health: is the stream alive, is data flowing, at what rate. You can build Databricks SQL alerts on top of them and get notified when throughput drops or streams die.

But that’s where it ends.

Beyond pipeline health

#

The system tables above give you pipeline health. You can see streams, throughput, gaps. For many data engineering use cases, that’s enough.

But security telemetry isn’t a normal data engineering use case.

When a Kafka-based pipeline broke, you had consumer lag, batch processing times, checkpoint drift, and dead letter queues, an entire ecosystem of signals telling you exactly where in the chain something went wrong. With Zerobus, you get two system tables showing aggregate stream health. If the pipeline goes silent, producing no errors but also no data, the system tables will eventually show a gap. But they won’t tell you why. The NiFi processor? The service principal credentials? A network policy change? The upstream source?

Databricks is the destination. It sees the symptom, not the cause.

Per-asset monitoring

#

The system tables tell you whether a stream is ingesting. They don’t tell you whether each expected source is reporting. Consider these scenarios:

• One of 200 EDR agents goes dark. The stream keeps ingesting from the other 199. Total throughput barely dips. No alert. One endpoint is now completely unmonitored.

• A branch office firewall stops sending logs. The Zerobus pipeline looks healthy, still receiving events from 50 other firewalls. But that specific branch now has zero network visibility.

• A cloud account’s audit logging gets silently disabled. The ingestion pipeline doesn’t know what should be coming. It only knows what is coming. Absence is invisible.

• A host rotates its TLS cert and the agent can’t phone home. From Databricks’ perspective, there’s simply less data today. Maybe it’s a quiet day. Maybe it’s an attacker who just killed the agent.

This isn’t a Zerobus problem. Zerobus does exactly what it promises: reliable, low-latency ingestion. But knowing that the pipe works doesn’t mean knowing that every tap is turned on.

Who’s watching whether every expected source is actually reporting?

Pipeline monitoring tells you the pipe works. Asset-level monitoring tells you nothing is missing.

I’ve already solved this.

#

If you’ve ever stared at a Splunk dashboard wondering “is this quiet because nothing’s happening, or because we lost visibility?” - you know the problem. It’s getting worse as pipelines get simpler and more abstracted.

I’ve been building something that addresses exactly this. Asset-level expected-vs-actual comparisons, per-source silence detection, and full pipeline visibility mapping. Zerobus was actually one of the first pipelines I tested it against, because when you strip away all the middleware, you need something else watching the gaps.

It works, it’s running locally, and it’ll go public very soon. More on that when it launches.

Zerobus Ingest fundamentally simplifies streaming ingestion into Databricks. Sub-200ms latency, zero infrastructure, and a clean SDK that gets out of your way. Building the NiFi processor took a weekend, and most of that was fighting classloader isolation, not the SDK itself.

If you’re running NiFi and want to try Zerobus, the processor is ready: us3r/nifi-zerobus-bundle (Apache 2.0). The Dockerfile handles the JNI quirks. The README documents every gotcha in this post and more.

The remaining question: when the pipeline goes silent, how do you know?

Links:

• nifi-zerobus-bundle - NiFi processor (Apache 2.0)
• Zerobus Ingest docs
• Databricks Zerobus Ingest overview
• Zerobus Ingest system tables reference