Lakeflow GA: The Unified Data Engineering Platform You've Been Waiting For

If you’ve been building pipelines on Databricks for any amount of time, you know the pain: DLT for transformations, partner tools for ingestion, notebooks for orchestration, and a prayer that everything stays in sync. Lakeflow consolidates all of that into a single, GA-ready platform.

This isn’t a rebrand. It’s a genuine unification of ingestion, transformation, and orchestration under one roof — with some significant new capabilities that change how you should think about data engineering on Databricks.

What Is Lakeflow?

#

Lakeflow is Databricks’ unified data engineering platform, now Generally Available. It brings three components under one umbrella:

The key insight: these aren’t three separate products bolted together. They share a common execution engine, unified monitoring, and a single API surface. Your Connect ingestion feeds directly into Declarative Pipelines transformations without intermediate landing zones or glue code.

Declarative Pipelines: DLT Grows Up

#

If you’ve used Delta Live Tables, Declarative Pipelines will feel familiar — because it is DLT, with a new name and significant improvements. The core programming model is the same: you declare what your data should look like, and the engine figures out how to get there.

What Changed from DLT

#

The rename isn’t cosmetic. Key differences:

1. Serverless by default. No more cluster management. Pipelines run on serverless compute with automatic scaling. You pay per DBU consumed, not for idle clusters.

2. Materialized Views and Streaming Tables are first-class. These are no longer DLT-only concepts — they’re Unity Catalog objects accessible from any Databricks SQL warehouse or notebook.

3. Row-level expectations with actions. You can now define data quality rules that drop, fail, or quarantine bad rows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

import dlt

@dlt.table(
    name="clean_transactions",
    comment="Validated transaction records"
)
@dlt.expect_all_or_drop({
    "valid_amount": "amount > 0",
    "valid_currency": "currency IN ('USD', 'EUR', 'GBP', 'PLN')",
    "not_null_id": "transaction_id IS NOT NULL"
})
def clean_transactions():
    return spark.readStream("STREAM raw_transactions")

4. Python and SQL parity. SQL pipelines now support the full feature set that was previously Python-only, including parameterized queries and complex flow control.

5. Built-in orchestration. Pipelines can be composed into multi-task jobs with dependencies, retries, and conditional execution — no need for external orchestrators.

Migration Checklist: DLT to Declarative Pipelines

#

If you have existing DLT pipelines, here’s what you need to do:

• Nothing breaks immediately. Existing DLT pipelines continue to run. The API endpoints are aliased.
• Update imports. import dlt remains valid. The dlt Python module is the canonical import.
• Review compute settings. Serverless is now the default for new pipelines. Existing pipelines retain their classic compute configuration until you explicitly migrate.
• Audit expectations. If you were using expect without an action, the default behavior (warn) hasn’t changed — but now is a good time to add explicit drop or fail actions.
• Check Unity Catalog registration. Materialized views and streaming tables created by Declarative Pipelines are automatically registered in Unity Catalog. Verify your governance policies cover these new objects.

1
2
3
4
5

-- Check what your pipeline has registered
SELECT table_name, table_type, created_by
FROM system.information_schema.tables
WHERE table_schema = 'your_schema'
  AND created_by LIKE '%pipeline%';

Lakeflow Connect: Free Ingestion That Actually Works

#

This is the game-changer for cost-conscious teams. Lakeflow Connect provides managed, no-code ingestion from popular sources — and the free tier gives you 100 DBU per day at no cost.

Supported Sources (GA)

#

The connector library keeps growing, but at GA these are production-ready:

SaaS Applications:

• Salesforce
• ServiceNow
• Microsoft Dynamics 365
• Workday
• Google Analytics
• HubSpot
• Zendesk

Databases (CDC):

• PostgreSQL
• MySQL
• SQL Server
• Oracle
• MongoDB

Cloud Storage:

• Amazon S3
• Azure Blob Storage / ADLS Gen2
• Google Cloud Storage

Setting Up a Connect Pipeline

#

Creating an ingestion pipeline is refreshingly simple. From the workspace UI:

1. Navigate to Data Engineering > Lakeflow Connect
2. Select your source type
3. Provide connection credentials (stored in Unity Catalog secrets)
4. Choose tables/objects to ingest
5. Select target catalog and schema
6. Configure sync mode: Full Refresh or Incremental (CDC)

For the programmatic types, the REST API works too:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

from databricks.sdk import WorkspaceClient
from databricks.sdk.service.pipelines import (
    IngestionConfig,
    TableSpecificConfig,
)

w = WorkspaceClient()

# Create a Salesforce ingestion pipeline via the Pipelines API
pipeline = w.pipelines.create(
    name="salesforce-ingestion",
    catalog="bronze",
    schema="salesforce_raw",
    ingestion_definition=IngestionConfig(
        connection_name="salesforce-prod",
        objects=[
            TableSpecificConfig(table_configuration={"Account": {}}),
            TableSpecificConfig(table_configuration={"Opportunity": {}}),
            TableSpecificConfig(table_configuration={"Contact": {}}),
        ]
    ),
    serverless=True,
)

The Free Tier Math

#

100 DBU/day translates to roughly:

• ~20-30 tables with hourly incremental sync from a typical SaaS source
• ~5-10 million rows/day from database CDC sources (depending on row size)
• Enough for most small-to-medium analytics teams’ ingestion needs

If you exceed the free tier, you pay standard serverless DBU rates. There’s no cliff — it’s a smooth transition.

Pro tip: Start with your highest-value, lowest-volume sources on Connect’s free tier. Use the savings to justify expanding to paid ingestion for larger sources.

Lakeflow Designer: No-Code for Real

#

Designer is the visual pipeline builder, and — hot take — it’s actually useful. Not just for non-technical users, but for prototyping complex transformations before writing production code.

What Designer Can Do

#

• Drag-and-drop transformations: Filter, join, aggregate, pivot, unpivot, and custom SQL — all in a visual canvas
• Live preview: See sample data at every stage of your pipeline without running the full job
• Code generation: Every Designer pipeline generates clean Python or SQL that you can export, version control, and extend
• Collaboration-friendly: Share visual pipelines with stakeholders who can understand the logic without reading code

When to Use Designer vs. Code

#

The sweet spot: use Designer to prototype and get stakeholder buy-in, then export to code for production. The generated code is clean enough to use as a starting point — you’re not fighting auto-generated spaghetti.

Designer Example: SCD Type 2

#

Here’s where Designer earns its keep. Slowly Changing Dimension Type 2 is one of those patterns that’s tedious to write from scratch every time. In Designer:

1. Drag in your source table
2. Add a “SCD Type 2” transformation block
3. Configure: business key, tracked columns, effective date column
4. Connect to your target table

Designer generates the merge logic, handles the surrogate keys, and manages the effective_from / effective_to / is_current columns. The generated SQL is standard MERGE INTO — no proprietary magic.

Putting It All Together: A Complete Pipeline

#

Here’s a realistic example that uses all three Lakeflow components:

Architecture

#

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

[Salesforce]  -->  Lakeflow Connect  -->  bronze.salesforce_raw.*
[PostgreSQL]  -->  Lakeflow Connect  -->  bronze.app_db_raw.*
                                              |
                                    Declarative Pipelines
                                              |
                                    silver.cleaned_opportunities
                                    silver.enriched_accounts
                                              |
                                    Declarative Pipelines
                                              |
                                    gold.revenue_dashboard
                                    gold.customer_360

Step 1: Ingestion with Connect

#

Set up two Connect pipelines — one for Salesforce (free tier), one for your application PostgreSQL (CDC):

1
2
3
4

-- These tables appear automatically in your bronze catalog
-- after Connect pipeline runs
SELECT * FROM bronze.salesforce_raw.opportunity LIMIT 10;
SELECT * FROM bronze.app_db_raw.customers LIMIT 10;

Step 2: Transform with Declarative Pipelines

#

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

import dlt

@dlt.table(
    name="cleaned_opportunities",
    comment="Opportunities with validated amounts and standardized stages",
    table_properties={"quality": "silver"}
)
@dlt.expect_or_drop("positive_amount", "amount > 0")
@dlt.expect_or_drop("valid_stage", "stage_name IS NOT NULL")
def cleaned_opportunities():
    return (
        spark.readStream("STREAM bronze.salesforce_raw.opportunity")
        .withColumn("amount_usd", 
            F.when(F.col("currency") != "USD", 
                   F.col("amount") * get_exchange_rate(F.col("currency")))
             .otherwise(F.col("amount")))
        .withColumn("stage_normalized", normalize_stage(F.col("stage_name")))
    )


@dlt.table(
    name="customer_360",
    comment="Unified customer view joining CRM and application data",
    table_properties={"quality": "gold"}
)
def customer_360():
    opportunities = dlt.read("cleaned_opportunities")
    customers = spark.read.table("bronze.app_db_raw.customers")
    
    return (
        customers
        .join(opportunities, customers.email == opportunities.contact_email, "left")
        .groupBy("customer_id", "email", "company_name")
        .agg(
            F.sum("amount_usd").alias("total_pipeline_value"),
            F.count("opportunity_id").alias("open_opportunities"),
            F.max("close_date").alias("latest_close_date")
        )
    )

Step 3: Orchestrate

#

Define a multi-task job that runs Connect ingestion first, then transformations:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

from databricks.sdk import WorkspaceClient
from databricks.sdk.service.jobs import *

w = WorkspaceClient()

job = w.jobs.create(
    name="daily-customer-pipeline",
    tasks=[
        Task(
            task_key="ingest_salesforce",
            pipeline_task=PipelineTask(pipeline_id="<connect-pipeline-id>")
        ),
        Task(
            task_key="ingest_app_db",
            pipeline_task=PipelineTask(pipeline_id="<connect-cdc-pipeline-id>")
        ),
        Task(
            task_key="transform",
            pipeline_task=PipelineTask(pipeline_id="<declarative-pipeline-id>"),
            depends_on=[
                TaskDependency(task_key="ingest_salesforce"),
                TaskDependency(task_key="ingest_app_db")
            ]
        )
    ],
    schedule=CronSchedule(
        quartz_cron_expression="0 0 6 * * ?",
        timezone_id="Europe/Warsaw"
    )
)

Cost Considerations

#

Lakeflow pricing is DBU-based, with some notable details:

Cost optimization tips:

1. Use Connect’s free tier strategically. 100 DBU/day is generous for ingestion. Don’t waste it on full refreshes when incremental CDC works.
2. Right-size pipeline triggers. Not every table needs real-time. Batch daily for dimensional data, stream only for truly time-sensitive sources.
3. Leverage expectations to fail fast. Bad data that makes it to gold costs more to fix than data that gets quarantined at silver.
4. Monitor with system tables. system.billing.usage gives you per-pipeline cost breakdowns.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

-- Check your Lakeflow costs for the last 30 days
SELECT 
    u.usage_metadata.pipeline_id,
    u.usage_metadata.pipeline_name,
    SUM(u.usage_quantity) as total_dbus,
    SUM(u.usage_quantity * p.pricing.default) as estimated_cost
FROM system.billing.usage u
LEFT JOIN system.billing.list_prices p
    ON u.cloud = p.cloud
    AND u.sku_name = p.sku_name
    AND u.usage_date BETWEEN p.price_start_time AND COALESCE(p.price_end_time, '2099-12-31')
WHERE u.usage_date >= DATEADD(DAY, -30, CURRENT_DATE())
  AND u.usage_metadata.pipeline_id IS NOT NULL
GROUP BY 1, 2
ORDER BY total_dbus DESC;

What’s Still Missing

#

Lakeflow is GA, but it’s not done. A few gaps to be aware of:

• Connect source coverage. The connector library is growing but still smaller than mature tools like Fivetran or Airbyte. If your source isn’t supported, you’re still writing custom ingestion.
• Designer complexity ceiling. For truly complex transformations (recursive CTEs, custom UDFs, multi-hop streaming), you’ll outgrow Designer quickly. That’s by design — it’s a prototyping tool, not a replacement for code.
• Cross-workspace pipelines. Lakeflow pipelines are workspace-scoped. If you need cross-workspace orchestration, you’re still using multi-workspace jobs or external orchestrators.
• Terraform support. The Databricks Terraform provider supports Declarative Pipelines, but Connect and Designer resources aren’t fully covered yet.

Bottom Line

#

Lakeflow is the first time Databricks has offered a genuinely unified data engineering experience. The free Connect tier lowers the barrier to entry, Designer makes pipelines accessible to non-engineers, and Declarative Pipelines (née DLT) is a mature, production-proven transformation engine.

If you’re already on Databricks, there’s no reason not to evaluate Lakeflow for new pipelines. If you’re evaluating Databricks, Lakeflow just made the platform significantly more compelling for data engineering workloads.

The migration from DLT is painless. The free ingestion tier is real. And Designer — surprisingly — doesn’t suck. That’s a win.

Have questions about Lakeflow or want to share your migration experience? Connect with me on LinkedIn.