Write to an external sink¶

Your pipeline builds a gold order_summary table, but the team that consumes it works out of a different Unity Catalog — one your pipeline doesn’t manage. You need to push a copy of the rows out to that external catalog as they land, without making the external table part of your pipeline’s own managed output.

You could hand-write that push: register the destination with dp.create_sink(...), then wire an @dp.append_flow that streams your view into it. Or you declare a type: sink write and let Lakehouse Plumber write that Lakeflow for you. That is the idea on every page: declare your ETL, don’t hand-write it.

Let’s route a gold order_summary table out to an external analytics catalog through a Delta sink, without touching PySpark.

Before you start¶

You need Lakehouse Plumber installed and a project to work in. If you followed the Get Started course you already have one; otherwise scaffold an empty project first. This guide assumes the gold order_summary table already exists in your pipeline’s catalog, and that the external analytics_shared catalog you’re exporting to already exists — a sink writes to a destination, it does not create the catalog or grant you access to it.

Declare the sink write¶

A pipeline in Lakehouse Plumber is a flowgroup: a short YAML file describing a sequence of actions. This one has two — a load that streams the gold table into a view, and a write whose write_target is a sink.

Create pipelines/order_summary_export.yaml:

pipelines/order_summary_export.yaml¶

# Route a gold table out of the pipeline-managed catalog into an external
# analytics catalog, via a Delta sink.
# Action chain: load (delta, streaming) -> write (sink: delta).
pipeline: export_analytics
flowgroup: order_summary_export

actions:
  # 1. Load: stream the in-pipeline gold order summary into a view.
  - name: load_order_summary
    type: load
    source:
      type: delta
      catalog: "${catalog}"
      schema: "${gold_schema}"
      table: order_summary
    readMode: stream
    target: v_order_summary

  # 2. Write: push the view out to an external Delta table via a sink,
  #    instead of hand-writing dp.create_sink + the append flow.
  - name: export_order_summary
    type: write
    source: v_order_summary
    write_target:
      type: sink
      sink_type: delta
      sink_name: order_summary_export
      options:
        tableName: "${analytics_catalog}.${analytics_schema}.order_summary"

The write action’s write_target is typed sink with sink_type: delta. sink_name is the identifier LHP registers the sink under. Everything under options is passed straight through to the Delta writer — here tableName names the external three-part table the rows land in. A Delta sink takes either tableName (a Unity Catalog table) or path (a file-based Delta table), never both.

delta is one of four sink kinds. A kafka sink streams rows to an Apache Kafka topic or Azure Event Hubs; a custom sink writes through a Python DataSink; and a foreachbatch sink runs custom Python per micro-batch for merges and multi-destination writes. This guide stays on the Delta sink — the write action reference documents the other three, end to end.

Point the tokens at your environment¶

The ${...} tokens resolve from a per-environment substitutions file. Create substitutions/dev.yaml:

substitutions/dev.yaml¶

# Development environment tokens.
# Referenced from flowgroups with ${token} syntax.
dev:
  catalog: dev_catalog
  gold_schema: gold
  analytics_catalog: analytics_shared
  analytics_schema: reporting

catalog and gold_schema locate the source table; analytics_catalog and analytics_schema name the external destination. This is the only place environment-specific values live, so exporting from a production catalog later is a prod.yaml with production values — the flowgroup itself never changes.

Generate the pipeline¶

Now compile the YAML into Lakeflow code. Validate first, then generate:

$ lhp validate --env dev
✓ discover (0.01s)
✓ preflight (0.00s)
✓ export_analytics  0 files
✓ validate (0.34s)
1 validated · 0.3s

$ lhp generate --env dev
✓ discover (0.01s)
✓ preflight (0.00s)
✓ export_analytics  1 file
✓ generate (0.34s)
✓ format (0.05s)
✓ monitoring (0.00s)
1 pipeline generated · 1 file · 0.4s

validate resolves every token and checks the actions before you commit to generating; generate writes the Python.

Read what Lakehouse Plumber wrote¶

Open generated/dev/export_analytics/order_summary_export.py. This is the entire output — nothing is hidden behind a runtime:

generated/dev/export_analytics/order_summary_export.py¶

# Generated by LakehousePlumber
# Pipeline: export_analytics
# FlowGroup: order_summary_export

from pyspark.sql import functions as F
from pyspark import pipelines as dp

# Pipeline Configuration
PIPELINE_ID = "export_analytics"
FLOWGROUP_ID = "order_summary_export"


# ============================================================================
# SOURCE VIEWS
# ============================================================================


@dp.temporary_view()
def v_order_summary():
    """Delta source: dev_catalog.gold.order_summary"""
    df = spark.readStream.table("dev_catalog.gold.order_summary")

    return df


# ============================================================================
# TARGET TABLES
# ============================================================================

# Delta sink to analytics_shared.reporting.order_summary

# Create Delta sink
dp.create_sink(
    name="order_summary_export",
    format="delta",
    options={"tableName": "analytics_shared.reporting.order_summary"},
)


# Define append flow(s)
@dp.append_flow(
    target="order_summary_export",
    name="f_order_summary_export_1",
    comment="Delta sink to analytics_shared.reporting.order_summary",
)
def f_order_summary_export_1():
    df = spark.readStream.table("v_order_summary")

    return df

It’s ordinary Lakeflow. The source view is a @dp.temporary_view wrapping spark.readStream.table("dev_catalog.gold.order_summary"). The sink itself is a dp.create_sink(name="order_summary_export", format="delta", options={...}) — the destination registered under your sink_name, with your options resolved to a plain dict. Then a generated @dp.append_flow streams the view into that sink. You described a source view and a destination table; LHP wrote the sink registration and the append flow that connect them.

Note

A sink is the opposite direction from a streaming-table write. The streaming-table write from Get Started emits dp.create_streaming_table for a table your pipeline owns and manages. A sink emits dp.create_sink plus an append flow to push rows out to a destination the pipeline does not manage. A Delta sink with tableName writes a real Unity Catalog table that downstream actions can read back; a path-based Delta sink or a Kafka sink is purely terminal — nothing in the pipeline reads it again.

What you just did¶

Twenty-one lines of YAML compiled to forty-nine lines of Lakeflow Python. Zero lines of the sink plumbing came from you — not the dp.create_sink registration, not the @dp.append_flow, not the readStream that feeds it. Change the destination table and regenerate, and LHP rewrites the registration to match — you never touch the Python.

The payoff compounds. Adding a second export, switching the destination catalog per environment, or swapping the Delta sink for a Kafka sink is a few more lines of YAML — not another forty-nine lines of Python each time.

What’s next¶

Stream to a message bus. A sink_type: kafka write pushes rows to an Apache Kafka topic or Azure Event Hubs instead of a Delta table. The source view must carry key and value columns; the write action reference shows how.
Run custom logic per batch. A foreachbatch sink runs your own Python on each micro-batch (merges, upserts, multi-destination writes), and a custom sink writes through a Python DataSink for any destination Databricks does not support directly.
See every sink option. The full option surface for each sink kind — and the tableName / path rule for Delta — is in the write action reference, with the constraints on each.