Explore the pipelines¶
The sample project is three pipelines built from eight flowgroups. This page reads the interesting ones so you can see how actions — the load / transform / write / test steps — compose into real pipelines. You don’t run anything here; you learn to read the YAML.
Every flowgroup follows the same shape: a pipeline and flowgroup name,
then a list of actions (or a use_template that expands into them). Each
action has a type and names a target view or table the next action reads.
Ingest: cloudfiles and delta¶
01_ingest/ reads raw data into bronze streaming tables two ways.
The cloudfiles (Auto Loader) ingests are driven by a template — one pattern instantiated twice, for JSON orders and CSV lineitem:
# Cloudfiles ingestion (JSON) via the cloudfiles_ingest template.
# schema_hints is a PATH → file-based hints (auto-detected by LHP).
pipeline: sample_ingest
flowgroup: orders_ingest
use_template: cloudfiles_ingest
operational_metadata: ["_processing_timestamp", "_source_file_path", "_source_file_size"]
template_parameters:
entity: orders
file_format: json
schema_hints: schemas/orders_hints.yaml # path → file-based hints (auto-detected)
Eight lines. use_template pulls in the ingestion pattern; the
template_parameters fill its {{ ... }} blanks. The
operational_metadata list opts this flowgroup into the columns defined in
lhp.yaml. Templates are how
one pattern serves many tables.
The delta ingest reads the read-only samples.tpch catalog directly — no
copy step, a cross-catalog load in a single flowgroup:
pipeline: sample_ingest
flowgroup: nation_region
presets:
- bronze_layer
operational_metadata: ["_processing_timestamp"]
actions:
- name: load_nation
type: load
source:
type: delta
catalog: samples # read-only sample catalog
schema: tpch
table: nation
readMode: stream
target: v_nation
- name: write_nation
type: write
Note the presets: [bronze_layer] line — a preset deep-merges shared
config (here, enabling Change Data Feed and a quality tag) into every
matching write. Presets, templates, and blueprints
are the three reuse tools, each removing a different kind of duplication.
Transform: four types in one chain¶
02_silver/orders_clean.yaml is the richest flowgroup — it chains all four
transform types between a load and a write:
- silver_quality
operational_metadata: ["_processing_timestamp"]
variables:
entity: orders
actions:
- name: "load_%{entity}_bronze"
type: load
source:
type: delta
catalog: "${catalog}" # fully qualified — a different pipeline wrote it
schema: "${bronze_schema}"
table: orders_raw
readMode: stream
target: "v_%{entity}_bronze"
- name: "sql_filter_%{entity}"
type: transform
transform_type: sql
source: "v_%{entity}_bronze"
sql: "SELECT * FROM stream(v_%{entity}_bronze) WHERE o_totalprice > 0"
target: "v_%{entity}_filtered"
- name: "schema_%{entity}"
type: transform
transform_type: schema
source: "v_%{entity}_filtered"
schema_file: schema_transforms/orders_typed.yaml
enforcement: permissive
Read the chain top to bottom:
a sql transform filters bad rows,
a schema transform renames and casts columns (
o_totalprice→total_price),a data_quality transform applies expectations from a file, and
a python transform localizes every timestamp column at runtime.
Each action’s target is the next action’s source. The %{entity}
tokens are local variables — a within-flowgroup substitution that names every
view consistently. Because the silver pipeline is separate from ingest, the load
reads the fully qualified bronze table (Lakeflow views are pipeline-scoped, so a
bare view name from another pipeline wouldn’t resolve).
Write: CDC, snapshot CDC, and a materialized view¶
The silver dimensions show the two change-data-capture write modes.
dim_customer reads a Change Data Feed and applies it as SCD2 history:
- name: write_dim_customer
type: write
source: v_customer_changes_clean
write_target:
type: streaming_table
catalog: "${catalog}"
schema: "${silver_schema}"
table: dim_customer
mode: cdc
cdc_config:
keys: [c_custkey]
sequence_by: _commit_version # CDF metadata column
scd_type: 2
apply_as_deletes: "_change_type = 'delete'"
except_column_list: [_change_type, _commit_version, _commit_timestamp]
dim_supplier applies periodic full snapshots instead, via a source
function — a write action with no upstream source:
actions:
- name: write_dim_supplier
type: write
write_target:
type: streaming_table
catalog: "${catalog}"
schema: "${silver_schema}"
table: dim_supplier
mode: snapshot_cdc
snapshot_cdc_config:
source_function:
file: functions/supplier_snapshot.py
function: next_supplier_snapshot
parameters: # resolved from substitutions, passed as kwargs
catalog: "${catalog}"
bronze_schema: "${bronze_schema}"
keys: [s_suppkey]
stored_as_scd_type: 2
track_history_column_list: [s_name, s_address, s_acctbal]
And 03_gold/sales_by_nation.yaml writes a materialized view from an
external SQL file:
flowgroup: sales_by_nation
operational_metadata: ["_processing_timestamp"]
actions:
- name: mv_sales_by_nation
type: write
write_target:
type: materialized_view
catalog: "${catalog}"
schema: "${gold_schema}"
table: sales_by_nation
sql_path: sql/sales_by_nation.sql # external SQL file (sql_path showcase)
Four write modes across three flowgroups — streaming table, CDC, snapshot CDC, materialized view. The Write guides explain when to reach for each.
What you just read¶
Eight flowgroups, every action kind, four write modes, two ingestion sources, and all three reuse tools — and not one line of PySpark. The YAML says what each pipeline is; Lakehouse Plumber writes the how. Next, watch it do exactly that.
Next¶
Turn this YAML into Lakeflow Python — Generate and inspect.