Glossary

Definitions of Lakehouse Plumber (LHP) terms used throughout this documentation.

Action

A single step inside a flowgroup. Actions have four top-level type values (load, transform, write, test), each with sub-types selected by an additional discriminator field.

Append flow

A Lakeflow construct that appends rows from a source into a streaming table. LHP emits one append flow per source when a write action lists multiple sources.

Blueprint

A higher-order pattern that instantiates multiple flowgroups at once. Where a template parameterizes actions inside one flowgroup, a blueprint parameterizes the flowgroups themselves. Blueprints declare parameters and a flowgroups array of flowgroup specs.

Blueprint instance

A YAML file that supplies parameter values to a blueprint via use_blueprint: plus a nested parameters: block. Each (spec, instance) pair expands into one generated flowgroup.

CDC

Change Data Capture — replicating row-level inserts, updates, and deletes from a source into a target. LHP exposes it through streaming-table write actions with mode: cdc (from a change feed) or mode: snapshot_cdc (from periodic full snapshots).

Environment token

A substitution of the form ${name} resolved against substitutions/<env>.yaml. The same flowgroup YAML generates different Python per environment because environment tokens differ between dev, staging, and prod. The bare {name} form is deprecated.

Event-log monitoring

A project-level feature configured under monitoring: in lhp.yaml. LHP generates a notebook that unions pipeline event logs, a Lakeflow pipeline of materialized views over that table, and a Databricks job that chains the two.

Expectation

A data-quality rule applied by a data_quality transform action. Expectations are loaded from an expectations_file and translated into Lakeflow @dp.expect-family decorators. Combined with quarantine mode they support dead-letter recycling.

FlowGroup

A logical slice of a pipeline — typically one source table or business entity — composed of an ordered list of actions. One YAML file can hold one or many flowgroups and declares its parent pipeline, its own name, optional job_name, optional local variables, applied presets, and an optional template reference.

Lakeflow

Short for Lakeflow Spark Declarative Pipelines (SDP) — the Databricks framework LHP generates Python for. Generated code uses the pyspark.pipelines API, imported as dp.

Load action

An action with type: load that reads external data into a temporary view. Sub-types: cloudfiles, delta, sql, python, jdbc, kafka, custom_datasource. One load action per data source.

Local variable

A flowgroup-scoped value declared under variables: and referenced as %{name}. Local variables resolve first, so their output can contain template parameters, environment tokens, or secret references that later layers resolve.

Materialized view

A Lakeflow target type for query-defined, engine-maintained datasets. Generated by write actions with write_target.type: materialized_view. The query is supplied by a source view, inline sql, or an external sql_path.

Operational metadata

Auto-injected columns (timestamps, source file paths, pipeline run IDs, and similar) added to write targets. Defined under operational_metadata in lhp.yaml as named columns and presets, then enabled per flowgroup or action.

Pipeline

A logical grouping label declared by every flowgroup as pipeline: <name>. All flowgroups sharing that name generate Python files into the same output folder and produce one Lakeflow pipeline resource per pipeline name. A pipeline is the deployment unit for a Databricks bundle.

Preset

A YAML file of default values deep-merged into actions matched by type and sub-type. Presets resolve before substitutions and before validation, and may extend other presets via extends:. A preset changes an action’s settings, never a flowgroup’s shape.

Quarantine

A data-quality mode (mode: quarantine) that splits a stream into passing and failing rows, writing failures to a configured dlq_table for replay rather than failing the pipeline.

SCD

Slowly Changing Dimension — a dimensional-modeling pattern for tracking historical attribute changes. LHP supports SCD Type 1 (overwrite) and Type 2 (history rows) through streaming-table CDC configuration.

Sink

A write action with write_target.type: sink that emits data to an external system rather than a Unity Catalog table. Sink types: delta, kafka, eventhubs, foreachbatch, custom.

Skill

The LHP skill installed by lhp skill install. The skill package ships inside LHP and is copied to .claude/skills/lhp/ (or ~/.claude/skills/lhp/ with --user); it provides agent-targeted context for authoring LHP configurations. A project install also writes a routing block into the project’s CLAUDE.md.

Snapshot CDC

A streaming-table mode where LHP applies point-in-time snapshots via Lakeflow’s apply-changes-from-snapshot API. Configured with write_target.mode: snapshot_cdc and snapshot_cdc_config, driven by a source table or a user-supplied source_function.

Streaming table

A Lakeflow target type for incremental, append-only or CDC datasets. Generated by write actions with write_target.type: streaming_table. Supports standard, CDC, and snapshot-CDC modes plus append flows from multiple sources.

Substitution

Token replacement in YAML resolved by LHP before code generation. Four syntaxes resolve in a fixed order: %{local_var} then {{ template_param }} then ${env_token} then ${secret:scope/key}. Each layer’s output can feed the next.

Template

A YAML file of parameterized actions applied to a flowgroup via use_template: and template_parameters:. LHP renders Jinja2 {{ }} placeholders inside the template and expands the rendered actions into the flowgroup.

Test action

An action with type: test that asserts a property of the data — row count, uniqueness, referential integrity, completeness, range, schema match, lookup coverage, custom SQL, or custom expectations. Test actions are only generated when lhp generate is invoked with --include-tests.

Transform action

An action with type: transform that reshapes or checks data already loaded into a view. Sub-types: sql, python, data_quality, schema, temp_table. Zero or many per flowgroup.

Write action

An action with type: write that persists the final dataset. Sub-types: streaming_table, materialized_view, sink. One write action per output table or sink.