======== Glossary ======== .. meta:: :description: Definitions of Lakehouse Plumber (LHP) terms — pipeline, flowgroup, action, preset, template, blueprint, substitution syntaxes, and Lakeflow target types. Definitions of Lakehouse Plumber (LHP) terms used throughout this documentation. .. glossary:: :sorted: Pipeline A logical grouping label declared by every flowgroup as ``pipeline: ``. All flowgroups sharing that name generate Python files into the same output folder and produce one Lakeflow pipeline resource per pipeline name. A pipeline is the deployment unit for a Databricks bundle. FlowGroup A logical slice of a pipeline — typically one source table or business entity — composed of an ordered list of actions. One YAML file can hold one or many flowgroups and declares its parent pipeline, its own name, optional ``job_name``, optional local ``variables``, applied presets, and an optional template reference. Action A single step inside a flowgroup. Actions have four top-level ``type`` values (``load``, ``transform``, ``write``, ``test``), each with sub-types selected by an additional discriminator field. Load action An action with ``type: load`` that reads external data into a temporary view. Sub-types: ``cloudfiles``, ``delta``, ``sql``, ``python``, ``jdbc``, ``kafka``, ``custom_datasource``. One load action per data source. Transform action An action with ``type: transform`` that reshapes or checks data already loaded into a view. Sub-types: ``sql``, ``python``, ``data_quality``, ``schema``, ``temp_table``. Zero or many per flowgroup. Write action An action with ``type: write`` that persists the final dataset. Sub-types: ``streaming_table``, ``materialized_view``, ``sink``. One write action per output table or sink. Test action An action with ``type: test`` that asserts a property of the data — row count, uniqueness, referential integrity, completeness, range, schema match, lookup coverage, custom SQL, or custom expectations. Test actions are only generated when ``lhp generate`` is invoked with ``--include-tests``. Preset A YAML file of default *values* deep-merged into actions matched by type and sub-type. Presets resolve before substitutions and before validation, and may extend other presets via ``extends:``. A preset changes an action's settings, never a flowgroup's shape. Template A YAML file of parameterized *actions* applied to a flowgroup via ``use_template:`` and ``template_parameters:``. LHP renders Jinja2 ``{{ }}`` placeholders inside the template and expands the rendered actions into the flowgroup. Blueprint A higher-order pattern that instantiates *multiple* flowgroups at once. Where a template parameterizes actions inside one flowgroup, a blueprint parameterizes the flowgroups themselves. Blueprints declare ``parameters`` and a ``flowgroups`` array of flowgroup specs. Blueprint instance A YAML file that supplies parameter values to a blueprint via ``use_blueprint:`` plus a nested ``parameters:`` block. Each ``(spec, instance)`` pair expands into one generated flowgroup. Substitution Token replacement in YAML resolved by LHP before code generation. Four syntaxes resolve in a fixed order: ``%{local_var}`` then ``{{ template_param }}`` then ``${env_token}`` then ``${secret:scope/key}``. Each layer's output can feed the next. Local variable A flowgroup-scoped value declared under ``variables:`` and referenced as ``%{name}``. Local variables resolve first, so their output can contain template parameters, environment tokens, or secret references that later layers resolve. Environment token A substitution of the form ``${name}`` resolved against ``substitutions/.yaml``. The same flowgroup YAML generates different Python per environment because environment tokens differ between ``dev``, ``staging``, and ``prod``. The bare ``{name}`` form is deprecated. Streaming table A Lakeflow target type for incremental, append-only or CDC datasets. Generated by write actions with ``write_target.type: streaming_table``. Supports standard, CDC, and snapshot-CDC modes plus append flows from multiple sources. Materialized view A Lakeflow target type for query-defined, engine-maintained datasets. Generated by write actions with ``write_target.type: materialized_view``. The query is supplied by a ``source`` view, inline ``sql``, or an external ``sql_path``. Sink A write action with ``write_target.type: sink`` that emits data to an external system rather than a Unity Catalog table. Sink types: ``delta``, ``kafka``, ``eventhubs``, ``foreachbatch``, ``custom``. Append flow A Lakeflow construct that appends rows from a source into a streaming table. LHP emits one append flow per source when a write action lists multiple sources. CDC Change Data Capture — replicating row-level inserts, updates, and deletes from a source into a target. LHP exposes it through streaming-table write actions with ``mode: cdc`` (from a change feed) or ``mode: snapshot_cdc`` (from periodic full snapshots). Snapshot CDC A streaming-table mode where LHP applies point-in-time snapshots via Lakeflow's apply-changes-from-snapshot API. Configured with ``write_target.mode: snapshot_cdc`` and ``snapshot_cdc_config``, driven by a ``source`` table or a user-supplied ``source_function``. SCD Slowly Changing Dimension — a dimensional-modeling pattern for tracking historical attribute changes. LHP supports SCD Type 1 (overwrite) and Type 2 (history rows) through streaming-table CDC configuration. Expectation A data-quality rule applied by a ``data_quality`` transform action. Expectations are loaded from an ``expectations_file`` and translated into Lakeflow ``@dp.expect``-family decorators. Combined with quarantine mode they support dead-letter recycling. Quarantine A data-quality mode (``mode: quarantine``) that splits a stream into passing and failing rows, writing failures to a configured ``dlq_table`` for replay rather than failing the pipeline. Operational metadata Auto-injected columns (timestamps, source file paths, pipeline run IDs, and similar) added to write targets. Defined under ``operational_metadata`` in ``lhp.yaml`` as named columns and presets, then enabled per flowgroup or action. Event-log monitoring A project-level feature configured under ``monitoring:`` in ``lhp.yaml``. LHP generates a notebook that unions pipeline event logs, a Lakeflow pipeline of materialized views over that table, and a Databricks job that chains the two. Skill The LHP skill installed by ``lhp skill install``. The skill package ships inside LHP and is copied to ``.claude/skills/lhp/`` (or ``~/.claude/skills/lhp/`` with ``--user``); it provides agent-targeted context for authoring LHP configurations. A project install also writes a routing block into the project's ``CLAUDE.md``. Lakeflow Short for Lakeflow Spark Declarative Pipelines (SDP) — the Databricks framework LHP generates Python for. Generated code uses the ``pyspark.pipelines`` API, imported as ``dp``.