Configure the project¶

Before you generate a line of code, learn where a Lakehouse Plumber project keeps its settings. Four files control everything; the sample ships them filled in, so this page is a guided tour — read each one, understand what it drives, and you’ll know exactly what to configure when you scaffold your own project.

Only two values actually need editing to run the sample. They’re flagged below with edit.

`lhp.yaml` — the project¶

The root lhp.yaml holds project-wide settings that apply to every flowgroup. Open it:

lhp.yaml¶

# LakehousePlumber Project Configuration — sample quickstart (TPC-H)
name: sample_lakehouse
version: "1.0"
description: "LHP sample quickstart: TPC-H medallion demo against samples.tpch"
author: ""
created_date: "2026-07-16T17:05:00.939324"

## Operational metadata columns available to flowgroups.
## Each sample flowgroup opts in explicitly with a list, e.g.:
##   operational_metadata: ["_processing_timestamp"]
## The file-lineage columns only make sense at the view (load) stage of
## cloudfiles ingestion, so their applies_to is restricted to ["view"].
## Note: the CDC flowgroups (dim_customer, dim_supplier) deliberately do NOT
## enable operational metadata — a per-run _processing_timestamp flowing
## through apply_changes would churn SCD2 history (see README).
operational_metadata:
  columns:
    _processing_timestamp:
      expression: "F.current_timestamp()"
      description: "When LHP processed the record"
      applies_to: ["view", "streaming_table", "materialized_view"]
    _source_file_path:
      expression: "F.col('_metadata.file_path')"
      description: "Source file (cloudfiles only)"
      applies_to: ["view"]
    _source_file_size:
      expression: "F.col('_metadata.file_size')"
      description: "Source file size in bytes (cloudfiles only)"
      applies_to: ["view"]

event_log:
  catalog: "${catalog}"
  schema: _meta
  name_suffix: "_event_log"

monitoring:
  checkpoint_path:  "/Volumes/${catalog}/_meta/checkpoints"
  job_config_path: "config/monitoring_job_config.yaml"

Three things live here in the sample:

``operational_metadata.columns`` — reusable column definitions (a processing timestamp, the source file path and size) that any flowgroup can opt into by name. Defining them once here keeps the expressions consistent across pipelines. See the operational-metadata reference.
``event_log`` — where Lakeflow writes each pipeline’s event log table.
``monitoring`` — checkpoint location and a job config for the generated monitoring pipeline (covered in the Operate guides).

Nothing here needs editing for the sample. When you start your own project, lhp.yaml is where project-level defaults belong.

`substitutions/dev.yaml` — per-environment values¶

Every ${...} token in your flowgroups resolves from the substitutions file for the environment you generate against. This is the one place environment-specific values live:

substitutions/dev.yaml¶

# ⚠️ IMPORTANT: Environment name 'dev' must match databricks.yml target name 'dev'
# This file: substitutions/dev.yaml → databricks.yml target: dev
# LHP generates resources under: resources/lhp/

dev:
  catalog: main                    # ← edit: a catalog you can CREATE in (quickstart step 2)
  bronze_schema: sample_bronze
  silver_schema: sample_silver
  gold_schema: sample_gold
  landing_volume: /Volumes/main/sample_bronze/landing   # keep in sync with catalog/bronze_schema above
  target_tz: America/New_York      # consumed by the python timezone transform

# Secret scope configuration — only needed if you activate
# examples_optional/jdbc_with_secret.yaml.txt (see README).
# secrets:
#   default_scope: sample-secrets
#   scopes:
#     db_secrets: my-db-secret-scope

Set the two highlighted values:

edit catalog — a Unity Catalog catalog you can CREATE in. The demo makes three schemas (sample_bronze, sample_silver, sample_gold) and a landing volume inside it.
edit landing_volume — keep it in sync with the catalog above: /Volumes/<your_catalog>/sample_bronze/landing.

prod.yaml sits alongside it with production values. The flowgroups never change between environments — only these files do. That is the whole point of substitution: logic separated from surroundings.

`config/pipeline_config.yaml` — catalog, schema, and compute¶

Because the sample is bundle-enabled, lhp generate needs to know the catalog, schema, and compute to stamp into each generated pipeline resource. That comes from a pipeline config, passed on every generate and validate with -pc:

config/pipeline_config.yaml¶

  serverless: true                # all three sample pipelines run serverless
  channel: CURRENT
  catalog: "${catalog}"           # from substitutions/<env>.yaml
  schema: "${bronze_schema}"      # default schema; overridden per pipeline below

---
pipeline: sample_silver
schema: "${silver_schema}"

---
pipeline: sample_gold
schema: "${gold_schema}"

It’s a multi-document YAML: the first document sets project-wide defaults (serverless compute, the catalog); each following document overrides one pipeline (silver and gold get their own schema). The catalog and schema are ${...} tokens, so this file needs no edits — it follows your substitutions/dev.yaml automatically.

`databricks.yml` — the bundle¶

A project becomes a bundle project the moment databricks.yml sits at its root. It defines the bundle name, where generated resources live, and one target per environment:

databricks.yml¶

# This is a Declarative Automation Bundles definition for sample_lakehouse.
# See https://docs.databricks.com/dev-tools/bundles/index.html for documentation.
bundle:
  name: sample_lakehouse
  uuid: 5bf55550-e950-4270-a425-dfa06459ec3f

include:
  # User-managed resource files (shared across all environments)
  - resources/*.yml
  - resources/lhp/*.yml

sync:
  include:
   - generated/${bundle.target}/**

variables:
  catalog:
    description: "UC catalog the demo writes to (you need CREATE on it)"
    default: main

targets:
  # ⚠️ IMPORTANT: Target names MUST match your LHP substitution environment names
  # - Each target name should have a corresponding substitutions/{env}.yaml file
  # LHP automatically adds environment-specific variables to each corresponding target below.
  #
  # Quickstart step 2: set the workspace host below AND the catalog — here
  # (variables.catalog) and in substitutions/dev.yaml. The bundle variable
  # drives the data-prep job parameter; the substitution drives LHP codegen.

  dev:
    # The default target uses 'mode: development' to create a development copy.
    # - Deployed resources get prefixed with '[dev my_user_name]'
    # - Any job schedules and triggers are paused by default.
    # See also https://docs.databricks.com/dev-tools/bundles/deployment-modes.html.
    mode: development
    default: true
    workspace:
      host: https://<your-workspace>.cloud.databricks.com   # ← edit (quickstart step 2)
      root_path: ~/.bundle/${bundle.name}/${bundle.target}

  prod:
    mode: production
    workspace:
      host: https://<your-workspace>.cloud.databricks.com
      # We explicitly deploy to a fixed workspace path to make sure we only have a single copy.
      root_path: /Workspace/<code_location>/.bundle/${bundle.name}/${bundle.target}
    permissions:
      - service_principal_name: <service_principal_id>
        level: CAN_MANAGE
    run_as:
      service_principal_name: <service_principal_id>

Two things to know, and one edit:

The catalog bundle variable is separate from the catalog substitution token. The variable feeds the data-prep notebook at run time; the token feeds lhp generate at build time. They must agree — so set the same catalog in both databricks.yml and substitutions/dev.yaml.
edit workspace.host under targets.dev — your workspace URL.
The sync.include glob ships generated/ to the workspace; include pulls in the bundle resources (both LHP-managed and hand-written).

The two edits, in one place

To run the sample you set exactly two things: your workspace.host in databricks.yml, and your catalog in both databricks.yml (the variable) and substitutions/dev.yaml (the token). Everything else is ready.

Next¶

You’ve seen where every setting lives. Now open the pipelines those settings feed — Explore the pipelines.

Configure the project¶

lhp.yaml — the project¶

substitutions/dev.yaml — per-environment values¶

config/pipeline_config.yaml — catalog, schema, and compute¶

databricks.yml — the bundle¶

Next¶

`lhp.yaml` — the project¶

`substitutions/dev.yaml` — per-environment values¶

`config/pipeline_config.yaml` — catalog, schema, and compute¶

`databricks.yml` — the bundle¶