Where to next¶

You’ve run the sample end to end. You know the four config files, the action model, the four write modes, and the deploy lifecycle. Two things are left: how to start your own project, and where to go for the features the sample didn’t cover.

Start your own project¶

Scaffold an empty project instead of the sample:

mkdir my_project
cd my_project
lhp init my_project          # add --no-bundle to skip Databricks packaging

Then, using what the sample showed you:

Set your environment values in substitutions/dev.yaml — the ${...} tokens your flowgroups reference.
Add flowgroups under pipelines/, one YAML file per pipeline.
If it’s a bundle project, keep config/pipeline_config.yaml and pass it with -pc on every generate.
Reach for presets, templates, and blueprints once a pattern starts repeating.

The Guides go deep on each task — every ingestion source, every write mode, every transform type.

Features the sample didn’t cover¶

The sample project is deliberately compact. Lakehouse Plumber does more — here is what you skipped, and where to learn it when you need it.

Data tests and reporting¶

Beyond the data_quality transform (which gates rows within a flow), LHP has a fourth action kind — test actions — for asserting things about your data: uniqueness, row counts, referential integrity, and more. Test results can be published to an external system like Azure DevOps Test Plans or a Delta audit table.

Monitoring and logging¶

The sample enabled monitoring in lhp.yaml and LHP generated a monitoring pipeline — but the course didn’t dig in. LHP can consolidate every pipeline’s event log into one place so you can watch health, throughput, and data-quality metrics across all your pipelines.

Keep bad data instead of dropping it¶

The sample drops rows that fail expectations (mode: dqe). When a failed row is something you can’t silently lose, quarantine it instead — route it to a dead-letter table you can inspect, fix, and recycle back into the pipeline.

Quarantine bad rows instead of dropping them

Develop faster¶

You edited YAML in a text editor and ran the CLI. LHP has richer authoring surfaces:

Develop in a sandbox — an isolated, scoped copy of your pipelines for safe iteration.
Develop in the local web IDE — a browser UI with a graph view, form editing, and live generation.
Get authoring help from the in-IDE assistant.
Drive Lakehouse Plumber from a coding agent.

Scale and ship¶

Fan one blueprint across tenants and regions — generate whole pipelines from a single pattern.
Validate and ship pipelines in CI.
Migrate an existing DLT pipeline to Lakehouse Plumber.

Understand the model¶

If you want the reasoning underneath what you just did, the Concepts section explains how Lakehouse Plumber compiles pipelines, why dependencies resolve the way they do, and how the reuse ladder is meant to be climbed.