Overview

If you have a specific data-engineering problem and want Lakehouse Plumber (LHP) to solve it, start here. Pick a category below and jump to the task.

Decide

• Decisions — Choose between Preset, Template, and Blueprint; pick a load source, write target, and write mode.

Ingest data

• Ingest with Auto Loader — Stream files from cloud object storage with Auto Loader (CloudFiles).

• Pipeline Patterns — Apply multi-source fan-in, path filtering, and other recipes from the patterns cookbook.

Operate and monitor

• Enable Monitoring — Set up centralized event-log monitoring across every pipeline in your project.

• Quarantine Records — Route failed rows to a Dead Letter Queue (DLQ) and recycle corrected rows back into the pipeline.

Reusability and patterns

• Multi-Flowgroup YAML Files — Combine multiple FlowGroups in a single YAML file with shared settings and inheritance.

• Dynamic Templates Guide — Use Jinja2 conditionals, loops, and filters in Templates for advanced parameter patterns.

Deploy

• Configure Bundles — Enable Databricks Asset Bundle (DAB) integration for environment-specific deployments.

• How to Set Up CI/CD for an LHP Project — Apply CI/CD patterns and DataOps practices for GitHub Actions, Azure DevOps, and Bitbucket Pipelines.

See also

• Architecture — Explanation of the LHP execution model and why generation works the way it does.

• Actions Reference — Reference catalog of every Load, Transform, Write, and Test action.