CLI Reference¶

The Lakehouse Plumber command-line interface provides project creation, validation, code generation and more. All commands are implemented with click so you can use the usual --help flags.

lhp¶

LakehousePlumber - Generate Lakeflow pipelines from YAML configs.

Usage

lhp [OPTIONS] COMMAND [ARGS]...

Options

--version¶: Show the version and exit.

-v, --verbose¶: Enable verbose logging

deps¶

Analyze and visualize pipeline dependencies for orchestration planning.

Usage

lhp deps [OPTIONS]

Options

-f, --format <format>¶

Output format(s) to generate (dot=GraphViz, json=structured data, text=readable report, job=orchestration job)

Options:: dot | json | text | job | all

-o, --output <output>¶: Output directory (defaults to .lhp/dependencies/)

-p, --pipeline <pipeline>¶: Analyze specific pipeline only

-j, --job-name <job_name>¶: Custom name for generated orchestration job (only used with job format)

-jc, --job-config <job_config>¶: Custom job config file path (relative to project root, defaults to templates/bundle/job_config.yaml)

-b, --bundle-output¶: Save job file to resources/ directory for Databricks bundle integration

--expand-blueprints¶: Emit one flowgroup per (blueprint x instance x spec) instead of deduping. Default behavior collapses synthetic flowgroups by (blueprint_name, spec_index) to keep the graph readable at scale.

--blueprint <blueprint_name>¶: Restrict the dependency graph to flowgroups expanded from the named blueprint. Combine with –expand-blueprints to see per-instance edges.

-v, --verbose¶: Enable verbose output

generate¶

Generate DLT pipeline code

Usage

lhp generate [OPTIONS]

Options

-e, --env <env>¶: Required Environment

-p, --pipeline <pipeline>¶: Specific pipeline to generate

-o, --output <output>¶: Output directory (defaults to generated/{env})

--dry-run¶: Preview without generating files

-f, --force¶: Force regeneration of all files, even if unchanged

--no-bundle¶: Disable bundle support even if databricks.yml exists

--include-tests¶: Include test actions in generation (skipped by default for faster builds)

-pc, --pipeline-config <pipeline_config>¶: Custom pipeline config file path (relative to project root)

--max-workers <max_workers>¶: Maximum worker processes. Default: ~80 percent of detected CPU count (honors cgroup limits on Linux), capped at the workload size. Override with the LHP_MAX_WORKERS env var. Use 1 for sequential.

--no-state¶: Deprecated no-op; retained for backwards compatibility.

info¶

Display project information and statistics.

Usage

lhp info [OPTIONS]

init¶

Initialize a new LakehousePlumber project in the current directory.

PROJECT_NAME is used for template rendering (e.g. bundle name, lhp.yaml). All files are created in the current working directory.

Usage

lhp init [OPTIONS] PROJECT_NAME

Options

--no-bundle¶: Skip Databricks Asset Bundle setup (bundle is enabled by default)

Arguments

PROJECT_NAME¶: Required argument

list-blueprints¶

List available blueprints with parameter and instance counts.

Usage

lhp list-blueprints [OPTIONS]

Options

-v, --verbose¶: Show each instance and the resolved pipelines it produces

list-presets¶

List available presets

Usage

lhp list-presets [OPTIONS]

list-templates¶

List available templates

Usage

lhp list-templates [OPTIONS]

show¶

Show resolved configuration for a flowgroup or blueprint instance.

Usage

lhp show [OPTIONS] [FLOWGROUP]

Options

-e, --env <env>¶: Environment

--instance <instance_path>¶: Path to a blueprint instance file; expands only that instance and displays the resolved flowgroups (M4).

Arguments

FLOWGROUP¶: Optional argument

skill¶

Manage the LHP Claude Code skill installation.

Usage

lhp skill [OPTIONS] COMMAND [ARGS]...

install¶

Install the LHP Claude Code skill.

Usage

lhp skill install [OPTIONS]

Options

--user¶: Install to ~/.claude/skills/lhp/ instead of <cwd>/.claude/skills/lhp/

--force¶: Overwrite an existing install without prompting

status¶

Show installed skill version and drift state.

Usage

lhp skill status [OPTIONS]

Options

--user¶: Check the install at ~/.claude/skills/lhp/

uninstall¶

Remove the LHP Claude Code skill installation.

Usage

lhp skill uninstall [OPTIONS]

Options

--user¶: Remove the install at ~/.claude/skills/lhp/

--force¶: Skip the confirmation prompt

update¶

Update the installed LHP skill to the current LHP version.

Usage

lhp skill update [OPTIONS]

Options

--user¶: Update the install at ~/.claude/skills/lhp/

--yes¶: Skip confirmation when installed version is newer than the CLI

stats¶

Display pipeline statistics and complexity metrics.

Usage

lhp stats [OPTIONS]

Options

-p, --pipeline <pipeline>¶: Specific pipeline to analyze

substitutions¶

Show available substitution tokens for an environment

Usage

lhp substitutions [OPTIONS]

Options

-e, --env <env>¶: Environment

validate¶

Validate pipeline configurations

Usage

lhp validate [OPTIONS]

Options

-e, --env <env>¶: Environment

-p, --pipeline <pipeline>¶: Specific pipeline to validate

-v, --verbose¶: Verbose output

--include-tests¶: Include test actions in validation (matches generate behavior)

--max-workers <max_workers>¶: Maximum worker processes. Default: ~80 percent of detected CPU count (honors cgroup limits on Linux), capped at the workload size. Override with the LHP_MAX_WORKERS env var. Use 1 for sequential.

Project Initialization¶

The lhp init command creates a new LakehousePlumber project with the complete directory structure and configuration templates.

Basic Usage¶

# Create a standard project
lhp init my_project

# Create a Databricks Asset Bundle project
lhp init my_project --bundle

Created Directory Structure:

my_project/
├── config/                    # Configuration templates
│   ├── job_config.yaml.tmpl
│   └── pipeline_config.yaml.tmpl
├── pipelines/                 # Pipeline flowgroups
├── templates/                 # Reusable action templates
├── presets/                   # Configuration presets
├── substitutions/            # Environment-specific values
├── schemas/                  # Table schema definitions
├── expectations/             # Data quality expectations
├── generated/                # Generated Python code (gitignored)
└── resources/                # Bundle resources (--bundle only)

Configuration Templates¶

The config/ directory contains template files for customizing Databricks job and pipeline settings:

job_config.yaml.tmpl

Template for customizing orchestration job configuration used with lhp deps command:

Job execution settings (max_concurrent_runs, timeout, performance_target)
Queue configuration
Email and webhook notifications
Scheduling (Quartz cron expressions)
Tags and permissions

Usage:

# 1. Copy and customize the template
cd my_project
cp config/job_config.yaml.tmpl config/job_config.yaml
# Edit config/job_config.yaml with your settings

# 2. Use with lhp deps command
lhp deps --job-config config/job_config.yaml --bundle-output

See Bundle Configuration Reference for detailed job configuration options.

pipeline_config.yaml.tmpl

Template for customizing Databricks Lakeflow Declarative Pipelines settings used with lhp generate command:

catalog and schema — required for bundle projects, defined either in project_defaults (project-wide) or per pipeline. See Configuring catalog and schema for pipelines for resolution rules.
Compute configuration (serverless vs. classic clusters)
Pipeline edition and runtime channel
Processing mode (continuous vs. triggered)
Notifications and tags
Event logging (can also be configured project-wide in lhp.yaml without requiring -pc)

Usage:

# 1. Copy and customize the template
cp config/pipeline_config.yaml.tmpl config/pipeline_config.yaml
# Edit config/pipeline_config.yaml with your settings (catalog, schema, …)

# 2. Pass the path on every generate invocation
lhp generate -e dev --pipeline-config config/pipeline_config.yaml

The --pipeline-config (-pc) flag tells LHP where to find the file that defines catalog and schema. Bundle projects require it on every lhp generate run; without it, every pipeline fails fast with a BundleResourceError pointing at Configuring catalog and schema for pipelines.

See Bundle Configuration Reference for detailed pipeline configuration options and Configuring catalog and schema for pipelines for the catalog/schema resolution order.

Note

Template files (*.tmpl) are provided as starting points. Copy and remove the .tmpl extension to activate them. This allows you to keep the original templates as reference while customizing your own versions.

Regeneration behavior¶

Deprecated since version The: --force flag on lhp generate and --no-state are retained for backwards compatibility but are no-ops. Every lhp generate run regenerates Python output unconditionally and wipes resources/lhp/ before rewriting one .pipeline.yml per pipeline.

Behavior:

Bare lhp generate -e <env>: regenerates Python code under generated/. Skip bundle sync with --no-bundle.
lhp generate -e <env> --pipeline-config <file>: regenerates Python code and rewrites every file under resources/lhp/ from the pipeline config.
Files outside resources/lhp/ are never touched, with one exception: the monitoring job YAML at resources/<name>.job.yml, identified by its # Generated by LakehousePlumber - Monitoring Job header.

Note

The lhp skill install --force flag is unrelated and remains active — it overwrites an existing skill install.

Skill Management¶

LHP ships a Claude Code skill (SKILL.md + reference markdown) that teaches Claude how to author LHP YAML configurations. The lhp skill command group installs and maintains this skill in your project (or your user-global Claude Code directory).

Subcommands¶

# Install the skill into <cwd>/.claude/skills/lhp/
lhp skill install

# Install for the current user (~/.claude/skills/lhp/)
lhp skill install --user

# Overwrite an existing install
lhp skill install --force

# Update an existing install to the current LHP version
lhp skill update

# Show installed version, current LHP version, and any drift
lhp skill status

# Remove the install (prompts for confirmation)
lhp skill uninstall

# Remove without prompting
lhp skill uninstall --force

Behavior¶

install errors if a skill is already present unless --force is given.
update errors if no install is found (use install instead). The skill is always overwritten — local modifications under .claude/skills/lhp/ will be replaced.
status reports one of: not installed, up-to-date, update available, installed-newer-than-CLI (suggests pip install -U lakehouse-plumber), or foreign-install (no marker file — suggests install --force).
An .lhp_skill_version marker file inside the install directory tracks which LHP version produced the content; this file is the source of truth for status and update.

When to use `--user`¶

Use --user when you want the skill available across every project on your machine. The default project-scoped install is best when each project might be pinned to a different LHP version.

By default, Lakehouse Plumber skips test actions during code generation for faster builds and cleaner production pipelines. Use the --include-tests flag to include data quality tests when needed.

Common Usage Patterns:

# Production builds (default) - fast, clean pipelines
lhp generate -e prod
lhp generate -e dev

# Development with data quality testing
lhp generate -e dev --include-tests

# Validate without test actions (default)
lhp validate -e dev

# Validate with test actions
lhp validate -e dev --include-tests

Flag Behavior:

Both commands respect --include-tests for per-flowgroup processing.

Without flag: Test actions are skipped during per-flowgroup processing in both validation and generation
With flag: Test actions are included — validated for configuration errors and generated as temporary DLT tables

Examples:

# Skip tests for faster CI/CD builds
lhp generate -e prod --dry-run

# Include tests for comprehensive validation
lhp generate -e dev --include-tests

# Preview test generation without writing files
lhp generate -e dev --include-tests --dry-run

Note

Test actions generate temporary DLT tables that persist test results for inspection and debugging while being automatically cleaned up when the pipeline completes.

Tip

When test_reporting is configured in lhp.yaml, the --include-tests flag also generates a test reporting event hook that publishes DQ results to external systems. See Test Result Reporting (Publishing).

CLI Reference¶

lhp¶

deps¶

generate¶

info¶

init¶

list-blueprints¶

list-presets¶

list-templates¶

show¶

skill¶

install¶

status¶

uninstall¶

update¶

stats¶

substitutions¶

validate¶

Project Initialization¶

Basic Usage¶

Configuration Templates¶

Regeneration behavior¶

Skill Management¶

Subcommands¶

Behavior¶

When to use --user¶

When to use `--user`¶