CLI Reference ============= .. meta:: :description: Command-line reference for lhp: generate, validate, deps, show, stats, and all available options. The **Lakehouse Plumber** command-line interface provides project creation, validation, code generation and more. All commands are implemented with `click `_ so you can use the usual ``--help`` flags. .. click:: lhp.cli.main:cli :prog: lhp :nested: full .. seealso:: For detailed information about dependency analysis and orchestration job generation, see :doc:`configure_bundles`. Project Initialization ---------------------- The ``lhp init`` command creates a new LakehousePlumber project with the complete directory structure and configuration templates. Basic Usage ~~~~~~~~~~~ .. code-block:: bash # Create a standard project lhp init my_project # Create a Databricks Asset Bundle project lhp init my_project --bundle **Created Directory Structure:** .. code-block:: text my_project/ ├── config/ # Configuration templates │ ├── job_config.yaml.tmpl │ └── pipeline_config.yaml.tmpl ├── pipelines/ # Pipeline flowgroups ├── templates/ # Reusable action templates ├── presets/ # Configuration presets ├── substitutions/ # Environment-specific values ├── schemas/ # Table schema definitions ├── expectations/ # Data quality expectations ├── generated/ # Generated Python code (gitignored) └── resources/ # Bundle resources (--bundle only) Configuration Templates ~~~~~~~~~~~~~~~~~~~~~~~ The ``config/`` directory contains template files for customizing Databricks job and pipeline settings: **job_config.yaml.tmpl** Template for customizing orchestration job configuration used with ``lhp deps`` command: - Job execution settings (max_concurrent_runs, timeout, performance_target) - Queue configuration - Email and webhook notifications - Scheduling (Quartz cron expressions) - Tags and permissions **Usage:** .. code-block:: bash # 1. Copy and customize the template cd my_project cp config/job_config.yaml.tmpl config/job_config.yaml # Edit config/job_config.yaml with your settings # 2. Use with lhp deps command lhp deps --job-config config/job_config.yaml --bundle-output See :doc:`bundle_config_reference` for detailed job configuration options. **pipeline_config.yaml.tmpl** Template for customizing Databricks Lakeflow Declarative Pipelines settings used with ``lhp generate`` command: - ``catalog`` and ``schema`` — required for bundle projects, defined either in ``project_defaults`` (project-wide) or per pipeline. See :doc:`configure_catalog_schema` for resolution rules. - Compute configuration (serverless vs. classic clusters) - Pipeline edition and runtime channel - Processing mode (continuous vs. triggered) - Notifications and tags - Event logging (can also be configured project-wide in ``lhp.yaml`` without requiring ``-pc``) **Usage:** .. code-block:: bash # 1. Copy and customize the template cp config/pipeline_config.yaml.tmpl config/pipeline_config.yaml # Edit config/pipeline_config.yaml with your settings (catalog, schema, …) # 2. Pass the path on every generate invocation lhp generate -e dev --pipeline-config config/pipeline_config.yaml The ``--pipeline-config`` (``-pc``) flag tells LHP where to find the file that defines ``catalog`` and ``schema``. Bundle projects require it on every ``lhp generate`` run; without it, every pipeline fails fast with a ``BundleResourceError`` pointing at :doc:`configure_catalog_schema`. See :doc:`bundle_config_reference` for detailed pipeline configuration options and :doc:`configure_catalog_schema` for the catalog/schema resolution order. .. note:: Template files (\*.tmpl) are provided as starting points. Copy and remove the ``.tmpl`` extension to activate them. This allows you to keep the original templates as reference while customizing your own versions. Regeneration behavior ~~~~~~~~~~~~~~~~~~~~~ .. deprecated:: The ``--force`` flag on ``lhp generate`` and ``--no-state`` are retained for backwards compatibility but are no-ops. Every ``lhp generate`` run regenerates Python output unconditionally and wipes ``resources/lhp/`` before rewriting one ``.pipeline.yml`` per pipeline. **Behavior:** - Bare ``lhp generate -e ``: regenerates Python code under ``generated/``. Skip bundle sync with ``--no-bundle``. - ``lhp generate -e --pipeline-config ``: regenerates Python code and rewrites every file under ``resources/lhp/`` from the pipeline config. - Files outside ``resources/lhp/`` are never touched, with one exception: the monitoring job YAML at ``resources/.job.yml``, identified by its ``# Generated by LakehousePlumber - Monitoring Job`` header. .. note:: The ``lhp skill install --force`` flag is unrelated and remains active — it overwrites an existing skill install. Skill Management ---------------- LHP ships a Claude Code skill (``SKILL.md`` + reference markdown) that teaches Claude how to author LHP YAML configurations. The ``lhp skill`` command group installs and maintains this skill in your project (or your user-global Claude Code directory). Subcommands ~~~~~~~~~~~ .. code-block:: bash # Install the skill into /.claude/skills/lhp/ lhp skill install # Install for the current user (~/.claude/skills/lhp/) lhp skill install --user # Overwrite an existing install lhp skill install --force # Update an existing install to the current LHP version lhp skill update # Show installed version, current LHP version, and any drift lhp skill status # Remove the install (prompts for confirmation) lhp skill uninstall # Remove without prompting lhp skill uninstall --force Behavior ~~~~~~~~ - ``install`` errors if a skill is already present unless ``--force`` is given. - ``update`` errors if no install is found (use ``install`` instead). The skill is always overwritten — local modifications under ``.claude/skills/lhp/`` will be replaced. - ``status`` reports one of: not installed, up-to-date, update available, installed-newer-than-CLI (suggests ``pip install -U lakehouse-plumber``), or foreign-install (no marker file — suggests ``install --force``). - An ``.lhp_skill_version`` marker file inside the install directory tracks which LHP version produced the content; this file is the source of truth for ``status`` and ``update``. When to use ``--user`` ~~~~~~~~~~~~~~~~~~~~~~ Use ``--user`` when you want the skill available across every project on your machine. The default project-scoped install is best when each project might be pinned to a different LHP version. By default, Lakehouse Plumber skips test actions during code generation for faster builds and cleaner production pipelines. Use the ``--include-tests`` flag to include data quality tests when needed. **Common Usage Patterns:** .. code-block:: bash # Production builds (default) - fast, clean pipelines lhp generate -e prod lhp generate -e dev # Development with data quality testing lhp generate -e dev --include-tests # Validate without test actions (default) lhp validate -e dev # Validate with test actions lhp validate -e dev --include-tests **Flag Behavior:** Both commands respect ``--include-tests`` for per-flowgroup processing. - **Without flag**: Test actions are skipped during per-flowgroup processing in both validation and generation - **With flag**: Test actions are included — validated for configuration errors and generated as temporary DLT tables **Examples:** .. code-block:: bash # Skip tests for faster CI/CD builds lhp generate -e prod --dry-run # Include tests for comprehensive validation lhp generate -e dev --include-tests # Preview test generation without writing files lhp generate -e dev --include-tests --dry-run .. note:: Test actions generate temporary DLT tables that persist test results for inspection and debugging while being automatically cleaned up when the pipeline completes. .. tip:: When ``test_reporting`` is configured in ``lhp.yaml``, the ``--include-tests`` flag also generates a test reporting event hook that publishes DQ results to external systems. See :doc:`actions/test_reporting`.