Deploy and run

You have generated Lakeflow code and bundle resources on disk. The last step is shipping them to a Databricks workspace and running the job. Lakehouse Plumber’s work ends at generate; the Databricks CLI takes it from here.

Before you deploy

You need:

• A Databricks workspace with Unity Catalog and serverless compute enabled. The demo reads the read-only samples.tpch catalog present on every UC workspace.

• The Databricks CLI, authenticated to your workspace (databricks auth login --host https://<your-workspace>).

• The two edits from Configure the project: your workspace.host and your catalog (set in both databricks.yml and substitutions/dev.yaml).

Deploy

From the project root, validate the bundle and deploy it:

$ databricks bundle validate --target dev
$ databricks bundle deploy   --target dev

deploy uploads the generated Python (the sync.include glob in databricks.yml ships generated/dev/) and registers the four pipelines and the job in your workspace.

Run the job

The sample ships a hand-authored job, sample_job, that orchestrates everything in order — a data-prep notebook first, then the pipelines:

resources/sample_job.yml

    sample_job:
      name: lhp_sample_quickstart
      tasks:
        - task_key: data_prep
          notebook_task:
            notebook_path: ../notebooks/data_prep.py
            base_parameters:
              catalog: ${var.catalog}
              round: "auto"          # notebook auto-advances; override per-run if needed
        - task_key: ingest
          depends_on:
            - task_key: data_prep
          pipeline_task:
            pipeline_id: ${resources.pipelines.sample_ingest_pipeline.id}
        - task_key: silver
          depends_on:
            - task_key: ingest
          pipeline_task:
            pipeline_id: ${resources.pipelines.sample_silver_pipeline.id}
        - task_key: gold
          depends_on:
            - task_key: silver
          pipeline_task:
            pipeline_id: ${resources.pipelines.sample_gold_pipeline.id}

Run it:

$ databricks bundle run sample_job --target dev

The data_prep task seeds the demo — it creates the schemas and landing volume, slices samples.tpch into files for Auto Loader, and builds the CDC and snapshot source tables. Then ingest → silver → gold run in sequence. When it finishes, query the results:

-- Bronze: ingested raw tables
SELECT count(*) FROM <your_catalog>.sample_bronze.orders_raw;

-- Silver: cleansed orders and the SCD2 customer dimension
SELECT count(*) FROM <your_catalog>.sample_silver.orders;
SELECT count(*) FROM <your_catalog>.sample_silver.dim_customer;

-- Gold: the aggregated materialized view
SELECT * FROM <your_catalog>.sample_gold.sales_by_nation ORDER BY revenue DESC;

Run it again to see incremental behavior

Re-run the same command — nothing else:

$ databricks bundle run sample_job --target dev

The data-prep notebook detects the previous run and advances the demo one round: new files land (Auto Loader ingests only the new ones), the customer change feed gets fresh updates and deletes (the SCD2 dimension grows history rows), and a new supplier snapshot arrives. That’s the incremental machinery you declared — running end to end, on real Databricks compute.

What you just did

You took a folder of YAML from an empty directory to a running, incrementally updating medallion pipeline on Databricks — without hand-writing a line of PySpark or bundle YAML. Everything you configured (the catalog, the workspace, the compute) is in four files you now understand. That’s the full lifecycle: declare, generate, deploy, run.

Tip

To tear the demo down: databricks bundle destroy --target dev, then DROP SCHEMA <your_catalog>.sample_bronze CASCADE (and the silver/gold schemas). The serverless job is small but not free.

Next

You’ve run the whole thing from the terminal. Now see the same project in the visual workbench — Explore LHP Web.