Fan one blueprint across tenants and regions¶

The Get Started course introduces a blueprint with a single parameter: one region fanned across three region files. Real platforms vary on more than one axis at once — a tenant and the region it runs in, a schema and the partitioning that schema needs. Bake those into copies of a pipeline and you maintain the same plumbing in triplicate: every schema tweak is a find-and-replace across tenants, and the region frozen into each copy is one more thing that drifts.

Let’s write the per-tenant bronze layer once as a blueprint that takes several parameters, then bind a tenant, its region, and its partition column per instance. One definition expands into a pipeline per tenant. This guide assumes the single-parameter intro from the Get Started course and goes deeper: multiple parameters per instance, required versus optional-with-default, and how specs times instances expand.

Before you start¶

You need a Lakehouse Plumber project and the blueprint basics from the Get Started course — what a blueprint is, and that instance files bind its parameters. This guide reuses the course’s shape (CSV cloudfiles loads into bronze streaming tables) and adds a second identity parameter and an optional config parameter on top.

Parameterize the pattern¶

A blueprint declares its parameters once, then uses %{parameter} placeholders wherever a value must vary per instance. Create blueprints/tenant_bronze.yaml:

blueprints/tenant_bronze.yaml¶

# A blueprint is a parametrised pattern written ONCE. Each instance file under
# pipelines/ binds the parameters, and the expander stamps out a full set of
# flowgroups for that binding.
#
# Fan-out: 2 flowgroup specs (orders, customers) x 3 tenant instances = 6 flowgroups.
#   %{tenant} / %{region} / %{partition_col}  -> resolved per instance
#   ${catalog} / ${bronze_schema} / ...       -> resolved per environment
name: tenant_bronze
version: "1.0"
description: "Per-tenant bronze ingest (orders + customers), fanned across tenants and regions."

parameters:
  # Two REQUIRED identity parameters: every instance must bind both.
  - name: tenant
    required: true
    description: "Tenant identifier; drives the pipeline name, flowgroup names, and table names."
  - name: region
    required: true
    description: "Region code; drives the landing path and is stamped into every table name."
  # One OPTIONAL config parameter with a default: instances may omit it.
  - name: partition_col
    required: false
    default: ingest_date
    description: "Column the bronze tables partition by. Defaults to ingest_date."

flowgroups:
  # ---- orders ingest ----
  - pipeline: "%{tenant}_bronze"
    flowgroup: "%{tenant}_orders_ingest"
    actions:
      - name: load_orders
        type: load
        source:
          type: cloudfiles
          path: "${landing_path}/%{tenant}/%{region}/orders/*.csv"
          format: csv
        target: v_orders_raw

      - name: write_orders
        type: write
        source: v_orders_raw
        write_target:
          type: streaming_table
          catalog: "${catalog}"
          schema: "${bronze_schema}"
          table: "%{tenant}_%{region}_orders"
          partition_columns:
            - "%{partition_col}"

  # ---- customers ingest ----
  - pipeline: "%{tenant}_bronze"
    flowgroup: "%{tenant}_customers_ingest"
    actions:
      - name: load_customers
        type: load
        source:
          type: cloudfiles
          path: "${landing_path}/%{tenant}/%{region}/customers/*.csv"
          format: csv
        target: v_customers_raw

      - name: write_customers
        type: write
        source: v_customers_raw
        write_target:
          type: streaming_table
          catalog: "${catalog}"
          schema: "${bronze_schema}"
          table: "%{tenant}_%{region}_customers"
          partition_columns:
            - "%{partition_col}"

Three parameters, of two kinds. tenant and region are required: true — every instance must bind both, or expansion fails. partition_col is required: false with a default: of ingest_date — an instance may supply it or leave it out. The two flowgroup specs, orders and customers, then reference the parameters:

%{tenant} drives the pipeline name (%{tenant}_bronze), both flowgroup names, and the leading segment of every table name.
%{region} drives a path segment and is stamped into every table name, so acme in us and acme in eu land in distinct tables.
%{partition_col} reaches a config value — partition_columns on the write target — not just a name. Parameters bind data-shaping config, not only identifiers.

One syntax rule carries the whole design, and it goes deeper than the course’s single-parameter case. Identity fields — pipeline: and flowgroup: — accept only the %{...} local-variable syntax, resolved as the blueprint expands. The ${...} environment tokens (${catalog}, ${bronze_schema}, ${landing_path}) are the opposite: they are allowed everywhere except those two fields, because they resolve later, per environment. So %{tenant} and %{region} name the pipelines and tables, while ${catalog} and ${landing_path} fill in the environment. Put a ${...} token in a pipeline: or flowgroup: string and expansion rejects it with LHP-VAL-044 — those names are the index the expander builds before environment substitution runs, so they must be stable across environments.

Bind each tenant¶

Each tenant is an instance — a small file that names the blueprint with use_blueprint: and binds the parameters under parameters:. Create pipelines/tenants/acme.yaml:

pipelines/tenants/acme.yaml¶

# One instance = one tenant. Both required parameters are bound; partition_col
# is overridden away from its default.
use_blueprint: tenant_bronze
parameters:
  tenant: acme
  region: us
  partition_col: order_date

acme binds all three parameters, overriding partition_col to order_date. The other two tenants are the same file with different values — and globex omits partition_col entirely:

pipelines/tenants/globex.yaml¶

# partition_col is omitted, so the blueprint default (ingest_date) applies.
use_blueprint: tenant_bronze
parameters:
  tenant: globex
  region: eu

Because globex supplies no partition_col, the blueprint default ingest_date applies. That is the effective-parameter rule: blueprint defaults first, then the instance’s parameters override them. The third tenant, initech in apac, is four lines with the same shape.

The instance format is strict, and the strictness is the payoff — it turns a typo into a build error instead of a silently missing pipeline:

A required parameter is enforced. Drop tenant or region from an instance and validation fails with LHP-VAL-042, naming the missing parameter.
An unknown parameter is rejected. Write tennant: and you get LHP-VAL-043 with a “did you mean” suggestion against the declared names.
An unknown blueprint is rejected. A use_blueprint: that names no blueprint fails with LHP-VAL-041, again with a suggestion.

Instance files live under the instance_include glob, which defaults to pipelines/**/*.yaml — the same tree as hand-written flowgroups. That overlap is intentional: Lakehouse Plumber routes by content, treating a file with a use_blueprint: key as an instance and everything else as a flowgroup, so instances and ordinary flowgroups coexist in pipelines/. Blueprint definitions default to blueprints/**/*.yaml. A flowgroup spec accepts more than the fields shown here — presets, use_template, variables, operational_metadata — and the blueprint reference catalogs every field and error code in full.

Expand into pipelines¶

Validate first, then generate. Validation resolves every parameter, checks the required ones, and expands the blueprint before it commits to writing code:

$ lhp validate --env dev
✓ discover (0.01s)
✓ preflight (0.00s)
✓ acme_bronze  0 files
✓ globex_bronze  0 files
✓ initech_bronze  0 files
✓ validate (0.55s)
3 validated · 0.6s

$ lhp generate --env dev
✓ discover (0.01s)
✓ preflight (0.00s)
✓ acme_bronze  2 files
✓ globex_bronze  2 files
✓ initech_bronze  2 files
✓ generate (0.52s)
✓ format (0.02s)
✓ monitoring (0.00s)
3 pipelines generated · 6 files · 0.6s

Two specs across three instances expanded into six flowgroups. That is the whole arithmetic of a blueprint: N specs times M instances equals N times M flowgroups, here 2 times 3 equals 6, grouped into one pipeline per tenant. lhp list blueprints --instances shows the expansion explicitly:

$ lhp list blueprints --instances
Blueprints
Name           Version  Params  Specs  Instances
tenant_bronze  1.0      3       2      3
Total blueprints: 1
Instances
  tenant_bronze (blueprints/tenant_bronze.yaml):
    - acme.yaml:    2 flowgroup(s) -> pipeline(s) ['acme_bronze']
    - globex.yaml:  2 flowgroup(s) -> pipeline(s) ['globex_bronze']
    - initech.yaml: 2 flowgroup(s) -> pipeline(s) ['initech_bronze']

Each instance produced two flowgroups and its own named pipeline. --instances is the current form of the command; the older lhp list-blueprints no longer exists.

Read what Lakehouse Plumber wrote¶

Open generated/dev/acme_bronze/acme_orders_ingest.py. It is an ordinary, complete pipeline module — the tenant and region baked into the names, and not a placeholder left:

generated/dev/acme_bronze/acme_orders_ingest.py¶

# Generated by LakehousePlumber
# Pipeline: acme_bronze
# FlowGroup: acme_orders_ingest

from pyspark import pipelines as dp

# Pipeline Configuration
PIPELINE_ID = "acme_bronze"
FLOWGROUP_ID = "acme_orders_ingest"


# ============================================================================
# SOURCE VIEWS
# ============================================================================


@dp.temporary_view()
def v_orders_raw():
    """Load data from csv files at /Volumes/dev_catalog/landing/files/acme/us/orders/*.csv"""
    df = (
        spark.readStream.format("cloudFiles")
        .option("cloudFiles.format", "csv")
        .load("/Volumes/dev_catalog/landing/files/acme/us/orders/*.csv")
    )

    return df


# ============================================================================
# TARGET TABLES
# ============================================================================

# Create the streaming table
dp.create_streaming_table(
    name="dev_catalog.bronze.acme_us_orders",
    comment="Streaming table: acme_us_orders",
    partition_cols=["order_date"],
)


# Define append flow(s)
@dp.append_flow(
    target="dev_catalog.bronze.acme_us_orders",
    name="f_orders",
    comment="Append flow to dev_catalog.bronze.acme_us_orders",
)
def f_orders():
    """Append flow to dev_catalog.bronze.acme_us_orders"""
    # Streaming flow
    df = spark.readStream.table("v_orders_raw")

    return df

Every parameter came through. %{tenant} and %{region} resolved into the table name dev_catalog.bronze.acme_us_orders and the landing path /Volumes/dev_catalog/landing/files/acme/us/orders/*.csv; the ${...} tokens resolved from substitutions/dev.yaml. And partition_col reached the config: partition_cols=["order_date"], because acme overrode the default. The other five modules are the same shape, one per tenant and table — and globex and initech, which omitted partition_col, carry partition_cols=["ingest_date"] from the blueprint default.

Note

An instance file and a hand-written flowgroup share the pipelines/ tree. Lakehouse Plumber tells them apart by content: a file with a use_blueprint: key is expanded as an instance, and every other file under the flowgroup glob is read as an ordinary flowgroup. You do not separate them by directory.

What you just did¶

One 71-line blueprint and three instance files — 16 lines in total — expanded into 6 flowgroups across 3 pipelines, 312 lines of Lakeflow Python in six modules. The pattern now scales on two axes at once. A new tenant is one four-line instance and inherits the whole bronze layer, region and partitioning included. A new table for every tenant is one more spec in the blueprint, and all three tenants get it in the same generate run — 2 specs became 3 the moment you add one, and 6 modules become 9.

That is the deep end of reuse: a preset removes duplicated configuration, a template removes duplicated structure, and a multi-parameter blueprint removes duplicated pipelines across every axis your platform repeats on. You describe the pattern once; Lakehouse Plumber writes it everywhere.

What’s next¶

See every blueprint field. The blueprint reference catalogs the full parameter schema, every flowgroup spec field (variables, presets, use_template, operational_metadata), the parameter-resolution phases, and the complete error-code list.
Trace the fan-out as a graph. lhp dag fully expands blueprint instances — one node per instance — so the dependency graph and job orchestration reflect every generated pipeline, not just one.
Deploy with bundles. You have generated pipelines on disk. The bundle step packages them for a Databricks workspace, tenants and all.