Blueprints¶

A blueprint is a parametrised collection of FlowGroup specs expanded once per instance file. Each (blueprint, instance, spec) triple yields one synthetic FlowGroup that joins the regular generation pipeline alongside hand-written FlowGroups.

This page catalogs the blueprint and instance file formats, parameter resolution rules, discovery behavior, generation behavior, CLI surface, and error codes. For when to use a blueprint versus a Preset or Template, see Decisions.

Blueprint file¶

Place blueprint files under the configured blueprint_include patterns (default blueprints/**/*.yaml and blueprints/**/*.yml). Each file is a single-document YAML matching the Blueprint model.

blueprints/erp_ingestion.yaml¶

name: erp_ingestion
version: "1.0"
description: "Standard ERP ingestion for all regional sites"

parameters:
  - name: site_name
    required: true
    description: "Site identifier"
  - name: site_id
    required: true
  - name: partition_key
    required: false
    default: order_date

flowgroups:
  - pipeline: "%{site_name}_erp_raw"
    flowgroup: "%{site_name}_orders_ingestion"
    actions:
      - name: load_orders
        type: load
        source:
          type: cloudfiles
          path: "/Volumes/raw/erp/%{site_name}/orders"
          format: json
        target: v_orders_raw
      - name: write_orders
        type: write
        source: v_orders_raw
        write_target:
          type: streaming_table
          catalog: ${catalog}
          schema: bronze
          table: "%{site_name}_orders_raw"
          partition_columns:
            - "%{partition_key}"

Top-level fields¶

Field	Type	Default	Notes
`name`	string	required	Unique across the project. Referenced by `use_blueprint:` in instance files. Duplicates raise `LHP-VAL-046`.
`version`	string	`"1.0"`	Free-form string for change tracking.
`description`	string	`null`	Surfaced by `lhp list-blueprints`.
`parameters`	list	`[]`	Parameter declarations (see below).
`flowgroups`	list	required	FlowGroup specs. Each entry is expanded once per instance. Empty list fails Pydantic validation.

Parameter declarations¶

Each entry in parameters: is a BlueprintParameter object.

Field	Type	Default	Notes
`name`	string	required	The key the instance file binds a value to. Strict-checked.
`required`	bool	`false`	If `true`, every instance must supply this parameter (`LHP-VAL-042` otherwise).
`default`	any	`null`	Applied when `required: false` and the instance omits the value.
`description`	string	`null`	Surfaced by `lhp list-blueprints --verbose`.

Unknown parameter keys in an instance file raise LHP-VAL-043 with a difflib “did you mean” suggestion against the declared parameter names.

FlowGroup spec fields¶

Each entry in flowgroups: is a BlueprintFlowgroupSpec. It accepts the same fields as a regular FlowGroup file.

Field	Required	Notes
`pipeline`	yes	Template string. Only `%{var}` permitted (see Parameter resolution).
`flowgroup`	yes	Template string. Same restriction as `pipeline`.
`job_name`	no	Optional job name; pass-through to the FlowGroup.
`variables`	no	Map of derived local variables. Spec `variables` win over instance parameters on key conflict.
`presets`	no	List of preset names applied to actions.
`use_template`	no	Template name. Renders into the spec’s action list.
`template_parameters`	no	Map of values for the referenced template. `%{var}` resolves at Step 0.5; Jinja2 `{{ var }}` resolves at Step 1.
`actions`	no	Inline action list.
`operational_metadata`	no	Bool or list of metadata column names.

A blueprint with N specs and M instances expands to N × M synthetic FlowGroups.

Instance file¶

Place instance files anywhere matched by instance_include (default pipelines/**/*.yaml and pipelines/**/*.yml). The instance file format is strict (extra="forbid"); unknown top-level keys are parse errors.

Preferred syntax¶

pipelines/sites/apac_sg.yaml¶

use_blueprint: erp_ingestion
parameters:
  site_name: apac_sg
  site_id: SG001

Field	Required	Notes
`use_blueprint`	yes	Name of the target blueprint. Unknown name raises `LHP-VAL-041`.
`parameters`	no	Flat map of `parameter_name: value`. Required parameters declared by the blueprint must appear here.
`overrides`	no	Reserved for future use. Accepted but unused.

Legacy syntax (deprecated)¶

Deprecated since version 0.8: The blueprint: + flat-parameters form is removed in V0.9. Migrate to use_blueprint: + nested parameters:.

The legacy form uses a top-level blueprint: key with parameter values as flat top-level siblings. A normalization validator rewrites legacy files to the preferred shape at parse time and emits a deprecation warning once per file path. Mixing both forms in the same file raises LHP-VAL-061.

Parameter resolution¶

Two substitution syntaxes apply to blueprints. They differ in scope and resolution phase.

Token	Where it works	When it resolves
`%{var}`	Anywhere in the blueprint.	In `pipeline:` / `flowgroup:` fields at Step 0 (expansion). Elsewhere at Step 0.5 (per-FlowGroup, after expansion).
`${env_token}`	Anywhere except `pipeline:` and `flowgroup:`.	Step 3 (environment substitution).
`${secret:scope/key}`	Same as `${env_token}`.	Step 3.

The expander rejects any ${...} token inside pipeline: or flowgroup: template strings with LHP-VAL-044. The match regex is \$\{[^}]+\}, so environment tokens and secret references are both rejected. The resolved (pipeline, flowgroup) tuple is used as the source-path index before Step 3 runs; allowing ${...} would make those keys unstable across environments.

Unresolved %{var} in pipeline: or flowgroup: raises LHP-VAL-055.

Effective parameter map¶

For each (instance, spec) pair the expander builds a merged variable map:

Blueprint parameter defaults.
Instance parameters values (override step 1).
Spec variables block (override step 2).

Spec variables win on key conflict — this protects blueprint-author intent against an instance that accidentally names a parameter the same as a derived spec variable.

Discovery¶

Default discovery globs:

blueprint_include: ["blueprints/**/*.yaml", "blueprints/**/*.yml"]
instance_include:  ["pipelines/**/*.yaml",  "pipelines/**/*.yml"]

Both instance_include and the regular include glob match pipelines/**/*.yaml by default. Discovery routes by content shape, not by path:

File has top-level use_blueprint: or legacy blueprint: key → instance.
File has top-level parameters: and flowgroups: and no actions: key → blueprint.
File matched by the flowgroup include glob but shaped like a blueprint → LHP-CFG-040 (move to blueprints/).
Anything else under a flowgroup glob → flowgroup.

Override blueprint_include or instance_include in lhp.yaml only to restrict discovery (for example, to gate a subset of sites during a rollout).

Generation behavior¶

Synthetic FlowGroups produced by expansion enter the regular processing pipeline. The expansion step is the only step exclusive to blueprints.

Step 0 — Blueprint expansion. %{var} resolved in pipeline: and flowgroup: fields; ${...} rejected there.
Step 0.5 — Local variables. %{var} resolved everywhere else against the merged parameter map.
Step 1 — Template render. Jinja2 {{ var }} substituted from template_parameters.
Steps 1.5 / 2 — Preset merge. Template-level then FlowGroup-level presets applied as deep-merge defaults.
Step 3 — Environment + secrets. ${env_token} and ${secret:scope/key} resolved.
Steps 3.5–5 — Validation. Unresolved tokens, FlowGroup shape, secret references.
Code generation. Python files written under generated/<env>/.

CLI commands¶

`lhp list-blueprints`¶

Lists discovered blueprints with parameter and instance counts.

lhp list-blueprints
lhp list-blueprints --verbose

--verbose runs the expander against each instance and prints every resolved (pipeline, flowgroup) tuple it produces.

`lhp show --instance`¶

Expands one instance and prints the resolved configuration. Mutually exclusive with the positional FLOWGROUP argument.

lhp show --instance pipelines/sites/apac_sg.yaml --env dev

Passing both forms raises LHP-CFG-057; passing neither raises LHP-CFG-058.

`lhp validate`¶

Runs blueprint discovery, expansion, and full FlowGroup validation.

lhp validate --env dev

`lhp deps`¶

By default the dependency graph deduplicates synthetic FlowGroups by (blueprint_name, spec_index) — one logical edge per spec, not per instance.

lhp deps
lhp deps --expand-blueprints
lhp deps --blueprint erp_ingestion

See Dependency Analysis & Job Generation for the full deps reference.

Error codes¶

Blueprint-related codes. The project-wide catalog lives in Error Reference.

Code	Category	Meaning
`LHP-CFG-040`	Config	Blueprint file matched by a flowgroup include glob. Move to `blueprints/` or adjust `include` / `blueprint_include`.
`LHP-VAL-041`	Validation	Instance references an unknown blueprint name. Error includes a “did you mean” suggestion.
`LHP-VAL-042`	Validation	Instance is missing a required parameter.
`LHP-VAL-043`	Validation	Instance has an unknown parameter key. Often a typo; error includes a suggestion.
`LHP-VAL-044`	Validation	`${...}` substitution token appears in a `pipeline:` or `flowgroup:` template string. Only `%{var}` permitted there.
`LHP-VAL-045`	Validation	Two instances expand to the same `(pipeline, flowgroup)` tuple. Both file paths in error context.
`LHP-VAL-046`	Validation	Two blueprint files declare the same `name`.
`LHP-CFG-047` … `050`	Config	Blueprint file shape errors: empty, multi-document, wrong shape, failed Pydantic validation.
`LHP-CFG-051` / `052` / `054`	Config	Instance file shape errors: empty, multi-document, invalid parse.
`LHP-VAL-053`	Validation	Instance file missing `use_blueprint:` / `blueprint:` key.
`LHP-VAL-055`	Validation	Unresolved `%{var}` in a `pipeline:` or `flowgroup:` template.
`LHP-CFG-057`	Config	`lhp show` called with both `FLOWGROUP` and `--instance`.
`LHP-CFG-058`	Config	`lhp show` called with neither `FLOWGROUP` nor `--instance`.
`LHP-IO-059`	I/O	Instance file path resolves outside the project root.
`LHP-VAL-061`	Validation	Conflicting instance syntax: both `use_blueprint:` and legacy `blueprint:`, or `use_blueprint:` plus unexpected top-level keys.