Ingest¶
A load action reads data into a pipeline. Which one you reach for depends on
where the data lives.
Your data is… |
Reach for |
Guide |
|---|---|---|
Files landing in cloud storage (JSON, CSV, Parquet) |
Auto Loader — incremental, exactly-once file ingestion |
|
An existing Delta table (any catalog) |
Delta load — batch or streaming, cross-catalog |
|
The result of a SQL query over existing tables |
SQL load |
|
In an operational database (Postgres, MySQL, …) |
JDBC load |
|
A streaming Kafka topic |
Kafka load |
|
Behind an API or protocol with no built-in loader |
A custom Python data source |