Kafka exporter

The Kafka exporter is a separate mode of SQLLoad. It runs continuously, reads Kafka topics and writes each topic into a table. Its only destination is Snowflake, through Snowpipe Streaming or staged files.

How it differs from a load project

	Load project	Kafka exporter
sqlload.toml	[connections], [targets] and [sources]	[source] with kind = "kafka", [destination] and [staging], plus the optional sections in Configuration
Runs	Once per sqlload load	Continuously: sqlload run exports until stopped, or for --max-duration
Destinations	Every target	Snowflake: Snowpipe Streaming, or files staged on a table stage or in S3
Commands	debug, plan, apply, load, copy, contract, logged-state	debug, tables, plan, apply, run, status

load, copy, contract and logged-state refuse an exporter configuration. The repository’s sqlload.example.toml is a complete exporter configuration.

[source]
kind = "kafka"
brokers = "${ENV:KAFKA_BROKERS}"
include = ["shop.*"]

[destination]
kind = "snowflake"
account = "myorg-myaccount"
user = "SQLLOAD"
auth = { method = "pat", token = "${ENV:SNOWFLAKE_TOKEN}" }
database = "RAW"
schema = "KAFKA"

[staging]
kind = "none"

Pages

Page	Covers
Run it	The commands in order, how rows reach Snowflake, and running several instances.
Configuration	Every section and key of the exporter’s sqlload.toml, with defaults.
Topic naming	How topics map to tables, stages, pipes and consumer groups.