Pipe

Pipe is the continuous data ingestion object in the Lakehouse. Once created via SQL DDL, it runs automatically, continuously reading data from object storage (OSS/COS/S3) or Kafka and writing it to a target table.

For a detailed introduction, see Pipe Object Model.

Chapter Contents

Page	Description
CREATE PIPE	Create an object storage Pipe or Kafka Pipe
ALTER PIPE	Pause, resume, or modify batch interval and other properties
DROP PIPE	Drop a Pipe (does not affect target table data)
SHOW PIPES	List all Pipes in the current Schema
SHOW CREATE PIPE	View the creation statement of a Pipe
DESC PIPE	View Pipe details including status, source, target, and latency

Common Operations

Create an Object Storage Pipe

-- LIST_PURGE mode: periodic polling, deletes source files after import CREATE PIPE orders_pipe VIRTUAL_CLUSTER = 'DEFAULT' INGEST_MODE = 'LIST_PURGE' AS COPY INTO orders FROM VOLUME orders_vol USING CSV OPTIONS('header' = 'true');

⚠️ LIST_PURGE mode permanently deletes source files from OSS after a successful import. This is irreversible. Use EVENT_NOTIFICATION mode if you need to retain the files.

Create a Kafka Pipe

CREATE PIPE kafka_orders_pipe VIRTUAL_CLUSTER = 'DEFAULT' BATCH_INTERVAL_IN_SECONDS = '60' AS COPY INTO orders_raw FROM ( SELECT CAST(value AS STRING) AS raw_msg FROM TABLE(READ_KAFKA( 'kafka-host:9092', 'orders_topic', '', 'pipe_orders_group', '', '', '', '', 'raw', 'raw', 0, map() )) );

Pause and Resume

-- Pause ALTER PIPE orders_pipe SET PIPE_EXECUTION_PAUSED = TRUE; -- Resume ALTER PIPE orders_pipe SET PIPE_EXECUTION_PAUSED = FALSE; -- Trigger an immediate scan ALTER PIPE orders_pipe REFRESH;

View and Drop

-- View all Pipes SHOW PIPES; -- View Pipe details DESC PIPE orders_pipe; -- Drop a Pipe DROP PIPE orders_pipe;

Document	Description
Pipe Object Model	Core concepts, comparison of two modes, deduplication mechanism, complete parameter reference
Object Storage Pipe Detailed Configuration	Complete configuration for EVENT_NOTIFICATION mode
Kafka Pipe Detailed Configuration	READ_KAFKA parameter reference, consumer offset management
Real-time Pipeline Selection Guide	Comparison and selection guide for Pipe / Stream / Dynamic Table