SnowDDL
  • 👋Introduction
  • 🚩Getting started
  • 📋Main features
  • 🪤SnowDDL vs. Declarative DCM
  • In-depth guides
    • 👓Object identifiers
    • 📐Data types
    • 📦Object types
    • 🎭Role hierarchy
    • 🚧Permission model
    • 🔦Other guides
      • Administration user
      • Integrations
      • Inbound shares
      • Object OWNERSHIP
      • Safe & unsafe DDL
      • Dependency management
      • Short hash explained
      • Env Prefix explained
      • Team workflow
      • Limitations & workarounds
      • Fivetran
      • Airbyte
      • Encrypt user passwords
      • Iceberg Tables
  • Basic usage (CLI + YAML)
    • 💻CLI interface
    • 📦YAML configs
      • ACCOUNT PARAMETER
      • ACCOUNT POLICY
      • AGGREGATION POLICY
      • ALERT
      • AUTHENTICATION POLICY
      • BUSINESS ROLE
      • DATABASE
      • DYNAMIC TABLE
      • EVENT TABLE
      • EXTERNAL ACCESS INTEGRATION
      • EXTERNAL FUNCTION
      • EXTERNAL TABLE
      • FILE FORMAT
      • FUNCTION
      • HYBRID TABLE
      • ICEBERG TABLE
      • MASKING POLICY
      • MATERIALIZED VIEW
      • NETWORK POLICY
      • NETWORK RULE
      • PERMISSION MODEL
      • PIPE
      • PLACEHOLDER
      • PROCEDURE
      • PROJECTION POLICY
      • RESOURCE MONITOR
      • ROW ACCESS POLICY
      • SCHEMA
      • SECRET
      • SEQUENCE
      • SHARE (outbound)
      • STAGE
      • STAGE FILE
      • STREAM
      • TABLE
      • TASK
      • TECHNICAL ROLE
      • USER
      • VIEW
      • WAREHOUSE
    • 🏷️YAML placeholders
    • 📬YAML tag !include
    • 🔐YAML tag !decrypt
  • Single DB
    • 🦀Overview
  • Advanced usage (Python)
    • ⚙️Programmatic config
    • 🐍Architecture overview
      • 🔵Blueprints
      • 🟣Config
      • 🟠Parsers
      • 🟢Resolvers
      • 🔴Engine
    • 🏗️Query builder & formatter
  • Breaking changes log
    • 0.45.0 - March 2025
    • 0.41.0 - January 2025
    • 0.37.0 - December 2024
    • 0.36.0 - November 2024
    • 0.33.0 - October 2024
    • 0.27.0 - May 2024
  • Links
    • GitHub repository
    • PyPI package
    • YouTube tutorials
    • Changelog
    • LinkedIn profile
Powered by GitBook
On this page
  • Schema
  • Usage notes
  • Additional privileges
  • Links
  1. Basic usage (CLI + YAML)
  2. YAML configs

PIPE

PreviousPERMISSION MODELNextPLACEHOLDER

Last updated 10 months ago

Config path: /<database>/<schema>/pipe/<name>.yaml

Example:

copy:
  table: test_table_2
  stage: test_external_stage
  transform:
    id: "GET($1, 'id')"
    name: "GET($1, 'name')"

auto_ingest: false

Schema

  • copy (dict)

    • table (ident) - target table for COPY INTO

    • stage (ident) - source stage for COPY INTO

    • path (str) - path prefix for files in stage

    • pattern (str) - regular expression to filter files in stage

    • file_format (ident) - for files in stage

    • match_by_column_name (str) - case_sensitive / case_insensitive

    • include_metadata (dict)

      • {key} (ident) - column name in target table

      • {value} (ident) - column name in metadata, e.g. METADATA$FILENAME

    • transform (dict)

      • {key} (ident) - column name in target table

      • {value} (str) - SQL expression to extract column value from source stage

    • options (dict)

      • {key} (ident) - COPY option name

      • {value} (bool, float, int, list, str) - COPY option value

  • auto_ingest (bool) - enable AUTO_INGEST

  • aws_sns_topic (str) - SNS topic for S3 bucket

  • integration (ident) - notification name for Azure

  • error_integration (ident) - notification integration to monitor ingestion errors

  • comment (str)

Usage notes

  1. file_format can only be specified by name referencing to FILE_FORMAT object.

Additional privileges

  • owner_integration_usage - please specify names of INTEGRATION objects used by pipes. Pipes may work without explicit INTEGRATION USAGE grant to OWNER role, but it is not guaranteed for Snowflake keep it this way forever.

Links

Re-creating pipes correctly is complicated. Make sure you read & fully understand . SnowDDL can only suggest or apply DDL queries to change pipes, but it cannot pause, monitor and refresh pipes for you.

Maintenance of active pipes implicitly incurs .

Notification integration mentioned in error_integration parameter must be additionally specified in parameter owner_integration_usage. Otherwise schema owner role will not be able to send notifications.

In order for PIPE objects to operate properly, the following additional grants should be added to OWNER role in :

📦
file format
integration
pipe recreation considerations
additional costs
schema
schema config
CREATE PIPE
ALTER PIPE
SHOW PIPES
Parser & JSON Schema (GitHub)