Fluid Forge
Why Forge
Concepts
Get Started
  • Consume a Data Product
  • See it run
  • Demos
  • Local (DuckDB)
  • Source-Aligned (Postgres → DuckDB)
  • AI Forge + Data Models
  • MCP Output Port — Serve to AI Agents
  • GCP (BigQuery)
  • Snowflake Team Collaboration
  • Declarative Airflow
  • Orchestration Export
  • Jenkins CI/CD
  • Universal Pipeline
  • 11-Stage Production Pipeline
  • Catalog Forge End-to-End
CLI Reference
  • Agent Policy (concept)
  • MCP Output Port — Serve to Agents
  • MCP deep-dive
  • AI-assisted authoring
  • LLM providers & backends
  • Overview
  • Quickstart
  • Examples
  • Your own CI
  • Your own scaffolding
  • Custom validator
  • Apply hook
  • Reference
  • Overview
  • Architecture
  • GCP (BigQuery)
  • AWS (S3 + Athena)
  • Snowflake
  • Local (DuckDB)
  • Custom Providers
  • Roadmap
GitHub
Why Forge
Concepts
Get Started
  • Consume a Data Product
  • See it run
  • Demos
  • Local (DuckDB)
  • Source-Aligned (Postgres → DuckDB)
  • AI Forge + Data Models
  • MCP Output Port — Serve to AI Agents
  • GCP (BigQuery)
  • Snowflake Team Collaboration
  • Declarative Airflow
  • Orchestration Export
  • Jenkins CI/CD
  • Universal Pipeline
  • 11-Stage Production Pipeline
  • Catalog Forge End-to-End
CLI Reference
  • Agent Policy (concept)
  • MCP Output Port — Serve to Agents
  • MCP deep-dive
  • AI-assisted authoring
  • LLM providers & backends
  • Overview
  • Quickstart
  • Examples
  • Your own CI
  • Your own scaffolding
  • Custom validator
  • Apply hook
  • Reference
  • Overview
  • Architecture
  • GCP (BigQuery)
  • AWS (S3 + Athena)
  • Snowflake
  • Local (DuckDB)
  • Custom Providers
  • Roadmap
GitHub
  • Introduction

    • Home
    • Why Fluid Forge
    • Getting Started
    • Snowflake Quickstart
    • See it run
    • Forge Data Model
    • Vision & Roadmap
    • Playground
    • FAQ
  • Concepts

    • Concepts
    • Builds, Exposes, Bindings
    • What is a contract?
    • Quality, SLAs & Lineage
    • Governance & Policy
    • Agent Policy (LLM/AI governance)
    • Providers vs Platforms
    • Fluid Forge vs alternatives
  • Data Products

    • Consume a Data Product
    • Product Types — SDP, ADP, CDP
  • Walkthroughs

    • Walkthrough: Local Development
    • Source-Aligned: Postgres → DuckDB → Parquet
    • AI Forge And Data-Model Journeys
    • Walkthrough: MCP Output Port
    • Walkthrough: Deploy to Google Cloud Platform
    • Walkthrough: Snowflake Team Collaboration
    • Declarative Airflow DAG Generation - The FLUID Way
    • Generating Orchestration Code from Contracts
    • Jenkins CI/CD for FLUID Data Products
    • Universal Pipeline
    • The 11-Stage Pipeline
    • End-to-End Walkthrough: Catalog → Contract → Transformation
  • CLI Reference

    • CLI Reference
    • Core workflow

      • fluid init
      • fluid demo
      • fluid forge
      • fluid validate
      • fluid plan
      • fluid apply
      • fluid diff
      • fluid status
    • Build & ship

      • fluid bundle
      • fluid generate
      • fluid generate artifacts
      • fluid validate-artifacts
      • fluid verify-signature
      • fluid generate iac
      • fluid generate-airflow
      • fluid generate-pipeline
      • fluid generate vector
      • fluid viz-graph
      • fluid publish
      • fluid ship
      • fluid rollback
      • fluid schedule-sync
    • AI & Agents

      • fluid ai
      • fluid agents
      • fluid mission
      • fluid mcp
      • fluid memory
      • fluid stats
      • fluid skills
    • Quality & governance

      • fluid test
      • fluid verify
      • fluid contract-tests
      • fluid contract-validation
      • fluid policy
      • fluid policy check
      • fluid policy compile
      • fluid policy apply
    • Standards & interoperability

      • fluid odps
      • fluid odps-bitol
      • fluid odcs
      • fluid export
      • fluid export-odps
      • fluid exporters
      • fluid import
      • fluid market
      • fluid datamesh-manager
    • Project & workspace

      • fluid product-new
      • fluid product-add
      • fluid workspace
      • fluid contract
      • fluid split
      • fluid config
      • fluid providers
      • fluid plugins
      • fluid provider-init
      • fluid auth
      • fluid secrets
      • fluid ide
      • fluid scaffold-ci
      • fluid scaffold-composer
      • fluid scaffold-ide
      • fluid docs
      • fluid runs
      • fluid retention
      • fluid describe
      • fluid doctor
      • fluid roadmap
      • fluid version
    • Catalog adapters

      • Source Catalog Integration (V1.5)
      • Publishing to a Catalog — Overview
      • BigQuery Catalog
      • Snowflake Horizon Catalog
      • Databricks Unity Catalog
      • Google Dataplex Catalog
      • AWS Glue Data Catalog
      • DataHub Catalog
      • Data Mesh Manager Catalog
      • OpenMetadata Catalog
    • CLI by task

      • CLI by task
      • Add quality rules
      • Add agent governance
      • Debug a failed pipeline run
      • Switch clouds with one line
  • Recipes

    • Recipes
    • Recipe — add a quality rule
    • Recipe — switch clouds with one line
    • Recipe — tag PII in your schema
    • Write a contract that consumes another contract
    • Generate per-environment overlays
  • SDK & Plugins

    • SDK & Plugins
    • Quickstart — your first plugin
    • Examples

      • Runnable examples
      • Example: hello-scaffold — the minimal viable plugin
      • Example: gitlab-ci-scaffold — generate a complete CI project
      • Example: steward-validator — a custom governance rule
      • Example: prod-key-guard — apply-time invariant check
    • Journeys

      • Journeys
      • Your own CI/CD

        • You have your own CI/CD setup, no problem
        • GitLab CI — the bundle template
        • GitHub Actions — the bundle template
        • Jenkins — the bundle template
        • CircleCI — the bundle template
      • You have a strict project layout, no problem
      • You have governance rules, no problem
      • You want a check at apply time, no problem
    • Reference

      • Reference
      • Roles reference
      • Entry points reference
      • Trust model
      • Packaging
      • Companion packages
  • Providers

    • Providers
    • Provider Architecture
    • GCP Provider
    • AWS Provider
    • Snowflake Provider
    • Local Provider
    • Creating Custom Providers
    • Provider Roadmap
  • AI & Agents

    • MCP Server
    • Built-in And Custom Forge Guidance
    • Forge Discovery Guide
    • Forge Memory Guide
    • Authoring Forge Tools
    • Guided fluid forge UX
    • LLM Providers
    • LiteLLM Backend
    • Capability Warnings
    • Cost Tracking
    • FLUID Forge Contract GPT Packet
    • Agentic Primitives
  • Operate & Deploy

    • Operating in CI
    • Production Troubleshooting
    • Airflow Integration
    • Blueprints
    • Source-Aligned Acquisition
  • Govern & Secure

    • Governance, Compliance & the Business Case
    • Governance & Compliance
    • Network Safety
    • Credential Resolver — Security Model
  • Configuration & Reference

    • Environment Variables
    • Typed Errors
    • Typed CLI Errors
    • API Stability — fluid_build.api
  • Architecture & Releases

    • V1.5 Catalog Integration — Architecture Deep-Dive
    • V1.5 + V2 Hardening — Release Notes
  • Project

    • Contributing to Fluid Forge
    • Fluid Forge Docs Baseline: CLI 0.14.0
    • Fluid Forge Docs Baseline: CLI 0.13.0
    • Fluid Forge Docs Baseline: CLI 0.12.0
    • Fluid Forge Docs Baseline: CLI 0.11.0
    • Fluid Forge Docs Baseline: CLI 0.10.0
    • Fluid Forge Docs Baseline: CLI 0.9.0
    • Fluid Forge Docs Baseline: CLI 0.8.11
    • Fluid Forge Docs Baseline: CLI 0.8.10
    • Fluid Forge Docs Baseline: CLI 0.8.9
    • Fluid Forge Docs Baseline: CLI 0.8.8
    • Fluid Forge Docs Baseline: CLI 0.8.7
    • Fluid Forge Docs Baseline: CLI 0.8.6
    • Fluid Forge Docs Baseline: CLI 0.8.5
    • Fluid Forge Docs Baseline: CLI 0.8.4
    • Fluid Forge Docs Baseline: CLI 0.8.3
    • Fluid Forge Docs Baseline: CLI 0.8.0
    • Fluid Forge Docs Baseline: CLI 0.7.11
    • Fluid Forge Docs Baseline: CLI 0.7.9
    • Fluid Forge v0.7.1 - Multi-Provider Export Release

fluid import

Convert existing tooling into FLUID contracts. Two modes: scan an existing dbt / Terraform / SQL directory, or import a foreign tool project (Meltano, Airbyte, dlt, Singer, dbt).

Syntax

fluid import [<engine> <source>] [options]

Mode 1 — directory scan

fluid import
fluid import --dir ./legacy-dbt
fluid import --provider snowflake
fluid import --yes
OptionDescription
--providerProvider for the generated contracts
--dir, -CDirectory to scan
--yes, -ySkip the confirmation prompt

This is the promoted migration path for existing Terraform or SQL projects. A dbt project that has a target/manifest.json is routed to the dbt manifest importer automatically.

Mode 2 — foreign tool importer

Convert an existing tool project into FLUID contracts (one per discovered tap / connector / source for the ingestion tools; one per project for dbt):

fluid import meltano <project-dir>       # Meltano project
fluid import airbyte <workspace-id>      # Airbyte OSS / Cloud workspace
fluid import dlt <pipeline-name>         # dlt pipeline
fluid import singer <tap-config.json>    # Singer tap + target
fluid import dbt <project-dir|manifest>  # dbt project (target/manifest.json)
OptionDescription
<engine> <source>Importer mode + source identifier
--out PATHOutput contract path (default: one contract.<id>.fluid.yaml per discovered source in cwd); with --split-by producing multiple products, --out is the output directory
--split-by {project|folder|group}dbt import product boundary: one contract per project (default), per top-level models/ subfolder, or per dbt group — cross-split ref()s become cross-product consumes[]. (since 0.13.1)
--provider {local|gcp|snowflake|aws|azure}Infrastructure provider for generated contracts. Default local.
--yes, -ySkip the confirmation prompt

What each importer does:

ImporterReadsEmits
meltanomeltano.yml + extract: blockOne engine: meltano acquisition contract per tap
airbyteWorkspace config from REST APIOne engine: airbyte acquisition contract per source
dlt@dlt.source modules in the pipelineOne engine: dlt acquisition contract per source
singerTap + target config filesOne engine: meltano acquisition contract (Meltano runs Singer protocol)
dbttarget/manifest.json (+ optional catalog.json)One contract per project by default; --split-by folder/group splits along product boundaries — see below

Secrets are auto-redacted to ${ENV_VAR} placeholders so the emitted contracts are safe to commit. Run fluid secrets login afterward to populate the keychain backend.

Example — migrating from Meltano

fluid import meltano ./my-meltano-project --provider local --yes
fluid validate *.fluid.yaml
fluid apply contract.tap_postgres.fluid.yaml --yes

The generated contract preserves Meltano's tap selections and the state/incremental mode mapping. See Source-Aligned Acquisition for engine-specific properties.

Importing a dbt project

(since 0.12.0) fluid import dbt performs a faithful brownfield conversion of a real dbt project by reading target/manifest.json — the artifact dbt parse produces without any warehouse access. The parse is stdlib-only (no dbt-core dependency) and emits one DataProduct contract per dbt project by default — see --split-by for multi-product splits.

cd my-dbt-project
dbt parse                                  # refresh target/manifest.json (no warehouse needed)
fluid import dbt . --provider snowflake    # or point at the manifest file directly
fluid validate contract.*.fluid.yaml

Run dbt parse first

The importer reads target/manifest.json, so make sure it is fresh — a stale manifest imports the project as it was when last parsed. dbt parse needs no warehouse connection. If target/catalog.json exists (from dbt docs generate), it is used as an overlay for warehouse-accurate column types.

What the importer recovers:

dbt artifactFLUID contract
Models / seeds / snapshots + depends_onOne expose each (no model cap); the ref()-derived DAG recorded per expose and as builds[].transformations
Generic tests (not_null, unique, accepted_values, …)dq.rules[] via the same shared test mapping the dbt generators use; relationships / range tests become column validationRules (FK + bounds)
Sources (+ source freshness)consumes[], freshness mapped to qosExpectations.freshnessMax
config.materializedExpose kind + build materialization hints; adapter-aware binding (snowflake / bigquery / redshift / databricks / duckdb)
Folder layout (staging / intermediate / marts)metadata.layer + metadata.productType (staging→Bronze/SDP, marts→Gold/CDP)
catalog.json / schema.yml data_typeColumn types (catalog overlay wins; schema.yml is the fallback)

Primary keys are inferred with the same precedence dbt itself uses (ModelNode.infer_primary_key); foreign keys are recovered from relationships tests.

Splitting one project into multiple products (--split-by, since 0.13.1)

By default (--split-by project) the importer emits one contract for the whole project — byte-stable with earlier releases. Two more boundaries split a monolithic dbt project along data-product lines:

BoundaryOne DataProduct per…
project (default)dbt project — the single-contract output above, unchanged
foldertop-level models/ subfolder (models outside any subfolder land in a reported root product)
groupdbt group — the dbt-mesh-native boundary; ungrouped models land in a reported ungrouped product, and a fully groupless manifest fails loudly suggesting folder mode

Cross-split ref()s are rewritten to cross-product consumes[] against the sibling product, so the inter-product DAG survives the split. With multiple products, --out names the output directory:

fluid import dbt . --split-by folder --out ./products/

Requirements: manifest schema v9 or newer (dbt-core 1.5, May 2023); the primary target is v12 (dbt 1.8+). Older manifests are rejected with a clear error. dbt >= 1.10 manifests (with arguments:-nested test params) are supported since 0.13.1 alongside the legacy flat-kwargs shape.

Like all importers, the command prints an import report alongside the written contract — what mapped 1:1, where defaults were used, and what is unsupported and must be re-authored. After importing, fluid generate transformation can regenerate a dbt project from the contract — test mappings are shared, so the round-trip stays symmetric.

Notes

  • Mode 1 (fluid import with no engine arg) is the existing migration path for dbt / Terraform / SQL projects.
  • Mode 2 (fluid import <engine> <source>) is the explicit tool importer — Meltano / Airbyte / dlt / Singer since 0.8.3, dbt since 0.12.0.
  • Since 0.14.0 the Snowflake round-trip is verified end-to-end: import dbt → apply creates no duplicate lowercase shadow namespace and a second apply is a no-op (+0 ~2 -0); fluid verify reads the deployed objects.
  • If you want a clean greenfield start instead, use fluid init or fluid forge.
  • For source-aligned ingestion from scratch (no existing tool project), fluid init --discover is the one-shot path.
Edit this page on GitHub
Last Updated: 7/28/26, 8:03 PM
Contributors: Jeff Watson, fas89, Claude Opus 4.7 (1M context)
Prev
fluid exporters
Next
fluid market