Fluid Forge
Why Forge
Concepts
Get Started
  • Consume a Data Product
  • See it run
  • Demos
  • Local (DuckDB)
  • Source-Aligned (Postgres → DuckDB)
  • AI Forge + Data Models
  • MCP Output Port — Serve to AI Agents
  • GCP (BigQuery)
  • Snowflake Team Collaboration
  • Declarative Airflow
  • Orchestration Export
  • Jenkins CI/CD
  • Universal Pipeline
  • 11-Stage Production Pipeline
  • Catalog Forge End-to-End
CLI Reference
  • Agent Policy (concept)
  • MCP Output Port — Serve to Agents
  • MCP deep-dive
  • AI-assisted authoring
  • LLM providers & backends
  • Overview
  • Quickstart
  • Examples
  • Your own CI
  • Your own scaffolding
  • Custom validator
  • Apply hook
  • Reference
  • Overview
  • Architecture
  • GCP (BigQuery)
  • AWS (S3 + Athena)
  • Snowflake
  • Local (DuckDB)
  • Custom Providers
  • Roadmap
GitHub
Why Forge
Concepts
Get Started
  • Consume a Data Product
  • See it run
  • Demos
  • Local (DuckDB)
  • Source-Aligned (Postgres → DuckDB)
  • AI Forge + Data Models
  • MCP Output Port — Serve to AI Agents
  • GCP (BigQuery)
  • Snowflake Team Collaboration
  • Declarative Airflow
  • Orchestration Export
  • Jenkins CI/CD
  • Universal Pipeline
  • 11-Stage Production Pipeline
  • Catalog Forge End-to-End
CLI Reference
  • Agent Policy (concept)
  • MCP Output Port — Serve to Agents
  • MCP deep-dive
  • AI-assisted authoring
  • LLM providers & backends
  • Overview
  • Quickstart
  • Examples
  • Your own CI
  • Your own scaffolding
  • Custom validator
  • Apply hook
  • Reference
  • Overview
  • Architecture
  • GCP (BigQuery)
  • AWS (S3 + Athena)
  • Snowflake
  • Local (DuckDB)
  • Custom Providers
  • Roadmap
GitHub
  • Introduction

    • Home
    • Why Fluid Forge
    • Getting Started
    • Snowflake Quickstart
    • See it run
    • Forge Data Model
    • Vision & Roadmap
    • Playground
    • FAQ
  • Concepts

    • Concepts
    • Builds, Exposes, Bindings
    • What is a contract?
    • Quality, SLAs & Lineage
    • Governance & Policy
    • Agent Policy (LLM/AI governance)
    • Providers vs Platforms
    • Fluid Forge vs alternatives
  • Data Products

    • Consume a Data Product
    • Product Types — SDP, ADP, CDP
  • Walkthroughs

    • Walkthrough: Local Development
    • Source-Aligned: Postgres → DuckDB → Parquet
    • AI Forge And Data-Model Journeys
    • Walkthrough: MCP Output Port
    • Walkthrough: Deploy to Google Cloud Platform
    • Walkthrough: Snowflake Team Collaboration
    • Declarative Airflow DAG Generation - The FLUID Way
    • Generating Orchestration Code from Contracts
    • Jenkins CI/CD for FLUID Data Products
    • Universal Pipeline
    • The 11-Stage Pipeline
    • End-to-End Walkthrough: Catalog → Contract → Transformation
  • CLI Reference

    • CLI Reference
    • Core workflow

      • fluid init
      • fluid demo
      • fluid forge
      • fluid validate
      • fluid plan
      • fluid apply
      • fluid diff
      • fluid status
    • Build & ship

      • fluid bundle
      • fluid generate
      • fluid generate artifacts
      • fluid validate-artifacts
      • fluid verify-signature
      • fluid generate iac
      • fluid generate-airflow
      • fluid generate-pipeline
      • fluid generate vector
      • fluid viz-graph
      • fluid publish
      • fluid ship
      • fluid rollback
      • fluid schedule-sync
    • AI & Agents

      • fluid ai
      • fluid agents
      • fluid mission
      • fluid mcp
      • fluid memory
      • fluid stats
      • fluid skills
    • Quality & governance

      • fluid test
      • fluid verify
      • fluid contract-tests
      • fluid contract-validation
      • fluid policy
      • fluid policy check
      • fluid policy compile
      • fluid policy apply
    • Standards & interoperability

      • fluid odps
      • fluid odps-bitol
      • fluid odcs
      • fluid export
      • fluid export-odps
      • fluid exporters
      • fluid import
      • fluid market
      • fluid datamesh-manager
    • Project & workspace

      • fluid product-new
      • fluid product-add
      • fluid workspace
      • fluid contract
      • fluid split
      • fluid config
      • fluid providers
      • fluid plugins
      • fluid provider-init
      • fluid auth
      • fluid secrets
      • fluid ide
      • fluid scaffold-ci
      • fluid scaffold-composer
      • fluid scaffold-ide
      • fluid docs
      • fluid runs
      • fluid retention
      • fluid describe
      • fluid doctor
      • fluid roadmap
      • fluid version
    • Catalog adapters

      • Source Catalog Integration (V1.5)
      • Publishing to a Catalog — Overview
      • BigQuery Catalog
      • Snowflake Horizon Catalog
      • Databricks Unity Catalog
      • Google Dataplex Catalog
      • AWS Glue Data Catalog
      • DataHub Catalog
      • Data Mesh Manager Catalog
      • OpenMetadata Catalog
    • CLI by task

      • CLI by task
      • Add quality rules
      • Add agent governance
      • Debug a failed pipeline run
      • Switch clouds with one line
  • Recipes

    • Recipes
    • Recipe — add a quality rule
    • Recipe — switch clouds with one line
    • Recipe — tag PII in your schema
    • Write a contract that consumes another contract
    • Generate per-environment overlays
  • SDK & Plugins

    • SDK & Plugins
    • Quickstart — your first plugin
    • Examples

      • Runnable examples
      • Example: hello-scaffold — the minimal viable plugin
      • Example: gitlab-ci-scaffold — generate a complete CI project
      • Example: steward-validator — a custom governance rule
      • Example: prod-key-guard — apply-time invariant check
    • Journeys

      • Journeys
      • Your own CI/CD

        • You have your own CI/CD setup, no problem
        • GitLab CI — the bundle template
        • GitHub Actions — the bundle template
        • Jenkins — the bundle template
        • CircleCI — the bundle template
      • You have a strict project layout, no problem
      • You have governance rules, no problem
      • You want a check at apply time, no problem
    • Reference

      • Reference
      • Roles reference
      • Entry points reference
      • Trust model
      • Packaging
      • Companion packages
  • Providers

    • Providers
    • Provider Architecture
    • GCP Provider
    • AWS Provider
    • Snowflake Provider
    • Local Provider
    • Creating Custom Providers
    • Provider Roadmap
  • AI & Agents

    • MCP Server
    • Built-in And Custom Forge Guidance
    • Forge Discovery Guide
    • Forge Memory Guide
    • Authoring Forge Tools
    • Guided fluid forge UX
    • LLM Providers
    • LiteLLM Backend
    • Capability Warnings
    • Cost Tracking
    • FLUID Forge Contract GPT Packet
    • Agentic Primitives
  • Operate & Deploy

    • Operating in CI
    • Production Troubleshooting
    • Airflow Integration
    • Blueprints
    • Source-Aligned Acquisition
  • Govern & Secure

    • Governance, Compliance & the Business Case
    • Governance & Compliance
    • Network Safety
    • Credential Resolver — Security Model
  • Configuration & Reference

    • Environment Variables
    • Typed Errors
    • Typed CLI Errors
    • API Stability — fluid_build.api
  • Architecture & Releases

    • V1.5 Catalog Integration — Architecture Deep-Dive
    • V1.5 + V2 Hardening — Release Notes
  • Project

    • Contributing to Fluid Forge
    • Fluid Forge Docs Baseline: CLI 0.14.0
    • Fluid Forge Docs Baseline: CLI 0.13.0
    • Fluid Forge Docs Baseline: CLI 0.12.0
    • Fluid Forge Docs Baseline: CLI 0.11.0
    • Fluid Forge Docs Baseline: CLI 0.10.0
    • Fluid Forge Docs Baseline: CLI 0.9.0
    • Fluid Forge Docs Baseline: CLI 0.8.11
    • Fluid Forge Docs Baseline: CLI 0.8.10
    • Fluid Forge Docs Baseline: CLI 0.8.9
    • Fluid Forge Docs Baseline: CLI 0.8.8
    • Fluid Forge Docs Baseline: CLI 0.8.7
    • Fluid Forge Docs Baseline: CLI 0.8.6
    • Fluid Forge Docs Baseline: CLI 0.8.5
    • Fluid Forge Docs Baseline: CLI 0.8.4
    • Fluid Forge Docs Baseline: CLI 0.8.3
    • Fluid Forge Docs Baseline: CLI 0.8.0
    • Fluid Forge Docs Baseline: CLI 0.7.11
    • Fluid Forge Docs Baseline: CLI 0.7.9
    • Fluid Forge v0.7.1 - Multi-Provider Export Release

fluid generate iac

Review-only emit of an OpenTofu module from a FLUID contract. fluid apply against a cloud provider auto-runs the same generator and hands the result to tofu apply — no flag needed; engine selection is per-provider (local keeps its native apply, aws / gcp / snowflake route through OpenTofu).

Available in 0.8.3 (PR #140)

v0.8.3 introduces the OpenTofu autogenerator: fluid apply for cloud providers now compiles the contract to a deterministic OpenTofu main.tf.json and delegates apply / state / drift / idempotency to the tofu binary. local keeps its native DuckDB apply — no tofu needed for the local-first onboarding path.

Syntax

fluid generate iac <contract> [--provider PROVIDER] [--out DIR] [--env NAME] [--shadow] [--validate]
OptionDescription
<contract>Path to FLUID contract file (YAML/JSON).
--provider {auto,aws,gcp,snowflake}Target cloud. Default auto — inferred from the contract.
--out, -o DIROutput directory for the emitted module. Default runtime/iac.
--env NAMEEnvironment overlay name (matches your contract's overlay block, e.g. dev / staging / prod).
--shadowAfter emitting, shadow-compare resource parity against the native planner. Catches a divergence between the OpenTofu emission and what fluid plan would have produced.
--validateAfter emitting, run tofu validate on the module (requires tofu on PATH).

Examples

# Review what fluid apply would emit for AWS
fluid generate iac contract.fluid.yaml --provider aws --out ./review

# Per-environment review
fluid generate iac contract.fluid.yaml --env staging --out ./review/staging

# Emit + validate against the planner + tofu validate in one pass
fluid generate iac contract.fluid.yaml --shadow --validate

The emitted directory contains main.tf.json (the compiled module), provider blocks, and a manifest.json describing how the contract mapped to resources. Inspect it before you run fluid apply.

Apply via OpenTofu

# Cloud providers auto-route through OpenTofu on v0.8.3 — no flag needed.
fluid apply contract.fluid.yaml --provider aws --yes
# → ... [iac] tofu init
# → ... [iac] tofu plan
# → ... [iac] plan-binding verified
# → ... [iac] tofu apply -auto-approve

Engine selection is automatic and per-provider (apply.py::resolve_apply_engine) — local keeps its native DuckDB apply; aws / gcp / snowflake route through OpenTofu. There is no --engine flag on fluid apply.

Supported providers

v0.8.3 ships built-in IaC plugins for three cloud targets. Each is a modular IacProviderPlugin (dbt-adapter pattern); third parties can register their own via the Provider SDK.

ProviderResources emitted
awsS3 buckets / IAM roles + policies / Glue databases + tables + column comments / Athena workgroups
gcpBigQuery datasets + tables / IAM bindings / GCS buckets / BigLake-Iceberg warehouse buckets (0.14.0)
snowflakeDatabases / schemas / tables / column comments (Horizon-aware) / file formats / stages / external volumes + Glue catalog integrations for Iceberg exposes (0.13.1)

Catalog metadata that previously lived in the retired glue and snowflake_horizon publish-side registrars is now emitted into aws_glue_catalog_table.parameters and snowflake_table column comments directly — one source of truth, drift-detected by tofu plan. See catalog overview.

Iceberg prerequisites and the anti-no-op gate

An Iceberg expose needs infrastructure dbt refuses to create, and the plugins derive it from the binding. The snowflake plugin emits the EXTERNAL VOLUME for a Snowflake-managed (Horizon) catalog and a Glue CATALOG INTEGRATION for location.catalog: glue (0.13.1, live-verified with a real tofu apply). The gcp plugin emits the GCS warehouse bucket backing a BigLake-Iceberg table (0.14.0) — a declared location.path means the product owns a prefix of a shared warehouse, so that bucket is emitted without force_destroy; an owned bucket (no path) keeps it.

The emitters are emit-when-derivable: an Iceberg expose missing a required input (an S3 warehouse without iam_role_arn, no derivable bucket name, an illegal external_volume override) used to emit nothing, silently — the failure surfaced only at dbt run. Since 0.14.0, fluid validate errors on exactly those cases before any emit. One --strict caveat: a Snowflake catalog whose auth is secret-bearing (polaris / unity / rest / nessie) draws a warning — FLUID does not emit those catalog integrations because the emitted module is credential-free — and fluid validate --strict promotes it to an error.

Packaging modes

Opt-in, new in 0.13.0

The packaging block lives in the 0.7.6 preview schema. 0.7.5 remains the stable default — a contract must declare fluidVersion: "0.7.6" to use it. A contract with no packaging block emits byte-identically to pre-0.13.0 releases (that path is a distinct LEGACY sentinel, golden-pinned — not a re-derivation), so nothing existing changes.

packaging declares, per infrastructure container, whether this product owns it or writes into a pre-existing, platform-owned pool:

fluidVersion: "0.7.6"
packaging:
  mode: shared                        # isolated | shared — blanket default for every kind
  pool: analytics-pool-eu             # REQUIRED whenever any container resolves 'shared'
  poolManifest: platform/pools.yaml   # optional; snapshotted into the bundle
  containers:                         # per-kind overrides win over `mode`
    warehouse: isolated               # hybrid tier: shared database, own warehouse
ModeEmits
isolatedAn owned OpenTofu resource — this product creates and can destroy the container. Today's exact emit.
sharedAn OpenTofu data source referencing the platform-owned pool, plus leaf-only owned resources (tables, prefixed objects, scoped grants). The product writes into the pool but structurally cannot destroy it.

Six container kinds are accepted:

KindResource
bucketaws_s3_bucket / google_storage_bucket
databasesnowflake_database and aws_glue_catalog_database
datasetgoogle_bigquery_dataset
schemasnowflake_schema
warehousesnowflake_warehouse — isolated gives per-product cost attribution
clusterConfluent environment/cluster — shared only in v1; the resolver rejects isolated (dedicated-cluster provisioning is not yet supported)

binding.packaging overrides the contract-wide block per exposure. Precedence is binding.packaging > top-level packaging > absent-LEGACY. pool propagates as the fluid_pool label/tag for cost attribution.

What shared changes per provider

ProviderReferenced (shared) behaviour
awsThe bucket becomes data.aws_s3_bucket with no force_destroy. A referenced Glue database is addressed by literal name (hashicorp/aws ships aws_glue_catalog_database as a resource only — no data source). Lake Formation registerLocation scopes to location.path, and registers nothing when a pooled bucket has no prefix; the bucket policy's ListBucket statement gains an s3:prefix condition.
gcpDataset and bucket become data sources. A shared dataset drops its authoritative access[] block (it would rewrite the pool's whole ACL and evict other tenants) in favour of per-table google_bigquery_table_iam_member. A shared bucket's IAM members gain an object-prefix CEL condition.
snowflakeA referenced database/schema emits neither a resource nor a data source (Snowflake's data sources are thin), so consumers inline the literal name. An isolated warehouse gets a dedicated snowflake_warehouse.

Ownership transitions

Changing a container's mode changes who owns it — but OpenTofu only sees a resource that left the configuration and plans a destroy. On a shared pool, that reaches every other tenant's data. So the ownership model is diffed against tofu state list before tofu plan, and a transition fails closed with copy-pasteable tofu state rm remediation.

TransitionBehaviour
isolated → shared (owned → referenced)Always blocked. There is no flag. Drop the resource from state first — tofu state rm touches zero bytes of infrastructure.
shared → isolated (referenced → owned)Requires fluid apply --adopt-shared-container, which emits a structured audit event.

Detection is deliberately conservative: over-flagging asks a human to look, under-flagging destroys a pool. LEGACY contracts resolve to the sentinel and can never transition, so the guard is a provable no-op for every pre-existing contract. Structured packaging_transition_blocked / packaging_adoption_override events are emitted for CI log scrapers.

Import adoption is gated too. Every provider plugin's discover_imports hook is gated on the packaging resolution, so a shared pool can never be tofu imported into a product's state. Leaf resources inside a pool stay importable.

Plan truthfulness and state key

  • Under packaging.mode: shared, plan.json no longer lists a create action for a container the product does not own, and carries a packaging summary — the digest-bound artifact a human approves now tells the truth.
  • A packaging-bearing contract defaults its backend state key to fluid/<safe-id>/terraform.tfstate. Contracts with no packaging block keep today's key exactly.

Operational requirements

  • tofu ≥ 1.6.0 on PATH. require_tofu_version() catches the silent terraform-on-PATH-as-tofu mixup at apply time.
  • Subprocess timeout: default 1800 seconds per tofu invocation. Override via FLUID_TOFU_TIMEOUT_SECONDS=<seconds>.

Security gates

Plan-binding integrity

The plan-binding gate from the native engine is replicated for OpenTofu — _apply_opentofu_engine.py::_verify_plan_binding_for_opentofu re-computes the bundleDigest and planDigest before any tofu apply. A tampered plan.json is rejected.

The emergency escape hatch is --no-verify-plan-binding; the apply logs at WARNING whenever it's used.

Destructive operations

--allow-data-loss is the override for destructive gate. When used, the apply emits a WARNING log line and a structured opentofu_destructive_gate_override event so CI log-scrapers can pick it up.

Brownfield: import existing resources

If the target cloud already has resources you want to fold into the IaC layer (rather than recreate), each provider plugin exposes a discover_imports hook that wires tofu import automatically. Run fluid generate iac --out ./review first, inspect the emitted resources against your live cloud, then cd review && tofu init && tofu import … for each pre-existing resource. The provider plugin's discover_imports produces the import commands for you.

What didn't change

  • The contract. No schema change — v0.8.3 contracts are byte-identical to v0.8.0 contracts.
  • The plan stage. fluid plan still produces the canonical Action list; the OpenTofu engine consumes that list and compiles to main.tf.json.
  • local provider. local keeps its native DuckDB apply path. fluid generate iac --provider local is a no-op.

See also

  • Catalog overview — where the retired Glue + Snowflake Horizon registrars now live
  • Providers vs Platforms — engine column added in v0.8.3
  • fluid apply — the user-facing apply command that auto-routes through OpenTofu
  • Network safety — outbound HTTP posture for cloud SDK calls
Edit this page on GitHub
Last Updated: 7/28/26, 8:03 PM
Contributors: fas89, Claude Opus 4.7 (1M context)
Prev
fluid verify-signature
Next
fluid generate-airflow