Fluid Forge
Contract-first data products — from local DuckDB to any cloud, trusted by your team and safe for your AI agents.
Local First
Install the CLI, scaffold a project, validate it, and run it locally before you touch cloud credentials.
Contract-Driven
Use one FLUID contract to describe the data product, then plan, test, verify, and publish from the same source of truth.
Promoted CLI Surface
These docs track the current "fluid --help" experience so new users are not sent down stale or deprecated command paths.
AI-Optional
Start with "fluid init" for a quickstart or use "fluid forge" when you want AI-assisted scaffolding and discovery.
Multi-Target Delivery
Build locally with DuckDB, then target GCP, AWS, Snowflake, or standards/export flows when you are ready.
Compatibility Aware
Legacy commands still exist in the docs where they matter, but primary pages lead with the current recommended workflow.
Fluid Forge is for data engineers who want to write a data product contract once and deploy it anywhere. Build and test locally with DuckDB, then push the same contract to BigQuery, Athena, or Snowflake — no pipeline glue code to maintain.
What you don't need to do
Fluid Forge replaces the five-tool stack most data teams currently maintain. With one contract.fluid.yaml:
- No Airflow DAG to write or maintain.
fluid generate schedule --scheduler airflow|dagster|prefectemits the right artifact. - No JVM heap tuning.
engine: duckdbruns embedded for dev; swap todlt/meltano/airbyte/kafka-connect/debeziumonly when you need them. - No Snowflake permission sprawl.
accessPolicy.grantscompiles to nativeGRANTstatements. - No Terraform for data-product IAM.
policy-applyemits BigQuery IAM bindings, Snowflake roles, S3 bucket policies — same source. - No 27 questions before you ship.
fluid forgeinfers from your local files; you answer 4. - No dbt project layout decisions. Forge wraps dbt; you write the contract, dbt does what it does best.
- No AI access surprises.
agentPolicydeclares which LLMs can read what, with audit logs, before any model gets a row. - No vendor lock.
binding.platform: snowflake→binding.platform: bigqueryis the only line that moves.
→ See the comparison page: Forge vs dbt / Dagster / Terraform / Snowpark for the honest breakdown of when Forge does and doesn't fit.
See it run
A 60-second walkthrough of the core move: write one contract.fluid.yaml, build and test it locally on DuckDB, then ship the same file to BigQuery and Snowflake — the only line that changes is binding.platform.
Use ←/→ to step scenes, space to pause, r to restart. See all reels → — quickstart, source-aligned Bronze, guided forge UX, day-2 ops, and more.
Start with the current workflow
pip install data-product-forge
fluid version
fluid doctor
fluid init my-project --quickstart
cd my-project
fluid validate contract.fluid.yaml
fluid plan contract.fluid.yaml
fluid apply contract.fluid.yaml --yes
This docs site currently tracks:
- CLI release
0.10.0 - Scaffolded contract examples using
fluidVersion: 0.7.5(older0.7.4/0.7.3/0.7.2contracts remain valid)
fluid version and fluidVersion are different things. The first is the CLI release you installed. The second is the schema version inside a contract.
Optional AI-assisted scaffolding
fluid forge
fluid forge --domain retail
fluid forge --llm-provider openai --llm-model gpt-4.1-mini
Use fluid forge when you want discovery, memory, and LLM-guided scaffolding. Use fluid init when you want the fastest deterministic quickstart.
For model-first work, forge from a business intent file and then generate dbt:
fluid forge data-model from-intent --example retail > intent.yaml
fluid forge data-model from-intent intent.yaml -o customer_orders.fluid.yaml
fluid generate transformation customer_orders.fluid.yaml -o ./dbt_customer_orders --dbt-validate
For every AI and data-model journey, including hosted provider strict mode, Ollama, DDL, source catalogs, review/diff/learn, and scheduling, see AI Forge And Data-Model Journeys.
Promoted command groups
| Group | Commands |
|---|---|
| Core Workflow | init, forge, forge data-model, validate, plan, apply, ship |
| Generate | generate transformation, generate dbt-tests, generate schedule, generate ci, generate standard |
| Integrations | publish, market, import |
| Quality & Governance | policy-check, diff, test, verify, contract |
| Day-2 Ops | runs, retention, secrets, stats |
| Utilities | config, split, bundle, auth, doctor, providers, memory, mcp, version |
Current release — 0.10.0, schema 0.7.5 GA
0.10.0 is the current release. Its headline is the plugin platform maturing: a new operator trust boundary — FLUID_PLUGINS_ALLOWLIST / FLUID_PLUGINS_BLOCKLIST gate every code-executing entry-point group before a plugin loads (a blocked plugin's code never runs) — plus two new inspection commands, fluid plugins (installed plugins per role, with allow/block status) and fluid exporters. 0.10.0 also reclassifies odps / odcs: they are spec exporters (contract → an open standard), not cloud providers, and now surface under fluid exporters rather than fluid providers — the export commands themselves are unchanged. The companion SDK reaches 0.10.0 (four typed role ABCs — CustomScaffold / Validator / InfraProvider / CatalogAdapter — typed Severity / ActionStatus / Phase domains, and an SDK↔CLI compatibility declaration), and the custom-scaffold engine reaches 0.4.0 with reproducible builds (a fluid-scaffold.lock lockfile, --pin, and copier-style --update). Schema 0.7.5 remains the GA default. The streaming Kafka → Iceberg sink (Kafka-Connect / Debezium connector config plus a managed Confluent Tableflow plugin landing data into Apache Iceberg) shipped in 0.9.0; recent releases also added pluggable modeling techniques + metadata-source adapters for fluid forge data-model (0.8.11), an extension-schema-aware copilot that generates and validates plugin contract.extensions.* blocks (0.8.9), and fluid describe --self capability introspection (0.8.10), all building on the MCP output-port gateway — runtime agentPolicy enforcement (model / use-case checks, row-level filtering, PII/PHI value redaction) with JWT-bearer + mTLS identity, introduced in 0.8.7 — see fluid mcp. The platform builds on the SDP / ADP / CDP Data Mesh vocabulary alongside the medallion Bronze / Silver / Gold layers, six ingestion engines (duckdb, dlt, meltano, airbyte, kafka-connect, debezium), the guided fluid forge UX (mode picker, welcome scan, slash commands, preview panel), and a companion SDK (data-product-forge-sdk). See SDK & Plugins, Source-Aligned Acquisition, and Product Types for the full picture.
Where to go next
- Getting Started for the local-first path
- SDK & Plugins — extend the CLI with your own scaffolds, validators, and apply-hooks
- Forge Data Model for intent, DDL, and catalog-driven model generation
- AI Forge And Data-Model Journeys for end-to-end AI-assisted and deterministic flows
- CLI Reference for the promoted command surface
- Providers for platform-specific guidance
- Walkthroughs for end-to-end examples
Compatibility note: fluid generate-airflow is still available, but primary docs now lead with fluid generate schedule --scheduler airflow.
Need help?
- Questions or ideas? Start a GitHub Discussion — we read every one.
- Bug or unexpected behavior? Open an issue with what you ran and what you saw.
- Want to contribute? See the contributing guide — we welcome doc fixes, examples, and providers.