Fluid Forge
Why Forge
Concepts
Get Started
  • Consume a Data Product
  • See it run
  • Demos
  • Local (DuckDB)
  • Source-Aligned (Postgres → DuckDB)
  • AI Forge + Data Models
  • MCP Output Port — Serve to AI Agents
  • GCP (BigQuery)
  • Snowflake Team Collaboration
  • Declarative Airflow
  • Orchestration Export
  • Jenkins CI/CD
  • Universal Pipeline
  • 11-Stage Production Pipeline
  • Catalog Forge End-to-End
CLI Reference
  • Agent Policy (concept)
  • MCP Output Port — Serve to Agents
  • MCP deep-dive
  • AI-assisted authoring
  • LLM providers & backends
  • Overview
  • Quickstart
  • Examples
  • Your own CI
  • Your own scaffolding
  • Custom validator
  • Apply hook
  • Reference
  • Overview
  • Architecture
  • GCP (BigQuery)
  • AWS (S3 + Athena)
  • Snowflake
  • Local (DuckDB)
  • Custom Providers
  • Roadmap
GitHub
Why Forge
Concepts
Get Started
  • Consume a Data Product
  • See it run
  • Demos
  • Local (DuckDB)
  • Source-Aligned (Postgres → DuckDB)
  • AI Forge + Data Models
  • MCP Output Port — Serve to AI Agents
  • GCP (BigQuery)
  • Snowflake Team Collaboration
  • Declarative Airflow
  • Orchestration Export
  • Jenkins CI/CD
  • Universal Pipeline
  • 11-Stage Production Pipeline
  • Catalog Forge End-to-End
CLI Reference
  • Agent Policy (concept)
  • MCP Output Port — Serve to Agents
  • MCP deep-dive
  • AI-assisted authoring
  • LLM providers & backends
  • Overview
  • Quickstart
  • Examples
  • Your own CI
  • Your own scaffolding
  • Custom validator
  • Apply hook
  • Reference
  • Overview
  • Architecture
  • GCP (BigQuery)
  • AWS (S3 + Athena)
  • Snowflake
  • Local (DuckDB)
  • Custom Providers
  • Roadmap
GitHub
  • Introduction

    • Home
    • Why Fluid Forge
    • Getting Started
    • Snowflake Quickstart
    • See it run
    • Forge Data Model
    • Vision & Roadmap
    • Playground
    • FAQ
  • Concepts

    • Concepts
    • Builds, Exposes, Bindings
    • What is a contract?
    • Quality, SLAs & Lineage
    • Governance & Policy
    • Agent Policy (LLM/AI governance)
    • Providers vs Platforms
    • Fluid Forge vs alternatives
  • Data Products

    • Consume a Data Product
    • Product Types — SDP, ADP, CDP
  • Walkthroughs

    • Walkthrough: Local Development
    • Source-Aligned: Postgres → DuckDB → Parquet
    • AI Forge And Data-Model Journeys
    • Walkthrough: MCP Output Port
    • Walkthrough: Deploy to Google Cloud Platform
    • Walkthrough: Snowflake Team Collaboration
    • Declarative Airflow DAG Generation - The FLUID Way
    • Generating Orchestration Code from Contracts
    • Jenkins CI/CD for FLUID Data Products
    • Universal Pipeline
    • The 11-Stage Pipeline
    • End-to-End Walkthrough: Catalog → Contract → Transformation
  • CLI Reference

    • CLI Reference
    • Core workflow

      • fluid init
      • fluid demo
      • fluid forge
      • fluid validate
      • fluid plan
      • fluid apply
      • fluid diff
      • fluid status
    • Build & ship

      • fluid bundle
      • fluid generate
      • fluid generate artifacts
      • fluid validate-artifacts
      • fluid verify-signature
      • fluid generate iac
      • fluid generate-airflow
      • fluid generate-pipeline
      • fluid viz-graph
      • fluid publish
      • fluid ship
      • fluid rollback
      • fluid schedule-sync
    • AI & Agents

      • fluid ai
      • fluid agents
      • fluid mcp
      • fluid memory
      • fluid stats
      • fluid skills
    • Quality & governance

      • fluid test
      • fluid verify
      • fluid contract-tests
      • fluid contract-validation
      • fluid policy
      • fluid policy check
      • fluid policy compile
      • fluid policy apply
    • Standards & interoperability

      • fluid odps
      • fluid odps-bitol
      • fluid odcs
      • fluid export
      • fluid export-odps
      • fluid exporters
      • fluid import
      • fluid market
      • fluid datamesh-manager
    • Project & workspace

      • fluid product-new
      • fluid product-add
      • fluid workspace
      • fluid contract
      • fluid split
      • fluid config
      • fluid providers
      • fluid plugins
      • fluid provider-init
      • fluid auth
      • fluid secrets
      • fluid ide
      • fluid scaffold-ci
      • fluid scaffold-composer
      • fluid scaffold-ide
      • fluid docs
      • fluid runs
      • fluid retention
      • fluid describe
      • fluid doctor
      • fluid roadmap
      • fluid version
    • Catalog adapters

      • Source Catalog Integration (V1.5)
      • Publishing to a Catalog — Overview
      • BigQuery Catalog
      • Snowflake Horizon Catalog
      • Databricks Unity Catalog
      • Google Dataplex Catalog
      • AWS Glue Data Catalog
      • DataHub Catalog
      • Data Mesh Manager Catalog
      • OpenMetadata Catalog
    • CLI by task

      • CLI by task
      • Add quality rules
      • Add agent governance
      • Debug a failed pipeline run
      • Switch clouds with one line
  • Recipes

    • Recipes
    • Recipe — add a quality rule
    • Recipe — switch clouds with one line
    • Recipe — tag PII in your schema
    • Write a contract that consumes another contract
    • Generate per-environment overlays
  • SDK & Plugins

    • SDK & Plugins
    • Quickstart — your first plugin
    • Examples

      • Runnable examples
      • Example: hello-scaffold — the minimal viable plugin
      • Example: gitlab-ci-scaffold — generate a complete CI project
      • Example: steward-validator — a custom governance rule
      • Example: prod-key-guard — apply-time invariant check
    • Journeys

      • Journeys
      • Your own CI/CD

        • You have your own CI/CD setup, no problem
        • GitLab CI — the bundle template
        • GitHub Actions — the bundle template
        • Jenkins — the bundle template
        • CircleCI — the bundle template
      • You have a strict project layout, no problem
      • You have governance rules, no problem
      • You want a check at apply time, no problem
    • Reference

      • Reference
      • Roles reference
      • Entry points reference
      • Trust model
      • Packaging
      • Companion packages
  • Providers

    • Providers
    • Provider Architecture
    • GCP Provider
    • AWS Provider
    • Snowflake Provider
    • Local Provider
    • Creating Custom Providers
    • Provider Roadmap
  • AI & Agents

    • MCP Server
    • Built-in And Custom Forge Guidance
    • Forge Discovery Guide
    • Forge Memory Guide
    • Authoring Forge Tools
    • Guided fluid forge UX
    • LLM Providers
    • LiteLLM Backend
    • Capability Warnings
    • Cost Tracking
    • FLUID Forge Contract GPT Packet
    • Agentic Primitives
  • Operate & Deploy

    • Airflow Integration
    • Blueprints
    • Source-Aligned Acquisition
  • Govern & Secure

    • Governance, Compliance & the Business Case
    • Governance & Compliance
    • Network Safety
    • Credential Resolver — Security Model
  • Configuration & Reference

    • Environment Variables
    • Typed Errors
    • Typed CLI Errors
    • API Stability — fluid_build.api
  • Architecture & Releases

    • V1.5 Catalog Integration — Architecture Deep-Dive
    • V1.5 + V2 Hardening — Release Notes
  • Project

    • Contributing to Fluid Forge
    • Fluid Forge Docs Baseline: CLI 0.9.0
    • Fluid Forge Docs Baseline: CLI 0.8.11
    • Fluid Forge Docs Baseline: CLI 0.8.10
    • Fluid Forge Docs Baseline: CLI 0.8.9
    • Fluid Forge Docs Baseline: CLI 0.8.8
    • Fluid Forge Docs Baseline: CLI 0.8.7
    • Fluid Forge Docs Baseline: CLI 0.8.6
    • Fluid Forge Docs Baseline: CLI 0.8.5
    • Fluid Forge Docs Baseline: CLI 0.8.4
    • Fluid Forge Docs Baseline: CLI 0.8.3
    • Fluid Forge Docs Baseline: CLI 0.8.0
    • Fluid Forge Docs Baseline: CLI 0.7.11
    • Fluid Forge Docs Baseline: CLI 0.7.9
    • Fluid Forge v0.7.1 - Multi-Provider Export Release

See Fluid Forge in action

Five short demos. Each one starts with the pain. Each one closes with a number you remember. Real numbers from production runs, rendered as frame-perfect SVGs (some live-captured, some faithfully scripted). Click play, read the takeaway banner, move on.

Pick your problem

Your painWatch thisTimePunchline
Writing 200 lines of Python per data source$0.03 per data product~50 sThree providers. Same contract. Free on Ollama.
Six months to set up Airbyte for one PostgresSix months → sixty seconds~40 sSix engines. One contract. Zero clusters.
Most CLIs ask 27 questions before they help23 questions, skipped~50 s47 ms scan. 23 of 27 questions auto-resolved.
3am Slack ping: pipeline broke, who knows whySkip the panic~75 s3am incident → ship in 90 seconds.
Your agent loop quietly spent $0.50 on tool history$0.50 → $0.05~40 sOne env var. 10× cheaper agent loops.

$0.03 per data product

Most data teams write 200 lines of Python per source. Or 8 lines of YAML and let the AI build the contract — same intent, three providers (Anthropic Haiku 4.5, OpenAI gpt-4.1-mini, local Ollama gemma2), real production cost figures on screen. Switch providers any time. The contract stays portable.

fluid forge data-model — same intent, three providers, real cost figures
Eight lines of intent YAML. Anthropic, OpenAI, Ollama — same flag swapped each time. All three emit the same valid contract, the same dbt project layout. Real production token counts and costs.

Pairs with Forge Data Model and LLM Providers. Long-form animated reel preserved at /reels/forge-in-action.html.


Six months → sixty seconds

Six months of Airbyte. Two weeks of Airflow DAGs. JVM heap tuning. For one Postgres source. Or fluid init --discover postgres://… and a 60-second flight to a working Bronze contract — switch engine: between duckdb, dlt, meltano, airbyte, kafka-connect, debezium when you outgrow embedded mode.

fluid init --discover postgres://... — Bronze contract in 60 seconds
Connect, scan information_schema (47 ms), infer 28 tables / 143 columns / 12 PII candidates / 8 foreign keys, emit a complete Bronze contract.fluid.yaml, validate, apply against embedded DuckDB. 6.2 seconds total. Then show the engine swap path.

Pairs with Source-Aligned Acquisition, Postgres → DuckDB walkthrough, and fluid init --discover. Long-form animated reel preserved at /reels/source-aligned-bronze.html.


23 questions, skipped

Most CLIs ask 27 questions before they help you. Forge asks four — the rest, it already knows. 47 ms welcome scan, 5-mode picker, inferences-first interview, slash commands at every prompt, cost-cap progress prefix in real time, and a pre-write panel so nothing surprises you.

fluid forge — guided UX in action
47 ms welcome scan finds 3 CSVs + 2 dbt models + 1 README, infers domain (finance) and PII (5 columns). 5-mode picker. 4 questions answered (most accept the inferred default with ↵). Cost-cap progress in real time ($0.000 → $0.021 of $0.050 cap). Pre-write panel shows exactly what will + won't change.

Pairs with Guided fluid forge UX and the fluid forge reference. Long-form animated reel preserved at /reels/guided-forge-ux.html.


Skip the panic

It's 3am. Pipeline broke. You have 90 seconds. fluid runs status (where), fluid runs logs --component dlq (why), fluid runs diff (what), one-line policy fix, fluid ship. Day-1 ships. Day-2 doesn't surprise.

3am Slack ping → ship in 90 seconds
PagerDuty: freshness SLA breached. fluid runs status shows 3 consecutive failures. fluid runs logs --component dlq surfaces the root cause (CHECK constraint NOT NULL, 47 partial-window customers). fluid runs diff shows what changed since the last OK run. One-line contract fix: NOT_NULL → NOT_NULL_WHERE customer_age_days >= 30. fluid ship. 87 seconds end-to-end. Recovered 12,361 rows.

Pairs with fluid runs, fluid retention, fluid secrets, fluid stats, and Typed CLI Errors. Long-form animated reel preserved at /reels/day2-ops.html.


$0.50 → $0.05

Long agent loops accumulate tool results — every turn rides on top of the last one. Without compaction, your bill grows super-linearly. Three strategies — truncate (free, default), summarize (LLM-backed, high recall), hybrid (cheap path first, summariser as safety net) — pick yours, set one env var, watch the bill drop 10×.

Agent-loop compaction — three strategies, real before/after costs
20-turn baseline: $0.503/run, super-linear context bloat (5K → 67K → 298K tokens). Three strategies side-by-side: truncate (5.8× cheaper), summarize (9.3× cheaper), hybrid (10.5× cheaper). One env var: FLUID_COMPACTION_STRATEGY=hybrid. No code changes.

Pairs with Agentic primitives → Token-budget pre-flight & compaction. Long-form animated reel preserved at /reels/compaction-and-warnings.html.


More demos in 30 seconds each

The full library — 14 frame-perfect SVG casts in all — covers AWS, Snowflake live-auth dry-run, agent-policy enforcement, blank scaffolds, policy compilation, and the canonical local quickstart:

→ Browse all demos


How these are sourced

Each cast above is a deterministic asciinema-rendered SVG generated by a Python pipeline (scripts/cast_builder.py → scrub-cast.py → svg-term). The narrative lives in scripts/demos/<name>.py — clone, edit, regenerate. Each cast plays inline on every browser without GIFs, JS frameworks, or third-party CDNs.

Token counts, durations, and costs in LLM-driven content (forge-multi-provider, agent-compaction) are real captures from production runs. Latency numbers in deterministic content (source-aligned-bronze, guided-forge-ux, day2-ops) are representative of the v0.10.0 stack against the bundled examples/ fixtures.

The original animated HTML reels are preserved at /reels/ for anyone who wants the longer-form pacing — each cast section above links through to its corresponding reel.

See also

  • Get Started — install, scaffold, validate, run locally
  • Source-Aligned Acquisition — the framework powering the Bronze cast
  • Product Types — SDP, ADP, CDP — the vocabulary used throughout
  • Guided fluid forge UX — the architecture behind the guided UX cast
  • Capability Warnings — the per-model coverage matrix
Edit this page on GitHub
Last Updated: 6/27/26, 4:58 PM
Contributors: fas89, Claude Opus 4.7 (1M context)
Prev
Snowflake Quickstart
Next
Forge Data Model