Fluid Forge
Why Forge
Concepts
Get Started
  • Consume a Data Product
  • See it run
  • Demos
  • Local (DuckDB)
  • Source-Aligned (Postgres → DuckDB)
  • AI Forge + Data Models
  • MCP Output Port — Serve to AI Agents
  • GCP (BigQuery)
  • Snowflake Team Collaboration
  • Declarative Airflow
  • Orchestration Export
  • Jenkins CI/CD
  • Universal Pipeline
  • 11-Stage Production Pipeline
  • Catalog Forge End-to-End
CLI Reference
  • Agent Policy (concept)
  • MCP Output Port — Serve to Agents
  • MCP deep-dive
  • AI-assisted authoring
  • LLM providers & backends
  • Overview
  • Quickstart
  • Examples
  • Your own CI
  • Your own scaffolding
  • Custom validator
  • Apply hook
  • Reference
  • Overview
  • Architecture
  • GCP (BigQuery)
  • AWS (S3 + Athena)
  • Snowflake
  • Local (DuckDB)
  • Custom Providers
  • Roadmap
GitHub
Why Forge
Concepts
Get Started
  • Consume a Data Product
  • See it run
  • Demos
  • Local (DuckDB)
  • Source-Aligned (Postgres → DuckDB)
  • AI Forge + Data Models
  • MCP Output Port — Serve to AI Agents
  • GCP (BigQuery)
  • Snowflake Team Collaboration
  • Declarative Airflow
  • Orchestration Export
  • Jenkins CI/CD
  • Universal Pipeline
  • 11-Stage Production Pipeline
  • Catalog Forge End-to-End
CLI Reference
  • Agent Policy (concept)
  • MCP Output Port — Serve to Agents
  • MCP deep-dive
  • AI-assisted authoring
  • LLM providers & backends
  • Overview
  • Quickstart
  • Examples
  • Your own CI
  • Your own scaffolding
  • Custom validator
  • Apply hook
  • Reference
  • Overview
  • Architecture
  • GCP (BigQuery)
  • AWS (S3 + Athena)
  • Snowflake
  • Local (DuckDB)
  • Custom Providers
  • Roadmap
GitHub
  • Introduction

    • Home
    • Why Fluid Forge
    • Getting Started
    • Snowflake Quickstart
    • See it run
    • Forge Data Model
    • Vision & Roadmap
    • Playground
    • FAQ
  • Concepts

    • Concepts
    • Builds, Exposes, Bindings
    • What is a contract?
    • Quality, SLAs & Lineage
    • Governance & Policy
    • Agent Policy (LLM/AI governance)
    • Providers vs Platforms
    • Fluid Forge vs alternatives
  • Data Products

    • Consume a Data Product
    • Product Types — SDP, ADP, CDP
  • Walkthroughs

    • Walkthrough: Local Development
    • Source-Aligned: Postgres → DuckDB → Parquet
    • AI Forge And Data-Model Journeys
    • Walkthrough: MCP Output Port
    • Walkthrough: Deploy to Google Cloud Platform
    • Walkthrough: Snowflake Team Collaboration
    • Declarative Airflow DAG Generation - The FLUID Way
    • Generating Orchestration Code from Contracts
    • Jenkins CI/CD for FLUID Data Products
    • Universal Pipeline
    • The 11-Stage Pipeline
    • End-to-End Walkthrough: Catalog → Contract → Transformation
  • CLI Reference

    • CLI Reference
    • Core workflow

      • fluid init
      • fluid demo
      • fluid forge
      • fluid validate
      • fluid plan
      • fluid apply
      • fluid diff
      • fluid status
    • Build & ship

      • fluid bundle
      • fluid generate
      • fluid generate artifacts
      • fluid validate-artifacts
      • fluid verify-signature
      • fluid generate iac
      • fluid generate-airflow
      • fluid generate-pipeline
      • fluid viz-graph
      • fluid publish
      • fluid ship
      • fluid rollback
      • fluid schedule-sync
    • AI & Agents

      • fluid ai
      • fluid agents
      • fluid mcp
      • fluid memory
      • fluid stats
      • fluid skills
    • Quality & governance

      • fluid test
      • fluid verify
      • fluid contract-tests
      • fluid contract-validation
      • fluid policy
      • fluid policy check
      • fluid policy compile
      • fluid policy apply
    • Standards & interoperability

      • fluid odps
      • fluid odps-bitol
      • fluid odcs
      • fluid export
      • fluid export-odps
      • fluid exporters
      • fluid import
      • fluid market
      • fluid datamesh-manager
    • Project & workspace

      • fluid product-new
      • fluid product-add
      • fluid workspace
      • fluid contract
      • fluid split
      • fluid config
      • fluid providers
      • fluid plugins
      • fluid provider-init
      • fluid auth
      • fluid secrets
      • fluid ide
      • fluid scaffold-ci
      • fluid scaffold-composer
      • fluid scaffold-ide
      • fluid docs
      • fluid runs
      • fluid retention
      • fluid describe
      • fluid doctor
      • fluid roadmap
      • fluid version
    • Catalog adapters

      • Source Catalog Integration (V1.5)
      • Publishing to a Catalog — Overview
      • BigQuery Catalog
      • Snowflake Horizon Catalog
      • Databricks Unity Catalog
      • Google Dataplex Catalog
      • AWS Glue Data Catalog
      • DataHub Catalog
      • Data Mesh Manager Catalog
      • OpenMetadata Catalog
    • CLI by task

      • CLI by task
      • Add quality rules
      • Add agent governance
      • Debug a failed pipeline run
      • Switch clouds with one line
  • Recipes

    • Recipes
    • Recipe — add a quality rule
    • Recipe — switch clouds with one line
    • Recipe — tag PII in your schema
    • Write a contract that consumes another contract
    • Generate per-environment overlays
  • SDK & Plugins

    • SDK & Plugins
    • Quickstart — your first plugin
    • Examples

      • Runnable examples
      • Example: hello-scaffold — the minimal viable plugin
      • Example: gitlab-ci-scaffold — generate a complete CI project
      • Example: steward-validator — a custom governance rule
      • Example: prod-key-guard — apply-time invariant check
    • Journeys

      • Journeys
      • Your own CI/CD

        • You have your own CI/CD setup, no problem
        • GitLab CI — the bundle template
        • GitHub Actions — the bundle template
        • Jenkins — the bundle template
        • CircleCI — the bundle template
      • You have a strict project layout, no problem
      • You have governance rules, no problem
      • You want a check at apply time, no problem
    • Reference

      • Reference
      • Roles reference
      • Entry points reference
      • Trust model
      • Packaging
      • Companion packages
  • Providers

    • Providers
    • Provider Architecture
    • GCP Provider
    • AWS Provider
    • Snowflake Provider
    • Local Provider
    • Creating Custom Providers
    • Provider Roadmap
  • AI & Agents

    • MCP Server
    • Built-in And Custom Forge Guidance
    • Forge Discovery Guide
    • Forge Memory Guide
    • Authoring Forge Tools
    • Guided fluid forge UX
    • LLM Providers
    • LiteLLM Backend
    • Capability Warnings
    • Cost Tracking
    • FLUID Forge Contract GPT Packet
    • Agentic Primitives
  • Operate & Deploy

    • Airflow Integration
    • Blueprints
    • Source-Aligned Acquisition
  • Govern & Secure

    • Governance, Compliance & the Business Case
    • Governance & Compliance
    • Network Safety
    • Credential Resolver — Security Model
  • Configuration & Reference

    • Environment Variables
    • Typed Errors
    • Typed CLI Errors
    • API Stability — fluid_build.api
  • Architecture & Releases

    • V1.5 Catalog Integration — Architecture Deep-Dive
    • V1.5 + V2 Hardening — Release Notes
  • Project

    • Contributing to Fluid Forge
    • Fluid Forge Docs Baseline: CLI 0.9.0
    • Fluid Forge Docs Baseline: CLI 0.8.11
    • Fluid Forge Docs Baseline: CLI 0.8.10
    • Fluid Forge Docs Baseline: CLI 0.8.9
    • Fluid Forge Docs Baseline: CLI 0.8.8
    • Fluid Forge Docs Baseline: CLI 0.8.7
    • Fluid Forge Docs Baseline: CLI 0.8.6
    • Fluid Forge Docs Baseline: CLI 0.8.5
    • Fluid Forge Docs Baseline: CLI 0.8.4
    • Fluid Forge Docs Baseline: CLI 0.8.3
    • Fluid Forge Docs Baseline: CLI 0.8.0
    • Fluid Forge Docs Baseline: CLI 0.7.11
    • Fluid Forge Docs Baseline: CLI 0.7.9
    • Fluid Forge v0.7.1 - Multi-Provider Export Release

🎬 CLI demos

14 frame-perfect SVG casts — install through deploy, local through Snowflake, the AI copilot, agentPolicy enforcement, day-2 incident response, and agent-loop compaction. Each one carries a takeaway popup with the punchline numbers. Click play — the SVG only animates after you opt in (no autoplay, no JS).

Convinced? → Install in 30 seconds. Want longer-form proof of specific workflows? → See it run (5 narrative scenarios, ~50 s each, with takeaway numbers).


Install + run, locally

Start here. No cloud account, no credit card, ~30 seconds end-to-end.

fluid init my-project --quickstart → validate → plan → apply
Install the CLI, scaffold the Customer 360 Analytics contract from the quickstart template, validate it against the 0.7.2 schema, preview the plan, and apply it against the local DuckDB provider. Two exposes — a master table and a high-value-customer view — land as Parquet under output/.

Same contract, different cloud

Swap binding.platform and re-deploy. The contract, schema, dq.rules, and the multi-stage build all stay byte-identical — only the cloud-specific binding fields change.

GCP / BigQuery

GCP quickstart — install, swap one line, deploy
From the local Customer 360 contract to a fully-deployed BigQuery dataset, in seconds. The `git diff` shows only the binding block changing — schema, dq.rules, and the 5-stage build all stay byte-identical.

AWS / Athena

AWS quickstart — S3 + Glue + Athena
Same Customer 360 contract, AWS provider extra installed, binding swapped to the AWS platform. S3 bucket provisioned, Glue catalog auto-created, Athena made queryable — all from fluid apply.

Snowflake

Snowflake quickstart — dry-run flow
The dry version: env-file credentials, contract validation, plan preview, and apply --mode dry-run rendering DDL without firing it. For the live-auth version see snowflake-real below.

AI copilot — full Gemini-powered flow

The fluid forge AI copilot generating a finance-domain contract end-to-end: project memory loaded, finance domain expertise pack applied (SOX + GDPR), local context discovered, a Gemini streaming call, and the contract emerging block-by-block with the agentPolicy gate. The hand-scripted version below mirrors the real-API flow at frame-perfect fidelity; a real-capture script (scripts/demos/forge_gemini_real_capture.py) is preserved for users who want to record an actual session.

fluid forge --domain finance --llm-provider gemini --llm-model gemini-2.5-flash
The full agent flow: memory → domain pack → discovery → streaming → contract emit (schema, dq.rules, accessPolicy, agentPolicy, sovereignty) → auto-validation → memory persist.

Snowflake live-auth dry-run

The snowflake-biz-lab flow at full fidelity: env credentials sourced, real validate --strict, plan against the live account, apply --mode dry-run rendering DDL without firing it, then policy-apply --mode check over the compiled IAM bindings.

Snowflake — validate → plan → apply --mode dry-run → policy-apply --mode check
Live auth (account=acme-demo placeholder; the scrubber substitutes the real account name). No DDL fires, no RBAC mutates — just the auth + connectivity + dry-render flow you'd run before a real production deploy.

AI copilot — interview shape only

fluid forge --blank skips the LLM call entirely and just scaffolds the structured stub for the chosen domain. Useful when you know what you want and don't need an LLM round-trip.

fluid forge --blank --domain finance
The blank skeleton with finance-domain defaults pre-seeded: SOX/GDPR regulatory framework, 'training' and 'fine_tuning' denied use cases, Gold layer assignment. Fill in the expose blocks yourself — no LLM call.

Policy + IAM compilation

The policy-check → generate artifacts → policy-apply --mode check triple. Validates the access policy, compiles to native cloud IAM (BigQuery/Snowflake/AWS), and runs the bindings in check-only mode (no live IAM mutations).

policy-check → generate artifacts → policy-apply --mode check
Three commands, full policy round-trip from declarative `accessPolicy.grants` in YAML to native cloud IAM JSON, then a dry-run that shows exactly which bindings would apply against the deployed state.

agentPolicy enforcement (LLM / AI governance)

Declare agentPolicy in YAML, validate it, see the enforcement summary, watch a replay of agent reads (allow/deny) against the policy.

agentPolicy — declare, validate, gate (validate → policy-check → audit)
The YAML block (allowedModels, deniedUseCases, canStore, auditRequired) → validate → policy-check enforcement summary → 4 replayed agent reads (gpt-4 allowed, claude-3 + training denied, unlisted model denied, gemini summarization allowed).

Long-form scenario casts

The 5 casts below pair with the See it run page — each tells a story (problem → CLI flow → punchline) at ~30-50 seconds with takeaway numbers.

$0.03 per data product — three providers, one contract

fluid forge data-model — same intent, three providers, real cost figures
Eight lines of intent YAML. Anthropic, OpenAI, Ollama — same flag swapped each time. All three emit the same valid contract, the same dbt project layout. Real production token counts and costs.

Six months → sixty seconds — source-aligned Bronze

fluid init --discover postgres://... — Bronze contract in 60 seconds
Connect, scan information_schema (47 ms), infer 28 tables / 143 columns / 12 PII candidates / 8 foreign keys, emit a complete Bronze contract.fluid.yaml, validate, apply against embedded DuckDB. 6.2 seconds total. Then show the engine swap path.

23 questions, skipped — guided UX

fluid forge — guided UX in action
47 ms welcome scan finds 3 CSVs + 2 dbt models + 1 README, infers domain (finance) and PII (5 columns). 5-mode picker. 4 questions answered (most accept the inferred default with ↵). Cost-cap progress in real time ($0.000 → $0.021 of $0.050 cap). Pre-write panel shows exactly what will + won't change.

3am Slack ping → ship in 90 seconds

3am Slack ping → ship in 90 seconds
PagerDuty: freshness SLA breached. fluid runs status shows 3 consecutive failures. fluid runs logs --component dlq surfaces the root cause. fluid runs diff shows what changed since the last OK run. One-line contract fix. fluid ship. 87 seconds end-to-end. Recovered 12,361 rows.

$0.50 → $0.05 — agent-loop compaction

Agent-loop compaction — three strategies, real before/after costs
20-turn baseline: $0.503/run, super-linear context bloat (5K → 67K → 298K tokens). Three strategies side-by-side: truncate (5.8× cheaper), summarize (9.3× cheaper), hybrid (10.5× cheaper). One env var: FLUID_COMPACTION_STRATEGY=hybrid.

How the casts are produced

The pipeline that built each SVG above:

   scripts/demos/<name>.py            ← cast generator
                ↓
        /tmp/casts/<name>.cast.raw    ← raw asciinema cast (gitignored)
                ↓
        scripts/cast-v3-to-v2.py      ← format conversion (asciinema 3.x → 2.x)
                ↓
        scripts/scrub-cast.py         ← strip API keys, JWTs, env-shaped secrets
                ↓
        svg-term --in <cast>          ← render to animated SVG (no --window;
                                          our <CliCast> component supplies
                                          the terminal chrome)
                ↓
   docs/.vuepress/public/demos/<name>.svg   ← the only file that gets committed

Two passes of secret-scanning happen:

  1. scrub-cast.py redacts known formats (AIza…, sk-…, sk-ant-…, JWTs, KEY=…/SECRET=…/PASSWORD=… 16+ char values) and substitutes literal $SNOWFLAKE_ACCOUNT/$SNOWFLAKE_USER/$GEMINI_API_KEY env values for friendly placeholders.
  2. Final-SVG grep in generate-demos.sh re-scans the output before keeping the file. If any leak pattern matches in the post-scrub SVG, the file is deleted and the build fails.

The .cast.raw working files live in /tmp/casts/ (gitignored) and are deleted at the end of each render.

To regenerate everything:

scripts/generate-demos.sh                 # regenerate every cast
scripts/generate-demos.sh --safe-only     # only the credential-free casts
scripts/generate-demos.sh forge-gemini    # one specific cast

Source for each cast lives at scripts/demos/<name>.py — review or fork freely.

Edit this page on GitHub
Last Updated: 5/29/26, 4:51 PM
Contributors: fas89, Claude Opus 4.7 (1M context)