Fluid Forge
Why Forge
Concepts
Get Started
  • Consume a Data Product
  • See it run
  • Demos
  • Local (DuckDB)
  • Source-Aligned (Postgres → DuckDB)
  • AI Forge + Data Models
  • MCP Output Port — Serve to AI Agents
  • GCP (BigQuery)
  • Snowflake Team Collaboration
  • Declarative Airflow
  • Orchestration Export
  • Jenkins CI/CD
  • Universal Pipeline
  • 11-Stage Production Pipeline
  • Catalog Forge End-to-End
CLI Reference
  • Agent Policy (concept)
  • MCP Output Port — Serve to Agents
  • MCP deep-dive
  • AI-assisted authoring
  • LLM providers & backends
  • Overview
  • Quickstart
  • Examples
  • Your own CI
  • Your own scaffolding
  • Custom validator
  • Apply hook
  • Reference
  • Overview
  • Architecture
  • GCP (BigQuery)
  • AWS (S3 + Athena)
  • Snowflake
  • Local (DuckDB)
  • Custom Providers
  • Roadmap
GitHub
Why Forge
Concepts
Get Started
  • Consume a Data Product
  • See it run
  • Demos
  • Local (DuckDB)
  • Source-Aligned (Postgres → DuckDB)
  • AI Forge + Data Models
  • MCP Output Port — Serve to AI Agents
  • GCP (BigQuery)
  • Snowflake Team Collaboration
  • Declarative Airflow
  • Orchestration Export
  • Jenkins CI/CD
  • Universal Pipeline
  • 11-Stage Production Pipeline
  • Catalog Forge End-to-End
CLI Reference
  • Agent Policy (concept)
  • MCP Output Port — Serve to Agents
  • MCP deep-dive
  • AI-assisted authoring
  • LLM providers & backends
  • Overview
  • Quickstart
  • Examples
  • Your own CI
  • Your own scaffolding
  • Custom validator
  • Apply hook
  • Reference
  • Overview
  • Architecture
  • GCP (BigQuery)
  • AWS (S3 + Athena)
  • Snowflake
  • Local (DuckDB)
  • Custom Providers
  • Roadmap
GitHub
  • Introduction

    • Home
    • Why Fluid Forge
    • Getting Started
    • Snowflake Quickstart
    • See it run
    • Forge Data Model
    • Vision & Roadmap
    • Playground
    • FAQ
  • Concepts

    • Concepts
    • Builds, Exposes, Bindings
    • What is a contract?
    • Quality, SLAs & Lineage
    • Governance & Policy
    • Agent Policy (LLM/AI governance)
    • Providers vs Platforms
    • Fluid Forge vs alternatives
  • Data Products

    • Consume a Data Product
    • Product Types — SDP, ADP, CDP
  • Walkthroughs

    • Walkthrough: Local Development
    • Source-Aligned: Postgres → DuckDB → Parquet
    • AI Forge And Data-Model Journeys
    • Walkthrough: MCP Output Port
    • Walkthrough: Deploy to Google Cloud Platform
    • Walkthrough: Snowflake Team Collaboration
    • Declarative Airflow DAG Generation - The FLUID Way
    • Generating Orchestration Code from Contracts
    • Jenkins CI/CD for FLUID Data Products
    • Universal Pipeline
    • The 11-Stage Pipeline
    • End-to-End Walkthrough: Catalog → Contract → Transformation
  • CLI Reference

    • CLI Reference
    • Core workflow

      • fluid init
      • fluid demo
      • fluid forge
      • fluid validate
      • fluid plan
      • fluid apply
      • fluid diff
      • fluid status
    • Build & ship

      • fluid bundle
      • fluid generate
      • fluid generate artifacts
      • fluid validate-artifacts
      • fluid verify-signature
      • fluid generate iac
      • fluid generate-airflow
      • fluid generate-pipeline
      • fluid viz-graph
      • fluid publish
      • fluid ship
      • fluid rollback
      • fluid schedule-sync
    • AI & Agents

      • fluid ai
      • fluid agents
      • fluid mcp
      • fluid memory
      • fluid stats
      • fluid skills
    • Quality & governance

      • fluid test
      • fluid verify
      • fluid contract-tests
      • fluid contract-validation
      • fluid policy
      • fluid policy check
      • fluid policy compile
      • fluid policy apply
    • Standards & interoperability

      • fluid odps
      • fluid odps-bitol
      • fluid odcs
      • fluid export
      • fluid export-odps
      • fluid exporters
      • fluid import
      • fluid market
      • fluid datamesh-manager
    • Project & workspace

      • fluid product-new
      • fluid product-add
      • fluid workspace
      • fluid contract
      • fluid split
      • fluid config
      • fluid providers
      • fluid plugins
      • fluid provider-init
      • fluid auth
      • fluid secrets
      • fluid ide
      • fluid scaffold-ci
      • fluid scaffold-composer
      • fluid scaffold-ide
      • fluid docs
      • fluid runs
      • fluid retention
      • fluid describe
      • fluid doctor
      • fluid roadmap
      • fluid version
    • Catalog adapters

      • Source Catalog Integration (V1.5)
      • Publishing to a Catalog — Overview
      • BigQuery Catalog
      • Snowflake Horizon Catalog
      • Databricks Unity Catalog
      • Google Dataplex Catalog
      • AWS Glue Data Catalog
      • DataHub Catalog
      • Data Mesh Manager Catalog
      • OpenMetadata Catalog
    • CLI by task

      • CLI by task
      • Add quality rules
      • Add agent governance
      • Debug a failed pipeline run
      • Switch clouds with one line
  • Recipes

    • Recipes
    • Recipe — add a quality rule
    • Recipe — switch clouds with one line
    • Recipe — tag PII in your schema
    • Write a contract that consumes another contract
    • Generate per-environment overlays
  • SDK & Plugins

    • SDK & Plugins
    • Quickstart — your first plugin
    • Examples

      • Runnable examples
      • Example: hello-scaffold — the minimal viable plugin
      • Example: gitlab-ci-scaffold — generate a complete CI project
      • Example: steward-validator — a custom governance rule
      • Example: prod-key-guard — apply-time invariant check
    • Journeys

      • Journeys
      • Your own CI/CD

        • You have your own CI/CD setup, no problem
        • GitLab CI — the bundle template
        • GitHub Actions — the bundle template
        • Jenkins — the bundle template
        • CircleCI — the bundle template
      • You have a strict project layout, no problem
      • You have governance rules, no problem
      • You want a check at apply time, no problem
    • Reference

      • Reference
      • Roles reference
      • Entry points reference
      • Trust model
      • Packaging
      • Companion packages
  • Providers

    • Providers
    • Provider Architecture
    • GCP Provider
    • AWS Provider
    • Snowflake Provider
    • Local Provider
    • Creating Custom Providers
    • Provider Roadmap
  • AI & Agents

    • MCP Server
    • Built-in And Custom Forge Guidance
    • Forge Discovery Guide
    • Forge Memory Guide
    • Authoring Forge Tools
    • Guided fluid forge UX
    • LLM Providers
    • LiteLLM Backend
    • Capability Warnings
    • Cost Tracking
    • FLUID Forge Contract GPT Packet
    • Agentic Primitives
  • Operate & Deploy

    • Airflow Integration
    • Blueprints
    • Source-Aligned Acquisition
  • Govern & Secure

    • Governance, Compliance & the Business Case
    • Governance & Compliance
    • Network Safety
    • Credential Resolver — Security Model
  • Configuration & Reference

    • Environment Variables
    • Typed Errors
    • Typed CLI Errors
    • API Stability — fluid_build.api
  • Architecture & Releases

    • V1.5 Catalog Integration — Architecture Deep-Dive
    • V1.5 + V2 Hardening — Release Notes
  • Project

    • Contributing to Fluid Forge
    • Fluid Forge Docs Baseline: CLI 0.9.0
    • Fluid Forge Docs Baseline: CLI 0.8.11
    • Fluid Forge Docs Baseline: CLI 0.8.10
    • Fluid Forge Docs Baseline: CLI 0.8.9
    • Fluid Forge Docs Baseline: CLI 0.8.8
    • Fluid Forge Docs Baseline: CLI 0.8.7
    • Fluid Forge Docs Baseline: CLI 0.8.6
    • Fluid Forge Docs Baseline: CLI 0.8.5
    • Fluid Forge Docs Baseline: CLI 0.8.4
    • Fluid Forge Docs Baseline: CLI 0.8.3
    • Fluid Forge Docs Baseline: CLI 0.8.0
    • Fluid Forge Docs Baseline: CLI 0.7.11
    • Fluid Forge Docs Baseline: CLI 0.7.9
    • Fluid Forge v0.7.1 - Multi-Provider Export Release

Consume a Data Product

You didn't build this product. You want to use it — and you want to use it safely, without spelunking through someone else's pipeline code to figure out what a column means, how fresh it is, or whether you're allowed to read it at all.

This page is the consumer front door. It shows you how to discover a published Fluid data product, read its contract to decide whether to trust it, and then consume it three ways — as a human with SQL or BI tools, as a downstream data product, or as an AI agent over MCP.

Why it matters The contract is how you trust a product before you query it. Schema, sensitivity, freshness, quality rules, lineage, and access rules all travel inside one versioned, validated artifact — so you can decide "is this safe to use?" by reading the contract, not by reverse-engineering the producer's code. And it's the same contract whether you're a person, a pipeline, or an agent: one source of truth, three consumers, the same declared rules.


Who this is for

You are…You want to…Jump to
An analyst / BI userRead the published data with SQL, a notebook, or a dashboardPattern 1 — Consume as a human
Building a downstream productPull this product into your own pipeline without re-typing its schemaPattern 2 — Consume as a downstream product
Wiring an AI agentLet an LLM query the product safely, under contract-declared rulesPattern 3 — Consume as an AI agent

Whichever row you're in, you start the same way: discover the product, then read its contract.


Step 1 — Discover: find a published product

Why it matters (analyst / consumer): Before you can use a product, you have to find it — and find out whether it's the right one. fluid market is your catalog search: it answers "what data products exist, who owns them, how good are they?" from the terminal, no portal login required.

Products are published to catalogs (stage 10 of the pipeline — see fluid publish). You discover them with fluid market, which searches and browses data products across your configured catalogs and blueprint sources.

# Browse everything
fluid market

# Narrow by what you care about
fluid market --search "customer analytics"
fluid market --domain finance --status active
fluid market --layer gold --min-quality 0.9
fluid market --owner growth-team

# Drill into one product
fluid market --product-id customer-360-v2 --detailed

Useful filters: --search / -s, --domain / -d, --owner / -o, --layer / -l, --status, --tags / -t, --min-quality, and --created-after / --created-before. Use --list-catalogs to see which catalog sources are configured, and --catalogs to restrict the search to specific ones.

Command Center enrichment is automatic and silent

If a FLUID Command Center instance is reachable, fluid market auto-detects it and enriches results with cross-organization catalog data — you don't configure anything for this to happen. If no endpoint is reached, it falls back to local catalog discovery transparently. This is detection-and-enrich, not a hosted UI you log into: the consumer surface here is the CLI.


Step 2 — Read the contract = read the trust story

Why it matters (analyst / consumer): "Can I trust this?" is the question that stalls most consumption. The contract answers it up front. Everything you need to evaluate a product — what's in it, how good it is, where it came from, and whether you're allowed to use it — is declared in one place and validated by the CLI, so it can't silently drift from the data.

Once fluid market points you at a product, open its contract. The contract is the single source of truth (why.md → the contract carries the context). Here's where each part of the trust story lives:

What you need to knowWhere it lives in the contract
What columns exist, their types, and what's sensitiveexposes[].contract.schema — each field has name, type, and an optional sensitivity (e.g. pii, phi, cleartext)
How good / fresh the data isexposes[].contract.dq.rules — freshness, completeness, uniqueness, valid_values, accuracy, schema, anomaly_detection, drift_detection — each with a severity (info / warn / error / critical)
The declared service-level targetsexposes[].qos — availability, freshness, latency, completeness, error budget
Where the data came from (lineage)Auto-derived from consumes[] (explicit upstream-product references), plus builds[].properties.sql and dbt repository references
Who may read itaccessPolicy.grants[] (humans / services, compiled to native cloud IAM) and exposes[].policy.agentPolicy (AI agents)

To inspect quality, SLA, and lineage in depth, see Quality, SLAs & Lineage.

Honest note on SLA monitoring

The qos targets you read in the contract are declared SLAs. Today they're published to catalogs (ODCS) and Data Mesh Manager so consumers can see the promise — but active monitoring against those thresholds is on the roadmap, not shipped. Treat qos as the producer's stated intent, not a live, alerting SLO yet.

The dq.rules, on the other hand, are enforced: they're checked by fluid validate, fluid test, and fluid verify, and an error / critical rule blocks the producer's deploy. So a published product has already passed its own declared quality gates.


Pattern 1 — Consume as a human (SQL / exports / BI)

Why it matters (analyst): You don't need to learn Forge to read a Forge product. Once you have access, the data lives in an ordinary table or view in the warehouse you already use. Forge got it there, governed; you query it with the SQL and BI tools you already know.

The contract's exposes[].binding tells you exactly where the data physically lives — the platform, and the table or view name. Once the producer grants you access via the top-level accessPolicy.grants (which compiles to native cloud IAM — BigQuery / Snowflake / S3), you read it with standard warehouse tooling:

-- BigQuery / Snowflake / Athena — whatever the binding names
SELECT segment, COUNT(*) AS customers
FROM analytics.customer_segments_v1
GROUP BY segment;

This consumption happens outside the Forge runtime. Forge declared the schema, applied the access grants from accessPolicy.grants, and deployed the table; from there you point Looker, Tableau, a notebook, or a SELECT straight at the bound object. The contract's schema is your data dictionary; its sensitivity tags tell you which columns are PII/PHI before you ever put them on a dashboard.


Pattern 2 — Consume as a downstream product (consumes[])

Why it matters (downstream data engineer): You shouldn't have to copy-paste an upstream product's schema into your pipeline and pray it stays in sync. Naming the upstream by product ID lets the planner resolve its bindings for you — when the upstream changes, your contract is checked against it at plan time, not at 3am in production.

A downstream product names its upstream by product ID in consumes[]. The planner resolves the binding, so your contract never re-types the upstream schema or transformation logic:

fluidVersion: "0.7.5"
kind: DataProduct
id: silver.orders_enriched
metadata:
  layer: Silver
  productType: ADP

consumes:
  - product: bronze.crm_orders      # named by ID — planner resolves the binding
    expose: orders
    alias: orders
  - product: bronze.crm_customers
    expose: customers
    alias: customers

builds:
  - name: enrich
    engine: sql
    sql: |
      SELECT o.order_id, o.amount_cents, c.region, c.segment
      FROM {{ orders }} o
      LEFT JOIN {{ customers }} c USING (customer_id)

Then drive it through the normal lifecycle:

fluid validate silver.orders_enriched.fluid.yaml
fluid plan     silver.orders_enriched.fluid.yaml
fluid apply    silver.orders_enriched.fluid.yaml --yes

fluid plan resolves each consumes[].product against the workspace registry, confirms the named expose exists, and validates the projected schema against your SQL {{ alias }} placeholders. A broken pointer never gets past plan.

Composition rules (enforced by the planner — see Product Types → Composition rules):

Product typeCan it consumes[]?
SDP (Bronze / source-aligned)No — SDPs are leaves; declaring consumes[] fails
ADP (Silver / aggregate)Yes — may consume SDP(s)
CDP (Gold / consumer-aligned)Yes — may consume SDP and/or ADP

For the full walkthrough, see the recipe: Write a contract that consumes another contract.


Pattern 3 — Consume as an AI agent (MCP output-port)

Why it matters (platform engineer onboarding an agent): Pointing an LLM at your warehouse is how PII leaks and runaway queries happen. The MCP output-port gives the agent a deliberately small, read-only surface — and enforces the contract's governance (allowed models, PII redaction) on every single call, with no extra glue code.

You serve a published product to an agent with fluid mcp output-port serve — the consumer-side data-access gate.

Use the right command

fluid mcp output-port serve is the consumer gate that serves data products to agents. fluid mcp serve is the producer / authoring server for data engineers — it does not serve published data to agents. They are different commands.

fluid mcp output-port serve ./contract.fluid.yaml

The agent gets exactly four bounded tools (full reference: fluid mcp):

ToolWhat it does
describeReturns schema + semantics + the agentPolicy block — no engine round-trip; how the agent orients itself
sampleReturns a few rows, with PII/PHI columns redacted
queryRuns a predeclared semantic aggregate (a metric/measure from semantics) — no raw SQL
query_sqlFree-form SELECT — OFF by default, advertised only with --allow-sql

The server is read-only, and free-form SQL stays off unless you explicitly opt in with --allow-sql.

Two contract-driven guarantees the gateway enforces on every call:

  1. agentPolicy gate. The allowlist lives per-expose at exposes[].policy.agentPolicy (a top-level agentPolicy fails validation). A caller whose model_id isn't in allowedModels is refused with a typed envelope:

    {
      "error": "AgentPolicyDenied",
      "tool": "sample",
      "reason": "not-in-allowedModels",
      "message": "denied by agentPolicy; see audit trail for the full decision."
    }
    
  2. Automatic PII/PHI redaction. A column marked sensitivity: pii keeps its name but its values come back as [REDACTED-PII] on every sample / query / query_sql result. The mask is alias-proof: even with --allow-sql, SELECT email AS x is rejected at compile time. The agent learns the field exists (so it can still write COUNT(DISTINCT email)) but never sees a real value.

For the hands-on version — serve a DuckDB-backed product, drive the tools, and watch a deny and a redaction happen — follow Walkthrough: MCP Output Port. For the policy concept in depth, see agentPolicy.

One nuance on token caps

agentPolicy.maxTokensPerRequest is a post-hoc throttle, not a pre-flight limit: the read executes, the response is measured, and it's withheld with TokenBudgetExceeded if it's over the cap. It bounds what the agent receives; it doesn't pre-block the query.


One contract, every consumer

The schema, sensitivity tags, qos, lineage, and dq.rules that earn human trust are the same declarations a downstream pipeline and an AI agent read to act safely — one artifact, no second governance system to keep in sync.

  • A person reads sensitivity: pii and keeps it off a dashboard; the agent gateway reads the same tag and redacts the value to [REDACTED-PII].
  • A person reads the top-level accessPolicy.grants to know they're allowed in; the agent is gated by agentPolicy.allowedModels on the same contract.
  • A downstream product doesn't re-read any of it by hand: its planner resolves the very exposes[].binding and schema the human and agent rely on, straight from consumes[] — so when the upstream changes, every consumer sees the change through the same source of truth.

One versioned artifact, three kinds of consumer, the same declared rules.


Next steps / See also

  • fluid market — discovery options, catalog filters, and Command Center enrichment.
  • Why Forge — the contract carries the context — what travels inside a contract and why it earns trust.
  • Quality, SLAs & Lineage — dq.rules, qos, and auto-derived lineage (with the SLA-monitoring roadmap note).
  • Consume one contract from another — the full Pattern 2 recipe.
  • Product Types → Composition rules — what can consume what.
  • Walkthrough: MCP Output Port — Pattern 3, hands-on.
  • fluid mcp reference — the four agent tools, every flag.
  • agentPolicy — the agent-access policy concept.
  • Governance, Compliance & ROI — the buyer / governance companion page, including the honest provider-enforcement matrix.
Edit this page on GitHub
Last Updated: 6/26/26, 10:55 AM
Contributors: fas89
Next
Product Types — SDP, ADP, CDP