Change a live data product

Time: 15 min · Skill: you have applied a contract before

A product is deployed and has consumers. This recipe adds a column to it the way a pipeline should: version the change, check it, make sure nobody changed the target by hand, apply, verify, and keep the record the next change needs. Every command below was run against the local provider with CLI 0.18.1.

Prerequisites: the build reads a CSV with DuckDB, which a plain pip install data-product-forge does not include. Install the local extra: pipx install "data-product-forge[local]". Without it, fluid apply stops with duckdb not installed. See the DuckDB sandbox.

Starting point

A product at version 1.0.0, bound to a local CSV:

orders/
├── contract.fluid.yaml
└── data/orders.csv

# orders/contract.fluid.yaml
fluidVersion: 0.7.5
kind: DataProduct
id: sales.orders_summary
name: Orders summary
domain: sales
metadata:
  layer: Silver
  owner:
    team: sales-analytics
    email: sales-analytics@example.com
builds:
  - id: summarise
    pattern: embedded-logic
    engine: sql
    properties:
      sql: >
        SELECT order_id, customer_id, amount
        FROM read_csv('data/orders.csv')
        WHERE status = 'completed'
exposes:
  - exposeId: orders_summary
    version: 1.0.0
    kind: table
    binding:
      platform: local
      format: csv
      location:
        path: out/orders_summary.csv
    contract:
      schemaPolicy: strict
      schema:
        - { name: order_id, type: INTEGER, required: true }
        - { name: customer_id, type: VARCHAR }
        - { name: amount, type: DECIMAL }

schemaPolicy: strict makes any column difference in the target count as drift. Without it, fluid diff judges the target by evolve_safe, the policy the build runs under when the contract names none, and reports a new or missing column as evolved rather than drift. (The schema's description calls strict the default; as of 0.18.1 the engine uses evolve_safe.)

Deploy through a saved plan, and keep it

Apply the plan file rather than the contract, and keep that plan as the record of what was last applied:

$ fluid plan contract.fluid.yaml --out runtime/plan.json
...
$ fluid apply runtime/plan.json --yes
...
✅ Data product deployed successfully
...
$ cp runtime/plan.json runtime/last-applied.plan.json

In CI, archive last-applied.plan.json with the build and restore it at the start of the next one. Jenkins pipelines generated by fluid generate ci do this when generated with --diff-last-applied (it needs the copyartifact plugin).

Record the schema consumers rely on

$ fluid contract-tests contract.fluid.yaml --write-baseline contract-baseline.json
✅ Baseline written to contract-baseline.json

Commit contract.fluid.yaml and contract-baseline.json together.

1. Make the change and bump the version

Add status to the query and the schema, and bump the expose's version. Adding a nullable column is a minor change (1.0.0 to 1.1.0); removing, renaming or retyping a column is a major one (2.0.0).

     properties:
       sql: >
-        SELECT order_id, customer_id, amount
+        SELECT order_id, customer_id, amount, status
         FROM read_csv('data/orders.csv')
         WHERE status = 'completed'
 exposes:
   - exposeId: orders_summary
-    version: 1.0.0
+    version: 1.1.0
...
         - { name: amount, type: DECIMAL }
+        - { name: status, type: VARCHAR }

2. Check the change against the baseline

$ fluid contract-tests contract.fluid.yaml --baseline contract-baseline.json
❌ Contract tests failed — 1 incompatibility(ies) found
   • orders_summary.status: column added
$ echo $?
2

fluid contract-tests fails on any column change, additive ones included: an added column, a removed column and a changed type are each listed. It does not decide whether a change is safe; it makes sure a person sees it. A removal and a type change read like this:

$ fluid contract-tests contract.fluid.yaml --baseline contract-baseline.json
❌ Contract tests failed — 2 incompatibility(ies) found
   • orders_summary.order_id: type changed INTEGER -> VARCHAR
   • orders_summary.amount: column removed

Once the change is reviewed, write the new baseline and commit it with the contract:

$ fluid contract-tests contract.fluid.yaml --write-baseline contract-baseline.json
✅ Baseline written to contract-baseline.json

Before 0.16.5, fluid contract-tests reported every contract as compatible. A pipeline that relied on it on an older CLI was not gated.

fluid diff --baseline does not see column changes (0.18.1)

fluid diff --baseline <old> --fail-on-breaking is meant to classify changes as breaking or not. As of 0.18.1 it reads columns from exposes[].schema, while a valid 0.7.x contract declares them under exposes[].contract.schema, so it compares no columns at all. Removing amount and retyping order_id between two valid contracts gives:

$ fluid diff contract.fluid.yaml --baseline previous.fluid.yaml --fail-on-breaking
No changes detected.
Summary: 0 breaking, 0 non-breaking, 0 info
$ echo $?
0

Use fluid contract-tests as the column gate until this is fixed. fluid plan does not compare a contract with its previous version either.

3. Gate on drift before you plan

fluid diff --exit-on-drift reads the live target and compares it with the contract. Run it before planning, so nothing is planned against a target someone changed by hand. Without the last-applied plan, it cannot tell your pending change from a hand edit:

$ fluid diff contract.fluid.yaml --exit-on-drift
...
Live drift check: 1 expose(s)
  orders_summary [local] drift  /work/orders/out/orders_summary.csv
      - status: declared (VARCHAR) but not in the target
      '.csv' files carry no column types; column names compared only
State drift check: not run (the contract runs on the local engine, which keeps no OpenTofu state). Drift comes from the live checks alone.
$ echo $?
1

With it, the comparison is three-way: a column the contract changed since the last apply, where the target still matches that apply, is pending:

$ fluid diff contract.fluid.yaml --exit-on-drift --last-applied runtime/last-applied.plan.json
...
  orders_summary [local] pending (apply will change it)  /work/orders/out/orders_summary.csv
      - status: declared (VARCHAR) but not in the target (pending: the contract changed it since the last apply)
...
$ echo $?
0

--exit-on-drift exits 1 on drift and 2 when a target could not be inspected. A --last-applied path that does not exist is ignored, so the first run of a new pipeline works without one.

4. Apply, verify, and move the record forward

$ fluid plan contract.fluid.yaml --out runtime/plan.json
$ fluid apply runtime/plan.json --yes
...
✅ Data product deployed successfully
$ cp runtime/plan.json runtime/last-applied.plan.json

fluid verify checks the deployed target against the contract:

$ fluid verify contract.fluid.yaml --strict
...
📋 Verifying: orders_summary
...
   📊 Rows: 2

   🔍 Dimension 1: Schema Structure
      ✅ PASS - All 4 declared columns present
...
✅ Match: 1
⚠️  Mismatch: 0
❌ Error: 0

Use --strict on the verify step of a pipeline. Without it, fluid verify exits 0 when it finds mismatches: it prints the Mismatch count and the job goes on. With it, a CRITICAL mismatch exits non-zero. Non-critical drift (a required column that went nullable, for example) is still only a warning; --fail-on-warning makes that fail too. See fluid verify.

The drift gate now reports a match:

$ fluid diff contract.fluid.yaml --exit-on-drift --last-applied runtime/last-applied.plan.json
...
  orders_summary [local] match  /work/orders/out/orders_summary.csv

Publish the new version to your catalog with fluid publish; see The CLI and the Command Center for what a Command Center records.

When the target changes outside the contract

If someone adds a column to the target by hand, the next gate fails, with or without --last-applied:

$ fluid diff contract.fluid.yaml --exit-on-drift --last-applied runtime/last-applied.plan.json
...
  orders_summary [local] drift  /work/orders/out/orders_summary.csv
      - note: in the target (VARCHAR) but not declared
...
$ echo $?
1

Either declare the column in the contract (and go through steps 1 to 4), or re-apply the contract to put the target back.

Breaking changes, `replace`, and rollback

Removing or retyping a column usually means rebuilding the target. --mode replace drops and recreates it. It refuses to run without --allow-data-loss unless the environment is dev and the target is known to be empty; as of 0.18.1 the target's row count is not measured, so in practice it refuses without the flag in every environment:

$ fluid apply contract.fluid.yaml --mode replace --yes
❌ apply_mode_data_loss_blocked  [ERR_APPLY_MODE_DATA_LOSS_BLOCKED]
  mode: replace
  env: None
  reason: --mode replace is destructive (env not set; target row count unknown (treating as
populated)). Pass --allow-data-loss to confirm the drop. The pre-replace table will be snapshotted
to <target>__backup_<ts> so `fluid rollback` can restore it.

Take your own backup before a replace (0.18.1)

fluid rollback restores from snapshots recorded in .fluid/rollback-state.json. As of 0.18.1, fluid apply writes none on its default engines:

On aws, gcp, snowflake and confluent, which apply through OpenTofu, the gate says so: NO SNAPSHOT WILL BE TAKEN.
On local, the gate says the table will be snapshotted (above), but after --allow-data-loss the apply records no snapshot, and fluid rollback finds nothing:

$ fluid apply contract.fluid.yaml --mode replace --allow-data-loss --yes
...
✅ Data product deployed successfully
$ fluid rollback --list
[rollback] no state file at /work/orders/.fluid/rollback-state.json. The file is
created on the first ``apply --mode replace`` / ``replace-and-build`` invocation.

Back the target up with the warehouse's own tools (a table clone or copy) before --allow-data-loss, and keep the previous contract in version control so you can re-apply it.

Housekeeping

fluid retention sweep deletes files under the state directory (./.fluid, or --state-root) by age. As of 0.18.1 it does not read any contract; it applies fixed horizons to each subdirectory, by file modification time:

Directory	Deleted after
`runs/`	30 days
`logs/`	90 days
`dlq/`	180 days
`lineage/`	365 days

With nothing past its horizon it reports zero:

$ fluid retention sweep
deleted_paths: []
bytes_freed: 0
by_category:
  run_state: 0
  run_logs: 0
  lineage: 0
  dlq: 0

The sequence, for a pipeline

fluid contract-tests contract.fluid.yaml --baseline contract-baseline.json
fluid diff contract.fluid.yaml --exit-on-drift --last-applied runtime/last-applied.plan.json
fluid plan contract.fluid.yaml --out runtime/plan.json
fluid apply runtime/plan.json --yes
fluid verify contract.fluid.yaml --strict
cp runtime/plan.json runtime/last-applied.plan.json

When the contract uses per-environment overlays, add --env <name> to each fluid command.

fluid diff: drift mode and version-diff mode.
fluid contract-tests
fluid apply: modes and safety gates.
What is a contract?: versioning.