fluid verify

Stage 9 of the 11-stage pipeline. fluid verify reads what the build produced and compares it with the contract: a local file, a BigQuery table, an S3 prefix behind a Glue table, or a Snowflake table. Which checks run depends on that target.

Syntax

fluid verify CONTRACT

CONTRACT is a contract YAML or a stage-1 bundle (.tgz). A relative binding.location.path resolves against the directory of the source contract, not the working directory, so the file the build wrote is the file verify reads wherever you run it from. A bundle records its source contract's directory.

Examples

fluid verify contract.fluid.yaml
fluid verify contract.fluid.yaml --strict
fluid verify contract.fluid.yaml --expose orders
fluid verify contract.fluid.yaml --show-diffs
fluid verify runtime/bundle.tgz --env aws --strict --out runtime/verify-report.json
fluid verify contract.fluid.yaml --env aws --strict --state-drift
fluid verify contract.fluid.yaml --reconcile-dbt --strict          # contract <-> dbt schema drift
fluid verify contract.fluid.yaml --reconcile-lineage --strict      # contract <-> observed/published lineage
fluid verify contract.fluid.yaml --reconcile-dbt --reconcile-lineage --warn-only   # report, never fail

What a failure looks like

A local CSV whose file has a column the contract does not declare:

fluid verify contract.fluid.yaml --strict
echo $?

📋 Verifying: orders
   Format: csv
   Target: /.../output/orders.csv

   🔴 Severity: CRITICAL (Impact: HIGH)
   ...
   📊 Rows: 1

   🔍 Dimension 1: Schema Structure
      ❌ FAIL - Schema structure mismatch
         ✅ Matching: 3/3
         ⚠️  Not declared in the contract: note

   ⚪ Data types, constraints, location: not checked for local files (column names, row count and
masked-value shapes only)
...
1

Without --strict the same run reports the drift and exits 0. Remove the extra column, or declare it in the contract, and the run exits 0 with Match: 1.

Key options

For the Athena options the flag wins over the environment variable.

What gets checked

fluid verify picks a verifier from each expose's binding. Each expose gets one status in the summary and the JSON report: a match, a mismatch, an error (the target could not be checked), or unsupported (no verifier for it).

What verify checks per platform

Dataset and table access grants (accessPolicy.grants) are not checked on any platform as of 0.18.1. Governance parity sets what each cloud emits beside what verify reads back.

An unsupported expose prints Skipped, is counted separately in the summary and never fails the run. It means fluid verify did not check it, not that the check passed:

📋 Verifying: events
   Format: pubsub_topic
...
   ⏭️  Skipped: No verifier for this GCP binding (format: pubsub_topic); `fluid apply` provisions
it, stage 9 does not reconcile it

This table lists the dimensions each verifier reports. A blank cell means the dimension is not checked for that target.

retention, encryption and columnRestrictions run only when the expose declares the matching field. Each one reads the live platform, so a check that cannot run is an error: it fails fluid verify with or without --strict.

Severity

Each mismatch is scored:

--strict fails on CRITICAL. --fail-on-warning also fails on WARNING and INFO. The JSON report carries the severity under results.<expose>.severity.level, so CI dashboards can key off it for red, amber and green.

Exit codes

A contract file that cannot be read, or an unknown --expose, exits 1 before any target is checked.

Local files

A local CSV or Parquet output has two checked dimensions: schema_structure and, when the expose declares policy.privacy.masking, masking. The row count is reported, not compared. The file is read with DuckDB, which needs the local extra.

• Any structure mismatch is CRITICAL, extra columns included. The file is the product, so a column the contract does not declare is a broken output. That includes the one-column placeholder that an amend apply lands: --strict fails on it.
• {{ env.NAME }} in a local path resolves as the build writer resolves it. A variable that is not set resolves to the empty string, on both sides.
• Verify reads the output file under the DuckDB sandbox's read rules. See the DuckDB sandbox.

BigQuery

The retention, encryption and column-restriction checks are explained, with the resources they read back, in Governance parity.

A BigQuery table is checked for the warehouse dimensions: structure, types, constraints and the dataset's region. {{ env.X }} in the binding resolves as fluid apply resolves it. A binding with no project uses the project from GOOGLE_PROJECT, GOOGLE_CLOUD_PROJECT, GCLOUD_PROJECT or CLOUDSDK_CORE_PROJECT, then the one Application Default Credentials carry. With none of them, the expose is an error that names the table.

Column types are compared after BigQuery's legacy names are folded into the standard ones (INTEGER and INT64, FLOAT and FLOAT64, BOOLEAN and BOOL, RECORD and STRUCT), so a correct BOOL column is not drift.

One GoogleSQL query then adds two dimensions:

The query needs bigquery.jobs.create on the project (roles/bigquery.jobUser) as well as read access to the table. A query that cannot run is an error. With BIGQUERY_EMULATOR_HOST set, the client talks to that host with anonymous credentials.

When the expose declares them, three governance dimensions read the live dataset and table:

A mismatch is CRITICAL. retention and encryption come from lifecycle.expire and binding.encryption.kms, which are in contract schema 0.7.6, a preview schema selected with fluidVersion: "0.7.6". columnRestrictions is in 0.7.5. The columnRestrictions check calls the Data Catalog API with Application Default Credentials and reads the taxonomy (Data Catalog GET) and each policy tag's IAM policy (getIamPolicy), so the credentials need the matching Data Catalog read permissions. No emulator serves Data Catalog, so under BIGQUERY_EMULATOR_HOST the readers of the tagged columns are not checked and the dimension reports unsupported.

On 4 October 2026, on 0.18.0, the FLUID team ran fluid verify against real Google Cloud for 11 products deployed from the same base contracts through a gcp overlay. Each passed its retention (DAY partitions with expiration_ms), encryption (a Cloud KMS key per dataset) and column-restriction (Data Catalog policy tags) checks. That is one run on one estate. In the same run, a principal outside the allowed readers who selected a restricted column was refused by BigQuery in this form:

User has neither fine-grained reader nor masked get permission to get data protected by policy tag "<taxonomy> : <tag>" on column <project>.<dataset>.<table>.<column>.

S3 and Glue (Athena)

A platform: aws binding with a bucket, path, database and table is provisioned by fluid apply as an S3 prefix plus a Glue table. Verify reads it in the binding's region. The retention, encryption and Lake Formation rows below are described with their emitted resources in Governance parity. The checks:

Glue declares no nullability, so the constraints dimension has nothing to compare. The Lake Formation check lists the table permissions the caller can see, so it must run as a Lake Formation administrator. When the contract's own grants are not in the listing, it reports an error rather than a pass.

The result location, first match wins:

1. A workgroup with managed query results, or one that enforces its own output location: the workgroup's.
2. --athena-output-location or FLUID_ATHENA_OUTPUT_LOCATION.
3. The workgroup's configured output location.
4. s3://<binding bucket>/.fluid/athena-results/.

A result location inside the table's own prefix is refused or replaced, so that Athena's result files never land among the table's data. The JSON report records the location used and why under results.<expose>.athena.output_location_source.

The identity running fluid verify needs:

• Glue: glue:GetTable and glue:GetDatabase, plus glue:GetPartitions for a partitioned table.
• Athena: athena:StartQueryExecution, GetQueryExecution, GetQueryResults and StopQueryExecution on the workgroup, and athena:GetWorkGroup to read the workgroup's configuration.
• S3: s3:GetObject on the data prefix, read and write on the results prefix, and s3:ListBucket, s3:GetBucketLocation and s3:ListBucketMultipartUploads on the bucket.
• s3:GetLifecycleConfiguration for a declared retention; kms:DescribeKey for a declared key, and kms:GenerateDataKey and kms:Decrypt when the result is encrypted with it.
• For a prefix registered with Lake Formation, lakeformation:GetDataAccess and a Lake Formation SELECT grant on the table.

boto3 comes from the aws extra. Without it, the check is an error that names the extra.

Row counts and run records

The count is compared with the run records the build wrote under .fluid/runs/<contract id>/<build id>/runs/ next to the contract. A CI pipeline that runs verify in a separate stage has to carry that directory over, or no comparison happens. A run counts as this table's when its record says it landed rows in it: facets.landed for an S3 prefix, facets.bigquery_load for a BigQuery table. A run that landed somewhere else, such as a local run from the same contract directory, is passed over. The newest run that is this table's sets the rule, by the mode it recorded:

The count is reported, and only an empty table fails, in cases such as: there is no run record, no run that landed in this table, a failed newest run, more than one build writes the expose (there is no single run to compare), or the newest run did not record its row count from the write. S3 also needs a build with pattern: acquisition; BigQuery also reads the run records of embedded-SQL builds.

Snowflake

A Snowflake table is checked for structure, types and constraints. The count of rows is reported in the metadata and is not compared with the build.

Reference-only contracts

For contracts where builds[].pattern is hybrid-reference, reference, or external-reference, the target tables are materialised by an externally-owned dbt or Airflow project, not by fluid apply. On the first pipeline run the external DAG has not run yet, so a missing table is expected state, not a verification failure.

fluid verify detects this mode from builds[].pattern and downgrades a missing target to INFO:

(contract is reference-only — missing tables will be reported as INFO,
not treated as verification failures)

📋 Verifying: subscriber360_core
   Format: snowflake_table
   Target: TELCO_LAB.TELCO_FLUID_DEMO.SUBSCRIBER360_CORE_V1
   🔵 INFO: Table not found: ... (reference-only — external pipeline owns creation)

The downgrade is narrow:

• Only a target that does not exist (result["exists"] is False) is downgraded, along with an empty table that has no run of its own to compare with.
• Authentication, configuration and connection errors still fail the run, whether or not --strict is set.
• Contracts with another pattern (imperative, declarative) keep the original behaviour: a missing table is an error.

Reconcile: contract and dbt schema (--reconcile-dbt)

A static, warehouse-free cross-check that the contract's promised columns (exposes[].contract.schema) agree with what the build's dbt project declares (models/**/schema.yml). It never connects to a warehouse and never runs dbt. It surfaces:

• a column the contract promises that dbt doesn't model,
• a dbt column the contract never exposes,
• a declared type that disagrees (compared conservatively through coarse type families, so NUMBER(38,0) vs integer is not flagged across adapters).

Drift exits non-zero unless --warn-only is set, independent of --strict: you asked for the check, so its drift gates on its own. The JSON report (--out) carries the detail under report["reconcile"].

Reconcile: contract and published lineage (--reconcile-lineage)

A local-only cross-check (no network) that the contract's declared lineage, consumes[] upstream refs and exposes[] output ports, agrees with:

1. Observed run evidence: the run records and cursor state the build runners persist under .fluid/ (which source streams were actually read).
2. The publish payload: the lineage edges the catalog registrar would push, rebuilt locally from the contract.

Critical drift fails the run under --strict unless --warn-only is set; soft drift never fails. dbt run records are handled separately: their streams are executed nodes, not upstream reads, so they are excluded from the undeclared-read check with a note. The JSON report carries the detail under report["reconcile_lineage"].

Apply state drift (--state-drift)

--state-drift runs the pass fluid diff runs over the state fluid apply keeps (which key, which backend): it refreshes the OpenTofu state and reports resources changed outside the apply. Run it from the directory fluid apply ran in, or pass --workspace-dir; for remote state, name the backend with --state-backend or FLUID_STATE_BACKEND. With no state in reach, the section prints a note and the run's drift comes from the live checks alone:

================================================================================
🧭 Apply State Drift
================================================================================
State drift check: not run (no apply state for this contract at .fluid/iac/gcp/gold_events_v1: run from the directory `fluid apply` ran in (or pass --workspace-dir), or name the remote state with --state-backend / FLUID_STATE_BACKEND). Drift comes from the live checks alone.

dbt test results (transformation checks)

When the product's build ran via dbt (fluid apply --mode amend-and-build), the runner parses target/run_results.json into run records, and fluid verify adds transformation-side checks next to the acquisition probes: dbt_tests_passed and no_error_severity_failures. Both are critical, so a failing contract-derived dbt test gates the exit code under --strict. See fluid runs status for the run-record view of the same data.

Notes

• Use verify after apply, or in CI, when you need contract-to-runtime confidence.
• Use fluid test when you want broader live-resource validation.
• The JSON report contains per-expose severity, the dimensional breakdown and remediation actions.