Fluid Forge Docs Baseline: CLI 0.17.0

Release dates: 0.16.3 on September 26, 0.16.4 on September 27, and 0.16.5, 0.16.6, 0.16.7 and 0.17.0 on September 28, 2026. Status: Superseded by 0.18.0, whose page also covers the current release, 0.18.1. Supersedes 0.16.0.

This baseline covers six CLI releases, documented together because none had a docs pass of its own. To move a project from an older release, use the upgrade guide.

Headline

One contract now deploys to AWS and to Google Cloud through its --env overlays, and is governed the same on both.

• Each policy is one contract field, enforced natively on each cloud. Retention, encryption at rest with a per-product key, column restrictions, masking and grants reach AWS (0.16.5–0.16.7) and Google Cloud (0.17.0), instead of being declared and dropped. fluid verify checks retention, encryption and column restrictions, and checks masking on Glue, Athena and BigQuery tables. On Google Cloud, fluid verify does not read dataset-level grants back; on AWS it checks the Lake Formation column grants of an expose that declares column restrictions.
• The two clouds keep two states. A product applied with --env aws and --env gcp used to share one OpenTofu state, so each cloud's plan read the other's resources as orphans to destroy.
• The generated 11-stage pipeline runs as generated (0.16.3), and its gates check what they claim: a tampered plan, a mismatched mode or env, and a live target that drifted all fail where they used to pass.
• A chained product reads what its consumes[] names (0.16.5), on DuckDB, from S3 or from BigQuery (0.17.0).

What has been run live

The forge-cli changelog for 0.17.0 measured the GCP work against moto and a BigQuery emulator, and left the real-cloud run open. On 4 October 2026, on 0.18.0, a demo lab applied two lineage chains of eleven products to real Google Cloud with fluid apply --env gcp, from generated Jenkins pipelines, as a deploy service account reached by Workload Identity Federation. In that run:

• each apply created its dataset, key ring and key, its table with daily partitions that expire after the retention, its dataset IAM members, and a policy tag on each restricted column;
• each build passed fluid verify against the live tables, including retention, encryption (Cloud KMS) and column restrictions (Data Catalog policy tags);
• silver and gold products built on GCP by reading their upstreams from BigQuery;
• querying as each persona, an analyst was refused each restricted column, a steward could read it, and a principal with no dataset grant was refused the table. BigQuery's refusal reads: User has neither fine-grained reader nor masked get permission to get data protected by policy tag "<taxonomy> : <tag>" on column <project>.<dataset>.<table>.<column>.

The same run found the drift-gate failure on 0.18.0 that 0.18.1 fixes.

Not proven: the Lake Formation half against a real AWS account as 0.17.0 derives it from columnRestrictions. The grant shape it emits (excluded columns beside wildcard) was applied and enforced on a real account from 0.16.6, written by hand in the overlay. See Governance parity.

Upgrade checklist

From 0.16.2. Steps 1–7 come from 0.16.3, 8–11 from 0.16.5, and 12–16 from 0.17.0. The upgrade guide has a check for each.

1. Relative local outputs land under the contract's directory, not the directory you ran the command from. Move anything that reads the old location.
2. fluid verify --strict fails a local output whose columns differ from the declared schema. It used to downgrade that to a warning.
3. fluid apply --build-id with a mode that runs no build is refused, and so is applying a plan.json in a different mode from the one it was planned in.
4. The drift gate can fire. fluid diff --exit-on-drift compares the live target: local files, DuckDB, Glue and BigQuery tables. Pass --last-applied to tell a contract change from drift. A target not created yet is not drift.
5. Command Center products are private unless the contract's classification is public.
6. Lake Formation bucket policies no longer name grantees in the same account. Those grantees could read the data straight from S3, around Lake Formation. To restore the old output, set binding.governance.lakeFormation.bucketPolicy: all-grantees (fluid-schema 0.7.6).
7. Regenerate committed pipelines with fluid generate ci. A generated Jenkinsfile installs the data-product-forge version that generated it, so a pipeline generated before the upgrade keeps running the old CLI.
8. A consumes[] entry that the build's SQL reads must resolve on the DuckDB embedded-SQL path: from the nearest fluid.workspace.yaml, a FLUID_UPSTREAM_CONTRACTS root, or an explicit input of the same name. It used to be skipped with a warning; it now fails with ConsumesResolutionError. Entries the SQL does not read stay lineage only.
9. fluid apply plan.json --env <other> fails with plan_env_mismatch. fluid plan --env X records the env in the plan, under its digest.
10. With AWS_ENDPOINT_URL or AWS_ENDPOINT_URL_S3 set, a DuckDB S3 secret that fails stops the run (ObjectStoreEndpointError), instead of sending reads and writes to AWS.
11. Masked columns land treated. The DuckDB runner now applies exposes[].policy.privacy.masking[]. hash needs a salt of at least 16 bytes in FLUID_PII_HASH_SECRET (or the variable params.saltEnv names), tokenize a key of at least 32 bytes, and encrypt a base64 key; an unset secret refuses the build. k_anonymity is refused. A treated column lands as a string, so declare it as one.
12. Run fluid schedule-sync once to retire each env's old DAGs. DAGs move from <product-id>/ with dag id <product>__<build> to <product-id>__<env>/ with dag id <product>__<env>__<build>. Left in place, the old DAG runs beside the new one, and both apply the same product against the same state.
13. Remote state moves to a per-provider key on the first apply: fluid/<id>/<provider>/terraform.tfstate. The apply copies the state with OpenTofu's own migration and leaves the old object in place. A key shared with another cloud's resources is refused (state_shared_with_another_provider).
14. GCP dataset grants stop being authoritative. They become google_bigquery_dataset_iam_member resources. The first apply revokes, once, the old access entries that no grant of the contract covers, and prints them.
15. --env <name> with no overlay is refused when the workspace's expected-environments lists that env for the product. It used to run the base contract.
16. A GCP binding with no region is refused, instead of landing in US.

What changed in 0.17.0

Governance parity on Google Cloud

Each field in this table is enforced natively on both clouds and checked by fluid verify:

lifecycle.expire, binding.encryption and binding.principals are in fluid-schema 0.7.6 only, so a contract needs fluidVersion: "0.7.6" to use them. Governance parity has the full table, including what fluid verify checks on each cloud.

binding.principals. An overlay maps the contract's logical principals to the identities they are on that cloud: one, a list, or []. With the block present, an unmapped principal is refused. On Google Cloud a placeholder principal (a reserved domain, or not an IAM member) is refused either way, because BigQuery refuses an access entry for an identity that does not exist.

Embedded-SQL builds read and land BigQuery

A consumes[] entry whose upstream is a gcp bigquery_table resolves to that table, and a build whose expose is one loads its result with a BigQuery load job (WRITE_TRUNCATE, CREATE_NEVER), failing on a short load. Naive timestamps load as UTC TIMESTAMP. fluid verify checks a BigQuery table's row_count against the build's run records, and masked columns, as it already did on Glue and Athena.

One contract on two clouds

• Per-provider state. See step 13. When the gcp apply finds the aws state at the old key, it leaves it alone. On the 4 October 2026 run, that first gcp apply printed a line of this form, where <old-state-location> is s3://<bucket>/<key> or gcs://<bucket>/<prefix>:

  state move:  <old-state-location> holds the aws provider's state (hashicorp/aws), not this provider's; left in place
  

  and wrote the gcp state to fluid/<id>/gcp/terraform.tfstate.

• DAGs per env. The env is in the dag id and the schedule scope, so the aws and gcp DAGs of one product no longer collide. See step 12.

• Overlays are enforced where the workspace expects them. See step 15.

• Sovereignty fails closed on GCP. A binding with no region is refused, fluid generate iac checks the region, location.location is checked like location.region, and stage 6 of a generated pipeline runs fluid plan --check-sovereignty.

fluid apply reports each run to the Command Center

When the Command Center publish configuration is present (FLUID_CC_ENDPOINT, an API key or bearer token, and an organization from FLUID_CC_ORG_ID or fluid.config.yaml), each fluid apply records a run: the product id, contract version and hash, env, provider, mode, state location, change counts, resource addresses and, for a build mode, each build's result. No credential, environment value or OpenTofu output is sent. Reporting is best effort: a Command Center that is down costs a warning line, never the apply's exit code. Set FLUID_COMMAND_CENTER_ENABLED=false to turn it off.

fluid publish upserts one Command Center product per contract id, so publishing from --env aws and from --env gcp leaves the platform and location of whichever ran last. A generated Jenkinsfile has its publish stage off by default. Pass --publish-stage-default to turn it on for the env that should publish. Only the Jenkins template reads this flag.

Fixed in 0.17.0

• Stage 8 of a gcp pipeline no longer crashes: fluid policy-apply raised a TypeError from its own log call.
• A BigQuery load runs in the table's location, never at a guessed US.

What changed in 0.16.5–0.16.7

The policies an AWS contract declares reach AWS.

• Retention and encryption at rest reach S3 (0.16.5). lifecycle.expire: true turns lifecycle.retention into an S3 lifecycle rule; binding.encryption.kms (product, none, alias/<name> or a key ARN) gives SSE-KMS with a bucket key. Both are written only on a bucket the product owns; a bucket that already has rules or SSE is adopted. fluid verify reads the live rules and key and fails on a difference.
• Masking is applied when the data lands (0.16.5). See step 11. The local file, the S3 object, the BigQuery staging file and the dead-letter queue all hold treated values, and fluid verify fails a masked column that landed in cleartext.
• Chained products (0.16.5). Each consumes[] entry the SQL reads becomes a view named by its exposeId, over the upstream contract's binding under the same --env overlay. AWS_ENDPOINT_URL_S3 / AWS_ENDPOINT_URL reach DuckDB, so a chain runs against MinIO or moto.
• fluid diff reads the apply's OpenTofu state (0.16.5). It runs tofu plan -detailed-exitcode against the apply's own state, so a setting changed by hand or a table deleted by hand fails --exit-on-drift. New flags: --state-backend, --workspace-dir, --no-state-drift, --ensure-opentofu.
• fluid generate ci --fluid-env-default <env> (0.16.5) sets the env a build on the defaults runs in, instead of dev.
• fluid contract-tests runs its comparison (0.16.5). It returned "compatible" for every contract.
• Lake Formation grants that hide columns plan and apply (0.16.6, 0.16.7). A grant with excludedColumns sets wildcard = true, and a column-limited grant carries SELECT only, which is what Lake Formation accepts. A column limit beside ALTER, DROP, INSERT, DELETE or ALL is refused when the module is emitted, and so are excluded columns the schema does not have.

What changed in 0.16.3 and 0.16.4

The generated 11-stage pipeline runs as generated.

• The generated Jenkinsfile runs end to end (0.16.3). CONTRACT defaults to the contract you generated from. Stage 0 installs into a workspace venv and pins data-product-forge to the generating version with the extras the contract needs (--fluid-package-spec overrides it). Every parameter's default is also its shell fallback, so a first build, or a build after a Jenkins restart, is a correct build. Stage 1 bundles with --env, and later stages read the bundle.
• The structural gates check what they claim (0.16.3). Plan digest and mode are checked before any build. The bundle records its source contract and env, and a stage asked for a different env refuses it.
• fluid verify checks an S3 + Glue binding (0.16.3): the Glue table's columns, and an Athena COUNT(*) against the rows the build landed.
• Scheduled builds get an Airflow 3 DAG that runs fluid apply (0.16.3), and fluid schedule-sync deletes only within the product's own folder.
• FLUID_STATE_BACKEND sets the default for fluid apply --state-backend (0.16.3); a bucket-only value keys each contract's state apart.
• Command Center publishing (0.16.3). fluid publish sends X-Organization-Id from FLUID_CC_ORG_ID (or the caller's single organization, or an organization slug in fluid.config.yaml), takes --env, records each contract version, and keeps the contract's classification. --dry-run shows the body it would send.
• fluid verify and fluid diff find a local file whose path uses {{ env.NAME }} (0.16.4), where the build wrote it.
• New fluid generate ci options (0.16.3): --apply-mode-default, --schedule-sync-default, --scheduler-default, --scheduler-destination-default, --diff-last-applied and --fluid-package-spec.

See also

• Upgrade guide
• 0.18.0 and 0.18.1 release notes
• 0.16.0 release notes
• One contract, two clouds, Per-environment overlays, Switch clouds
• OpenTofu state, Governance parity, The Command Center, Federated upstreams
• fluid apply, fluid diff, fluid verify, fluid schedule-sync
• 11-stage pipeline walkthrough
• forge-cli CHANGELOG