Upgrading
This page is the checklist for moving a project to a newer CLI. Each release's notes say why something changed; this page says what you do about it, in order.
Upgrade and pin the CLI
pip install --upgrade "data-product-forge[local]"
fluid version
$ fluid version
╭─────────────────────────── 📦 Version Information ───────────────────────────╮
│ FLUID CLI │
│ Version: 0.35.0 │
│ API: v1 │
│ │
│ Supported Specifications: │
│ • FLUID 0.7.1, 0.7.2, 0.7.3, 0.7.4, 0.7.5, 0.7.6, 0.7.7 (preview) │
│ • Default: 0.7.6 │
│ • Latest (stable): 0.7.6 │
╰──────────────────────────────────────────────────────────────────────────────╯
...
Install the extras your targets need: local (DuckDB and pandas), aws, gcp, snowflake and databricks (fluid apply on Databricks needs it for metric views and dbt job runs since 0.30.0). For AI forge, MCP and lineage (litellm, mcp, lineage, standard) see Changes in 0.28.0. Since 0.18.0 the local extra requires duckdb>=1.5.0, and the engine refuses an older DuckDB.
Pin an exact version in CI and production:
pip install "data-product-forge[local]==0.35.0"
A range such as >=0.16,<0.17 is not safe for CLI behaviour. Patch releases have changed results on purpose: 0.16.3 moved relative outputs and made verify --strict fail where it had passed. The SemVer promise covers the fluid_build.api Python package only (see API Stability). Upgrade by changing the pin and working through the checklist below.
A generated pipeline pins its own version. A Jenkinsfile written by fluid generate ci installs the version that generated it, through its FLUID_PACKAGE_SPEC parameter:
$ fluid generate ci contract.fluid.yaml --system jenkins
...
|- FLUID_PACKAGE_SPEC default: data-product-forge[local]==0.35.0
Upgrading the CLI on your machine does not upgrade that pipeline. Regenerate it, or set FLUID_PACKAGE_SPEC when you start a build.
Contracts do not need a new fluidVersion
Upgrading the CLI does not change the schema your contracts declare. 0.29.0 reads fluidVersion 0.7.1 to 0.7.6, and 0.7.7 as a preview (the list fluid version prints). 0.7.6 is stable since 0.26.0: it is the default, and the contracts forge-cli writes declare it. A contract that declares 0.7.5 still validates as 0.7.5.
Some fields exist only in 0.7.6: lifecycle.expire, binding.encryption, binding.principals, binding.governance.lakeFormation.bucketPolicy, and consumes[].upstreamWorkspace / upstreamDigest among them; the 0.7.6 field reference lists every field, and Federated upstreams covers the last two. Before you move a contract to 0.7.6, give each entry of an overlay's builds, exposes and consumes an id (see Changes in 0.25.0 and 0.26.0). To try what lands next, declare fluidVersion: "0.7.7".
A safe order for any upgrade
Upgrade the CLI in a branch, and run
fluid version.Validate every contract, once per overlay you deploy:
fluid validate contract.fluid.yaml --env awsPreview each cloud apply without applying. A dry run runs the pre-plan checks this page refers to (state key, shared state, region, GCP grants), then plans:
fluid apply contract.fluid.yaml --env aws --dry-runIt ends with
dry-run: plan only — not applying.and writes no state.Regenerate committed pipelines, and review the diff:
fluid generate ci contract.fluid.yaml --system jenkinsWork through the section for each version you cross. The table below says which ones.
Find the row for the version you run now, and read it and every row after it, up to the version you move to. The last two rows are topic sections that cut across versions: read them as well when they apply.
| You run | Read |
|---|---|
0.15.x | From 0.15.x to 0.16 |
0.16.x | From 0.16 to 0.17 |
0.17.x | From 0.17 to 0.18.2 |
0.18.x | Changes in 0.19.0. On 0.18.0 or 0.18.1, also the fluid diff row of From 0.17 to 0.18.2 |
0.19.x | Changes in 0.20.0, which fixes six security advisories |
0.20.x | Changes in 0.21.0 |
0.21.x | Changes in 0.22.0 |
0.22.x | Changes in 0.23.0 |
0.23.x | Changes in 0.24.0 |
0.24.x, 0.25.x | Changes in 0.25.0 and 0.26.0 |
0.26.x | Changes in 0.27.0 and Extras in 0.27.0 |
0.27.x | Changes in 0.28.0 |
0.28.x | Changes in 0.29.0 |
0.29.x | Changes in 0.30.0 |
0.30.0 | Changes in 0.30.1 and Changes in 0.31.0 |
0.30.1 | Changes in 0.31.0 |
Anything before 0.24.0, if you read forge's ODCS or ODPS output | Read this first if you consume the exported artifacts (0.23.0 and 0.24.0) |
Anything before 0.23.0, and any DuckDB upgrade | DuckDB 1.x to 2.x (0.22.0 and 0.23.0) |
Read this first if you consume the exported artifacts
New in 0.23.0 and 0.24.0
Exported ODCS and ODPS documents change shape. Every change below corrects output that was wrong, but the old values may be baked into whatever reads them, so this is the section to read before you upgrade rather than after. The work is forge-cli #753 (0.23.0) and forge-cli #756 (0.24.0).
Nothing here changes a FLUID contract. If you do not consume fluid generate standard, fluid odcs export, fluid odps export or fluid generate artifacts output, skip it.
| Change | What to do | How to check |
|---|---|---|
A Kafka column's ODCS physicalType is an Avro type name. bigint now exports as long, where it exported integer | Re-derive and re-register any Avro schema built from a forge ODCS export of a Kafka topic with a 64-bit column | Export the contract and read physicalType on that column |
A per-port ODCS contract's name is the port's, not the product's; the product name moves to a fluidProductName custom property | Re-key anything that keys files, registry subjects or display names on the contract name; read the product from fluidProductName | Export a product with more than one expose: the two documents no longer share a name |
ODPS inputPorts[].contractId is <productId>.<exposeId>, not the bare upstream product id | Point any resolver that looks an input port up by that id at the port-scoped form | Export a contract with a consumes[] entry and read inputPorts[].contractId |
A Kafka server's ODCS format becomes a message format once the contract declares binding.serialization | Nothing, until you adopt that field | A contract without it exports the same bytes as before |
Derived quality rules carry an id, and rules declared in dq.rules are exported as ODCS quality | Nothing. A DQ engine that labels results by rule id stops refusing the document | quality[].id is present on every rule the exporter mints |
fluid odps import refuses to write a contract that does not validate, and its pass-through bucket moved from metadata.odps_passthrough to the root extensions.odps | Read the bucket at its new address. Pass --lenient if you need the old write-anyway behaviour | An import that used to print a success tick and write an invalid file now fails and writes nothing |
Breaking. Two exposes resolving to one contract id fail fluid validate and every export | Give each expose its own exposes[].contract.id, or rename the expose whose id is derived | fluid validate names both exposes and the shared id |
0.24.0: an ODPS export of a contract with consumes[] gains outputPorts[].inputContracts, and an input port's version may differ from 1.0.0 where the contract states a version range (forge-cli #756) | Update a consumer that compares exported documents byte for byte, or one that expects 1.0.0 on every input port | Export a contract with a consumes[] entry that has a versionConstraint: the input port's version is the lowest version the range accepts, and outputPorts[].inputContracts lists the upstream |
0.24.0: an ODCS sql quality rule derived from exposes[].contract.quality[].expression is a counting query (SELECT COUNT(*) FROM ${table} WHERE (<expression>) IS NOT TRUE) with mustBe: 0, not the bare expression with mustBe: true; its description changes with it (forge-cli #756) | Update a DQ configuration or test that pins the old query string or the boolean mustBe. An expression already written as a SELECT or WITH query is unchanged | Export the contract and read query and mustBe on the rule |
The Kafka physicalType change is the urgent one
This was silent data corruption, not a cosmetic fix. Kafka had no physical-type table, so fluid_to_physical fell back to the ODCS logicalType — and ODCS's own schema defaults an unqualified integer to i32. A tool deriving an Avro schema from the exported contract therefore read 32 bits for a column a producer such as Flink BIGINT writes as 64. The registered schema and the data disagreed, and nothing signalled it.
The physical type is now the Avro type name, from a table measured against the Avro 1.12 specification and against datacontract-cli's own ODCS→Avro exporter. The integer family is split by width, because Avro has no 8- or 16-bit integer: int, integer, int2, int4 and serial are int, while bigint, long, int8, int64, longint and bigserial are long. decimal carries its precision and scale in customProperties.
If you build Avro schemas from these exports, treat every 64-bit column as having been wrong and re-register.
The per-port ODCS name
A per-port export — fluid generate standard --format odcs, and the per-expose documents fluid generate artifacts writes — used to name every document after the product, so a product with two exposes published two contracts both called "Orders Product". The id was already per-port; the name was not, so a tool that keys files, registry subjects or display names on the name could not tell them apart.
The name is now the port's title, falling back to its exposeId, and the product's name is in a fluidProductName custom property.
There is deliberately no special case for a single-port product: making the exported name depend on how many siblings a port has would change that name the day a second port is added. So a single-expose product's exported name changes too, whenever its port's title (or exposeId) differs from the product's name. The ODCS id is unchanged in every case, and a whole-contract export is untouched.
Input ports, and the version constraint that used to be dropped
inputPorts[].contractId carried the bare upstream product id, which is not the id anything publishes: an output port is published under <productId>.<exposeId>. It is now that port-scoped form — or the upstream's own id when the contract names one in the new consumes[].contractId.
A consumes[].versionConstraint was dropped outright, because ODCS's input-port object has no field for it. It now rides in a fluidVersionConstraint custom property.
The Kafka server format
A Kafka binding declares format: kafka_topic, which says nothing about whether the values on the topic are Avro or JSON — the one thing a consumer has to know to read it at all. With the new binding.serialization.value, the exported ODCS server format is the message format ODCS means by that field: avro, json, protobuf or xml, with avro-confluent and json-confluent collapsing onto the wire format they carry. Whether a schema registry is involved is not lost — binding.serialization rides along verbatim in the fluidExtras custom property, unless ODCS_FLUID_EXTRAS=false opts out of that blob.
A contract that has not adopted the field exports exactly as before. The exporter will not guess a wire format from a contract that never stated one: binding.format passes through as it always has, and where a contract sets neither, the pre-existing json default still applies. So this changes a Kafka export only once the contract says what its messages are.
Quality rules
Rules the exporter derives from required, the primary-key tag and validationRules now carry an id. The ids are derived rather than counted, so two renders of one contract produce the same ids and an export is not a spurious diff. That matters twice: a DQ engine that labels its results by rule id refuses a whole contract whose rules have none, and DataHub's ODCS ingestion seeds its assertion URNs from quality.id, so a stable id is what keeps an assertion's identity across a rename or a reorder.
Rules an author declared in dq.rules used to be parked in the fluidExtras custom property and nowhere else, so a consumer running the published contract's quality saw only the checks forge had synthesised. They are now exported as ODCS quality for the four types ODCS can express:
dq.rules type | Exported as |
|---|---|
completeness | library metric nullValues, dimension completeness |
uniqueness | library metric duplicateValues, dimension uniqueness |
valid_values | library metric invalidValues, dimension conformity |
freshness | a text rule — ODCS has no library metric for it |
The other four — accuracy, schema, anomaly_detection and drift_detection — have no executable ODCS form, so each is logged by name with the reason and still carried in fluidExtras. Silence was the bug; a rule with no ODCS form is now reported rather than dropped without a word.
fluid odps import
The command's own help said it "always emits one validated FLUID contract" while never running the validator, so importing its own export printed a success tick and wrote a file fluid validate then rejected with ten errors. It now validates before writing:
fluid odps import product.odps.yaml -o contract.fluid.yaml
A document that does not produce a valid contract fails, names the problem and writes nothing. --lenient writes it anyway — note that this widens a flag that already existed: it used to mean only "downgrade output-port resolution failures to warnings", and it now also means "write the contract even when it does not validate".
The pass-through bucket the importer writes for everything ODPS carries that FLUID has no field for moved from metadata.odps_passthrough to the root extensions.odps. metadata is additionalProperties: false, so the old address could never validate; root extensions is the one object the FLUID schema declares open. Root extensions arrived in FLUID 0.7.3, so an import whose source document records an older fluidVersion warns and declares the default instead, because the older schema cannot hold the bucket.
The breaking one: two ports under one contract id
A contract whose exposes[].contract.id collides with a sibling's resolved id — pinned or derived — now fails fluid validate and every export door. The check compares the ids the exporters will actually mint, so a pin equal to a sibling's derived <productId>.<exposeId> is caught too:
fluid validate contract.fluid.yaml
The message names both exposes and the shared id, and the advice fits which side pinned it: change the contract.id where one is pinned, rename the expose where the id is derived.
That export used to succeed. It wrote one per-port ODCS document for two output ports, so the losing port's schema was absent from the bundle and both ports resolved to the survivor — and fluid generate artifacts sealed it under a MANIFEST merkle root. The failure is the fix.
From 0.15.x to 0.16
Release notes: 0.16.0 (covers 0.15.1 to 0.16.2).
| Change | What to do | How to check |
|---|---|---|
Generated SQL scripts create views (CREATE OR REPLACE VIEW); on Snowflake and Databricks that drops grants on an existing view | Re-grant after the script runs, or rename an output that collides with an existing table | Each generated file's header says -- Materialises: <name> (CREATE OR REPLACE VIEW) or -- Not materialised: <reason> |
dbt multi-stage models are named after stages[].outputs, and fluid_* sentinel tests now fire | Update references to the old stage_* model names | Regenerate the dbt project and diff models/ |
| The federation digest gate warns and applies | If CI must block on a drifted pin, fail on the log line | Search the apply log for apply_consumes_drift |
| The AWS region comes from the binding | Set location.region to where the resources are, or move them | The dry run refuses with opentofu_region_moved, naming both regions |
Athena could not read Glue Parquet tables FLUID created before 0.16.2 | New tables declare the Hive input format and SerDe. Whether a re-apply updates an old table was not checked here: run fluid apply --dry-run and look for an in-place change to the table's storage descriptor | SELECT COUNT(*) in Athena does not fail with HIVE_UNSUPPORTED_FORMAT |
fluid import airbyte needs a server | Pass --server-url or set FLUID_IMPORT_AIRBYTE_URL | The import refuses by name when neither is set |
The container image has no 0.16.0 tag; 0.16.1 is the first image after 0.15.3.
From 0.16 to 0.17
Release notes: 0.17.0 (covers 0.16.3 to 0.17.0).
Results that change on purpose
| Change | What to do | How to check |
|---|---|---|
Relative local outputs land under the contract's directory (0.16.3) | Move readers of the old location | Run a local apply and look under the contract's directory |
fluid verify --strict fails a local output whose columns differ from the schema (0.16.3) | Fix the schema or the build | Run fluid verify --strict before the upgrade lands in CI |
--build-id with a mode that runs no build, and a plan applied in another mode, are refused (0.16.3) | Plan in the mode you apply | The apply fails before any build |
The drift gate compares the live target (0.16.3), and reads the apply's OpenTofu state (0.16.5) | Use --last-applied to separate a contract change from drift | fluid diff --exit-on-drift exits non-zero on real drift |
Command Center products are private unless the contract is classified public (0.16.3) | Classify public products as public | fluid publish --dry-run shows the body it would send |
Lake Formation bucket policies omit same-account grantees (0.16.3) | Set bucketPolicy: all-grantees (fluid-schema 0.7.6) only if you need the old output | The dry run's plan changes the bucket policy |
A consumes[] entry the SQL reads must resolve (0.16.5) | Put the upstream under the same fluid.workspace.yaml, list its root in FLUID_UPSTREAM_CONTRACTS, or bind an explicit input of the same name | The build fails with ConsumesResolutionError naming the entry |
A plan records its env (0.16.5) | Apply with the same --env you planned with | A different --env fails with plan_env_mismatch |
A failed DuckDB S3 secret stops the run when AWS_ENDPOINT_URL is set (0.16.5) | Fix the endpoint or credentials | The run fails with ObjectStoreEndpointError |
Masked columns land treated (0.16.5) | Set FLUID_PII_HASH_SECRET (16 bytes or more) for hash, the key variables for tokenize and encrypt, and declare treated columns as strings | An unset secret refuses the build; fluid verify fails a column that landed in cleartext |
The plan-env check, in practice:
fluid plan contract.fluid.yaml --env aws --out plan.json
fluid apply plan.json --env aws
Retire the old Airflow DAGs
0.17.0 puts the env into the DAG directory and id: <product-id>__<env>/ with dag id <product>__<env>__<build>, where 0.16.7 and earlier wrote <product-id>/ with <product>__<build>. The first sync after the upgrade must retire the old DAGs, or Airflow runs both, and both apply the same product against the same state.
fluid schedule-sync --scheduler airflow \
--dags-dir dist/artifacts/schedule/ \
--destination /opt/airflow/dags/ \
--env prod --report sync-report.json
- With
--delete-scope product(the default) and a local orgit+sshdestination, the sync deletes, from<product-id>/, only the DAGs rendered for the same product and env under the old id. Other files stay. - For any other destination (S3, GCS, MWAA, Composer, Astronomer and the rest) it cannot read the destination, so it prints a note instead:
[schedule-sync] note: <product>__<env>/ now holds <product>'s DAGs for env <env>. …Delete the old DAG files for that env at the destination once. --delete-scope destinationremoves the old directory along with everything else the sync does not ship. Use it only for a destination this product owns alone.
Check: each entry of superseded_scopes in sync-report.json names the new scope, the product it replaces, the env, and old_dags_retired. false means the deletion is still yours to do.
Remote state moves to a per-provider key
How state keys, backends and the commands that read state fit together is in OpenTofu state; deploying one contract to two clouds is in One contract, two clouds.
The default per-contract state key gains the provider: fluid/<id>/<provider>/terraform.tfstate (for GCS, the prefix fluid/<id>/<provider>). This applies to contracts that use a per-contract key: one with a packaging block, or any contract whose backend comes from a bucket-only FLUID_STATE_BACKEND. An explicit key in --state-backend, and the shared legacy key fluid/terraform.tfstate, do not move.
The first real apply copies the state with OpenTofu's own migration (tofu init -force-copy), checks the copy, and prints:
state move: moved <n> resource(s) from <old> to <new> with `tofu init -migrate-state`; the old object is left in place
A dry run never moves state. While the move is pending, it plans against the old key and prints state move: read from <old>: `fluid apply` moves it to <new>.
When the apply for one cloud finds the other cloud's state at the old key, it leaves it alone. On a live run on 4 October 2026, the first --env gcp apply printed that the old key holds the aws provider's state (hashicorp/aws), not this provider's; left in place, and wrote the gcp state to fluid/<id>/gcp/terraform.tfstate.
Two refusals to expect:
state_shared_with_another_provider: the key does not name the provider and already holds another cloud's resources, so this plan would destroy them. Give each provider its own key, for example--state-backend s3://<bucket>/fluid/<id>/<provider>/terraform.tfstate, or use a bucket-onlyFLUID_STATE_BACKEND.- A state that mixes providers is not moved; the apply stops with an error naming both keys, so you decide.
The apply does not delete the old object. Once the apply for each provider has moved its state, nothing reads the old key, and you can remove it.
GCP dataset grants become member resources
The policy table for AWS and GCP, and what fluid verify checks on each, is in Governance parity.
Dataset grants are google_bigquery_dataset_iam_member resources, which add to a dataset's access instead of replacing it. On the first apply to a dataset whose state holds the old authoritative access list, the apply revokes, once, the entries no grant of the contract covers, and prints them:
dataset <dataset>: its access list was written by an older forge-cli; this apply revokes the entries no grant of the contract covers: <entries>
For that one apply the old list is still authoritative, so an entry added by hand since the last apply is revoked too. Special groups, views and routines are kept.
Check before applying: fluid diff makes the same change, so its plan shows the revocation, and the dry run prints the line above.
fluid diff contract.fluid.yaml --env gcp
If every entry of a dataset's list would be revoked, the apply is refused with opentofu_dataset_access_unreconciled. Revoke those entries yourself (bq update or the console), or keep one of them in the contract for this apply, then re-run.
Overlays a workspace expects
--env <name> with no overlay is refused when the workspace's fluid.workspace.yaml lists that env for the product under expected-environments. Before 0.17.0 the base contract was used as if it were that env. See Environments and overlays and Workspaces.
# fluid.workspace.yaml
expected-environments:
orders: [gcp]
The refusal is skipped for dev and for an env the base contract already binds to. The CLI wraps the message at the terminal width; this was run at 100 columns:
$ fluid validate orders/contract.fluid.yaml --env gcp
❌ Validation error: contract_load_failed
error: --env 'gcp' has no overlay, but fluid.workspace.yaml expected-environments (orders)
declares 'gcp' an environment of this product, so the base contract (bound to local) would be used
as if it were 'gcp'. Add overlays/gcp.yaml, or remove 'gcp' from fluid.workspace.yaml
expected-environments (orders)
[ERR_CONTRACT_LOAD_FAILED]
...
Without that workspace entry, the same command only warns (overlay_not_found) and uses the base contract. A contract's own environments block warns but does not refuse.
Also check
- A GCP binding with no region is refused, instead of landing in
US. - Regenerate committed pipelines. Stage 6 now runs
fluid plan --check-sovereignty, and the DAG ids changed. - Command Center run reports. If the publish configuration is set, each
fluid applyreports its run to the Command Center (what it sends). SetFLUID_COMMAND_CENTER_ENABLED=falseto turn it off.
From 0.17 to 0.18.2
Release notes: 0.18.0 to 0.18.2. That page has a "Do I need to change anything?" table; the short version:
| Change | What to do | How to check |
|---|---|---|
$ref is confined to the root contract's directory tree; URL, file:// and absolute refs are refused | Widen with FLUID_REF_ROOT per command, or copy the fragment in | fluid validate fails with a message that the ref escapes the ref root |
Contract SQL runs in DuckDB's sandbox: no URLs other than declared s3://, no SET or PRAGMA that change a setting, no autoloaded extensions | Allow a directory with FLUID_DUCKDB_ALLOWED_DIRS, land remote data first, remove SET / PRAGMA | Run each local build once: fluid apply contract.fluid.yaml --mode amend-and-build --yes |
OpenAPI documents in a bundle may not $ref another file or URL | Inline the schemas under components | fluid validate on the bundle reports OAS-REF-EXTERNAL |
The local extra needs duckdb>=1.5.0 | pip install -U "data-product-forge[local]" | The engine refuses an older DuckDB by version |
fluid diff resolves {{ env.* }} in a binding. 0.18.1 fixed only the live check (forge-cli #692): fluid diff still failed with diff_failed for every silver and gold whose gcp binding names its project through {{ env.* }}, until 0.18.2 (forge-cli #696) | Upgrade to 0.18.2, not 0.18.1; no contract change | Stage 5 (fluid diff --exit-on-drift) passes on a silver or gold whose gcp binding names its project through an env placeholder |
Check a bundle the way CI builds it:
fluid bundle contract.fluid.yaml --format tgz --out runtime/bundle.tgz
fluid validate runtime/bundle.tgz
Background: Composing a contract with $ref, DuckDB sandbox for contract SQL, Contract loading API (new in fluid_build.api 1.1).
Changes in 0.19.0
0.19.0 moves the AWS provider to terraform-provider-aws 6, and adds column masking and row filters (forge-cli #700). The first row can stop an AWS apply that worked before.
| Change | What to do | How to check |
|---|---|---|
terraform-provider-aws is pinned to ~> 6.0. Provider 6.23 and later read a bucket's tags through S3 Control, which needs s3:ListTagsForResource | Grant s3:ListTagsForResource to the identity that applies AWS products | Run fluid apply contract.fluid.yaml --env aws --dry-run as that identity on an already applied product: the plan shows no change. With the pin moved and nothing else, eleven applied AWS products planned no change on provider 6.67 |
A full-refresh BigQuery load uses WRITE_TRUNCATE_DATA, where it used WRITE_TRUNCATE | Nothing. WRITE_TRUNCATE removed every row access policy of the table it wrote, so each build undid the row filters the apply had made. WRITE_TRUNCATE_DATA replaces the rows and keeps the table's policies, policy tags and partitioning. An emulator keeps WRITE_TRUNCATE | After a full-refresh build, the table still has its row access policies |
A contract that sets neither columnRestrictions with access: mask nor policy.authz.rowFilters is not affected by the rest of the release. A contract that sets them is refused where its binding cannot enforce them, with an unsupported_binding kind that names the fix:
column-mask-unenforceable: an AWS binding masks a column and declares nogovernance.lakeFormation.maskedViews(definerandvalidationConnection). Lake Formation cannot mask, so a protected Glue view returns the masked columns. Declare the field, or deny the columns instead of masking them.row-filter-unenforceable: an AWS binding filters rows with no Lake Formation grant that lets the filtered principal read the table. GrantSELECT, or remove the filter.row-filter-view: a row filter on a BigQuery view. Row access policies attach to tables, so filter the table the view reads.
Changes in 0.20.0
These changes shipped in forge-cli 0.20.0, after the 0.19.0 the section above covers. It fixes six security advisories: they come after the first table. Each row of the first table names the pull request that made it.
| Change | What to do | How to check |
|---|---|---|
fluid apply validates the contract against its schema on both engines and in every mode (forge-cli #714). A cloud apply (aws, gcp, snowflake, confluent) or a build mode skipped the check, so a contract fluid validate rejects could apply with exit 0. It now stops before planning with apply_contract_invalid, or contract_version_unsupported below fluidVersion 0.7 | Run fluid validate <contract> --env <env> for each overlay you apply, and fix what it reports. No flag skips the check | fluid apply <contract> --env <env> --dry-run stops before tofu plan with the event |
A plan.json that embeds no contract is refused with apply_plan_missing_contract (forge-cli #714). It used to run its actions unchecked on the native engine | Re-run fluid plan and apply the new plan | The apply fails before anything runs |
An aws contract with orchestration.engine: airflow or mwaa and an orchestration.schedule fails fluid apply and fluid generate iac with unsupported_binding, kind mwaa-environment-not-emitted (forge-cli #715). The apply reported success and created no MWAA environment | Set orchestration.engine: none to schedule through EventBridge Scheduler, or run the DAGs on an MWAA environment you own and drop orchestration.schedule. See Scheduling on AWS | fluid generate iac contract.fluid.yaml refuses with the kind and writes no module |
An AWS Iceberg expose with a location.catalog other than glue (rest, iceberg_rest, polaris, unity, nessie, lakekeeper) holds a Glue database and table that earlier releases created for it. A Snowflake Iceberg expose with catalog: lakekeeper, bigquery, iceberg-rest, hive, jdbc, hadoop or dynamodb holds an EXTERNAL VOLUME 0.19.0 and earlier created. fluid apply stops before planning with iceberg_catalog_move_blocked (forge-cli #707), where it planned to destroy them. Destroying a Glue database deletes every table in it | Run the tofu state rm commands the refusal prints, then apply again. The resources stay where they are | fluid apply <contract> --env <env> --dry-run stops with the event and prints the commands |
A Kafka Connect or embedded Debezium build whose Iceberg sink config forge-cli derives is refused when its expose names no table namespace (location.database, or location.dataset on GCP). A Kafka Connect build is also refused when its expose names no location.table (forge-cli #710). The derived config used to name the table None.<table>, or leave Debezium Server on its default namespace | Set the field on the expose. An override that sets iceberg.tables or table-namespace itself is not affected | The build is refused before it runs, with a message that the expose's binding names no table namespace, or no table |
A key the schema does not have is one way a contract fails the first row. With fluid apply --dry-run, a GCP contract that still uses the deprecated metadata.policies went on to tofu plan before 0.20.0, and now stops:
❌ apply_contract_invalid [ERR_APPLY_CONTRACT_INVALID]
command: apply
error_count: 1
errors: ["metadata: Additional properties are not allowed ('policies' was unexpected)"]
message: Contract validation failed with 1 error(s); ``fluid apply`` will not proceed on a schema-invalid contract. Run ``fluid validate`` for the full report.
Background: Contract validation.
Security advisories
0.20.0 fixes six security advisories. Each changes what a project file, an MCP client or a proxy may do, so a setup that relied on the old behaviour stops working until you move the setting. The full upgrade notes are in the forge-cli CHANGELOG.
| Change | What to do | How to check |
|---|---|---|
Project-config catalog endpoints are no longer trusted (GHSA-vw85-xrwh-3p3x). A catalog endpoint set by .fluidrc.yaml, .fluidrc, fluid.config.yaml or .fluid/config.yaml in the checkout (Command Center, Data Mesh Manager, DataHub, OpenMetadata) gets only a key that the same file carries. It no longer gets FLUID_API_KEY, DMM_API_KEY or any other key from the environment, /etc/fluid or ~/.fluid. ${VAR} placeholders in it are kept as written. The fluid apply Command Center report needs https:// for every host that is not loopback or on FLUID_COMMAND_CENTER_HOST_ALLOWLIST | Set the endpoint yourself: FLUID_CC_ENDPOINT, DMM_API_URL, DATAHUB_GMS_URL or OPENMETADATA_SERVER_URL, the endpoint in ~/.fluid/config.yaml, or fluid publish --target <name>:<endpoint>. Move ${VAR} placeholders out of project configs, and quote credentials | fluid publish fails, or fluid apply prints run not reported (...). The message names what was withheld, never its value |
A workspace root above the project no longer widens the DuckDB sandbox (GHSA-58px-v64p-6ppm). A fluid.workspace.yaml at $HOME, above it, in a home-holding directory (/home, /Users), in a temp directory, at /, or above the contract's repository (.git, .hg) is found but not granted to contract SQL | Move the workspace file into the directory that holds the products. For products cloned side by side, list their common directory in FLUID_DUCKDB_ALLOWED_DIRS and declare the file. Move a contract saved straight in $HOME or /tmp into a directory of its own | The build runs with the contract's own directory and logs local_workspace_not_granted, naming the directory and the fix |
A project .env can no longer set operator-only variables (same advisory). FLUID_COMMAND_CENTER_URL, DMM_API_URL, HTTPS_PROXY, FLUID_STATE_BACKEND, FLUID_DUCKDB_ALLOWED_DIRS, FLUID_SECRETS_FILE, PGHOSTADDR, PATH and git transport variables are among them. If CI or your shell exports a credential while a committed .env names the matching endpoint (OPENAI_BASE_URL, PGHOST, SNOWFLAKE_ACCOUNT), that endpoint is ignored | Export the variable in the shell or the CI job, or put it in a FLUID_SECRETS_FILE you export. See What a project's .env may set | fluid prints Ignored ... from project dotenv files and runs without the variable |
An MCP client can no longer use the host's cloud identity (GHSA-m7j5-jcrq-cmxm). The source-catalog tools honour a client's allow_metadata_service: true only when the server runs with --allow-metadata-service (or FLUID_ALLOW_METADATA_SERVICE=1). A catalog read that relied on a bare AWS_REGION, GOOGLE_CLOUD_PROJECT or similar variable now needs the same opt-in, fluid forge data-model from-source included | Add --allow-metadata-service to the server's args in your MCP client config if you rely on instance-role, workload-identity or ADC auth. See Credential resolver | The tool call fails with an error that names the flag, or a catalog read fails with "No credentials resolved" |
Federation credentials need an operator allowlist (GHSA-m288-j7hc-rhrj). A federated upstream whose auth.secret_ref attaches a credential is refused unless the operator lists it in FLUID_FEDERATION_SECRET_REF_ALLOWLIST, except GITHUB_TOKEN or GH_TOKEN sent to a GitHub host. An http:// or git:// endpoint that carries one is refused | Set FLUID_FEDERATION_SECRET_REF_ALLOWLIST (for example CATALOG_TOKEN@catalog.acme.com, and GITHUB_TOKEN@<your host> for GitHub Enterprise Server) in the shell, the CI environment or a FLUID_SECRETS_FILE outside the project, with hosts written without a port. Move the endpoint to https:// (ssh:// for git) | The upstream is reported as unreachable. The federation gate warns and does not stop fluid apply |
The MCP output port reads a client certificate only from a trusted proxy (GHSA-x4fj-f8p7-p9f3). X-Client-CN and X-Client-Fingerprint are read only after shared-token or jwt auth, from a peer listed in FLUID_MCP_TRUSTED_PROXIES. fluid mcp output-port serve exits 2 over stdio for an expose whose row filter reads ${caller.client_cn} or ${caller.client_fingerprint}, and over HTTP without gateway auth | Set FLUID_MCP_TRUSTED_PROXIES to the proxy's address if you run behind an mTLS proxy: 127.0.0.1 for the shipped nginx and Caddy templates. List only the proxy, or the narrowest CIDR that covers its replicas. Serve such an expose over HTTP with gateway auth | Calls a certificate filter applies to fail with RowFilterIdentityMissing, and the WARNING names the address to add |
fluid init --discover keeps passwords out of contracts (GHSA-25mh-f2f4-9vxp). A contract from a Postgres or MySQL/MariaDB URI that holds a password carries secretRef: env://PGPASSWORD or env://MYSQL_PWD, not password:. A URL contract keeps no query string, fragment or user:password@ part | Export PGPASSWORD or MYSQL_PWD before fluid apply. Add back by hand any query parameter the source needs that is not a credential. Against a remote host the env:// reference also needs an allowlist since 0.23.0: see Changes in 0.23.0 | The contract has no password: line, and the closing line of --discover lists the URL parameters it dropped |
Changes in 0.21.0
0.21.0 caps how far a YAML file may expand, merges overlay lists by id from fluidVersion0.7.6, and stops running Flink SQL as SQL of forge-cli's own.
| Change | What to do | How to check |
|---|---|---|
The YAML guard refuses a file, FLUID contracts included, that nests more than 128 levels, or whose aliases add more than 100,000 nodes or 5,242,880 characters of text, or that grows more than 100-fold past 1,000 nodes (forge-cli #725). The 5 MiB size cap counts UTF-8 bytes. ODCS and ODPS files now go through the guard too, so fluid odps import and validate, fluid odcs validate and import, fluid odps-bitol validate and fluid forge --seed-from refuse a file over 5 MiB or with more than 50 aliases | Write the repeated blocks out instead of reusing them through anchors | The error names the sizes. A bundle directory whose ODPS document the guard refuses, or that does not parse, fails instead of importing as ODCS-only |
A file the resolver probes for a port's contractId must declare that contractId as its id (forge-cli #725) | Make the port's contractId and the ODCS id agree | A strict import fails, or skips the port under --lenient (an input port is always skipped) |
In a contract whose fluidVersion is 0.7.6 or later, an overlay patches builds, exposes and consumes by id, not by position (forge-cli #732). An overlay entry with no id, or two naming the same id, fails the load where the base's entries carry unique ids | Give each overlay entry the id of the entry it patches. See Changes in 0.25.0 and 0.26.0 for the rule, since 0.7.6 became stable in 0.26.0. A 0.7.5 contract still merges by position | fluid validate contract.fluid.yaml --env <env> fails with contract_load_failed for an id-less entry. An entry whose id the base lacks is appended and logged as overlay_entry_added |
A build with properties.language: flink_sql is no longer run as SQL (forge-cli #731). fluid apply --mode amend-and-build ran its Flink SQL on DuckDB or refused it, and the local, AWS, GCP and Snowflake planners planned it as a DuckDB, Athena, BigQuery or Snowflake statement. It is now reported as run outside forge-cli, planned without SQL, and left out of generated schedules | Run the job on the platform that runs Flink. Declare engine: flink with fluidVersion: "0.7.6" to name the engine too | fluid apply --mode amend-and-build counts the build on its own summary line as run outside forge-cli |
Changes in 0.22.0
0.22.0 makes --strict fail on a retargeted overlay patch, changes the operators in generated Airflow DAGs, and lets the local engine run on DuckDB 2.x. The DuckDB changes are in DuckDB 1.x to 2.x, which follows this section.
| Change | What to do | How to check |
|---|---|---|
fluid validate --env <env> --strict (one contract or a whole workspace), fluid contract-validation --env <env> --strict and fluid test --env <env> --strict fail when an overlay patch lands on an entry that names another id (overlay_patch_retargeted), which an overlay merged by position can do (forge-cli #734). fluid validate <bundle>.tgz --strict fails on one the bundle recorded (forge-cli #735), and stage 2 of the generated pipelines runs --strict | Put the overlay's entries in the base's order, or set fluidVersion: "0.7.6" with a unique id on every base entry, so overlays patch by id. Rebuild a bundle made before 0.22.0: it records no such warning and passes as before. fluid contract-validation and fluid test do not read the bundle's record, so run them on the contract file with --env | fluid validate contract.fluid.yaml --env <env> --strict fails. Without --strict the finding is a warning |
Generated Airflow DAGs run Snowflake SQL through SQLExecuteQueryOperator with conn_id, because apache-airflow-providers-snowflake 6.0 removed SnowflakeOperator. A GCP create_table task uses BigQueryCreateTableOperator where the google provider has it (13 and later), and BigQueryCreateEmptyTableOperator before that (forge-cli #736) | Regenerate each committed DAG: a DAG generated before does not load on providers-snowflake 6.0 | Generate the schedule again and diff the DAG: the operator names change |
The new operators render more of their arguments as Jinja: SQLExecuteQueryOperator renders conn_id and hook_params (the warehouse and database), and BigQueryCreateTableOperator renders table_resource (the schema). A literal {{, {% or {# in a warehouse or database name, or in a schema field such as a description, now fails the task when it runs (forge-cli #736) | Escape it as {{ '{{' }}, {{ '{%' }} or {{ '{#' }}, or wrap the value in {% raw %} ... {% endraw %} | The task fails at run time, not at generation |
fluid scaffold-composer writes no DAG for a contract whose every build runs outside forge-cli (engine: flink, properties.language: flink_sql) or continuously and that declares no orchestration.tasks, where it used to write one. It takes the DAG's cron from the first build a scheduler starts (forge-cli #737) | Delete a DAG it wrote for such a contract before, from --out-dir and from Cloud Composer if you deployed it. The command names the file when it finds it under the name it writes now | fluid scaffold-composer writes no file for such a contract |
A product whose every build becomes continuous (or runs outside forge-cli) gets no schedule/ artifacts, so pipeline stage 11 skips fluid schedule-sync (forge-cli #734) | Remove by hand a DAG deployed for it earlier: it stays in the scheduler | fluid generate schedule writes no files for the contract |
An environment overlay cannot turn a scheduled build continuous (trigger.type: continuous, a 0.7.6 preview): overlays cannot delete the base's cron or schedule, which a continuous trigger refuses (forge-cli #734) | Declare the trigger in each environment's overlay instead of in the base | The schema error names the field and the trigger type that takes it |
DuckDB 1.x to 2.x
0.22.0 raised the local engine's DuckDB range to duckdb>=1.5.0 with no upper bound, so pip install -U "data-product-forge[local]" can now resolve DuckDB 2.x. DuckDB 2.0 needs Python 3.11, so a Python 3.10 install stays on 1.5.x and nothing below changes for it.
Nothing here asks you to upgrade DuckDB. Read it before you do, and before anything else in your stack does it for you — pip install -U on a fresh environment, a rebuilt CI image, or another tool in the same environment pulling DuckDB 2.x in.
python -c "import duckdb; print(duckdb.__version__)"
| Change | What to do | How to check |
|---|---|---|
A DuckDB database file another 2.x tool wrote cannot be opened on 1.5. That covers dbt-duckdb, dlt, and forge-cli 0.21.0 or older running on DuckDB 2.0 | Convert it, or keep every reader of that file on 2.x. The conversion is below | fluid apply on 1.5.x fails to open the file |
Contract SQL runs on the DuckDB that is installed, and 2.0 changed some SQL: x -> x + 1 is an error, and integer // and % by zero raise instead of returning NULL | Write lambda x: x + 1; wrap a division that can divide by zero in TRY(...) | Run each local build once: fluid apply contract.fluid.yaml --mode amend-and-build --yes |
A DuckDB acquisition build now resolves a Postgres source's connection.secretRef (0.22.0). It used to ignore it, and libpq's PGPASSWORD or ~/.pgpass supplied the password | In 0.22.0, remove the secretRef to keep using PGPASSWORD. From 0.23.0 an acquisition build no longer sees PGPASSWORD from the environment (see Changes in 0.23.0): declare the credential on the connection. An inline password still wins over a secretRef | The build fails, naming the reference, where the secretRef does not resolve on the host that runs fluid apply |
| An incremental timestamp cursor written on 1.5 and read on 2.0 could silently drop or duplicate rows | Nothing: no state file needs rewriting. Upgrade to the release below before you move an incremental build across DuckDB versions | A run that lands fewer rows than are due, at exit code 0, with the stream recorded succeeded |
A pipeline discovered on 1.5 failed SchemaDriftError on 2.0 on a false positive | Nothing, in the release below | fluid apply fails the drift gate after a DuckDB upgrade, with no contract edit and no new data |
A file forge-cli itself creates on 2.x — fluid demo's .fluid/db.duckdb, the local provider's session and persist databases, a Meltano acquisition's DuckDB destination — is written in the format DuckDB 1.5 writes, which DuckDB 1.0 and later open. A column type that needs a newer format, such as VARIANT, is refused in it on 2.x exactly as it is on 1.5. A file that already exists keeps the format it has.
To convert a file that only opens on 2.x, run this on 2.x:
ATTACH 'old.duckdb' AS src (READ_ONLY);
ATTACH 'new.duckdb' AS dst (STORAGE_VERSION 'v1.0.0');
COPY FROM DATABASE src TO dst;
DuckDB's own 2.0 migration notes cover the SQL changes in full.
The two drift defects, and what fixes them
Fixed in 0.23.0
Both fixes shipped in 0.23.0, in forge-cli #749.
An incremental cursor written on 1.5 and read on 2.0 silently dropped or duplicated rows. A cursor is stored as text and pasted into WHERE <field> > '<text>' as an untyped literal, so DuckDB coerces it to the column's type — and the two versions disagree about that type. read_json_auto over a timestamp carrying Z gives TIMESTAMP on 1.5 and TIMESTAMP WITH TIME ZONE on 2.0, where a naive literal is read in the session time zone, which is the host's. On a host at a non-zero UTC offset the > boundary therefore moved. An end-to-end run that wrote its cursor on 1.5 and continued on 2.0 landed 5 of 6 rows and advanced the cursor past the row it skipped — exit code 0, stream recorded succeeded. Both versions satisfy duckdb>=1.5.0, so one install can write state another reads. CSV sources were never affected: read_csv_auto already sniffed TIMESTAMPTZ on 1.5.
A timestamp cursor now compares instants (epoch_us), which no session time zone can shift, and the stored text is normalised to a naive UTC instant.
- No state file needs rewriting. A cursor a shipped 2.0 install already wrote carries an explicit offset, and is normalised when it is read.
- The instant comparison is used only for a column whose type
epoch_usaccepts (TIMESTAMP,TIMESTAMP WITH TIME ZONE,TIMESTAMPTZ,TIMESTAMP_S,TIMESTAMP_MS,TIMESTAMP_NS,TIMESTAMPTZ_NS,DATE,TIME). Integer, date-as-integer and opaque string cursors keep the plain comparison, becauseepoch_usraisesBinderExceptionon them. The choice is made from the column's type, never from the cursor text's shape: aVARCHARcolumn of timestamp-looking strings yields a cursor indistinguishable from a real timestamp's. - Pinning the session time zone to UTC would also have worked, and is deliberately not used.
TimeZoneis an ICU-backed setting, the DuckDB sandbox forbids autoloading extensions, and where ICU is absent DuckDB does not recognise the option at all — so a pin would fail the connection in exactly the hardened environments the sandbox exists for. - The accepted cost: wrapping the column in a function can stop the predicate being pushed into the Parquet or JSON reader, so a large source scans more than it would have.
A pipeline discovered on 1.5 failed SchemaDriftError on 2.0. The drift gate compared type names, and 2.0 infers HUGEINT where 1.5 infers DOUBLE for the same integer literal, and TIMESTAMP WITH TIME ZONE where 1.5 infers TIMESTAMP for the same JSON timestamp. Upgrading DuckDB — no contract edit, no new data — broke a working pipeline on a false positive: the inferrer drifted, the data did not. Those pairs are now treated as one physical type under two names.
- The widening and narrowing classification is unchanged. The family check runs after it, so
type_widenedandtype_narrowedstill fire for every ordered numeric pair. DOUBLEandHUGEINTare not value-identical: for the same literal 1.5 landed a lossy 17-significant-digit float where 2.0 lands the exact 128-bit integer. So the non-failing path says the physical type moved rather than passing silently, with a line that ends "the same physical type under two names ... The stored width does change, so a value's precision may differ between the two."
fluid init --discover emitted contracts its own fluid validate rejected. The emitter wrote the source's type name straight through, so a discover on DuckDB 2.0 produced type: hugeint and type: '"null"', and fluid validate then exited 1 on the file the tool had just written. Validating every name in DuckDB's own duckdb_types() against the bundled schema showed 35 rejected names, not the two reported: uinteger and integer[] were rejected on 1.5 as well, so this was never only a 2.0 problem. Discovered types are now canonicalised at the emitter, each rejected name mapping to the nearest one the schema accepts and every already-valid name left untouched. An unsigned or 128-bit type maps to a width that holds its range (uinteger → bigint, hugeint → numeric) rather than to the similar-sounding name. A contract already emitted with an invalid type stays invalid — the fix is at emit time — but such a contract already failed validation, so nothing working changes.
Background: DuckDB sandbox for contract SQL, Source-aligned acquisition.
Changes in 0.23.0
0.23.0 changes what forge exports, fixes three DuckDB 2.0 defects, and stops a contract from choosing where an environment variable is sent.
| Change | What to do | How to check |
|---|---|---|
| Exported ODCS and ODPS documents change shape (forge-cli #753) | Read Read this first if you consume the exported artifacts before you upgrade | See that section's table |
Three DuckDB 2.0 defects are fixed: an incremental cursor written on 1.5 and read on 2.0 dropped or duplicated rows, a pipeline discovered on 1.5 failed SchemaDriftError on 2.0, and fluid init --discover wrote types fluid validate rejected (forge-cli #749) | Upgrade to 0.23.0 before you move an incremental build across DuckDB versions. See The two drift defects, and what fixes them | See that section |
A contract that uses env:// in connection.secretRef against a remote host fails until the operator allows it (forge-cli #751). All six acquisition runners (airbyte, debezium, dlt, duckdb, kafka_connect, meltano) refuse to resolve the reference unless every destination the connection block carries is loopback | Set FLUID_ACQUISITION_SECRET_REF_ALLOWLIST to the variable's own name: PG_PASSWORD@db.example.com for that host only, or PG_PASSWORD for any host. Write the host without a port. Set it in the shell, in CI or in a FLUID_SECRETS_FILE outside the project, not in a project .env, where it is ignored. localhost, 127.0.0.0/8 and ::1 need nothing. For a production credential, use vault://, aws://, gcp://, azure:// or file://, which are not policed. See Environment variables | The build fails before the secret is read, with a refusal that names the variable and the host, never the value |
fluid init --discover against a remote host writes a contract with env:// | Set the same allowlist before fluid apply. The discover hint does not print it yet: the refusal from apply does | fluid apply refuses the contract until the variable is allowlisted |
An acquisition source no longer receives this host's ambient database credentials: PGPASSWORD, PGPASSFILE, PGSERVICE, MYSQL_PWD and ten more variables are hidden while a postgres or mysql source is attached (forge-cli #750). A source whose declared connection carries its own credentials is unaffected | Put the credential on the connection (secretRef or an inline password) | A build that took its password from PGPASSWORD or MYSQL_PWD alone now fails to authenticate |
Changes in 0.24.0
0.24.0 fixes three more exports: lineage, versions and quality SQL (forge-cli #756). Exported documents change shape again, so read the 0.24.0 rows of Read this first if you consume the exported artifacts before you upgrade:
- An ODPS export of a contract with
consumes[]gainsoutputPorts[].inputContracts, and an input port'sversionmay differ from1.0.0. - An ODCS
sqlquality rule becomes a counting query withmustBe: 0.
Nothing else in this release asks you to act. consumes[].contractVersion is new and optional (0.7.6): the exact upstream version a product was built against, within versionConstraint. It wins over the lowest version of the range as the version an exported input port claims.
Changes in 0.25.0 and 0.26.0
0.25.0 added contract fragments ($ref). 0.26.0 made schema 0.7.6 stable and added the Databricks provider. Each item names the pull request that made the change; the forge-cli CHANGELOG carries the full upgrade notes of both releases.
0.25.0
- A
$refthat 0.24.0 left in the contract now composes or fails (forge-cli #760). A ref with keys beside it, a same-file#/pointer, or a ref inside an overlay used to stay a literal key. It now resolves, and one that cannot be resolved fails every command, naming the file and the place. A contract that embeds a document with its own$refkeys (a JSON Schema or OpenAPI fragment insideparametersor an extension) now fails too: keep such a payload as a string. - Contracts that use the new forms need 0.25.0 or later. Text refs, list-root fragments, keys beside a
$ref, pointers, refs inside overlays and a workspaceref_rootfail on 0.24.0, or load differently there. Pindata-product-forge>=0.25.0wherever such contracts are validated or planned. - Overlays are held to the ref root, and
--envnames an environment. Anoverlays/directory or overlay file that links outside the contract's tree is refused: widen the root withref_rootinfluid.workspace.yaml, orFLUID_REF_ROOT, to share one.--env ../xand--env /abs/xare refused by every command. - A root contract must be a regular file of at most 5 MiB, and a composed contract nests at most 128 levels. A contract read through a pipe (
fluid validate <(git show HEAD:contract.fluid.yaml)) is refused: write it to a file, or move its large parts into fragments. fluid splitkeeps the root's text and comments instead of re-dumping it, and writes no file through a symlink. The fragments are the same as before.- A contract whose orchestrator is on a public host fails until the operator allows it (forge-cli #758). Set
FLUID_ACQUISITION_DEPLOYMENT_HOST_ALLOWLISTto the host or its parent domain, without a port, in the shell, in CI, or in aFLUID_SECRETS_FILEoutside the project. A project.envcannot set it. Deployments onlocalhost, loopback or a private network need nothing.
0.26.0
- Schema 0.7.6 is stable, and 0.7.7 is the preview (forge-cli #768). What forge-cli writes now declares
0.7.6: the importers,fluid product-new,fluid init --discoverand thefluid forge --scaffoldtemplates. Runningfluid importagain over a contract it wrote changes itsfluidVersionfrom0.7.3to0.7.6. A0.7.6contract merges an overlay'sbuilds,exposesandconsumesby id, so an overlay entry that names no id fails to load where the base list has more than one entry, or where another overlay entry names an id. fluid bundleno longer writescontract.bundled.fluid.yamlon its own (forge-cli #763). Without--out,yamlandjsongo to stdout and atgzto<contract id>.fluid.bundle.tgz. A script that read that file must pass--out contract.bundled.fluid.yaml. Delete acontract.bundled.fluid.yamlan earlier run left beside a contract: fromfluid shipit is a tarball, fromfluid bundlea copy that may be stale.fluid applychecks every platform the contract declares (forge-cli #766). It stops withapply_platform_unsupported(exit 2) where a platform has no provider;provider_unknownnow means a--providervalue no provider has. A contract with an expose on such a platform beside the exposes that chose the provider used to apply with exit 0, and now stops: pass--providerto apply it as before. With thelocalprovider, a later build declared for a platform no provider serves (composer,kubernetes, ...) now stops the run: declare it as the job it is, or drop the platform.- A contract can no longer name the host a dbt build's credential goes to (forge-cli #773). A dbt build on Databricks, Redshift or Postgres that sets
builds[].execution.runtime.resources.hostnow fails before dbt starts, withDbtProfileErrornaming the field. Remove it and setDATABRICKS_HOST,REDSHIFT_HOSTorPGHOST. A Databricks catalog or schema must be a plain identifier, or come from a variable whose name ends in_CATALOGor_SCHEMA; or leave the field out and setDATABRICKS_CATALOG/DATABRICKS_SCHEMA. - A dbt build with no
execution.runtime.platformtakes its platform from the first expose's binding, for the generatedprofiles.ymland the model-contract column types. It used to get a DuckDB profile whatever the binding said. - A dbt build on Databricks runs in the workspace (forge-cli #774). With
execution.runtime.platform: databricks,fluid applyprovisions it as a job through OpenTofu, and--mode amend-and-buildno longer runs dbt on this machine. See Databricks. fluid auth logout databricksremoves only the profile you name (forge-cli #772):fluid auth logout databricks <profile>. It never deletes~/.databrickscfg. With no profile named, it leaves the file alone, lists the profiles and exits 1.- ODCS and ODPS round trips: an ODCS export records
fluidVersioninfluidExtraswhatever the version, andfluid odcs import/fluid odps importof a document that records no version declare0.7.6. Re-importing such a document writes a different file.
Changes in 0.27.0
These changes shipped in 0.27.0. Most fail closed where a contract used to apply without them, so read them before upgrading from 0.26.0 or earlier. The Databricks ones come first; see Databricks. The OpenTofu, AWS and GCP ones follow, then fluid diff --baseline.
- A Databricks contract with a
sovereigntyblock and nobinding.location.regionis now a finding (forge-cli #778): an error underenforcementMode: strict, a warning underadvisory.fluid applyandfluid generate iacnow hold the policy, and refuse a forbidden placement withSovereigntyViolationError. Setbinding.location.regionto the region of the workspaceDATABRICKS_HOSTnames. It was refused as a field the module would drop; it is read now. Write it as a plain lower-case name (databricks-region-invalidotherwise), and declare one region per contract (databricks-region-conflict).tofu planfails when the workspace's metastore is in another region. - A contract's labels become Unity Catalog tags (#779). They were dropped silently. A key's
-becomes_. A contract whose labels collide once-is read as_(databricks-tag-collision), or that puts more than 50 tags on one schema or table (databricks-tags-too-many), is now refused.fluid verifychecks the tags. fluid publishsendsmetadata.workspace_hostfor a Databricks product (#780), fromDATABRICKS_HOST. Nothing to do.- Column restrictions and row filters are accepted on Databricks (#785).
policy.authz.columnRestrictionsandrowFilterswere refused (databricks-governance-unmapped).fluid applynow binds them on the table in Unity Catalog, after creatingfluid_-prefixed SQL functions, and needs the[databricks]extra andDATABRICKS_WAREHOUSE_IDto do it. A denied column readsNULL. New refusals:access: allow, a text mask on a non-string column, a principal with nogroup:,user:orserviceAccount:prefix, and a predicate outside the allowlist.fluid generate iacnow refuses such a contract (databricks-enforcement-outside-module): usefluid apply. A dbt job that writes a table not yet protected is created paused, and the nextfluid applyunpauses it. - Databricks retention, bronze landing and scoped grants (forge-cli #801, since 0.27.0).
lifecycle.expirewas refused (databricks-retention-unmapped). It is accepted now for a table rebuilt in full on every run, whenlifecycle.retentioncovers the Delta history (30 days by default);fluid verifychecks it. A DuckDB acquisition that writes a Databricks expose now uploads Parquet to the volumeDATABRICKS_STAGING_VOLUMEnames and loads the table, where it wrote a local file; it needsDATABRICKS_WAREHOUSE_IDtoo, and refuses a source mode other thanfull_refresh(databricks-landing-unsupported).policy.privacy.maskingis accepted for such an expose. AnaccessPolicygrant withresourceswas refused whatever it named (databricks-grant-scope-unmapped). The three shapes forge-cli can evaluate are placed now, only on the tables they name; any other path is still refused. A supersededfluid_function is dropped after a successful apply. See Retention and Landing a DuckDB acquisition. - S3 state is locked (forge-cli #787), when the installed OpenTofu is 1.10 or newer. Before you upgrade, give every role that runs
fluid apply(a--dry-runincluded),fluid difforfluid verify --state-driftagainst S3 states3:GetObject,s3:PutObjectands3:DeleteObjecton<key>.tflock, or setFLUID_TOFU_S3_LOCKFILE=0in the job's environment (a project.envcannot set it). A plan-only CI role fails without one of the two. A run killed hard can leave the lock behind:tofu force-unlock <ID>in the contract's workdir clears it. An older OpenTofu logss3_state_unlockedand keeps working, unlocked. See Locking. - A column type Glue or BigQuery cannot hold is refused (#793). A column typed
array,struct,map,object,recordorrowfailsfluid validate,fluid generate iacandfluid applywithcolumn-type; declare itjsonorstring, or, on AWS with a 0.7.7 contract, declare the column'sfields,element, orkeyandvalue(see Nested columns). On Glue,geography,geometry,geom,point,hll, a decimal Hive rejects andbit(n)over one bit are refused too. Some types are still written with a fallback and now warn, sofluid validate --strictfails until you declare the type you mean: on Gluejson,jsonb,variant,super,time,intervalandbignumeric(written asstring), on BigQuerygeometry,geomand a decimal wider than aNUMERICholds. The first apply on a Glue table an earlier release created changes the types of columns that werestring(smallint,timestamptzand the rest now get their Hive type), andfluid applyrefuses until--allow-data-loss: Parquet written under the old schema may stop reading until it is rewritten, and reverting the column type restores reads. Until that apply,fluid verifyandfluid diffreport the mismatch. See AWS column types and BigQuery column types. - An AWS schedule is no longer dropped silently (#794). A dotted or long contract id gets a valid name, with a hash when it had to change. Any AWS plan the plan validator refuses now fails
fluid generate iacandfluid applyeven for a contract with no schedule, where it used to emit the module and exit0;fluid planalready refused these, so run it before you upgrade. A plan that emits the scheduled Lambda warnsschedule_placeholder_lambda: the function only logs its tasks, and the rolesfluid-workflow-executionandEventBridgeSchedulerRolemust already exist. See Schedule names and refusals. - A build mode refuses an id the build phase cannot use, before anything is applied (#803).
fluid apply --mode amend-and-buildandreplace-and-buildcheckidand everybuilds[].idbefore the apply hooks,tofuand any build. Each id must start with a letter or_, use only letters, digits,_,.and-, and be at most 128 characters. Such an id (1orders, or a longer one) passesfluid validate. On the OpenTofu engine the product used to be provisioned and the run then stopped with "Unexpected error". Now the run stops first withapply_build_identifier_invalid. Rename the id. A renamedidgets a new.fluid/iac/<provider>/<id>workdir, and its builds start without their cursors and watermarks. See Build ids. - A build mode refuses a
--build-idthat names no build, before anything is applied (#808).fluid apply --mode amend-and-build --build-id <id>(andreplace-and-build) checks the id against the contract'sbuilds[].idbefore the apply hooks,tofuand any build. Ids match exactly, case and spaces included. On the OpenTofu engine a mistyped id used to provision the product, and the run then stopped withBuild not found. Now it stops first withapply_build_id_not_found, which lists the contract's build ids. The exit code is still1. A Command Center run records the eventapply_build_id_not_foundwhere it recordedbuild_not_found:<id>. See Build ids. fluid validateandfluid planwarn about an id the build phase refuses (#813). For a contract withbuilds, anidorbuilds[].idthat starts with a digit or is longer than 128 characters is a validate warning, sofluid validate --strictfails on it.fluid planprints it and still writes the plan. Rename the id:fluid applyin a build mode refuses it anyway. See An id the build phase refuses.fluid doctor --strictand a readable failed apply (#786).fluid doctor --strictexits1when the summary says "Action needed", copilot readiness included; the default exit code does not change. A failed OpenTofu apply names the first error, its resource and the provider's detail, where it used to print the raw tail oftofu's-jsonstream; a script that matched that text needs the new one. Seefluid doctorand A failed OpenTofu apply.fluid diff --baselinecompares a 0.7.x contract's columns (forge-cli #789). It read onlyexposes[].schema, so a removed column, a type change that is not a widening and an addedrequired: truecolumn passed--fail-on-breaking. A pipeline that runs it can start exiting 1. See Contract version diff.fluid diff --baselinerefuses a contract its schema does not accept (forge-cli #805): exit 2,diff_contract_invalid. Runfluid validateon both files first. A pre-0.7 baseline is refused too. It also reports more breaking changes in a 0.7.x contract: a quality rule whose severity becomeserrororcritical, and a key column (aprimary-keytag,semanticType: identifier, ...) that stops beingrequired. Asensitivitychange is an info finding. A bot that parses--format markdownnow finds the path and the description each in a code span. See Both contracts must be valid.- A provider plugin that replaces a built-in provider now needs the operator's grant (forge-cli #812). A
fluid_build.providersentry point that claimsaws,gcp,snowflake,databricksorlocalis skipped without being imported unlessFLUID_PLUGINS_ALLOWLISTgrants it, for exampleacme-fluid-gcp:gcp.fluid providersnow lists providers without importing them, andazureis no longer accepted as a provider name anywhere. - An IaC plugin that replaces a built-in platform now needs the operator's grant (forge-cli #807). A package on the
fluid_build.iac_providersentry point is imported only when a contract'sbinding.platformneeds it. An entry point that claimsaws,gcp,snowflake,confluent,databricks, an alias such asglueorbigquery, orlocalis skipped without being imported, with a WARNING. To keep an override you rely on, add a distribution-qualified entry toFLUID_PLUGINS_ALLOWLIST, for exampleacme-fluid-aws:aws; it restricts no other plugin.redshiftanddatamesh-managercan never be granted, and two distributions that ship the same platform name are both refused. Allow and block names now match case-insensitively with-equal to_. The global--providerflag accepts built-in providers only; a plugin platform is reached throughbinding.platform. See Shipping an IaC plugin. - Redirects and CA bundles (forge-cli #804). The Data Mesh Manager client refuses a redirect to another origin instead of forwarding its API key; the error names the new origin and says to update
DMM_API_URL. AREQUESTS_CA_BUNDLEorCURL_CA_BUNDLEthat names a path that cannot be loaded switches the Command Center reporter off for the run with a warning; TLS is never relaxed. Seefluid datamesh-managerand Command Center reporting. redshiftanddatamesh-managerare no longer providers.--providerorFLUID_PROVIDERnaming either one now exits 2. Forredshiftthe message points to--provider aws; for Data Mesh Manager, tofluid publish --target datamesh-manager. A contract'sruntime.platform: redshiftis served byaws, so it needs no change. A plugin cannot claim either name. See Retired provider names.- A Kafka Connect or embedded Debezium Iceberg sink's
catalogAuthneeds the operator's say (forge-cli #818). A sink whosecatalogAuthreaches a non-loopback host fails its run until the operator pairs the variable with the host asNAME@hostinFLUID_ACQUISITION_SECRET_REF_ALLOWLISTand, for a hostname, lists it inFLUID_ACQUISITION_DEPLOYMENT_HOST_ALLOWLIST. A catalog or token endpoint onhttp://, a loopback host included, failsfluid validate; move it tohttps://.fluid validatewarns, with theexportline the run needs, when only an entry is missing. A derived sink whosebinding.locationholds text that starts withenv:, or holds${or a control character, or aregionthat is not one label, now failsfluid validateand the run,catalogAuthor not; write the value itself. See Operator allowlists. - A value that ends in a line break is refused (forge-cli #797, #799, #811). Whole-string validators and schema
patterns no longer let a trailing newline through, which anid: |block scalar, a.envvalue or a%0Ain a URL produces.fluid validaterefusesid: "orders\n"under every bundled schema version, with the hint(it ends in a line break, which a YAML | or > block keeps and |- or >- drops); write the value with|-or>-, or plainly. Strip the value where an environment variable or file sets it. An AWS binding region with whitespace around it is refused (aws-region-whitespace). See A value that ends in a line break. - Cloud-metadata addresses are refused even when listed (forge-cli #823). The acquisition orchestrator guard (kafka_connect, debezium, airbyte) and the OpenLineage emitter now refuse
fd00:ec2::254,fd00:ec2::23,fd20:ce::254,fd00:c1::a9fe:a9feand100.100.100.200, and any IPv4-mapped or NAT64 address that carries a link-local or metadata address, where they allowed them like any private host. A Tailscale tailnet or an IPv6 cluster on ULA addresses needs no change. See Network safety. - The orchestrator guard judges the address the connection is pinned to (forge-cli #814). For kafka_connect, debezium and airbyte, the check on a contract-named
deployment.server_urlno longer resolves the hostname a second time. The name still decides the allowlist and the pinned address decides public or private; a link-local address is refused even for a listed name, and the refusal names both, as inhost 'connect.example.com', connecting to '93.184.216.34', is public. A URL it cannot parse is refused with "it has no host".
Extras in 0.27.0
(since 0.27.0) Four extras were added next to the base install, so you could switch before 0.28.0 took their packages out of it. Changes in 0.28.0 says what that change does.
| Extra | Installs | For |
|---|---|---|
litellm, alias ai | litellm | The AI backend of fluid forge, fluid forge data-model with an API-key provider (`--llm-provider openai |
mcp | mcp | The MCP server SDK: fluid mcp serve, MCP output ports, fluid scaffold-ide and the keyless mcp-sampling provider |
lineage | openlineage-python | Lineage events when OPENLINEAGE_URL is set |
standard | litellm and mcp | The AI and MCP behaviour a base install had through 0.27.0 |
The extras that already existed are all, aws, catalogs, copilot, databricks, datahub, dev, docs, gcp, local, odcs-strict, odps, opds, postgres, sdk, snowflake, test-emulators, test-emulators-heavy, tui, vector and viz. Since 0.28.0 viz, tui and odps are empty aliases: nothing imports graphviz, textual or ruamel.yaml, and pip install 'data-product-forge[viz]' still works.
(since 0.27.0) When an optional package is missing, the CLI prints a message naming the feature, the package and the extra, then exits 1 with no traceback; see Production troubleshooting. fluid doctor reads the litellm version from package metadata instead of importing it.
Changes in 0.28.0
These changes shipped in 0.28.0. The first is a breaking change to the install, not to any contract or command: read it before you move a pipeline, a Docker image or a CI install line. After it come the pins that changed with it, the dependency list, the Debezium .env names, the Iceberg catalogAuth checks, Databricks and the smaller changes.
The base install
(since 0.28.0, forge-cli #837) litellm, mcp and openlineage-python are no longer in the base install; with litellm went boto3, aiohttp, openai, tokenizers and about fifty more packages. Counted with uv pip compile for a base install with no extras, the number of packages goes from 88 to 41 (Linux, Python 3.12), from 86 to 45 (Linux, Python 3.10) and from 89 to 40 (Windows, Python 3.12); litellm alone was 30 of them. packaging is now a declared base dependency, since it used to arrive through litellm.
The base install keeps the core pipeline: validate, plan, apply, verify, publish, generate iac, bundle, diff, doctor, and template or blank forge. If that is all you run, you need nothing.
If you use AI forge, MCP or lineage, name the extra. These install lines also work on 0.27.0, where they change nothing.
pip install "data-product-forge[litellm]" # AI forge; [ai] is the same extra
pip install "data-product-forge[mcp]"
pip install "data-product-forge[lineage]"
pip install "data-product-forge[standard]" # litellm and mcp: the pre-0.28.0 AI and MCP behaviour
| You run | You need |
|---|---|
fluid forge with an API-key provider (--llm-provider openai, anthropic, gemini, ollama), fluid forge data-model with an API-key provider (`--llm-provider openai | anthropic |
fluid mcp serve, fluid mcp output-port serve, fluid scaffold-ide | mcp |
OPENLINEAGE_URL set on fluid apply through the OpenTofu engine, or on an acquisition build | lineage |
A native AWS plan that must resolve the account id (a contract that declares orchestration), live AWS reads, the Glue catalog import blocks of apply | aws |
litellm and mcp together, the AI and MCP packages a 0.27.0 base install had | standard |
[dev] and [all] include all three. The Docker image profiles already install [standard] and [lineage].
The CLI stops before it does any work. An explicit --llm-provider naming an API-key provider (fluid forge, fluid forge data-model, --agent-loop), the fluid ai setup wizard and the MCP commands (mcp serve, mcp output-port serve, scaffold-ide) print the exact pip install line and exit 1, with no traceback. --llm-model alone resolves to the provider saved with fluid ai setup, and to ollama when none is saved. For AI forge the check runs before the interview reads any context or memory, and before the fluid ai setup wizard reads the OS keyring or asks for a key. The forge message names --llm-provider claude-code (your Claude Code login) and --template starter; the --agent-loop message names only --template starter; ai setup and init name claude-code, codex, cursor, kiro, mcp-sampling and --template starter. Only Claude Code needs no setup of its own: codex, cursor and kiro reuse their own CLI login or key, and mcp-sampling runs inside fluid mcp serve, which needs [mcp]. fluid forge data-model with no provider flags runs offline. fluid init does not exit 1 for a missing extra: its AI hand-off prints the line and the options, then continues in template mode (it picks Claude Code if claude is on PATH). Since 0.30.0 it does exit 1 in one case: with litellm installed and keys for two or more providers set, with no provider named, the AI path stops instead of writing a template (forge-cli #857; see Changes in 0.30.0). Bare interactive fluid forge with no provider uses Claude Code when it is installed, and otherwise prints the hint and offers guided mode. Template and blank forge are unchanged. fluid scaffold-ide writes nothing without mcp, because every entry it writes launches fluid mcp serve.
❌ The MCP server needs mcp, which is not installed.
pip install 'data-product-forge[mcp]'
[ERR_MCP_SDK_UNAVAILABLE]
Lineage is the one that does not stop. Lineage is emitted only by fluid apply through the OpenTofu engine and by acquisition builds; a native local apply never emits it. Without lineage, those print one warning naming the extra on the console (and log it) and carry on without sending events.
AWS needs aws. boto3 used to arrive through litellm. Without it, a native AWS plan that cannot resolve the account id fails with The AWS account id could not be resolved: boto3 is not installed, so sts:GetCallerIdentity cannot be called, followed by the install line, instead of the misleading bucket-name error it used to give. Only a contract that declares orchestration (orchestration.schedule, triggers or tasks) reaches that plan; other contracts skip it. fluid apply without boto3 instead warns that the Glue catalog import blocks are omitted. Setting AWS_ACCOUNT_ID avoids the lookup.
fluid doctor no longer reports a configured provider as ready when litellm is missing; its Next steps panel names the extra.
Before you move, put the extra in FLUID_PACKAGE_SPEC in CI, or in your own install step.
Pins that changed with it
(since 0.28.0, forge-cli #837) The mcp extra now needs mcp>=1.30,!=2.0.*,!=2.1.*,<3.0 (it was mcp>=1.22), because earlier releases have published advisories, among them an SSE session that was not bound to the principal that opened it; fluid mcp output-port serve uses that transport. The litellm extra excludes litellm 1.107.0, a malicious release since removed from PyPI (MAL-2026-2144).
The dependency list now matches the imports
(since 0.28.0, forge-cli #829) typing-extensions and the Windows-only colorama leave the base install, because nothing imports them (pydantic and click still bring them where needed). anyio is named in it. The [gcp] extra drops google-auth-oauthlib and google-cloud-iam. The GCP project-access probe, which is the one user of google-api-python-client (still not in [gcp]), reports "not checked" when that package is absent, where it used to report "no access". fluid odcs validate --roundtrip needs deepdiff, which now belongs to the [odcs-strict] extra; a missing one says so.
A project .env can no longer configure the embedded Debezium Server
(since 0.28.0, forge-cli #833) If you set any of these in a project .env, .env.<environment> or .env.local, move it to the shell or the CI job: it is ignored now, with the usual warning. The names are DEBEZIUM_*, QUARKUS_*, SMALLRYE_* and MP_CONFIG_* in the forms SmallRye Config matches, JDK_JAVA_OPTIONS, JAVA_OPTS, JAVA_HOME, and the launcher's EXTRA_CONNECTOR, JMX_HOST, JMX_PORT, JMX_EXPORTER_PORT, JMX_EXPORTER_CONFIG and OTEL_JAVAAGENT_*. See Project dotenv trust.
Iceberg sink catalogAuth checks are stricter
(since 0.28.0, forge-cli #828 and #830) On a Kafka Connect or embedded Debezium build with sink.catalogAuth, fluid validate and the run's preflight now refuse a catalog or token endpoint at a cloud metadata address (AWS, GCP, Oracle and Alibaba, IPv6 and NAT64 forms included) even when you list it, and a catalog URL that holds ${, starts with env: or holds a control character, wherever it is set (binding.location.uri, iceberg_catalog_overrides, the iceberg.* keys of sink_connector_config except iceberg.kafka.*, server.sink.config). A build without catalogAuth is unchanged. See Operator allowlists and Network safety.
Databricks tables get tags and comments
(since 0.28.0, forge-cli #831) fluid apply writes the table comment and column comments from the contract (on a table a dbt build writes, the comments stay dbt's), and every Databricks table gets the product's identity tags (fluid_product_id, fluid_layer, fluid_owner_team and others), which count toward Unity Catalog's 50 tags per securable together with your labels. A contract that used to apply may now be refused: a label that becomes one of those keys (databricks-tag-collision), a tag value over 256 characters (databricks-tag-value-too-long), more than 1,000 column tags on a table (databricks-tags-too-many), and a description holding a backslash or ${ (databricks-comment-unsafe). A table is certified through Unity Catalog's system.certification_status: certified when the label certification: certified is set, deprecated when lifecycle.state is deprecated; any other value of the label is refused (databricks-certification-invalid). Setting it needs the ASSIGN permission on that governed tag for the deploy identity (from Databricks' documentation; not verified on a live workspace), and the plan names the tables. After a load, (since 0.28.0, forge-cli #835) the bronze landing deletes the local masked Parquet copy under .fluid/staging/<build>/ and fails the run if it cannot. See Databricks.
Other changes
- (since 0.28.0, forge-cli #840) For a local-provider deployment,
fluid publishreports a relativelocation.pathanchored at the contract's directory, so the Command Center gets the file the engine writes. The contract itself is unchanged. - (since 0.28.0, forge-cli #835)
fluid generate iacprints the reason, fix and doc of a sovereignty refusal, asfluid applydoes. - (since 0.28.0, forge-cli #837) The CLI prints install hints as written, and masks GitHub, Anthropic and OpenAI keys, JWTs and
password=...pairs in the error text it prints. - (since 0.28.0, forge-cli #836)
fluid --helpand each command import only the module of the command being run: about 300 modules on a bare install instead of about 857. The command list and every help line are unchanged. - (since 0.28.0, forge-cli #832) Nine internal modules that nothing imported are removed:
fluid_build.logging_utils,fluid_build.cli.init_samples,fluid_build.copilot.schemas.table_metadata,fluid_build.providers.local.ducksql,.mocksand.samples, andfluid_build.runtimes.dataform_gcp,.dbt_bigqueryand.dbt_duckdb. No command or contract behaves differently. It matters only to code that imported one of them directly.
Changes in 0.29.0
These changes to AI authoring can change what a script, a CI job or an IDE agent sees.
Which provider and which key
- (since 0.29.0, forge-cli #850) A run with nothing configured and no terminal picks an LLM. It uses a running local Ollama, else the first coding-agent CLI on
PATH, where it used to fall back togemini. Name the provider to keep a job from picking one. See When nothing is configured. - (since 0.29.0, forge-cli #852) No
geminifallback. When nothing selects a provider, forge stops withcopilot_no_llm_providerand says why. It used to fall back togemini, and sentFLUID_LLM_API_KEYthere whatever key it was. Pass--llm-provider, setFLUID_LLM_PROVIDERor the key of exactly one provider, or save a provider withfluid ai setup. See No LLM provider is selected. - (since 0.29.0, forge-cli #854) Runs without a terminal use the provider
fluid ai setupsaved, with its model and Ollama host, andfluid doctorwalks the same ladder. A saved coding agent is passed over when a run states an endpoint or a model, or runs the agent loop. See Which provider a run uses. - (since 0.29.0, forge-cli #853)
FLUID_LLM_API_KEYgoes only to a provider you name. A job that used it with a provider forge picked, such as a running Ollama or the provider of the one key variable set, now sends that provider's own key, or none, and forge warns. Name the provider:--llm-provider <name>orFLUID_LLM_PROVIDER=<name>. See Which key a provider gets. - (since 0.29.0, forge-cli #853) A project
.envendpoint for OpenAI, Anthropic, Gemini or GitHub Models needs the provider's own key beside it.FLUID_LLM_API_KEYno longer pairs it, and interactive setup no longer copies your saved key over the project's key. See LLM endpoints in a project file. - (since 0.29.0, forge-cli #850)
claude -pno longer reads your Claude Code settings. Authentication kept there (anapiKeyHelper, or Bedrock or Vertex variables insettings.json) must come from the shell environment. See Claude Code.
Ollama
- (since 0.29.0, forge-cli #850) Ollama calls reach the daemon. They go to
--llm-endpoint, elseOLLAMA_HOST, elselocalhost:11434, and a remoteOLLAMA_HOSTis honoured. See The Ollama endpoint. - (since 0.29.0, forge-cli #852)
OLLAMA_HOSTmay name any machine everywhere. The probe for a running Ollama, the default-model pick andfluid ai setupno longer swap a non-local host forlocalhost, andfluid ai test --endpointtakes an Ollama address on any host. - (since 0.29.0, forge-cli #852) Forge looks for Ollama models where it sends the call. With
--llm-endpointorFLUID_LLM_ENDPOINT, the default-model pick and the model check ask that daemon, and each--tieredstage calls it. Auser:password@in the address goes as basic auth to the call and to the model lookups. See One address for every Ollama read.
fluid forge --agent, fluid ai test and provider code
- (since 0.29.0, forge-cli #850)
fluid forge --agenthonours the AI flags. With--llm-provider,--llm-model,--llm-endpointor--context, it authors with forge's AI instead of writing the blank TODO contract, and a failed generation exits non-zero. See Headless agent mode. - (since 0.29.0, forge-cli #852) In AI mode,
fluid forge --agentreports the contract for any--target-dir. See Headless agent mode. - (since 0.29.0, forge-cli #854)
fluid forge --agentstdout is JSON Lines only. Panels, other output and the missing-extra message go to stderr. - (since 0.29.0, forge-cli #850)
fluid ai testprobes the real call. A rejected key exits3, a model the provider does not have exits4, and other failed calls exit5. Seeai test. - (since 0.29.0, forge-cli #852)
LlmProvider.list_available_modelstakes anendpointkeyword, the address the call will go to. Forge passes it only when an endpoint is stated, so code that subclassesLlmProviderand overrides the method with two arguments keeps working until then. With an endpoint stated and the model check on (--require-llm,--tieredorFLUID_LLM_MODEL_PREFLIGHT), the check fails withcopilot_llm_model_preflight_failed; add the keyword to avoid it.
Changes in 0.30.0
These changes shipped in 0.30.0. The first can stop a plan that used to pass: read it before you upgrade a pipeline.
fluid plan checks sovereignty by default
(forge-cli #868, since 0.30.0) fluid plan now refuses a contract that fluid validate refuses for its sovereignty policy. It exits 1 with ERR_SOVEREIGNTY_VIOLATION and writes no plan. Up to 0.29.0 plain fluid plan wrote a plan for such a contract and exited 0; only --check-sovereignty checked. Under enforcementMode: strict, the schema default, the refused contracts include one with a binding outside allowedRegions, in another jurisdiction, or on AWS, GCP, Azure or Databricks with no region declared. In every mode they include one with a binding in a region listed in deniedRegions.
- A plan step can now fail. Before you upgrade, run
fluid validateon each contract: the sovereignty errors it reports are the onesplannow refuses. To write the plan anyway, pass--no-check-sovereignty. It prints a warning and logs a WARNING (sovereignty_check_skipped), andfluid applykeeps its own checks. - A failing plan stage writes no plan file. The pipelines
fluid generate ciwrites already pass--check-sovereignty. On 0.29.0 a failing stage still wroteplan.json; now it writes none, and a file an earlier run left at--outstays as it was. A provider hook that reports nothing no longer passes a contractfluid validaterefuses, because the same check runs first.
See Sovereignty gate.
Databricks
- (forge-cli #865, since 0.30.0) A Databricks expose with
semanticsnow gets a metric view, and the apply needsdatabricks-sdk.fluid applycreates<table>_semantic_modelbeside the table. Withoutpip install 'data-product-forge[databricks]', the apply is refused before anything is applied. Asemanticsblock the view cannot hold (a derived metric, say) now failsfluid validatefor a Databricks binding: express it as a ratio, or remove it. See Metric views. - (forge-cli #873, since 0.30.0) A schema the contract stops using is now destroyed, not left failing the apply. When a table moves to another schema or a schema's last expose is removed,
fluid applydrops the metric views and the mask and row filter functions this product made there, when the deploy identity owns them, so OpenTofu can destroy the schema. Anything else the schema holds is named and left.--mode dry-runlists the drops and runs none. See A schema the contract stops using. - (forge-cli #868, since 0.30.0) Unity Catalog tag values are written as the contract states them. A tag value is now the contract's text less surrounding whitespace. Up to 0.29.0 it was the GCP label slug: lower-cased, with each character other than a letter, a digit,
_or-turned into-, and cut at 63 characters. Sofluid_jurisdictionisUSwhere it wasus, andfluid_regulationisGDPR, CCPAwhere it wasgdpr_ccpa. Tag keys are unchanged, and so are GCP labels and AWS tags. See Governance tags.- The next
fluid applyupdates the tags in place. The plan shows each changed tag as~. Nothing is replaced or destroyed, and no--allow-data-lossis needed. Until that apply,fluid verifyreports each old value as a changed tag (CRITICAL). On Databricks Free Edition, a product carrying the old values planned+0 ~17 -0. - A value over 256 characters is refused as
databricks-tag-value-too-longbyfluid validate,fluid generate iacandfluid apply. Up to 0.29.0 it was cut to 63 characters. Shorten the value before you upgrade. - Anything that reads a tag value sees the new text, for example a query on
information_schema.table_tagsthat matches'us'.
- The next
- (since 0.30.0, forge-cli #861)
fluid apply --mode amend-and-buildon Databricks runs each dbt job build now, and waits for it. Up to 0.29.0 it only provisioned the job, so a manual job never ran. A pipeline that applies withamend-and-buildorreplace-and-buildnow waits for each run, up toFLUID_DATABRICKS_RUN_TIMEOUT_SECONDS(by default an hour, or the job's own limits plus 15 minutes), and fails when a run fails. It needsdatabricks-sdk: withoutpip install 'data-product-forge[databricks]', the apply is refused before anything is applied. To provision a job without running it, apply with--mode amend. See Running the job fromfluid apply. - (since 0.30.0, forge-cli #864) A Databricks expose with a freshness or completeness SLO turns on data quality monitoring at the next
fluid apply. A contract that already statesqos.freshnessSLO,qos.completenessTarget, anobservability.defaultSLIsfreshness or completeness block, or acontract.dq.rulesentry of typefreshness,completenessoranomaly_detectiongets adatabricks_data_quality_monitoron the schema it owns. Its scans run on serverless compute, billed asDATA_QUALITY_MONITORING. On Databricks Free Edition, expect them to count toward the daily compute quota (not measured), and a workspace over it stops all compute for the rest of the day. If Databricks refuses the monitor (it documents that monitoring needs serverless compute andMANAGEon the schema or its catalog), the apply now fails after the tables are created. To keep an expose unwatched, setcontract.dq.monitoring.enabled: falseon it. See Data quality monitoring. - (since 0.30.0, forge-cli #871) A Databricks dbt job runs the contract's
dq.rulesas dbt tests at the nextfluid apply, and a rule that cannot run is refused. Up to 0.29.0 the job ignored these rules, andfluid validateaccepted them whatever they said. The upload now gains the generated test files, one or more per rule, in afluid_dq/folder under the first of the project'stest-paths, anddbt buildruns each test right after its model. A rule of severitycriticalorerrorthat fails now fails the run, skips the models below it, and fails the--mode amend-and-buildapply that started it. Awarnorinforule is reported and does not. Before you upgrade, runfluid validateon each contract. On an expose a Databricks dbt job writes, it now refuses adrift_detectionrule, ananomaly_detectionrule on an expose with monitoring turned off, avalid_valuesrule with no allowed values, a rule that names a column the schema does not declare, a build whoserepositoryis a Git URL while an expose it writes has rules, and a project that already holds a test with a rule's name, or a file at the path of a generated test (<test path>/fluid_dq/<model>/<name>.sql). To keep a rule from failing runs, set itsseveritytowarn. See Contract rules as dbt tests.
Other changes
- (forge-cli #868, since 0.30.0) Logs and printed errors no longer mask a fully qualified name such as
catalog.schema.table:fluid applyprinteddatabricks.load table=***REDACTED***, and now prints the table's name. A JWT or JWE is still masked, recognised by its JOSE header (a first part that decodes to a JSON object).redact_secret_text()no longer takes thekeep_dotted_nameskeyword, so code that passes it gets aTypeError. - (forge-cli #857, since 0.30.0) Keys for two or more providers stop the run, even with Ollama running. With
OPENAI_API_KEYandANTHROPIC_API_KEYboth set (or one provider's key andOLLAMA_HOST) and nothing naming a provider, a run without a terminal used a running local Ollama, and interactive setup used the first key it found. Forge now stops withcopilot_no_llm_providerand names the keys, without asking Ollama.fluid ai testandfluid doctorsay the same, andfluid init's AI path exits 1 where it used to write a template. Afluid ai setupchoice and a single keyring key still come first, andGEMINI_API_KEYwithGOOGLE_API_KEYis one provider. Pick one with--llm-providerorFLUID_LLM_PROVIDER. See No LLM provider is selected.
Changes in 0.30.1
0.30.1 fixes one regression of 0.30.0. Use 0.30.1, not 0.30.0.
| Change | What to do | How to check |
|---|---|---|
fluid validate --strict no longer fails a dbt-built Databricks product that describes its tables. 0.30.0 warned on every Databricks expose a dbt build writes when the contract has a table or column description, because dbt's CREATE OR REPLACE sets the comments and fluid apply sets none. --strict turned that warning into an error, so stage 2 of the generated pipelines (VALIDATE_STRICT defaults to true) failed for every such product. The message is now a note with the same text | Upgrade to 0.30.1, not 0.30.0 | fluid validate --strict passes. The note is under "Validation Notes:", --format json lists it under notes, and a bundle reports it as an info issue |
Changes in 0.31.0
These changes are in 0.31.0. Several remove a command, flag, tool or variable: read them before you upgrade a pipeline.
(since 0.31.0, forge-cli #879)
fluid mcp output-port serve --transport httpspeaks MCP Streamable HTTP at/mcp. The HTTP+SSE endpoints/sseand/messages/are gone. The MCP specification deprecated HTTP+SSE in its 2025-03-26 revision. Point each client athttp://<host>:<port>/mcpand set its transport to Streamable HTTP,--transport httpinclaude mcp addfor example. A client still set up for SSE fails:/sseanswers 404 once a request passes auth, and at/mcpitsGETgets 400 (Missing session ID). Behind the shipped proxy templates the URL becomeshttps://<your domain>/mcp, and their settings for streamed responses stay as they are.- An MCP server without auth no longer listens beyond loopback. A
--hostother than127.0.0.0/8,::1orlocalhostnow stops at startup, with exit 2 and the fix, when auth is off: the defaultshared-tokenmode with noFLUID_MCP_AUTH_TOKEN,jwtwithout its three settings, orFLUID_MCP_AUTH_MODE=none. Up to 0.30.1 the output port warned and served. Off loopback, a shared token must also be at least 32 characters:openssl rand -hex 32makes one. - Without auth, a request's
Hostmust be127.0.0.1,localhostor[::1](or the bound address) with a port, or it gets 421. So a gateway behind a reverse proxy that forwards its public name needsFLUID_MCP_AUTH_TOKEN(orjwt) set on the gateway itself. - A browser request from a page not served from this machine gets 403, with or without auth: an
Originthat is nothttporhttpson a loopback host,Origin: nullincluded. With auth off, theOriginmust also be127.0.0.1,localhostor[::1](or the bound address) with a port. Clients that are not browsers send noOriginheader and are not affected. fluid mcp servegains--transport http, which serves the authoring tools the same way, by default athttp://127.0.0.1:8766/mcp. stdio stays the default. See Serve over HTTP.
(since 0.31.0, forge-cli #880)
fluid validate --strictfails on placeholder emails, starter-template values and TODO markers, on an SDP build that joins, and on an SDP that consumes a product whose type the workspace cannot read. Each is a new warning, sofluid validatewithout--strictstill passes. A CI job that runs--strict, stage 2 of a generated pipeline among them, now fails on a contract that still has an owner email atexample.comorcompany.com, aTODO, or the values of an unedited template: a contract scaffolded from a template (fluid init --quickstart,fluid init --template,fluid forge --template) has a placeholder owner email until you edit it. Replace each value with the real one, or delete an optional key such asmetadata.owner.email. See Placeholders and starter values and What an SDP does not do.fluid validate --format jsonaddsissues, with a code, a JSON pointer, a fix and an example for each finding. It is there only when there is a finding, anderrors,warningsandnotesare unchanged. A bundle'sCONTRACT-RULEfindings gainrule,pointer,fix,exampleanddocs_url. See Finding codes andissues.
(since 0.31.0, forge-cli #881)
fluid forgeAI mode runs on the four FLUID Agent Skills, and the embedded copilot is removed. With a coding agent and a terminal, forge opens the agent in the workspace with the skills loaded; with no terminal, it drives Claude Code headless, and Codex, Cursor and Kiro take the one-call path. Without a coding agent, it makes one LLM call with the skills' text and sendsfluid validate --offlineerrors back, for at most 3 attempts. AI mode no longer overwrites a contract: use--refineto change one. A run with no terminal needs--context, and--forge-agent-modedefaults toagentic. See AI mode in 0.31.0.fluid memory,fluid mission,fluid stats --judge,fluid init --agent,fluid init --show-workand 30fluid forgeflags are removed. A command that still passes one exits 2, withunrecognized arguments, or withinvalid choicefor the word after a flag that took a value. Take it out: Removed flags lists each forge flag with what to use instead.fluid forge --no-memoryis still accepted and ignored.fluid mcp serveloses five tools.forge_run,forge_from_source,score_contract_quality,enrich_contract_suggestionsandsearch_semantic_memoryare removed, and so is--llm-provider mcp-sampling. Take them out of--allow-toolsand--deny-toolslists and agent instructions. See Authoring tools.- The store's
postgresandvectorbackends and the[vector]extra are removed.FLUID_STORE_BACKEND=postgresorvectorlogs a warning and uses the file store, andfluid doctorasks you to unset it. Drop[vector]from install specs. - Variables only the copilot read are ignored, among them
FLUID_PROMPT_PROFILE,FLUID_PROMPT_OVERLAYS,FLUID_AGENT_WEB_TOOLS,FLUID_FORGE_AUTO_RESUME,FLUID_INTERVIEW_LEGACYandFLUID_LLM_FALLBACK_CHAIN. Environment variables marks each one. fluid forge data-modelis unchanged. Every subcommand and flag stays,--tieredand the routing-model flags included, and so doesfluid stats --json. Command Center'sfrom-intentpath, with--deterministicor--llm-provider, runs on them as before.
(since 0.31.0, forge-cli #884)
fluid mcp servereads source catalogs with one tool,source_catalog, and serves 7 tools.list_source_adapters,list_source_tables,inspect_source_table,list_source_lineageandlist_source_glossaryare nowsource_catalogwithkindset toadapters,tables,table,lineageorglossary. The other arguments keep their names, and no alias keeps the old tool names: rename the calls in agent instructions and scripts. Every result names itskind, and atableorlineageresult now sits under it, as in{"kind": "table", "table": {...}}. A result isconciseunless the call asks for"response_format": "detailed", which returns every field, as the five tools did. Seesource_catalog.- An unknown tool name in
--allow-toolsor--deny-toolsstopsfluid mcp servewith exit 2. A name that is not a tool now fails at startup: a typo, one of the five tools, or a tool that forge-cli #881 removed. In 0.30.1 those tools were still served, and a typo was ignored. Replace the five names withsource_catalog. An IDE config that an olderfluid scaffold-idewrote lists the five old tools underautoApprove, so the client does not auto-approvesource_catalog: add it to the list by hand, or runfluid scaffold-ide --force, which writes the scaffold's files again over any edits. See Access controls. - A result estimated at more than 6,000 tokens comes back as a resource. The call returns a short envelope and a
resource_linktofluid://artifacts/<id>, which the client reads withresources/read. Over stdio the envelope also names a file holding the same JSON. A client that reads only a result's text sees the envelope where it used to see the result.FLUID_MCP_INLINE_TOKEN_LIMIT=0keeps every result inline. See Large results. - A catalog listing with a scope now works through an MCP client. On MCP SDK 2.x,
list_source_tables(nowsource_catalogwithkindset totables) and a scopedlist_source_glossaryfailed withError executing toolfor an ordinary scope such as{"database": "DB", "schema": "SALES"}: the server sent the scope's unsettablesback asnull. Unset fields are now left out.
(since 0.31.0, forge-cli #887)
- An unknown tool name in
--allow-toolsor--deny-toolsstopsfluid mcp output-port servewith exit 2. The check runs before either transport binds, as it does forfluid mcp serve. In 0.30.1 a mistyped name was accepted: a deny list denied nothing and the tool stayed callable, and a mistyped allow list served fewer tools without a word. The tools aredescribe,query,query_sqlandsample, and matching is exact and case-sensitive. Fix the name in the launcher script or IDE config that starts the gateway. Seeserveflags.
(since 0.31.0, forge-cli #888)
FLUID_TELEMETRY,DO_NOT_TRACKandtelemetry.enabledin~/.fluid/config.yamldo nothing. The UX telemetry and its consent are removed, and nothing emits theforge.invocationspan fields.fluid doctor --envno longer lists the two variables or prints the telemetry line, andfluid doctor --env --jsonis{"env": [...]}without thetelemetrykey. A script that reads that key gets nothing: stop reading it.FLUID_COPILOT_AGENT_LOOPno longer stops forge from picking a saved or detected coding agent. The variable does nothing, likeFLUID_COPILOT_FORCE_INTERVIEW,FLUID_COPILOT_JUDGE,FLUID_COPILOT_SELF_EVALandFLUID_COPILOT_SEMANTIC_MEMORY, which nothing reads either. Drop all of them from env files and CI settings.
(since 0.31.0, forge-cli #891)
- A contract that pins
sovereignty.jurisdictionstartsfluid mcp output-port serve --transport httponly with JWT auth that maps a claim tojurisdiction. In 0.30.1 it also started undershared-token, underjwtwithout its issuer, audience or JWKS URL, and under aFLUID_MCP_JWT_CLAIM_MAPPINGthat maps the claim away, then denied every call asmissing-caller-jurisdiction. Each now exits 2 before it listens and says why. Such a gateway answered no call, so nothing that worked stops working: switch it toFLUID_MCP_AUTH_MODE=jwt, or setsovereignty.crossBorderTransfer: trueif the data may leave. See Caller-jurisdiction enforcement.
(since 0.31.0, forge-cli #906)
fluid validate --strictfails on two new warnings about dbt MetricFlow names: a dimension with the name of an entity or another dimension of its expose, and a name that is a dimension of one expose and an entity of another, whichdbt parserejects whenever the project declares a metric. Rename one of the two. See Names dbt MetricFlow holds once.- A contract that
fluid forge data-modelwrites with one expose per dataset gives each relationship's foreign entity to itsfromexpose only. Every expose used to get the foreign entity of every relationship, most of them on a column the expose does not have, andfluid generate transformationexported them to dbt. Forge the contract again, then generate its dbt project again. fluid generate transformationreads asemanticscollection that is not a list as empty, with a warning, where it stopped with aTypeError.
(since 0.31.0, forge-cli #905)
- BREAKING:
tofuruns with an environment allowlist, not a copy of your environment.fluid apply,fluid plan --tofu,fluid diff,fluid verify --state-driftandfluid generate iac --validatepasstofuthe process basics, proxies and CA bundles, OpenTofu's settings by name plusTF_TOKEN_*,TOFU_CLI_PATHandGITHUB_OUTPUT, the credential families of the plugin (AWS_,GOOGLE_,SNOWFLAKE_,DATABRICKS_and the others), and the cloud of the state backend and of the encryption key.TF_CLI_ARGS*,TF_VAR_*,TF_ENCRYPTION,TF_INPUTandTF_REATTACH_PROVIDERSno longer pass, and neither do model keys such asGOOGLE_API_KEYorAZURE_OPENAI_API_KEY. A job that set one of them to change whattofudoes stops doing so: atofu_env_droppedwarning names the variables. An IaC plugin that lives outside forge declares what its provider reads incredential_env_varsandcredential_env_prefixes. See Whattofugets from the environment. - BREAKING:
PipelineConfig.oidc_providerandgenerate_pipeline_template(oidc_provider=)are replaced bycloud_auth. No CLI flag ever set them, so their OIDC steps never reached a generated pipeline, and the Azure login step went with them. Code that calls the library passescloud_auth, a tuple ofaws,gcp,snowflake,databricksandopenbao, whichfluid generate cifills from the contract. See Keyless cloud login. - A Databricks run with an OIDC auth type needs
DATABRICKS_CLIENT_ID.github-oidc,azure-devops-oidc,env-oidcandfile-oidcexchange the job's token for the service principal the client id names. forge used to blank it for any type butoauth-m2m. See Credentials. - New, opt-in:
FLUID_STATE_ENCRYPTIONencrypts the OpenTofu state and saved plan. Nothing changes until you set it, and it needs OpenTofu 1.13 or newer. See State and plan encryption. - New: generated GitHub Actions and GitLab pipelines log in to the cloud with OIDC, and Snowflake accepts workload identity federation. Regenerate a committed pipeline with
fluid generate ciand create the cloud-side trust its header lists. See Keyless cloud login and Workload identity federation.
(since 0.31.0, forge-cli #909, forge-cli #925, forge-cli #946, forge-cli #949)
fluid profileis a new command. It reads one expose's records where they live and writes a FLUX profile document of output-checked aggregates, which a Simulation pins by digest. It reads every value and forms every statistic withflux-stats, the registry FLUX's fidelity check uses. Nothing that exists changes. The[profile]extra adds DuckDB and flux-stats (>=0.6.0,<0.7) from PyPI;fluid profilesays what to install when flux-stats is missing. Seefluid profile.
(since 0.31.0, forge-cli #951)
fluid profilewrites FLUX'smean/1statistic for each number column, so a run checked against the profile can passfluxsim verifyandfluxsim ship. A profile you wrote before keeps its digest. Written again, it carriesmean/1and gets a new digest, so pin the new one.--fractional-bitssets the grid, 16 by default, and it must equal the fidelity block'sencoding.fractionalBits. See Themean/1statistic.
(since 0.31.0, forge-cli #911)
- 0.7.7 (preview) contracts can state rules about valid data, and
fluid validatechecks them.exposes[].contract.invariants[]holds checks, derived columns and dependencies, written in FLUID SQL/1. Columns gainidentifierFormat,placeholdersandsharedAttribute, and relationships gainsharedColumns. Every field is optional. A 0.7.7 contract that uses none of them gets no new errors, though it may get notes, which--strictnever counts. See Rules about valid data.
(since 0.31.0, forge-cli #917)
fluid shipruns the 11-stage pipeline in place of the four-stage macro. It bundles, validates, generates and validates artifacts, checks drift, plans, applies the reviewedruntime/plan.jsonagainstruntime/bundle.tgz, applies policy and verifies;--publishand--schedule-syncadd stages 10 and 11. A script that runsfluid shipnow also runsdiff,policy-applyandverify, so a drifted target or a failed verify stops it. See The 11-stage runner in 0.31.0.validateandvalidate-artifactsrun with--strictby default. A contract with a warning, a placeholder owner email among them, now stops at stage 2.--no-strictdrops it.--skip-planis removed and exits 2;--skip-bundleis accepted and ignored. Use--stageor--untilto run part of the pipeline.- The exit status is ship's own: 0 passed, 1 a stage failed, 2 ship refused its arguments, 3 a stage failed after apply changed something, 130 interrupted. Through 0.30.1 ship exited with the failing stage's code, which is now in
runtime/ship-report.json. - Ship refuses before anything is written, with exit 2: an environment name a stage would refuse (it must start with a letter, at most 64 characters), a contract that is missing, symlinked or under a system path, and an unknown stage.
- A contract outside the current directory runs in its own directory, so
runtime/anddist/land beside it.
(since 0.31.0, forge-cli #928)
- The templates and scaffolds write no stand-in owner email. The templates bundled for
fluid initcarry none, andfluid init,fluid market --instantiate,fluid forge --template,fluid product-newandfluid importleavemetadata.owner.emailout rather than write one (--owner-emailgives one tofluid init --blueprintand--discover).fluid init --template csv-basicsand thefluid marketblueprints' default no longer write the starter teamdata-team. A contract you scaffolded before keeps its placeholder: replace it with the team's real address, or delete it, beforefluid validate --strict,fluid shipor a generated pipeline's stage 2.
(since 0.31.0, forge-cli #929)
fluid shipreads the generated pipelines' switches and stage knobs. ARUN_STAGE_<N>_<NAME>=falsein the environment switches that stage off,--stageincluded, andVALIDATE_STRICT,VERIFY_STRICT,GENERATE_EMIT,DIFF_EXIT_ON_DRIFT,DIFF_LAST_APPLIED,PLAN_HTMLandPOLICY_APPLY_MODEreach their stage. A job that exports one of them for another tool now changes what ship runs: unset it in that step (for a knob, its flag also beats the variable). After a dry-run apply, stage 8 checks the bindings only when stage 7 is in the pipeline; without it, stage 8 enforces them. See Stage switches and knobs.
(since 0.31.0, forge-cli #941)
fluid scaffold-ciandfluid generate-pipelineare removed. Both exit 2 as unknown commands. Runfluid generate ci --system <system>instead (github,gitlab,jenkins,azure,bitbucket,circleciortekton). A script or CI job that calls either one needs the change;fluid forge --ci <provider>still writes CI files.
(since 0.31.0, forge-cli #942)
fluid generate ci --complexityexits 2. One pipeline shape replaces the four tiers. Drop the flag from scripts and add what the advanced and enterprise tiers carried with--add security,--add complianceand--add sbom(repeatable). Seefluid generate ci.- GitHub Actions: the file is
.github/workflows/fluid-pipeline.yml. It replacesfluid-basic.yml,fluid-standard.yml,fluid-advanced.ymlandfluid-enterprise.yml. The old tier files and the oldsecurity.ymlandcompliance.ymlstill run until you delete them.fluid generate ciandfluid forgename each one they find and do not delete it. - Give the GitHub environment required reviewers before you regenerate. An apply (stages 7 to 11) now runs on a push to the default branch, in the environment named after
FLUID_ENV(devunless you set it). With no reviewers on that environment nothing waits for a person. Settings > Environments is where they go. The other systems have their own gate: see Operating in CI. - The install settings are repository or project variables.
FLUID_PACKAGE_SPEC,FLUID_PIP_INDEX_URL,FLUID_PIP_EXTRA_INDEX_URLandFLUID_ALLOW_PRERELEASEare no longer a run-time input on GitHub Actions, GitLab CI, Azure DevOps, Bitbucket or CircleCI. A pipeline that passed one when it started a run now sets it as a variable in the repository or project settings. Jenkins and Tekton still take them as build parameters. On GitLab CI, keep the minimum role to use pipeline variables at Maintainer or No one allowed (Settings > CI/CD > Variables), since the Run pipeline form takes any variable. --publish-include-envis removed.fluid shipalways passes--envtofluid publish, so drop--publish-include-envand--no-publish-include-envfrom scripts.fluid forge --ci-complexityis removed. Forge asks for the CI provider only, and.fluid/ci-state.jsonrecords the provider and the add-ons. A file written by an earlier forge still loads.- New:
--default-branch BRANCHnames the branch whose pushes apply. The default is the remote's default branch (origin/HEAD), elsemain; GitLab CI reads its own$CI_DEFAULT_BRANCH. A pull request or merge request against it, or a manual run on another branch, runs stages 1 to 6 only. fluid shipstage 11 writesruntime/schedule-sync-report.jsonwhen it runs. See Stage switches and knobs.
(since 0.31.0, forge-cli #897, forge-cli #900, forge-cli #907)
- BREAKING: three kinds of AWS contract that used to be accepted are refused. Run
fluid validateon each AWS contract before you upgrade, because the same checks run atfluid generate iacandfluid apply.- A row filter or a column restriction whose principal also holds a grant that could undo it.
row-filter-unenforceableorcolumn-restriction-unenforceablenames the grant. The causes:SELECTorALLinpermissionsWithGrantOptionfor a filtered role, a read grant toIAM_ALLOWED_PRINCIPALS, to the filtered role's account or:rootARN, or to an organization, two filters on one role, and a second grant for the same role in another spelling beside a column limit. Principals are compared by IAM identity: the path is dropped and the name is read in lower case. See Who a grant, a row filter and a column restriction name. - A contract that plans
iam.bind_glue_database,iam.bind_s3_bucket,athena.create_view,redshift.ensure_schema,redshift.ensure_tableorevents.ensure_schedulefails withaws-op-not-emitted. Through 0.30.1fluid generate iacwrote the module without them andfluid applyexited 0. Create the view, schema or table in a build step, and for a schedule setorchestration.schedulewith an engine other thanairflowormwaa. See Planned operations the module does not create. governance.lakeFormation.adminsis refused in every contract, andtagDefinitionsunless the contract declarespackaging.containers.lfTagVocabulary: isolated. Move the admins to your platform's bootstrap state. A product's state used to own the account's Lake Formation settings, and destroying the product left the account with no Lake Formation admin. A product whose state already holdsaws_lakeformation_data_lake_settingsneeds care: withadminsremoved, the next apply plans to destroy that resource, which empties the account's admin list. The data-loss gate stops it, because the type is not one of the access grants it exempts. Put the admins in your platform's state first, and then release the resource from the product's state withtofu -chdir=.fluid/iac/aws/<safe-id> state rm <address>, as Packaging modes shows for another resource, rather than passing--allow-data-loss. See Account-wide Lake Formation objects.- Every data cells filter now writes
excluded_column_names, so a filter that only restricts rows applies on real AWS, where it failed withProvider produced inconsistent result after apply. A masked view is built only for a grant that reads, so the later<table>_masked_Nnames of a contract that had aDESCRIBE-only grant renumber, and the next apply changes them in state.
- A row filter or a column restriction whose principal also holds a grant that could undo it.
(since 0.31.0, forge-cli #898)
- New, opt-in:
fluid plan --tofusaves the OpenTofu plan, andfluid apply plan.jsonruns exactly that plan. Nothing changes until a pipeline passes--tofu. A pipeline that does needs a remote state both jobs share, protected plan artifacts, and a plan role that can write the state and its lock. See Saving the OpenTofu plan and The OpenTofu plan is what gets approved.
(since 0.31.0, forge-cli #899)
- The OpenTofu floor is 1.12.0, and
--ensure-opentofuinstalls 1.13.9. A runner withtofu1.6 to 1.11 stops withtofu X.Y.Z is older than the required minimum 1.12.0. Upgrade the runner image, or runfluid apply --ensure-opentofu, which now replaces an oldertofu. On 1.11, adding retention to a live GCP table plans an in-place update that BigQuery refuses. The generated module declaresrequired_version = ">= 1.12".
(since 0.31.0, forge-cli #901)
--state-backend s3://<bucket>/<key>?region=<region>names the state bucket's region. Only that exact tail is read as a region, so no existing key changes. After a state move to the provider key, the summary prints the commands that remove the old state object and its lock; forge never deletes state. See Remote state.- The Snowflake provider pin moves from
~> 2.0to~> 2.23. 2.23 adds thetoken_file_pathfield. The nexttofu inittakes a provider of 2.23 or newer.
(since 0.31.0, forge-cli #915)
- A new Glue Iceberg table gets its first metadata from Glue, and an existing table is never replaced. A table that an earlier release created has no metadata, so Athena still refuses it until you replace it. See Iceberg tables in Glue for the check and the two ways to replace one that is empty.
(since 0.31.0, forge-cli #910, forge-cli #921)
- The ODCS export mirrors
dq.rulesthresholds. A FLUID threshold is the share of rows that pass, and ODCSnullValues,duplicateValuesandinvalidValuescount rows that fail. Through 0.30.1 the threshold was copied across, sothreshold: 1.0with>=came out as "at least one null". A full share ismustBe: 0, a partial completeness share is the null share as a percentage,valid_valuesismustBe: 0, and a partial uniqueness share is not exported. A rule now lands on the columns it checks. Re-export every contract and re-publish the ones a catalog or a quality engine reads. See Quality rules in the export.
(since 0.31.0, forge-cli #908, forge-cli #914, forge-cli #916, forge-cli #920)
- Both Data Mesh Manager publish paths send lineage Entropy draws, retire stale contracts and send the archetype. Publish again to get the product-to-product and source-system edges, the archetype, and the retirement of contracts no port names any more. Retired contracts are never deleted.
- A product with an upper-case letter in its id is published under a lower-case id by both paths. The registrar used the id as written. After the first publish the old product stays in Entropy until you delete it there.
fluid publish --target datamesh-managerrefuses plain http to a non-local host unlessDMM_ALLOW_INSECURE_HTTP=true, and warns about settings it does not use. Seefluid datamesh-manager.
(since 0.31.0, forge-cli #923)
- One renderer writes every Airflow DAG a contract's
orchestrationsection describes. A DAG id drops.and-for AWS too, a bare-word schedule such asdailyis the Airflow keyword for AWS (00:00, where it ran at 02:00), a task with no operator fails closed for every provider, and a value that did not resolve is refused, where it used to rendermy-projectorUNKNOWN_DB. Regenerate and review each committed DAG.fluid generate-airflowis an alias offluid generate schedule --scheduler airflow, and must run from the project directory for a contract with scheduled builds. Seefluid generate-airflow.
(since 0.31.0, forge-cli #902, forge-cli #903, forge-cli #904, forge-cli #912, forge-cli #913, forge-cli #918, forge-cli #927, forge-cli #932, forge-cli #936, forge-cli #938, forge-cli #939, forge-cli #950)
- New, opt-in, 0.7.7 preview: Spark to Iceberg on AWS and the schema registry.
fluid applystarts a Spark build's run and keeps a streaming one running; a scheduled one runs throughfluid apply --mode amend-and-build --build-id <id>, as every scheduled build does. There is nofluid sparkcommand. A contract that uses none of the new fields is unaffected.fluid planchecks a consumed topic against its registry, andfluid generate iacregisters a produced one, only where the contract names a registry. See Spark to Iceberg on AWS, Scheduled Spark builds, Schema registry andfluid import schema-registry. - A running Glue Avro stream restarts once on the release that adds the pinned jars to its config hash. Plan and apply need the same
FLUID_SPARK_GLUE_JARS_URL.
(since 0.31.0, forge-cli #893)
- A
secretRefis one grammar,<scheme>://<locator>[#<field>][?version=<n>], andfluid validaterefuses what does not follow it. It also refuses a scheme the operator has not enabled, a locator outside the allowed prefixes and a connection host the allowlist does not cover, and its message names the setting that allows it. A contract that validated on 0.30.1 can fail now. AsecretRefthat may be a pasted secret (no://, or a URL withuser:password@) is refused, and never printed. Runfluid secrets checkon every contract before you upgrade a pipeline. See What is refused. file://takes an absolute path, andfile://nameis gone. The lookup in~/.fluid/secrets/<name>no longer exists. Writefile:///run/secrets/name. The path has to sit under an allowed prefix,/run/secrets/by default, and a symlink is checked again. Mount the secret there, or add your directory toFLUID_SECRET_PREFIXES. See Schemes.vault://needs#field.vault://kv/orders/pgis refused: name the key, as invault://kv/orders/pg#db_pass. A reference reads KV v2 on OpenBao or Vault, with the address inBAO_ADDRorVAULT_ADDR. See Vault and OpenBao.aws://isaws-sm://, andgcp://andazure://need a resolver pack.aws-sm://needs the[aws]extra.gcp-sm://andazure-kv://resolve only when a pack for the scheme is installed and the operator lists the scheme inFLUID_SECRET_SCHEMES. On the default policy agcp-sm://orazure-kv://reference is refused asnot enabled, with a warning that no resolver is installed. Once the scheme is listed,fluid validateonly warns, andfluid applystops with the install hint. See Schemes.- The destination policy covers every scheme, not only
env://. A value fromvault://,aws-sm://,file://or a pack is sent to a host that leaves the machine only whenFLUID_ACQUISITION_SECRET_REF_ALLOWLISTallows it, with<reference>@<host>,<reference>@*or<prefix>*@<host>, beside theNAMEandNAME@hostentries that worked before. A pipeline that reads a non-envsecret for a remote source now fails until you set it, in the shell or CI job that runsfluid apply. SeeFLUID_ACQUISITION_SECRET_REF_ALLOWLIST. - The operator now decides which stores, which locators and which stages.
FLUID_SECRET_SCHEMES(defaultenv,file,vault,aws-sm),FLUID_SECRET_PREFIXES(default:fileunder/run/secrets/) andFLUID_SECRET_STAGES(defaultapply) are read only from the real environment, and a project.envcannot set them.validate,plan,bundleanddiffnever fetch a value. A pipeline that needs a value in another stage has to list that stage. See The operator's policy. - A Debezium
extra_configmay not set a host or credential key (database.hostname,mongodb.connection.stringand the like) when the connection has asecretRef. Name the host in the connection block. SeeFLUID_ACQUISITION_SECRET_REF_ALLOWLIST. fluid_build.secrets.SecretManagerandbuild_runners._secret_policyare removed. Thefluid_build.secretspackage replaces them, with no compatibility shim. Library code that imports either one moves to the package'sresolve_secret. See How a value is fetched.- The Fernet credential file store and plaintext credentials in
~/.fluidrc.yamlare removed.~/.fluid/credentials.enc,FLUID_ENCRYPTION_KEY,FLUID_ENCRYPTION_PASSPHRASE,~/.fluid/.keyand thefluid auth doctorcheck of it are gone. Move each credential to the OS keyring, a secret store or the environment. See Provider credentials from a store. - The credential chain asks a secret store only when
FLUID_CREDENTIAL_STOREnames one. SettingGCP_PROJECTorAWS_REGIONno longer sends credential lookups to a cloud secret manager. To keep reading provider credentials from a store, setFLUID_CREDENTIAL_STORE=vault://<mount>/<prefix>oraws-sm://<prefix>, and store each credential at<prefix>/<provider>#<key>. See Provider credentials from a store. - New:
fluid secrets check, the run-secrets file and the Vault JWT login.fluid secrets check <contract>lints references and, with--resolve, proves they exist without reading a value.FLUID_RUN_SECRETS_FILEhands a run exactly the secrets it may use.FLUID_VAULT_JWT_ROLElogs a CI job in to OpenBao or Vault. Nothing changes until you use them. Seefluid secrets check, The run-secrets file and JWT login for CI.
Changes in 0.32.0
These changes are in 0.32.0. Two change what a contract or a generated file looks like: read them before you upgrade a pipeline.
(since 0.32.0, forge-cli #952)
- In the 0.7.7 preview, an
enumlevel may hold any character, and an empty item or an unescaped quote is now refused. A level is a quoted string with its own quote doubled, as SQL writes a string literal, soenum('Dr.', '>10000', 'O''Brien', 'a, b')is valid. An unquoted word list such asenum(IE, United Kingdom)reads as before. An empty item, such asenum(a,,b), and a level with an unescaped quote, such asenum('it's'), now failfluid validateon a contract that declaresfluidVersion: "0.7.7". Fix them before you upgrade. The schemas for 0.7.6 and earlier are unchanged. - The 0.7.7 schema file changed, and 0.7.1 to 0.7.6 keep their bytes. If you copy schema files out of the package, copy 0.7.7 again.
- Generated files stop carrying an enum's levels. dbt model contracts and casts wrote
varchar('a','b')for an enum column, and the ODCSphysicalTypewroteVARCHAR('a','b'). Each now gives the plain string type, and ODCS no longer readsenum(10)as amaxLengthof 10. Regenerate what you commit and expect that one diff.
(since 0.32.0, forge-cli #953)
fluid generate transformationkeeps contract values as data in the Glue, Dataflow and Spark job scripts. A column name or type, the expose id, the contract name, a stage name or embedded SQL that held a quote, a line break or"""could end its place in the script and add a statement that runs with it. Each value is now written as a string literal, a one-line comment or an escaped block. Regenerate and review the diff: the string literals these scripts write now use single quotes, the Glue script's SQL summary comment is one line, and two stages whose names sanitise alike, such asa-banda.b, get two functions,stage_a_bandstage_a_b_2, where the second used to replace the first.
(since 0.32.0, forge-cli #959)
fluid profile --bundle DIRanswers a Simulation's fidelity block. The profile then carries the statistic each declared property is compared with, grouped means, histograms, supports and quantiles among them, so the engine scores every property but a correlation. Without--bundle, a profile holds what it held before. The[profile]extra needs flux-stats 0.6.2 or newer, whichpip install -U 'data-product-forge[profile]'brings. See Answering a fidelity block.
Changes in 0.33.0
These changes are in 0.33.0. Several change what a plan or a generated file does: read them before you upgrade a pipeline. A BigQuery decimal with a precision replaces its live table once, behind --allow-data-loss; a Snowflake column whose type changes needs its table migrated; a Glue table that declares partitionBy is replaced the first time it is partitioned; and a generated CI file changes only when you run fluid generate ci again.
(since 0.33.0, forge-cli #965, forge-cli #967)
fluid profile --bundlewritesCfor every withheld group and cell of a histogram or a quantile. In 0.32.0 each kept the code of the check that withheld it,Afor a small count, and a code that names the check tells a reader a range. NowCmarks them all, whatever withheld them. The summary still counts each by its check, and a new line counts the histogram cells withheld inside groups that were published. Every other statistic keeps the code of its check. A profile you wrote before is unchanged and still reads. Profile again to get these checks, and pin the new digest. See What leaves: the output checks.- A quantile is published only when at least 10 records lie below its value and at least 10 lie above it. In 0.32.0 a quantile was published when 10 records held its value, so its ranks could show that fewer than 10 records lie below it or above it. No record below or above counts as a zero, which
acro-defaultwithholds, so a quantile at the smallest or the largest value of its records is withheld. A column'squantilesmetric is checked as before. See The checks on each group. - A quantile is published only as its column's one histogram or quantile. A block that declares a median and a histogram of one column now gets the histogram and no median, and a block that declares two quantiles of one column gets neither. To keep a quantile, declare no other histogram or quantile on its column.
- A group withheld for its size withholds its siblings in a grouped histogram or quantile. When a group of a
groupByholds fewer than 10 records with a value in a column, every group of that column's histograms and quantiles by thatgroupByis withheld too, as the groups of a grouped mean and a grouped support already were. The summary counts a sibling that passed its own checks asD. - The checks still leave one gap. A reader can still subtract one histogram's published counts from another's, or from a level histogram's group size, and read a withheld cell, a zero included, to within the rounding. forge-cli plans to close that with FLUX's
check_edge_counts, which a flux-stats release has to carry first, and to ship it in a later release. That would change what a bundle profile publishes again.
(since 0.33.0, forge-cli #978)
fluid profile --helpand the profile page say what a publishedminormaxleaves readable. A publishedminshows that no record lies below it, and a publishedmaxthat none lies above it. No check withholds aminor amaxfor that. A quantile whose value is the smallest or the largest of its records shows the same, and underacro-defaultit is withheld for that. Nothing the profiler writes changes. See The checks on each group.
(since 0.33.0, forge-cli #966)
fluid generate schedulerendersbuilds[].dependenciesand dataset triggers. Builds thatbuilds[].dependencieslink now run in one Airflow DAG, onefluid apply --build-idtask per build, each after the builds it names, and a dependency'sconditionbecomes the task's trigger rule. If you gave those builds cron triggers of their own to fake the order, keep one trigger and regenerate with--overwrite, which now removes the DAG files the contract no longer makes. Two different triggers in one linked group stop generation. See Build graphs and dataset triggers.datasetandschedule_and_datasetbuilds get a DAG. Adatasetbuild runs when the datasets intrigger.datasetsare updated (all of them, or any one withdatasetsOperator: any), andschedule_and_datasetruns on its cron and on those updates. A task now declares the exposes its build writes asoutlets, where an expose has an asset URI, so a regenerated one-build DAG gains anAssetimport and anoutlets=list when its build writes such an expose. The files need Airflow 2.6 or later (2.9 foranyandschedule_and_dataset), or Airflow 3.- A contract whose only builds are streams that
fluid applykeeps running now getsNo build needs a DAGfromfluid generate schedule, where it used to getNo scheduler engine specified(when the contract set no engine) or an error that the orchestration section has no task to render (withorchestration.engine: airflow).fluid validatewarns about an Airflow task'sbuildStepRef, which no generated DAG runs: list the build inbuilds[].dependenciesinstead.
(since 0.33.0, forge-cli #972)
- The bundle packs the dbt project a build runs, and
fluid applyrefuses a checkout whose dbt files changed.fluid bundle --format tgzpacks each dbt build's project undersources/dbt/<build id>/:dbt_project.yml, the package and selector files,.dbtignoreand the files under the*-pathsdirectories, neverprofiles.yml,target/,logs/ordbt_packages/. It refuses what it cannot bind: a symbolic link, a model in a hidden or__pycache__directory, a path key outside the project or written as Jinja, and a file over 64 MiB.fluid planrecords the files, andfluid applycompares the checkout with them before anything runs. The dbt runner compares again just before dbt starts. A changed, added or removed file, or a bundle made before this change that packs no project for a dbt build, fails withapply_plan_digest_bundle_mismatch. Runfluid generate dbt-testsbeforefluid bundle, install packages frompackage-lock.ymlin the job that applies (installed packages are not bound), and plan and apply in the same checkout path.--no-verify-plan-bindingskips the check and logs a warning. See The dbt project of a dbt build and The dbt project.
(since 0.33.0, forge-cli #985)
- A bundle's bytes no longer depend on where it was written.
fluid bundle --format tgzused to record the source contract's path, relative to--out, inMANIFEST.json, so the same checkout bundled to two paths gave two archives with one digest. Those paths (source.contractandsource.overlay) now go to<bundle>.provenance.jsonbeside the archive. It names the archive's sha256 and is ignored when it names other bytes.MANIFEST.jsonkeepssource.envand the overlay warnings. Copy the provenance file with the bundle, and keep the contract at the same relative path: without the file, relative paths anchor at the bundle's own directory, and a bundle that packs a dbt project is refused at apply. Paths inside an older bundle's MANIFEST are not read, so bundle again. See The provenance file. fluid bundleprints a packet digest, and a plan is bound to it. The newpacket:line is the sha256 of the.tgzfile, anddigest:stays the merkle digest of the resolved contract and its files.fluid plan <bundle>records the packet aspacketDigest.fluid applyrefuses a bundle with other bytes (apply_plan_digest_packet_mismatch) and a bound plan that records none (apply_plan_digest_packet_missing), so runfluid planagain on a plan made by 0.32.0 or earlier. See The packet digest.fluid bundle --attestnames the bundle file as its subject. The in-toto subject digest is the sha256 of the bundle file in bare hex, sosha256sum bundle.tgzmatches it. It used to be the merkle root with asha256:prefix. The merkle root, the packet digest and the checkout paths are in the statement's external parameters. A verifier that matched the old subject has to match the file's hash.
(since 0.33.0, forge-cli #979)
fluid generate artifactswrites each Spark build'sjob.json. The newsparkemit key is on by default, so a contract with a forge Spark build getsspark/<build id>/job.jsonand one more entry in the artifactsMANIFEST.json, and a generated Jenkinsfile listssparkinGENERATE_EMIT. A Spark build that the compiler refuses, or that the upload would refuse, now fails stage 3 withgenerate_artifacts_failedinstead of waiting forfluid apply. The operator's rules for contract SQL and hosts are still enforced only byfluid apply. A contract with no Spark build is unchanged. See Spark job configs.
(since 0.33.0, forge-cli #983)
- A Spark job's role can look up the directories it is granted. On 0.32.0 and earlier, a stream on AWS failed within minutes of starting, because S3 refused the lookup of its checkpoint directory with a 403. The role forge creates, and the policy for an EMR on EC2 cluster, now read each granted directory's own key and list it by its exact name. Run
fluid applyto update the policy. A role you name withexecutionRoleArnis yours, and needs the same two permissions. See What the job's role may do in S3.
(since 0.33.0, forge-cli #984)
- Each EMR runtime takes its own
releaseLabelform.emr-serverlessandemr-ec2keep the plainemr-7.12.0.emr-ekstakesemr-X.Y.Z-latestor a datedemr-X.Y.Z-YYYYMMDD, with an optional variant such asemr-7.12.0-java11-latest, andfluid validatenow refuses the plain form there. Anemr-eksbuild with noreleaseLabelruns onemr-7.12.0-latest, so its config hash changes: runfluid planagain. The config hash of anemr-serverlessoremr-ec2build does not change. Change anemr-eksbuild that namesemr-7.12.0toemr-7.12.0-latestwhen you upgrade: 0.32.0 and earlier refuse that label. See Release labels.
(since 0.33.0, forge-cli #990)
fluid validatestops warning about SQL the operator listed. The warning that a build's ownproperties.sqlruns only if the operator lists it was printed for every Spark and managed Flink build with SQL, whateverFLUID_SPARK_CONTRACT_SQL_ALLOWLISTorFLUID_FLINK_CONTRACT_SQL_ALLOWLISTheld, sofluid validate --strictfailed such a build even after the list named it. Validate now reads the list, under the same environment asfluid apply, and warns only while apply would refuse the build. An entry that pins other SQL gets a warning that says the SQL changed after review. A build listed by product, by build or with the digest of its SQL is quiet, so--strictpasses when nothing else warns. Export the variable in the job that runsfluid validateas well as in the one that runsfluid apply: a validate job without it keeps the warning, and a pipeline whose stage 2 isfluid validate --strictfails on it. See Contract SQL andfluid validate.
(since 0.33.0, forge-cli #994)
fluid planandfluid applywarn when an EMR on EC2 cluster can lose a Spark build to Spot. For the cluster of eachemr-ec2build they read its instance groups or fleets once, with the default AWS credential chain, and warn, never refuse, when the primary node runs on Spot, when core nodes run on Spot, or when core or task nodes run on Spot and theyarn-sitedoes not keep YARN application masters on On-Demand nodes. Each warning names its fix. A cluster they cannot read prints nothing,fluid apply --dry-runreads nothing, andplan.jsondoes not change. Give the role that runs plan and apply, and the scheduler's role for a scheduled build,elasticmapreduce:DescribeCluster,ListInstanceGroupsandListInstanceFleetson the cluster, or the check stays silent. See What plan and apply check on the cluster.
(since 0.33.0, forge-cli #973, forge-cli #968)
- A BigQuery table is partitioned and clustered as its binding declares, and a layout BigQuery cannot honour is refused.
binding.location.partitionByandbinding.properties.cluster_bywere dropped without a word, andfluid validate --strictpassed. Runfluid validateon each BigQuery contract with the new CLI: the old one passed them. ApartitionBywith more than one column or a non-time column, acluster_bywith more than four columns or a column BigQuery cannot cluster by, and either key on a view now fail, naming the field. BigQuery cannot partition an existing table, so a live table whose contract already declaredpartitionBywithout retention plans a replacement, whichfluid applyrefuses without--allow-data-loss. See Partitioning and clustering. - New: a 0.7.7 column that declares its
element,fields, orkeyandvalueis written to BigQuery as REPEATED and RECORD fields. A nullable array, a nullable element and a map are written but warn, so--strictfails on them. Nothing changes for a contract that declares no nested column. See Nested columns.
(since 0.33.0, forge-cli #971, forge-cli #980)
fluid shipgenerates artifacts for a reference-only contract, and its report changes shape. Stage 3 used to skip a contract with ahybrid-reference,referenceorexternal-referencebuild, and the run exited 0 with noMANIFEST.json. It now runs, and stage 4 fails where it used to skip when stage 3 was to writedist/artifacts/and did not. Inship-report.json,"schema"isfluid-ship-report/2, a stage'sskippedstatus is nowskipped-by-policy,unsupportedornot-run, and an interrupted stage isunknown, andskip_reasonandfailure_reasonare onereason. A script that reads the report should read the new statuses; a version 1 record still merges. See Reports.- New, opt-in:
fluid ship --require-artifacts KEYSand--release-dir DIR. The first fails the run unless stage 3 writes what it promised. The second publishes a run's bundle, artifacts, plan and reports as a release directory once every stage it ran passed. Nothing changes until you pass them. See Requiring artifacts and Publishing a release.
(since 0.33.0, forge-cli #970)
- The ODCS export of a 0.7.7 contract writes its declared invariants as executable quality rules. A contract with
invariants,identifierFormat,placeholders,sharedColumnsorsharedAttributenow has morequalityentries, on the schema object and on the property, each markedfluidGenerated. Where a column declares placeholders, its existing enum and regex rules change: the placeholders join an enum'svalidValuesand extend the pattern. Re-export and re-publish a contract a catalog or a quality engine reads. A contract that declares none of them exports as before. See Declared invariants as quality rules.
(since 0.33.0, forge-cli #969, forge-cli #981)
- New, opt-in:
fluid validate --diagnosticsand--format sarif, andfluid diff --baseline --impactwith--plan. The first puts each finding at the file, line and column that wrote it, and writes a SARIF 2.1.0 log for code scanning. The second adds an impact report to the version diff for a pull request. Without the flags, the output of both commands is unchanged. See Findings at the line that wrote them and Impact report for a pull request.
(since 0.33.0, forge-cli #986)
- New, opt-in:
fluid verify --require-every-port.--strictfails a run that verified nothing, so a run that verified one port passed while the others were skipped. With the flag, each skipped expose fails the run and is named with the reason. Nothing changes until you pass it. See Requiring every port. fluid ship --release-dirpasses it to stage 9. A release run now fails at stage 9, and publishes nothing, when an expose was skipped (no verifier for its binding, or a reference-only table that is not built yet), when an entry has no id or repeats one, or when the contract declares no expose.--skip-stage verifytakes stage 9 out of the run. See Publishing a release.
(since 0.33.0, forge-cli #991)
fluid generate dbt-testswrites afluid_ruletest for each declared rule, and a macro. On a 0.7.7 contract, each invariant, identifier format, bounded placeholder declaration, shared column and shared attribute becomes a model-level test, andfluid_rule.sqlgoes to<project>/<first macro-paths entry>/fluid/. The command now fails withdbt_tests_no_macro_dirwhen the contract declares such rules and nodbt_project.ymlis above--out: write--outinside the project, or pass the new--macros-dir. It never overwrites a macro file it did not write. A column's placeholders are also exempt from itsaccepted_valuesand range tests. A contract that declares none of these generates what it did before. See Declared rules as dbt tests.- A
schema.ymlno longer carries contract text that dbt would run as Jinja. A column name, description, enum value or test argument that holds{{,{%or{#, or looks like a call toenv_var,var,sourceordoc, or torefwith anything but a plain model name, is written as a literal of itself. Runfluid generate dbt-testsagain to rewrite a file made before. Any other string is written as before.
(since 0.33.0, forge-cli #987)
- The GitLab pipeline from
fluid generate ci --system gitlabhas a Run pipeline form and handsfluid-applya release packet. The file starts withspec:inputs(GitLab 17.11 or later): the 11 stage switches and nine knobs, 20 inputs in all, each closed by options or a pattern. On the default branchfluid-planrunsfluid ship --stage 1-6 --release-dir release, andfluid-applyuses only that job's artifacts and stops withoutrelease/release.json. A merge request plans without a packet and gets afluid-impactjob instead of an apply. A failedfluid-plankeeps its reports and removes its bundle, plan and artifacts. The pipeline runs--release-dirandfluid diff --impact, which 0.32.0 lacks: with aFLUID_PACKAGE_SPECthat installs 0.32.0 or older, the default-branchfluid-planfails and a merge request'sfluid-impactfails without blocking the merge. A file written by 0.32.0 or older has no form and none of this; runfluid generate ci --system gitlabagain to get it. See GitLab CI: form, release packet and impact report.
(since 0.33.0, forge-cli #999)
- A saved OpenTofu plan is bound to its target and applies on another runner.
fluid plan --tofunow records the cloud account and region (AWS), the project and region (GCP), the organization and account (Snowflake) or the host (Databricks), plus the state backend, thetofuversion and the provider versions, and printstarget: ....fluid apply plan.jsonreads them again aftertofu initand before it applies or changes anything, the state-key migration and hooks included, and refuses another target, an unreadable recorded field, anothertofuor provider version, another backend, or a plan with no target. With a state backend that both runners reach, a plan made on one runner applies on another, at another checkout path. A plan made against existing local state that the applying runner does not have is refused asapply_plan_digest_tofu_plan_state_missing. See The target a plan is bound to. - The provider lock travels with the plan. When
tofu initwrote one,fluid plan --tofuwrites<stem>.terraform.lock.hclnext toplan.jsonand binds its sha256. Pass it on with the plan:fluid applyinstalls it beforetofu init, and refuses a missing or changed one. - A plan saved by 0.32.0 or earlier records no target.
fluid applyin 0.33.0 refuses it withapply_plan_digest_tofu_plan_target_missing: runfluid plan --tofuagain. Reading the AWS account needsboto3(pip install 'data-product-forge[aws]') on both runners.
(since 0.33.0, forge-cli #997)
- New:
fluid generate dbt-tests --coverage,--coverage-out,--require-rule,--require-all-rulesand--adapter.--coverageprints which declared rules each engine can run, and the reason where one cannot, and--coverage-outwrites the report as JSON.--require-rule ID(repeatable) and--require-all-rulesfail the command, before it writes any test, when a selected adapter cannot run a required rule, when a rule named by--require-rulehas nothing to test, and when the contract declares rules below 0.7.7.--adapter TYPEselects the adapters, by default one for each platform the exposes bind to (neversparkortrino). See Which engines run each rule. - A Databricks dbt job tests the declared rules too.
fluid applyadds each declared rule that has a bound, on an expose the build writes, to the uploaded project as a singular test,<test path>/fluid_dq/<model>/fluid_dq_<id>.sql, and the run report names it. A rule with no exact Databricks SQL, or one that reads an expose the build does not write, is refused when the build is planned, and so is a declared rule on a build whoserepositoryis a Git URL. A contract that declares such a rule and builds a Databricks dbt job now fails at plan, where 0.32.0 ignored the rule. See Declared rules about valid data.
(since 0.33.0, forge-cli #1002)
- A saved-plan target field that was not bound is logged when the plan is applied, and a provider with no target reader is named. The WARNING
apply_tofu_plan_target_field_unboundnames the provider, the field and the reason. A provider that forge reads no target for (confluent, a plugin installed through an entry point) is named at plan and at apply, asplan_tofu_target_provider_unboundandapply_tofu_plan_target_provider_unbound. - The AWS region falls back to the profile's, and the profile is chosen in the order the AWS provider's SDK reads them. A region set only in the named profile's shared config binds. The profile is
AWS_PROFILE, elseAWS_DEFAULT_PROFILE, in the order the AWS provider's SDK reads them. See The target a plan is bound to.
(since 0.33.0, forge-cli #993)
fluid policy applygrants a Lakekeeper table in Lakekeeper. An Iceberg expose whosebinding.location.catalogislakekeeperused to compile to a warning that said to enforce access in Lakekeeper itself, and stage 8 granted nothing there.fluid policy compilenow writes oneprovider: lakekeeperbinding for each principal that has access to the table, fromaccessPolicy.grants, the deprecatedmetadata.policiesand the expose'spolicy.authz.readersandwriters.fluid policy apply --mode enforcegrants them in Lakekeeper, and revokes any other directdescribe,selectormodifygrant on that table, whoever added it. Ownership and the right to grant are never touched.--mode checkreads the grants and plans, and changes nothing: run it on a contract with a Lakekeeper table and read the revokes before you enforce. A table its writer has not created yet is reported and left for a later run, and a principal that Lakekeeper does not know leaves its table's grants as they are. See Lakekeeper tables andfluid policy compile.- Stage 8 needs the operator's Lakekeeper settings, or a pipeline that grants nothing today fails. By default
fluid shiprunsfluid policy-apply --mode enforce, which exits1on a Lakekeeper table when no credential is configured or the namespace is not allowed, where it used to exit0with a warning. After a dry-run apply, or with--policy-mode check, it runs--mode checkinstead. That exits0when no credential is configured, and reports the tables as not checked. It still exits1for a namespace that is not allowed or for any table it cannot check. ExportFLUID_LAKEKEEPER_CREDENTIAL_REFwithFLUID_LAKEKEEPER_OAUTH2_SERVER_URIandFLUID_LAKEKEEPER_SCOPE, orFLUID_LAKEKEEPER_TOKEN_REF, andFLUID_LAKEKEEPER_NAMESPACE_ALLOWLISTin the job that runs stage 8, or take the stage out with--skip-stage policy-apply. A project.envcannot set these names. The credential's reference has to be allowed for the catalog's host inFLUID_ACQUISITION_SECRET_REF_ALLOWLIST, andbinding.location.uriand the token endpoint have to behttps://URLs. See The operator's settings.
(since 0.33.0, forge-cli #1000)
fluid generate ciruns stage 3 for a contract with reference builds. Through 0.32.0 it switchedRUN_STAGE_3_GENERATE_ARTIFACTSoff by default when any build washybrid-reference,referenceorexternal-reference, so stage 4 had nothing to check and stages 8 and 11 ran without the policy bindings and schedule that stage 3 writes. Only--no-generate-artifactsturns it off now. A file generated by 0.32.0 keeps the old default until you generate it again.--install-mode dev-sourceputsfluidonPATH. On GitHub Actions, GitLab CI, Azure DevOps, Bitbucket, CircleCI and Tekton the install step ranexport PYTHONPATH=/forge-cli-src, which ended with the step and never provided the command, so the install step failed atfluid --versionwithcommand not found, and nofluid shipstep ran. It now runspip install -e /forge-cli-src, and a missing checkout or a failed install stops the job with aFATALline. Jenkins is unchanged.- Generated pipelines pin their container images by digest, and
fluid generate ci --system tekton --image IMAGEsets the Task image. Tekton, GitLab, Bitbucket and CircleCI namedpython:3.12-slimandcimg/python:3.12by tag (and, onmain,python:3.12for the GitLabfluid-impactjob). See Container images.
(since 0.33.0, forge-cli #988)
- A BigQuery
decimal(p,s)is written asNUMERIC(p,s), orBIGNUMERIC(p,s)where only that holds the range, and the first plan replaces each live table that declares one. In 0.32.0 it was a bareNUMERIC, which isNUMERIC(38,9), while dbt wrotenumeric(p,s). BigQuery cannot give an existingNUMERICcolumn a precision, so the first plan after the upgrade replaces each live table with a parameterised decimal, once, andfluid applyrefuses it without--allow-data-loss. Declare the columndecimalto keep the table as it is. Each contract with a BigQuery table gains oneterraform_dataresource, so that plan adds one resource. See Decimal precision and scale. - A decimal's arguments are a precision and scale its engine holds, or the column is refused (on BigQuery, a decimal wider than a
BIGNUMERICis written asBIGNUMERICwith a warning).decimal(1;drop table x)went into the Snowflake DDL asNUMBER(1;drop table x), anddecimal(40,20)became aNUMBERthat Snowflake rejects at apply. The arguments are digits, a precision of at least 1 and a scale no larger, within the limits of Snowflake NUMBER (precision to 38, scale to 37), Databricks DECIMAL (precision to 38) and BigQuery.fluid validaterefuses the rest on Snowflake, as on Databricks and BigQuery, andfluid generate iacrefuses it. - The dbt engine and the IaC emitters type a column alike on BigQuery and Snowflake. A dbt model builds into the table
fluid applycreates, and the two wrote many types differently. On Snowflakesmallint,money,json,geography,timestamp_ntz,timestamptzandtimestamp_ltzwereVARCHAR; they are now typed from one table. A Snowflake table applied before keeps its types, and Snowflake cannot change a live column to another type, so migrate the table of a column whose type changes. See Column types.
(since 0.33.0, forge-cli #992)
- New:
fluid doctor --contract,--operation,--live,--provider,--state-backendand--compare. A preflight that says whether an operation can run here, offline by default: each capability isexact,lossy,unsupportedorrequires-live-proof, and the exit code is1when one isunsupported.--liveadds read-only identity and state-backend checks.fluid doctor --envalone still lists the runtime switches. Seefluid doctor --contract.
(since 0.33.0, forge-cli #1004)
- A project
.envcan no longer set a name that is not a shell identifier, aBASH_FUNC_name, orBASH_XTRACEFD,ZDOTDIR,FPATH,NULLCMDorREADNULLCMD. Bash readsBASH_FUNC_<name>%%as the function<name>, so a project file could replace a command, orexec, in a bash script the CLI starts, such as Debezium Server'srun.sh, with your credentials in scope. The names go with the shell start-up settings that were already refused. A name has to be letters, digits and underscores, and not start with a digit: a key such asmy.setting,A-Bor1ABCin.env,.env.<environment>or.env.localis now ignored too.fluidnames each ignored variable and file with the usual warning, and never a value. Search the project's dotenv files for such a key, and set what you need in the shell or the CI job. See What a project's.envmay set.
(since 0.33.0, forge-cli #996)
- A Glue table is partitioned as
binding.location.partitionBydeclares. Through 0.32.0partitionByandbinding.properties.cluster_bynever reached the Glue table, and both passedfluid validate --strict. On a Hive table thepartitionBycolumns becomepartition_keys, with Athena partition projection when the contract gives every key's values. A live table whose keys or projection change, or that is partitioned for the first time, is replaced, andfluid applyrefuses that without--allow-data-loss.partitionByon an Iceberg or Delta table in Glue andcluster_byon any Glue table are now refused atfluid validate, so a contract that set either fails it. See Partitioned Glue tables.
(since 0.33.0, forge-cli #1001)
- The dbt engine writes no contract text into
schema.yml,sources.ymlorsemantic_models.ymlas Jinja. A name, description or value that holds{{,{%or{#, or looks like anenv_var(...)orref(...)call, ran when dbt parsed a project generated by 0.32.0. It is now written as a literal of itself, asfluid generate dbt-testswrites it. A model, source or table name or a version that holds a quote, a backslash, a brace or a control character is refused when the project is generated. Anaccepted_valuestest with a quote or backslash in a value is written withquote: falseand each adapter's SQL literal. The SQL of the models is not escaped: it is dbt Jinja, and only its{{ env.X }}placeholders are rewritten. See Contract text in a generated dbt project.
(since 0.33.0, forge-cli #1005)
fluid applystarts no EMR on EC2 step until the build's policy is attached to the cluster's instance-profile role. A step runs as that role, which the cluster's owner holds, and forge creates the build's least-privilege policy for the owner to attach. The first apply of a stream used to start it before anyone could, so, unless the role already held the job's permissions, the step failed at once and was not started again with the same config. Now the apply refuses, names the policy and prints theaws iam attach-role-policycommand, and ends with exit code1. Nothing was started, so the apply after the attach starts the stream. When forge cannot read the cluster's role, or the build names its ownexecutionRoleArn, the step starts as before. Give the identity that runsfluid apply, and a scheduler's role,elasticmapreduce:DescribeCluster,iam:GetInstanceProfileandiam:ListAttachedRolePolicies: the generatedfluid-<product>-spark-callerpolicy, which a scheduled build's scheduler attaches, now carries them. See An EMR on EC2 step starts once its policy is attached.
(since 0.33.0, forge-cli #975)
- A Spark build can run on the machine
fluid applyruns on:parameters.spark.runtime: local.fluid applywrites the build's job config and the runner under a root of its own, and startsspark-submiton that machine, on a laptop or a lab host. The output is a table in an Iceberg REST catalog onplatform: local. The operator sets whichspark-submitruns (FLUID_SPARK_SUBMIT_EXECUTABLE), the Spark properties file that declares the catalog (FLUID_SPARK_SUBMIT_PROPERTIES_FILE), the artifact root (FLUID_SPARK_LOCAL_ROOT) and the namespaces a local run may write (FLUID_SPARK_LOCAL_NAMESPACE_ALLOWLIST). A project.envcannot set them, and an unset list of namespaces allows none. Nothing changes for a contract with noruntime: localbuild. See Running a build on this machine. - Each Spark stream that runs on AWS restarts once on its next
fluid apply. The runner file changed, and a build's config hash covers the runner's sha256, so every Spark stream gets a new hash: apply stops the running job and starts it again on the same checkpoint. A build that runs to completion is not restarted. See Starting and reconciling runs. - A Spark stream can send the records it cannot land to dead letters:
properties.stream.deadLetter(0.7.7 preview). A record whose value or key is not the declared JSON, or that lacks a required field, goes to a table beside the output (<table>__dead_letterby default) or to a Kafka topic, and the stream keeps running until more thanmaxRecordsBeforeAbortdead letters (10000 by default) have gone out since the job started. It then stops after the batch that crossed the limit, andfluid applystarts it again only when the build changes. A topic is written only once the operator lists it inFLUID_SPARK_TARGET_ALLOWLIST, asmsk-write:<cluster ARN>/<topic>orkafka-write:<host:port>/<topic>, on every runtime. An Avro stream, the managed Flink lane, and a table beside a bucket-root output, a location forge registers with Lake Formation or an output whose lifecycle expires its objects are refused. A stream with nodeadLetterblock keeps its job config, and restarts once with the new runner, as the entry above says. See Dead letters. - A project
.envcan no longer set a name that starts withSPARK_,PYSPARK_,HADOOP_orYARN_.spark-submitreads these to choose the program, the JVM options, the Python and the cluster it uses. The names are ignored, in any letter case, with the usual warning. Set what you need in the shell or the CI job. See What a project's.envmay set.
(since 0.33.0, forge-cli #1003)
- New, opt-in:
fluid ship --journal-dirkeeps a stage journal, and every--release-dirrun keeps one. Each stage writes astartedevent before it runs and afinishedevent after, with its status and the digests of what it wrote. Before anapplythat is not a dry run, an enforcingpolicy-apply,publishor aschedule-syncthat is not a dry run, ship refuses the stage when an earlier attempt of it, or ofapply, has no known outcome, until you inspect the target and pass--acknowledge-inspected STAGE. Anapplythat did not pass is refused until stage 6 has planned again.--explainshows what the journal would refuse. Without either flag no journal is kept and no stage is refused, and the run report records a stage killed by a signal (exit 129 to 159) asunknowninstead offailed. See The stage journal.
Changes in 0.34.0
These changes are in 0.34.0.
(since 0.34.0, forge-cli #1008)
fluid applystops a removed Spark build's runs on AWS before it destroys what they run with. When a plan deletes a build's resources, because the build left the contract or the product went, the apply cancels the build's active runs and waits for them (FLUID_SPARK_WAIT_SECONDS, 900 seconds by default), and empties and stops the product's own EMR Serverless application. It refuses first, with theaws iam detach-*-policycommands, when an IAM policy that the plan destroys is still attached, and a run that is still going after the wait refuses the apply before anything is destroyed.--dry-runsays what it would cancel. The first apply after the upgrade adds three tags,fluid_runtime,fluid_run_targetandfluid_run_region, in place to each Spark build's log group and to an EMR on EC2 build's job policy. The identity that runsfluid applyneedsiam:ListEntitiesForPolicy, the list, get and cancel actions of the runtime that a removed build ran on, and, for the product's own application,emr-serverless:GetApplicationandemr-serverless:StopApplicationbesides the EMR Serverless list, get and cancel actions. See When a build leaves the contract.
(since 0.34.0, forge-cli #1007)
- Removing a Spark build on AWS needs no
--allow-data-loss, unless the build has a dead-letter table. The data-loss gate counted what a removed build ran with (its runner and job config objects, role and policies, log group and the key its logs are encrypted with, Glue job and connection, and the product's own EMR Serverless application) as data, so a plan that drops a build was refused unless the apply was allowed to destroy data, and that allowance also let a real data loss in the same plan through. They are now listed as removed and not gated. An output table, a bucket and a bucket's own key still count, and so does a stream's dead-letter table (sink: table), which goes with the build, so a plan that removes one of them needs the flag as before. A pipeline that passes it only to remove a build that has no dead-letter table no longer needs to. See When a build leaves the contract.
(since 0.34.0, forge-cli #1015)
- A Spark stream can bound what one batch reads:
properties.stream.maxOffsetsPerTrigger(0.7.7 preview). A whole number from 1 to 9223372036854775807, passed to Spark's Kafka source, so a stream that catches up after a stop, or starts fromearliest, lands in batches of bounded size,trigger.availableNowruns included. Unset, a batch reads everything that is waiting and the job config has no key. Managed Flink refuses the key. Setting it changes the job config, so a running stream restarts once on its next apply. See Bounding a batch. - Dead letters sent to a topic are keyed by where they came from. The key was the record's own key. It is
<topic>-<partition>@<offset>now, and the record's own key is in the headerfluid.dead-letter.raw-key, null for a record without a key and for a build that masks fields. A consumer that reads the key reads the header instead. A batch sent again after a failure carries the same keys, so a consumer drops repeats by key. Table dead letters are unchanged. See The topic. - The runner sets
failOnDataLosstotrueon every Kafka source. That is Spark's default, so a stream whose offsets the topic no longer holds fails, as before, and does not skip records. - Each running Spark stream restarts once on its next apply, on its checkpoint. The runner changed, and its sha256 is part of the config hash, so the hash of a build whose contract did not change is another one.
(since 0.34.0, forge-cli #1014, forge-cli #1019)
- A Spark run starts only on the job config that
fluid applycompiled. Every runtime, local included, passes the config's digest at submit (--fluid-job-sha256), and forge's runner refuses any other config before it reads anything. On AWS,fluid applyfirst reads the job config and the runner back from S3, and starts nothing unless they are the ones this apply compiled: the build fails withor its runner is not the one this apply compiled, andfluid applyends with exit code1. A running job of another config is not cancelled, and keeps going: runfluid applyagain, and it uploads both and starts the run. The identity that runsfluid applyneedss3:GetObjecton both objects, andkms:Decryptwhere the bucket encrypts them with a KMS key (the scheduler'sfluid-<product>-spark-callerpolicy holds both). Write access to the artifact bucket stays the boundary. The runner changed, so each running Spark stream restarts once on its next apply, on its checkpoint. See A run starts only on the config this apply compiled.
(since 0.34.0, forge-cli #1009)
fluid applyruns a dbt or Python build whatever its trigger. A dbt or Python build of typedataset,schedule_and_dataset,event,dependencyortimetablefailed underfluid applywithUnknown trigger type. That included the task of the DAG thatfluid generate schedulerenders for adatasetorschedule_and_datasetbuild (fluid apply --build-id). The dbt and Python runners now run such a build once.manualkeeps its iterations,continuousis still refused, and atrigger.typethat is none of the names the schema knows is refused. See What the DAG does.fluid applyfinds the dbt that has the build's adapter. An engine named after its package,dbt-athena-community, was looked for asathena-community, so a local dbt with the Athena plugin was never used. forge now asks for dbt's plugin name (athena), reads the adapter of an engine without one (dbt,dbt-core) from the build's platform, and matches thePlugins:list ofdbt --versionexactly.- The dbt container fallback is pinned, and refuses what it cannot pin or authenticate. It runs
python:3.12-slimpinned by digest and installsdbt-core==1.11.17and an exact adapter package from wheels only, with pip run isolated against PyPI.DBT_ADAPTER_PACKAGEmust list exact pins,DBT_DOCKER_IMAGEandDBT_BOOTSTRAP_IMAGEmust be pinned by digest, and an adapter with no pin of its own (glue, for one) runs in a container only withDBT_ADAPTER_PACKAGEorDBT_DOCKER_IMAGE. A build of an AWS adapter (athena,glue,redshift) whose dbt profile names an AWS profile, or ofathena,glueor a Redshift IAM build that has no AWS keys in the environment or the dbt profile, is refused beforedockerstarts, with the command that installs the adapter next to dbt or exports short-lived keys. forge mounts nothing of~/.aws. See Which dbtfluid applyruns.
(since 0.34.0, forge-cli #1010)
- A Spark job on AWS can write its own table where Lake Formation governs new tables. forge granted a job's role Lake Formation access to its table only when the output's binding declared a non-empty
governance.lakeFormationblock, so in an account that removed theIAM_ALLOWED_PRINCIPALSdefault for new tables, a stream without that block could not see its own table. The grant (INSERT,ALTER,DESCRIBE,SELECT, andDELETEfor a batch job) is now emitted for such an output and for a stream's dead-letter table beside it, with acountthat the plan decides from the account's settings: 1 where Lake Formation governs new tables, 0 where IAM alone decides, so an account that keeps the default creates nothing new. The identity that runsfluid planandfluid applyneedslakeformation:GetDataLakeSettings. An output with a non-emptygovernance.lakeFormationblock keeps its grants, and an EMR on EC2 job, which runs as the cluster's own role, gets none. See Lake Formation writes.
(since 0.34.0, forge-cli #1013, forge-cli #1019)
fluid apply --mode reconcilekeeps the Spark streams running and deploys nothing. It runs notofu, no DDL and no build, and it takes no plan. It starts a stream that died again on its checkpoint, within its restart budget, when what the last apply deployed is what the contract compiles to. When it is not, it starts nothing, exits1and names what to run.--build-id <stream>picks one stream, and a keepalive, a scheduler task every few minutes, is what runs it. See Reconcile mode.- A Spark stream that died starts again within a restart budget, on a plain
fluid applytoo. On EMR on EC2, Glue and a local build, a stream whose last run endedFAILEDwas an error until the build changed. It is started again on its checkpoint now, up toexecution.retries.maxAttemptstimes per rolling hour (5 when unset, at most 10), each restart carrying--fluid-restart-of <run id>, and a spent budget is the error. EMR Serverless and EMR on EKS keep their own retries, and forge starts nothing on top of them. A run cancelled through the runtime is left stopped by--mode reconcileand started again by a plainfluid apply.execution.retriesis no longer part of the config hash on EMR on EC2 and Glue, so a stream there that declares it restarts once on its next apply. A local run whose supervisor got a signal from outside forge, a shutdown of the machine say, reads as failed, not cancelled. Of two copies of one config the older is kept, where the newer was. See The restart budget. - The scheduler policy
fluid-<product>-spark-callercovers always-on streams, and its statements change shape. It lets a role start, watch and stop a stream's runs and read its job config and runner, so a keepalive can run on MWAA. It holds one statement for each runtime, with every build's target, role and artifacts, where it held one for each build, which reached the 6,144-character limit of a managed policy at about six builds. It holdskms:Decryptthrough S3 only, for those artifacts, so a bucket that encrypts them with a KMS key needs no IAM addition for the scheduler's role (the key's own policy must still allow it, unless it leaves that to the account's IAM policies). See The caller's permissions.
(since 0.34.0, forge-cli #1017)
- A destroy waits for an EMR Serverless application that is still being created or started. A
CREATINGorSTARTINGapplication is waited for until it settles: a created one is deleted as it is, a started one is stopped first, and one still moving afterFLUID_SPARK_WAIT_SECONDSrefuses the apply. The refusal's hint namesiam:ListEntitiesForPolicy,emr-serverless:GetApplicationandemr-serverless:StopApplication, which the identity that runs the destroy needs beyond the runtime's list and cancel actions. See When a build leaves the contract.
(since 0.34.0, forge-cli #1012)
- An Iceberg table on Glue plans no change after an engine commits to it. forge emitted the table's location with a trailing slash. Iceberg keeps it without one and writes it back into Glue on every commit, so each
fluid applyafter a commit planned an update (~1) for each table that had been written, and sent anUpdateTablewhile a stream kept committing. The location is now emitted as Iceberg keeps it. The first apply after the upgrade updates once, in place, the location of each Iceberg table that Glue still holds with the trailing slash, and a table in a Hive format keeps its trailing slash. See Iceberg tables in Glue.
(since 0.34.0, forge-cli #1018)
fluid generate schedulewrites a keepalive DAG for an always-on stream that declaresexecution.retries. On Airflow, a Spark stream that is nottrigger.availableNowand nottrigger.type: continuousgets a DAG whose one task,fluid_keepalive, runsfluid apply <contract> --mode reconcile --build-id <stream> --yesevery 5 minutes, or on the build'sexecution.orchestration.airflow.dagConfig.schedule. A stream that died starts again on its checkpoint, within the restart budget thatexecution.retriessets, and a run fails when the stream cannot be kept running. A stream that already declaresexecution.retriesgets the DAG from its nextfluid generate scheduleorfluid generate artifacts. A stream withoutexecution.retriesgets no DAG, as before. A contract whose only builds are such streams gets this DAG instead ofNo build needs a DAG; Prefect, Dagster andfluid scaffold-composerstill write nothing for it. The worker needs theFLUID_SPARK_*settings and the forge release that the CI apply has. See Keeping a stream running.
Changes in 0.35.0
These changes are in 0.35.0.
(since 0.35.0, forge-cli #1022)
- A plan made from a
fluid ship --release-dirrelease can be applied when the contract has a dbt build. The release now holdsbundle.tgz.provenance.json, listed inrelease.jsonfilesand outsideidentity. Through 0.34.0fluid applyrefused such a plan withapply_plan_digest_bundle_mismatch.fluid shipwarnsrelease_provenance_missingwhen it cannot write the file. See Publishing a release. - A plan can be applied in another checkout of the same tree. When the paths the plan records are gone,
fluid applyfinds the contract through the provenance file beside the bundle it verified the plan against. The dbt files must still match the packed copy. See The dbt project.
(since 0.35.0, forge-cli #1021)
data-product-forge[profile]installs flux-stats 0.7.0. The extra asked forflux-stats>=0.6.2,<0.7in 0.34.0 and asks forflux-stats[yaml]>=0.7.0,<0.8now. Upgrade the extra withpip install -U 'data-product-forge[profile]'.fluid profilestops withprofile_statistics_unavailableon an older flux-stats. See fluid profile.
Older releases
Upgrading from before 0.15.0? Start with the 0.15.0 checklist, then continue from From 0.15.x to 0.16. Step 11 there, about federation digests, now warns instead of failing the apply.