fluid generate
Unified artifact generation from FLUID contracts.
Syntax
fluid generate <transformation|speed-transformation|dbt|dbt-tests|schedule|ci|standard|artifacts|iac|vector|flink>
fluid generate flink <contract> --out <dir>
Subcommands
fluid generate artifacts
Stage 3 of the 11-stage pipeline. One call writes the catalog and execution artifacts for a contract:
- ODCS (v3.1.0) contract files under
odcs/ - ODPS-Bitol (v1.0.0) product files under
odps-bitol/ - OPDS (v4.1, LF/ODPI) product file under
opds/ - Airflow DAGs under
schedule/, when the contract has anorchestration.engineor a scheduled build - Policy bindings under
policy/bindings.json(IAM / GRANT) - A
MANIFEST.json(SHA-256 per file and a merkle root) that stage 4 verifies
fluid generate artifacts runtime/bundle.tgz --out dist/artifacts/ # the pipeline path
fluid generate artifacts contract.fluid.yaml --out dist/artifacts/ # local shortcut
fluid generate artifacts contract.fluid.yaml --emit odcs,odps-bitol # catalog only
fluid generate artifacts runtime/bundle.tgz --env aws \
--contract-path contracts/orders/contract.fluid.yaml # DAGs that run `fluid apply --env aws`
Options: CONTRACT, --out, --emit, --manifest, --env, --contract-path. The output tree, the emit keys, the scheduled-build DAGs and the fragment-layout limitation are on the generate artifacts page. Verify the emitted tree with fluid validate-artifacts (stage 4).
fluid generate flink
(since 0.27.0, forge-cli #784) Compiles the managed Flink builds of a 0.7.7 contract into what fluid apply would deploy, and writes it for review. Nothing contacts a cluster.
fluid generate flink contract.fluid.yaml --out runtime/flink
It takes the contract, --out (-o) for the directory (default runtime/flink), and --env for an environment overlay. It prints Generated 1 managed Flink application(s) in <dir>, and <dir> holds a manifest.json and, under a directory named for each build's deployment namespace (data-products/ in the example), a .sql file (the Flink SQL) and a .yaml file (the ConfigMap and the FlinkDeployment) for each build. Managed Flink to Iceberg covers the whole lane.
fluid generate transformation
Generate transformation artifacts from the contract's builds. The engine is read from builds[].engine; fluid generate transformation --list prints the engines the CLI has:
dataflow patterns: hybrid-reference, embedded-logic
dataform patterns: hybrid-reference, embedded-logic, multi-stage
dbt patterns: hybrid-reference, embedded-logic, multi-stage
glue patterns: hybrid-reference, embedded-logic
spark patterns: hybrid-reference, embedded-logic, multi-stage
sql patterns: embedded-logic, multi-stage
fluid generate transformation
fluid generate transformation contract.fluid.yaml -o ./dbt_project
fluid generate transformation contract.fluid.yaml -o ./dbt_project --dbt-validate
fluid generate transformation contract.fluid.yaml --all-builds -o ./out
Key options:
contract(default:contract.fluid.yamlin the current directory)--output,-o— output directory (default: the build'srepository, else./<engine>_project)--build-index— which build to generate for (default0)--all-builds— generate for every build in the contract--concurrency N— how many builds--all-buildsgenerates in parallel (default4)--model PATH— a forged logical sidecar (*.model.json); see below for the automatic case--overwrite--env--model-contracts— emit enforced dbt model contracts on every expose model (contract: {enforced: true}plus per-columndata_typeandconstraintsderived fromexposes[].contract.schema[]), sodbt buildfails in producer CI when the model's output drifts from the contract. Opt-in; dbt-only (dbt-core ≥ 1.5).--dbt-tests-key {auto,tests,data_tests}— which YAML key generated dbt data tests attach under. The defaultauto(also$FLUID_DBT_TESTS_KEY) detects the dbt binary you would run: Fusion and dbt-core ≥ 1.8 emitdata_tests:, dbt-core before 1.8 and "no dbt found" emit the legacytests:. Pass an explicit value in CI generators that have no dbt binary. dbt-only.--mesh-hub HUB_PROJECT_NAME— declare a dbt Mesh hub: emitsdependencies.ymlwithprojects: [{name: HUB_PROJECT_NAME}](dbt-core ≥ 1.6). dbt-only. See Mesh hub.--dbt-validate— after generating a dbt project, rundbt parseagainst it. Skipped without a dbt binary, and for other engines.--list,--verbose,--quiet(-q)
When the contract was produced by fluid forge data-model, the generator loads the logical sidecar named by labels.modelSidecar and emits deterministic SQL from the forged logical model. For dbt output, zero generated models/**/*.sql files is a hard failure rather than a quiet success.
fluid generate speed-transformation and fluid generate dbt are aliases for this path.
What the sql engine emits
Behaviour change in 0.16.0
The sql engine used to emit one inert SELECT per stage. Each stage of a multi-stage build now runs CREATE OR REPLACE VIEW. Regenerate and re-run any scripts you committed from an earlier version.
fluid generate transformation contract.fluid.yaml -o gen_sql
consumes_not_wired: build customer_360_pipeline uses the 'sql' engine, which never reads consumes[], ...
Generated 5 files (sql engine):
gen_sql/01_stage_1_customer_base.sql
gen_sql/02_stage_2_purchase_summary.sql
gen_sql/03_stage_3_engagement_summary.sql
gen_sql/04_stage_4_rfm_calculation.sql
gen_sql/05_stage_5_high_value_filter.sql
gen_sql/01_stage_1_customer_base.sql:
-- Generated by fluid generate — do not edit manually.
-- Regenerate with: fluid generate
-- Contract: gold.customer.analytics_360_v1
-- Stage: stage_1_customer_base (1/5)
-- Materialises: customer_base (CREATE OR REPLACE VIEW)
CREATE OR REPLACE VIEW customer_base AS
SELECT
customer_id,
...
Files are
NN_<stage>.sql, numbered by the stage's position instages[]. A runner that executes them in filename order runs the stages in declaration order, so declare stages in dependency order; the generator warns (sql_stage_order_not_topological) when a stage precedes one it depends on.The view is named after the stage's declared output, the first entry of
stages[].outputs, because a contract's SQL refers to stages by output name (FROM customer_base). A stage with no output of its own, or whose output name another stage also claims, gets-- Not materialised: <reason>in the header instead, and its SQL is emitted as a bare statement. SQL that already writes to a target of its own (INSERT,CREATE,MERGEand the other statements that name a sink) is also emitted as it is, with a-- Not materialised:header.Declared inputs are bound.
builds[].properties.parameters.inputsentries (name,path,format) become00_inputs.sql, which sorts first:-- customers_raw <- data/customers.csv CREATE OR REPLACE VIEW customers_raw AS SELECT * FROM read_csv_auto('data/customers.csv', AUTO_DETECT:=true, DELIM:=',');This is the statement
fluid applyruns for the local provider. Paths resolve relative to the directory the script runs in. The file is written only for a local (local/duckdb) build platform; for any other platform the generator warnssql_inputs_not_bound_for_platformand writes nothing.A single-stage
embedded-logicbuild emits one bareSELECTnamed after the build id (clean_customers.sql), with no view around it.
A re-run of CREATE OR REPLACE VIEW is not harmless. It drops the grants on the view in Snowflake and Databricks and makes a Snowflake stream over the view stale, and it fails when a table of the same name already exists. Review the generated scripts before pointing them at a shared schema.
How dbt models are named
(since 0.16.0) A multi-stage build emits one model per stage, and the model file is named after the stage's declared output, not the stage's name:
fluid generate transformation dbt.fluid.yaml -o gen_dbt
Generated 10 files (dbt engine):
gen_dbt/dbt_project.yml
gen_dbt/packages.yml
gen_dbt/profiles.yml
gen_dbt/models/
sources.yml
gen_dbt/models/marts/
customer_360_master.sql
customer_base.sql
engagement_summary.sql
high_value_customers.sql
purchase_summary.sql
schema.yml
For the stage named stage_4_rfm_calculation with outputs: [customer_360_master], the file is customer_360_master.sql.
- The first usable entry of
outputsnames the model. A stage with no output keeps its own name, and a name two stages both claim goes to neither (the generator logsdbt_stage_model_name_collision). dependsOnentries are rewritten to model names, soref()calls resolve.- The directory still comes from the stage name: a name containing
staging,stg_,extractorrawgoes tomodels/staging/; one containingintermediate,int_orprepgoes tomodels/intermediate/; any other goes tomodels/marts/.
A project generated before 0.16.0 has different model and file names, and regenerating renames them. The contract's data tests now attach to models that exist, so a data-quality rule with no dbt equivalent appears as a fail-loud fluid_* sentinel test (for example fluid_valid_values_email), where before it attached to a model that was not there.
Two warnings cover a generated file that names a model the run did not emit:
| Warning | What happens |
|---|---|
dbt_schema_yml_model_missing | A schema.yml entry whose model was not emitted is kept, and the warning says its tests and column docs attach to nothing. It is kept so that fluid verify --reconcile-dbt, which reconciles the contract's exposes against schema.yml, does not start failing with model_missing_in_dbt. |
semantic_models: dropping ... | A semantic model or metric whose ref target was not emitted is dropped from semantic_models.yml. |
Engines and consumes[]
Only the dbt engine turns consumes[] into something: it emits models/sources.yml. The other engines (sql, spark, glue, dataform, dataflow) do not read consumes[], and say so once per build, naming the upstreams, with exit code 0:
consumes_not_wired: build customer_360_pipeline uses the 'sql' engine, which never reads consumes[], so nothing it generates is derived from the 3 declared upstream(s): bronze.customer.raw_customers_v1.raw_customer_data, bronze.sales.raw_orders_v1.raw_order_data, bronze.support.raw_interactions_v1.raw_interaction_data. Whether the emitted code happens to reference them depends entirely on what the build's own SQL names. Point the build at them by hand, or use the dbt engine, which emits them as sources.
The customer-360 template triggers it. Wire the upstreams by hand in the build's SQL, or use the dbt engine.
For a dbt build, an upstream the generator cannot find in the workspace gets a source whose database and schema fall back through environment variables, so dbt parse does not fail on a missing variable:
database: '{{ env_var(''FLUID_SOURCE_DATABASE'', target.database) }}'
schema: '{{ env_var(''FLUID_SOURCE_SCHEMA'', target.schema) }}'
For a Snowflake, BigQuery (gcp), Redshift or Athena (aws) binding, a platform variable sits between the two: SNOWFLAKE_DATABASE / SNOWFLAKE_STAGE_SCHEMA, GCP_PROJECT / BIGQUERY_DATASET, REDSHIFT_DATABASE / REDSHIFT_SCHEMA, ATHENA_DATABASE / ATHENA_SCHEMA. The generated project's output above is for a local binding, which has none.
Contract text in a generated dbt project
(since 0.33.0, forge-cli #1001)
dbt renders the names, descriptions and test arguments of a properties file as Jinja, and it runs a test argument that looks like env_var(...), var(...), ref(...), source(...) or doc(...) even without braces. Through 0.32.0 the dbt engine wrote a contract's text into schema.yml, sources.yml and semantic_models.yml as it was, so a description of {{ exceptions.raise_compiler_error(env_var('HOME')) }} ran when dbt parsed the project. The engine now writes each such string as a Jinja literal of itself, the way fluid generate dbt-tests does, and dbt reads it back as the same text:
# contract: description: "{{ exceptions.raise_compiler_error(env_var('HOME')) }}"
- name: orders
description: '{{ "{{ exceptions.raise_compiler_error(env_var(''HOME'')) }}" }}'
What is escaped. A string that holds
{{,{%or{#, or looks like one of those calls. Every other string is written as before, so the project of a contract with plain names is unchanged. Aref('<model>')that the engine writes itself for a relationship test stays as it is.Source placeholders. In
sources.yml, a{{ env.X }},${X}or dbtenv_var(...)placeholder, with or without a default, still becomes anenv_var(...)that dbt resolves at parse time. The text around a placeholder is escaped where it holds a brace.Semantic models.
semantic_models.ymlgets the same escaping. A metric'sfilteris written as it is, because dbt never renders it.Refused names. dbt runs a model's tests against
ref('<name>', version='<v>')and a source table's againstsource('<source>', '<id>'), Jinja that it builds by pasting the name in, so no literal of the name helps. A model name, source name, table name or version that holds a quote, a backslash, a brace or a control character is refused when the project is generated. The command exits1and the error names the value:exposeId "orders') }}{{ env_var('SECRET'" cannot name a dbt node: dbt writes it into Jinja it runs, so it must hold no quote, backslash, brace or control characterAn integer contract version stays an integer.
accepted_valueswith a quote or a backslash. dbt pastes each value between single quotes, so a value such asit'sor a backslash broke the test. A test with such a value is written withquote: false, and each value is written as the SQL string literal of each adapter, chosen bytarget.type. The values that need nothing are written as before, hereA:- accepted_values: values: - '''A''' - '{% if target.type in ["duckdb"] %}{{ "''it''''s''" }}{% elif target.type in ["bigquery"] %}{{ "''it\\''s''" }}{% else %}...{% endif %}' quote: false
This covers the three properties files. It does not cover the SQL of the models. (since 0.33.0, forge-cli #1011) The SQL of a generated model, and a contract's own SQL, is dbt Jinja and still renders as dbt Jinja: {{ ref('upstream') }} and {{ source('raw', 't') }} in a stage's SQL run as dbt reads them. Only the {{ env.X }} and ${X} placeholders in it are rewritten, to env_var('X'), as in 0.32.0. So a contract author is trusted for dbt: whoever can write the contract's SQL can run Jinja when dbt parses the project.
dbt compatibility
The generator reads dbt --version from $DBT_EXECUTABLE or PATH, and chooses the YAML shapes the detected dbt understands:
| Generated shape | Emitted when dbt-core is | Otherwise |
|---|---|---|
Generic-test inputs nested under arguments: | 1.10.8 or later, or dbt v2 (Fusion) | Flat inputs. |
Source freshness and loaded_at_field under config: | 1.10.5 or later, or dbt v2 (Fusion) | At the top of the source table. |
Model access is written under config: whatever version is detected.
If no dbt is detectable, the generator emits the older shapes. dbt v2 rejects those, so a CI job that generates without dbt installed and then runs dbt v2 fails at parse. Install the dbt you will run in the job that runs generate, or pass --dbt-tests-key explicitly for the tests key. forge-cli's source records the floors above as measured on dbt-core 1.8.10, 1.9.11, 1.10.0, 1.10.5, 1.10.8, 1.12.5 and 2.0.4.
Up to 0.33.0, when fluid apply runs a dbt build in a container, the fallback when the local dbt cannot run the build's adapter, the container installs dbt-<adapter><2 and dbt-core<2 into python:3.12-slim. Three variables change that: DBT_BOOTSTRAP_IMAGE replaces the base image, DBT_ADAPTER_PACKAGE replaces the install (and drops the dbt-core<2 cap, which is how you opt into v2), and DBT_DOCKER_IMAGE runs a prebuilt image that already has dbt. (since 0.34.0, forge-cli #1009) The fallback installs pinned packages into a pinned image, and the variables take pinned values only: see Which dbt fluid apply runs.
Which dbt fluid apply runs
(since 0.34.0, forge-cli #1009) fluid apply runs a dbt build with the dbt that $DBT_EXECUTABLE names, as it is and whatever adapter it has. Without $DBT_EXECUTABLE, it uses dbt on PATH, or the dbt next to the Python that runs forge, when that dbt has the build's adapter. Otherwise, with Docker available, it runs the build in a container.
The adapter. The build's engine names it, as dbt-snowflake names snowflake. The engines dbt and dbt-core name none, so the build's execution.runtime.platform does: snowflake, bigquery (and gcp), duckdb, redshift (and aws), postgres (and postgresql), spark, athena (and aws-athena), glue (and aws-glue) and databricks. dbt lists an adapter under its plugin name, which can differ from its package's: dbt-athena-community installs the plugin athena. For dbt-core v1, forge matches the plugin name exactly against the names listed under Plugins: in dbt --version. A dbt Fusion (v2), which lists none, is used for any adapter. In 0.33.0 the engine dbt-athena-community was looked for as athena-community, so a dbt that had the Athena plugin, found without $DBT_EXECUTABLE, was never used, and the build went to a container. A build that names no adapter runs on the local dbt as it is.
The container. With no local dbt that has the adapter, forge runs docker run on an image pinned by digest, installs the packages in it from wheels only, and runs dbt. This is the command that fluid apply prints for an Athena build, with the project's path shortened:
docker run --rm -v <project>:/workspace/project -w /workspace/project -v <project>:/workspace/profiles -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY python:3.12-slim@sha256:5e2bdb1b0d542d5039cffb8181c65930093949194a65c365dfb1b0fe13a4fa5b sh -lc PIP_CONFIG_FILE=/dev/null python -I -m pip install --isolated --index-url https://pypi.org/simple --quiet --no-cache-dir --disable-pip-version-check --only-binary=:all: dbt-core==1.11.17 dbt-athena-community==1.11.1 && dbt build --no-partial-parse --project-dir /workspace/project --profiles-dir /workspace/project --profile lake
The environment goes in by name (-e NAME), so no value is on the command line. pip runs isolated: -I keeps the mounted project off its path, the index is PyPI and no pip configuration file loads, and no PIP_* setting is passed into the container, so a project .env cannot point pip at another index. A PYTHON* setting reaches the container only when forge sets it itself or FLUID_DBT_FORWARD_ENV names it. forge pins dbt-core and one package for each of these adapters. Their dependencies resolve within those packages' own ranges, and nothing is hash-locked.
| Adapter | Package pinned |
|---|---|
athena | dbt-athena-community==1.11.1 |
bigquery | dbt-bigquery==1.12.4 |
clickhouse | dbt-clickhouse==1.10.3 |
databricks | dbt-databricks==1.13.0 |
duckdb | dbt-duckdb==1.11.1 |
postgres | dbt-postgres==1.11.0 |
redshift | dbt-redshift==1.11.2 |
snowflake | dbt-snowflake==1.12.5 |
spark | dbt-spark==1.11.1 |
trino | dbt-trino==1.11.0 |
dbt-core is pinned at dbt-core==1.11.17. An adapter that is not in the table, glue for one, runs in a container only when DBT_ADAPTER_PACKAGE or DBT_DOCKER_IMAGE is set. Without either, and once the AWS checks below pass, the build is refused with forge has no pinned package for the dbt adapter 'glue', so it will not install one in a container. Install the adapter next to your dbt, or set DBT_ADAPTER_PACKAGE to an exact pin (for example dbt-<name>==1.2.3). Without Docker, a local dbt that lacks the adapter refuses the build too:
❌ Unable to prepare dbt build 'marts': dbt executable '/venv/bin/dbt' has no 'athena' plugin. Install the adapter next to it (pip install 'dbt-athena-community==1.11.1'), set DBT_EXECUTABLE to a dbt that has it, or make Docker available.
The variables. Each takes only a pinned value, and refuses the rest before docker starts:
DBT_BOOTSTRAP_IMAGEis the base image that dbt is installed into, pinned by digest (name[:tag]@sha256:<64 hex digits>). A tag is refused:DBT_BOOTSTRAP_IMAGE='python:3.12-slim' names a tag, and a tag can move: pin it by digest (python:3.12-slim@sha256:<digest>, fromdocker buildx imagetools inspect python:3.12-slim).DBT_ADAPTER_PACKAGEis one or more exact requirements (name==version), separated by spaces or commas. They replace the adapter's pin, and forge addsdbt-core==1.11.17unless the list names dbt-core, which is how a build runs another dbt-core, v2 included. A range is refused:DBT_ADAPTER_PACKAGE must list exact pins (name==version), not 'dbt-glue>=1.0': the dbt container installs only pinned packages.DBT_DOCKER_IMAGEis a prebuilt image that has dbt, pinned by digest. forge installs nothing in it. A tag is refused as for the base image.
AWS adapters. The adapters athena, glue and redshift authenticate to AWS, and a container has no access to the host's AWS profiles: forge mounts the project and the profiles directory and nothing of ~/.aws, and it does not pass in AWS_PROFILE, AWS_DEFAULT_PROFILE, AWS_CONFIG_FILE, AWS_SHARED_CREDENTIALS_FILE, AWS_CA_BUNDLE or the two token-file paths (AWS_WEB_IDENTITY_TOKEN_FILE, AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE). Before docker starts, forge refuses a build that would find no credentials: when the dbt profile names an AWS profile (aws_profile_name or iam_profile), and when neither the environment (AWS_ACCESS_KEY_ID with AWS_SECRET_ACCESS_KEY, or the container credentials URI) nor the dbt profile holds keys. A Redshift build that does not use IAM needs none. The first message names the profile's field and never its value, and both give the two ways out:
❌ Unable to prepare dbt build 'marts': the local dbt has no 'athena' plugin, so this build would run in a container, and the dbt profile names an AWS profile (aws_profile_name), which a container cannot read. Install the adapter next to dbt (pip install 'dbt-athena-community==1.11.1'), or export that profile's short-lived keys (eval "$(aws configure export-credentials --profile <that profile> --format env)") and run without naming a profile (unset AWS_PROFILE, or drop aws_profile_name from the dbt profile)
❌ Unable to prepare dbt build 'marts': the local dbt has no 'athena' plugin, so this build would run in a container, and neither the environment nor the dbt profile holds AWS keys, and a container cannot read the host's AWS profiles. Install the adapter next to dbt (pip install 'dbt-athena-community==1.11.1'), or export short-lived keys (eval "$(aws configure export-credentials --format env)")
These outputs are from forge-cli's dbt runner, the code fluid apply calls for a dbt build, with a stand-in for the local dbt (its --version output) and for docker: no container ran and no AWS call was made. The project's path is <project>.
Mesh hub
fluid generate transformation dbt.fluid.yaml -o gen_mesh --mesh-hub central_hub
writes dependencies.yml:
projects:
- name: central_hub
packages:
- package: dbt-labs/dbt_utils
version:
- '>=1.4.0'
- <1.5.0
dbt does not allow packages.yml and dependencies.yml in one project, so the package pins that packages.yml would carry are folded into dependencies.yml, and no packages.yml is written. --dbt-validate runs dbt deps before dbt parse when either file exists and dbt_packages/ does not.
Errors
Before it writes a file, the generator resolves each generated path and checks it against the output directory. A contract field that becomes a filename, such as builds[].properties.stages[].name, passes fluid validate with any string, so it can hold a path:
generated_path_outside_output_dir: {'path': '01_../../../../ESCAPED.sql', 'output_dir': 'gen_evil', 'hint': "A generated file path escapes the output directory. This normally means a contract field that becomes a filename (e.g. builds[].properties.stages[].name) contains path separators or '..'."}
The command exits 1 and writes nothing. generated_path_collision is the same refusal for two keys that resolve to one file (02_z/../01_a.sql normalises onto 01_a.sql), which would otherwise leave one stage's SQL silently replaced. Both checks run before any engine writes a file. Stage names become filenames: keep path separators and .. out of them.
fluid generate dbt-tests
Read the contract's exposes[].contract.dq.rules[] and emit a dbt schema.yml so your data-quality rules run as native dbt tests.
fluid generate dbt-tests
fluid generate dbt-tests contract.fluid.yaml -o dbt/models/schema.yml
fluid generate dbt-tests --env prod
Key options:
| Option | Description |
|---|---|
contract | Contract path. Defaults to contract.fluid.yaml in the current directory. |
--out, -o | Output path for the dbt schema.yml (default ./schema.yml). Drop it into your dbt project's models/<schema>/ directory. |
--env | Environment overlay (dev / test / prod). |
--dbt-tests-key {auto,tests,data_tests} | Which YAML key the tests attach under; same rule as generate transformation. Default auto, also $FLUID_DBT_TESTS_KEY. |
--macros-dir DIR | (since 0.33.0, forge-cli #991) Where the fluid_rule macro goes, as DIR/fluid/fluid_rule.sql. Default: the first macro-paths entry of the nearest dbt_project.yml above --out, or macros when it names none. A relative DIR is relative to the current directory. Only a contract that declares rules uses it. See Declared rules as dbt tests. |
--coverage | (since 0.33.0, forge-cli #997) Prints which declared rules each engine can run, and why not where one cannot. See Which engines run each rule. |
--coverage-out PATH | (since 0.33.0, forge-cli #997) Writes the same report as JSON to PATH. Missing parent directories are created. |
--require-rule ID | (since 0.33.0, forge-cli #997) Repeatable. Fails, and writes no tests, when a selected adapter cannot run that rule, or nothing tests it (a placeholder declaration with no bound). |
--require-all-rules | (since 0.33.0, forge-cli #997) The same check for every declared rule that has something to test. |
--adapter TYPE | (since 0.33.0, forge-cli #997) Repeatable. An adapter the required rules must run on: duckdb, snowflake, bigquery, databricks, spark, postgres, athena or trino. Default: the adapter of each platform the exposes bind to (never spark or trino: name those with --adapter), or all eight when none maps to one. It changes nothing without a requirement. |
- Column-level rules map to dbt's built-in tests —
not_null,unique,accepted_values— withdbt-utilsplaceholders emitted for rules that have no native dbt equivalent. - Model-level rules map to
dbt-utils:anomaly_detection/drift_detectionrules becomedbt-utilsexpressions, and afreshnessrule becomesdbt_utils.recency. - The generated
dbt-utils-based tests require thedbt-utilspackage to be installed in your dbt project. - A generated file starts with
# managed-by: fluid. Re-running the command regenerates a file that has the header, so edits to it are lost. A file without the header is not overwritten: the command fails withdbt_tests_refusing_overwriteand tells you to delete the file or pick another--out.
Declared rules as dbt tests
(since 0.33.0, forge-cli #991)
On a fluidVersion: 0.7.7 contract, the command also writes the rules the contract declares in Rules about valid data. Each invariant, identifier format, bounded placeholder declaration, shared column and shared attribute becomes one model-level fluid_rule test on the model of the expose that declares it. A placeholder declaration with no operator is not bounded, and gets no test. The test is named fluid_ plus the rule's id, with each character outside letters, digits and _ turned into _ and a repeated name given _2, _3: fluid_gross_from_net for an invariant, fluid_product_code_identifierFormat and fluid_product_code_placeholders for a column, fluid_account_status for a shared column of the account relationship, and fluid_customer_segment_orders_segment for the segment column of orders in the shared attribute customer_segment. A dependency with bothWays writes a second test with _reverse added. A contract of an earlier version gets no fluid_rule test and no macro.
The tests need the fluid_rule macro, which the command writes to <macro folder>/fluid/fluid_rule.sql in the same run. For a project whose dbt_project.yml names no macro-paths:
dbt/
├── dbt_project.yml
├── macros/fluid/fluid_rule.sql
└── models/schema.yml # fluid generate dbt-tests -o dbt/models/schema.yml
fluid generate dbt-tests contract.fluid.yaml -o dbt/models/schema.yml
fluid generate dbt-tests contract.fluid.yaml -o schema.yml --macros-dir dbt/macros # --out is outside the project
The macro file starts with -- managed-by: fluid. A run replaces a macro file that starts with it and never one that does not: it fails with dbt_tests_refusing_overwrite and refusing to overwrite <path>: it is not managed by fluid (missing the '-- managed-by: fluid' first line). Delete it or pass another --macros-dir., before it writes schema.yml. Three more refusals are dbt_tests_no_macro_dir: no dbt_project.yml above --out, a dbt_project.yml that cannot be read, and a first macro-paths entry that is absolute or has .. in it. The first and the last name --macros-dir as the way out. None of them writes a file.
One project, every adapter. Each test carries the SQL of the rule for each adapter forge writes, and the macro picks the one for target.type: duckdb, snowflake, bigquery, databricks, spark, postgres, athena and trino. A project that tests on DuckDB locally runs the same tests on Athena in production. On an adapter the rule has no SQL for, the test fails when it runs, with an error that names the rule and the adapter, and dbt parse still succeeds. The command warns at generation which adapters those are. A rule that no adapter can run is logged as a warning with its reason, gets no test, and is not approximated. fluid validate says what to change.
The band is compared exactly. With unit: rows the test compares the violating rows with the bound. With unit: percent it compares 100 * violating with value * covered, so a boundary row is never lost to a division. severity: error fails the test, and warning and info warn.
Placeholders. A column's placeholder values and prefixes are exempt from the column's own accepted_values and range tests, through a where config with one branch for each SQL dialect (spark shares the branch of databricks, and trino that of athena). An adapter with no branch keeps every row. A column name, placeholder value or prefix that holds a backslash gets no where, so its placeholders are not exempt.
Contract text is data. dbt renders a test's arguments as Jinja, and it evaluates one that looks like env_var(...) even without braces. Every piece of contract text in a fluid_rule test is written as a Jinja string literal of itself, so an invariant id of {{ exceptions.raise_compiler_error(env_var('HOME')) }} stays the text. The rest of schema.yml follows the same rule: a column name, description, enum value or test argument that holds {{, {% or {#, or looks like a call to env_var, var, source or doc, or to ref with anything but a plain model name, is written as a literal, and any other string is written as before. ref('orders') is left as it is.
Which engines run each rule
(since 0.33.0, forge-cli #997)
--coverage prints one row for each declared rule and one column for each engine forge writes SQL for, before it generates anything else, and the command then carries on and writes its files:
rule expose duckdb snowflake bigquery databricks postgres athena
----------------------- -------- ------ --------- -------- ---------- -------- ------
gross_from_net invoices yes yes yes yes yes yes
bad invoices no no no no no no
group_name.placeholders invoices - - - - - -
yes: runs exactly; no: not run (reason below); -: declared, nothing to test
Why not:
bad (invoices) on every engine: the validator refuses it (... FLUID SQL/1 has no pattern matching ...)
group_name.placeholders (invoices) on every engine: declared, and no operator bounds it
yes is a rule the engine runs exactly. no is a rule it cannot run, with the reason below the table: the validator refuses the rule, or a construct has no exact SQL on that engine. - is a placeholder declaration that no operator bounds, so there is nothing to test. spark uses the SQL of databricks, and trino that of athena, so they have no column. A contract that declares no rules, or declares them below fluidVersion 0.7.7, prints No declared rules about valid data (exposes[].contract.invariants[] and friends).
--coverage-out PATH writes the same report as JSON: engines, the list of engines, and rules, one object for each rule with id, expose, kind, column and engines. Each engine entry has a status (executable, unsupported or skipped) and, where there is one, a reason.
Rules that must run. --require-rule ID (repeatable) and --require-all-rules turn the report into a gate. The ID is the first column of the table. The command stops before it writes schema.yml or the macro, with exit 1 and one of these errors:
| Error | Raised when |
|---|---|
dbt_tests_required_rule_unsupported | A selected adapter cannot run a required rule, or a rule named by --require-rule has nothing to test (a placeholder declaration with no bound). The message lists each one as <rule> on <adapter>: <reason>. --require-all-rules passes over declarations with nothing to test |
dbt_tests_unknown_required_rule | --require-rule names an id the contract does not declare. The message lists the declared ids |
dbt_tests_unknown_adapter | --adapter names a type forge writes no SQL for. The message lists the eight |
dbt_tests_rules_need_preview | The contract declares rules about valid data below fluidVersion 0.7.7, where none of them runs, so no requirement can be met |
fluid generate dbt-tests contract.fluid.yaml -o dbt/models/schema.yml \
--require-rule gross_from_net --adapter snowflake --adapter athena
fluid generate dbt-tests contract.fluid.yaml -o dbt/models/schema.yml --require-all-rules
A coverage file asked for with --coverage-out is written before the requirement is checked, so a refused run still leaves it.
No dbt was available to run dbt test on a generated project. The generated YAML and its quoting come from the CLI's output, read back with a YAML and a Jinja parser.
After generating, run dbt test in your dbt project to execute the rules.
fluid generate schedule
Generate orchestration artifacts: Airflow DAGs, Dagster pipelines or Prefect flows. The scheduler is read from orchestration.engine, or set with --scheduler.
fluid generate schedule contract.fluid.yaml --scheduler airflow -o dags
fluid generate schedule contracts/orders/contract.fluid.yaml --env aws
fluid generate schedule --list
Key options:
contract--output,-o— output directory (default./dags,./pipelinesor./flows, by scheduler)--scheduler—airflow,dagsterorprefect--overwrite— overwrite an existing output directory. (since 0.33.0, forge-cli #966) For afluid applyDAG it also removes stale DAG files, as the build-graph list below says.--env— environment overlay. For afluid applyDAG it is also the--enveach scheduled run passes tofluid apply.--contract-path PATH— the contract's path relative to$FLUID_PROJECT_DIRon the Airflow worker, for afluid applyDAG. Default: thecontractargument relative to the current directory.--dag-id ID,--schedule SCHEDULE— (since 0.31.0, forge-cli #923) Override the DAG id and the schedule (a cron expression or an Airflow keyword) of the one DAG rendered from anorchestrationsection. The contract id stays in the DAG's header and tags. A DAG that comes from a scheduled build takes both from its trigger, so these options are refused there, withgenerate_schedule_override_not_supportedand exit 2.--list,--verbose
Builds that declare execution.trigger.schedule (or cron) and no explicit orchestration.tasks get Airflow DAGs, with Airflow as the default engine. Each DAG task runs its build through fluid apply --mode amend-and-build:
fluid generate schedule sched.fluid.yaml --env aws -o dags_out
Generated 1 files (airflow scheduler):
dags_out/customer_360_pipeline_dag.py
The DAG's contents, its worker requirements and the environment it passes to fluid are on the generate artifacts page. The DAG id carries the env: <product>__<env>__<build>.
(since 0.33.0, forge-cli #966) generate schedule also renders the build graph:
- Builds that
builds[].dependencieslink run in one DAG, onefluid apply --build-idtask per build, each after the builds it names. A dependency'scondition(success,completion,always) becomes the task's trigger rule (all_success,all_done,always). - A build with
trigger.type: datasetgets a DAG that runs when the datasets intrigger.datasetsare updated (all of them, or any one withdatasetsOperator: any).schedule_and_datasetruns on its cron and on those updates. A dataset is an Airflow asset URI, or{productId, exposeId}of an expose of the contract. - Each task declares the exposes its build writes as
outlets, so another DAG's dataset trigger can start on them. - A cycle, a dependency on the build itself or on no build, and the other graph errors stop generation with
generate_schedule_invalid_scheduleand exit 2. - A contract whose only builds are streams that
fluid applykeeps running printsNo build needs a DAGand writes no files. In 0.32.0 and earlier it printedNo scheduler engine specifiedwhen the contract set no engine. (since 0.34.0, forge-cli #1018) A stream that declaresexecution.retriesis the exception on Airflow:generate schedulewrites it a keepalive DAG, whose one task runsfluid apply --mode reconcile --build-id <stream> --yesevery 5 minutes. Prefect, Dagster andfluid scaffold-composerwrite nothing for it. See Keeping a stream running. - With
--overwrite, DAG files an earlier run wrote into-ofor the same product and env that the contract no longer makes are removed.
The Airflow integration has a worked example, the asset URI of each platform and the errors.
generate schedule writes the DAG file straight into -o, with no per-product directory. fluid schedule-sync with its default --delete-scope product refuses a directory that is not one directory per product, so write to -o <dags-dir>/<product-id>/; see What gets deleted. A DAG id that changed in 0.17.0 needs the one-time retirement of the old DAG described under Upgrading from 0.16.7 and earlier.
This is the promoted path for orchestration generation.
(since 0.31.0, forge-cli #923) One renderer writes every Airflow DAG a contract's orchestration section describes: fluid generate schedule, fluid generate-airflow and fluid export --engine airflow all call fluid_build.schedulers.airflow.render_dag, for AWS, GCP and Snowflake. See fluid generate-airflow for what that changed, and the Airflow integration for a scheduled Spark build, which gets the standard fluid apply DAG.
generate schedule must run from the project directory when the contract has scheduled builds: a contract path outside the current directory stops with generate_schedule_contract_outside_project and exit 2. It resolves the settings a DAG needs the way the provider does, and never writes a placeholder: see fluid generate-airflow.
fluid generate ci
Generate CI/CD pipeline configuration for Jenkins, GitHub Actions, GitLab CI, Azure DevOps, Bitbucket, CircleCI or Tekton.
(since 0.31.0, forge-cli #942) Every system gets a thin wrapper: GitHub Actions and Jenkins run each stage as a step, fluid ship [<contract>] --stage <N>, and the other systems run fluid ship --stage 1-6 and fluid ship --stage 7-11. So the 11 stages have one implementation, fluid ship, and the file holds the triggers, the gates and the install. A pull or merge request runs stages 1 to 6 and keeps runtime/ as an artifact. A push to the default branch also runs stages 7 to 11 in a second job, behind the system's own gate (an environment: on GitHub Actions, a manual job on GitLab CI, an input approval on Jenkins, and so on). What a generated pipeline runs has the gate of each system. --complexity is gone: see --add below.
fluid generate ci --system github
fluid generate ci contracts/orders/contract.fluid.yaml --system jenkins --out Jenkinsfile
Jenkins example for a product chained behind another and applied in aws:
fluid generate ci contracts/orders/contract.fluid.yaml \
--system jenkins \
--fluid-env-default aws \
--apply-mode-default amend-and-build \
--default-publish-target fluid-command-center \
--out Jenkinsfile
Jenkins example for an Entropy Data / Data Mesh Manager first run:
fluid generate ci contract.fluid.yaml \
--system jenkins \
--install-mode pypi \
--default-publish-target datamesh-manager \
--no-verify-strict-default \
--publish-stage-default \
--out Jenkinsfile
Options:
| Option | Description |
|---|---|
contract | Contract path. Defaults to contract.fluid.yaml when omitted. The path as given becomes the pipeline's CONTRACT default, so contracts/orders/contract.fluid.yaml generates a pipeline that builds that file. |
--system | Target CI system: jenkins, github, gitlab, azure, bitbucket, circleci, or tekton. Default gitlab. Hyphen/underscore variants (github-actions, azure-devops, gitlab-ci, circle-ci, …) are also accepted. |
--complexity {basic,standard,advanced,enterprise} | 0.30.1: pipeline complexity. Default standard. basic is validate + apply, standard adds testing, advanced multi-env and approvals, enterprise governance and compliance steps. (since 0.31.0, forge-cli #942) Removed: the flag exits 2 with a pointer to --add. |
--add {security,compliance,sbom} | (since 0.31.0, forge-cli #942) Repeatable. Add a job beside the pipeline: security is a daily OSV-Scanner scan plus fluid policy-check --strict, compliance a weekly policy report and lineage check, sbom an SBOM (GitHub Actions: anchore SPDX with an attestation; elsewhere syft when the runner has it, advisory). See Add-ons. |
--default-branch BRANCH | (since 0.31.0, forge-cli #942) The branch whose pushes run stages 7 to 11; a pull or merge request against it, or a manual run on another branch, runs stages 1 to 6. Default: the remote's default branch (origin/HEAD), else main. GitLab CI reads its own $CI_DEFAULT_BRANCH. |
--out PATH | Override the primary output path. Multi-file systems still write their supporting files to canonical locations. |
--no-generate-artifacts | Skip the transformation and schedule stages, for reference-only contracts. Through 0.32.0 it was also set automatically when any build's pattern was hybrid-reference, reference or external-reference. It sets stage 3's default to off on every system. (since 0.33.0, forge-cli #1000) Nothing sets it automatically. A contract with reference builds, or only reference builds, still runs stage 3. |
--install-mode {pypi,dev-source} | 0.30.1: Jenkins and Tekton. pypi (default): the install step installs the package spec (Jenkins: FLUID_PACKAGE_SPEC into a virtual environment in the workspace). dev-source installs from a /forge-cli-src bind mount for contributor labs and fails if the mount is missing; on Tekton it also drops the fluid-package-spec and pip index parameters from pipeline.yaml. (since 0.31.0, forge-cli #942) Every system reads it. (since 0.33.0, forge-cli #1000) dev-source runs pip install -e /forge-cli-src, which puts fluid on PATH, instead of exporting PYTHONPATH. On the six other systems it stops with a FATAL line when /forge-cli-src/pyproject.toml is missing, and when the install fails. The runner needs a package index, or setuptools and setuptools-scm installed with PIP_NO_BUILD_ISOLATION=0 (pip reads it inverted). Jenkins keeps the agent's installed fluid with PYTHONPATH=/forge-cli-src. In 0.32.0 every other system exported PYTHONPATH=/forge-cli-src in the install step, which ended with the step and provided no fluid command, so the install step failed at fluid --version. |
--image IMAGE | (since 0.33.0, forge-cli #1000) Tekton only: the image the Task steps run. Default python:3.12-slim, pinned by digest. It takes [registry[:port]/]name[:tag][@sha256:<digest>], and a value that is not an image reference is refused with generate_ci_invalid_option. Pin yours by digest too (name:tag@sha256:<digest>): a tag can move. Another system refuses the flag with generate_ci_image_unsupported. |
--fluid-package-spec SPEC | Default of the FLUID_PACKAGE_SPEC parameter. Default: the version that generated the file, with the extras the contract's bindings need across its base and every overlay, such as data-product-forge[local]==0.35.0 or data-product-forge[aws,local]==0.35.0. |
--fluid-env-default ENV | Default of FLUID_ENV, the overlay the stages pass to --env, and the fallback the stages read, so a build with no parameters runs in ENV. Letters, digits, ., _ and -, starting with a letter. Default dev. |
--apply-mode-default MODE | Default of APPLY_MODE, the mode stage 6 plans and stage 7 applies: dry-run, create-only, amend, amend-and-build, replace or replace-and-build. Default dry-run for Jenkins, amend for Tekton. |
--default-publish-target TARGET | Default of PUBLISH_TARGETS, and the fallback stage 10 reads when a build has no parameters. Default datamesh-manager. Pick the catalog your team publishes to, for example fluid-command-center, horizon, datahub or collibra. |
--[no-]verify-strict-default | Jenkins only. Default of VERIFY_STRICT. Default true. |
--[no-]publish-stage-default | Default of RUN_STAGE_10_PUBLISH. Default false. |
--[no-]publish-include-env | 0.30.1, Jenkins only. Whether stage 10 appends --env "${FLUID_ENV:-dev}" to fluid publish. Default true since 0.16.3. (since 0.31.0, forge-cli #942) Removed: fluid ship always passes --env to fluid publish. |
--[no-]schedule-sync-default | Default of RUN_STAGE_11_SCHEDULE_SYNC. Default false. |
--scheduler-default {airflow,mwaa,composer,astronomer,prefect,dagster} | Default of SCHEDULER, the scheduler stage 11 syncs to. Default blank, which makes stage 11 a no-op. |
--scheduler-destination-default URL | Default of SCHEDULER_DESTINATION: a shared DAG root (s3://, gs://, az://, ssh://, scp://, git+ssh://, file:// or an absolute path). Stage 11 writes this product into <root>/<contract id>/ and deletes only there, so several products' pipelines can share one root. |
--[no-]diff-last-applied | Jenkins only. Give stage 5 the plan the last successful build applied (fluid diff --last-applied), copied from that build's artifacts, so a schema change the contract makes shows as pending and not as drift. Needs the copyartifact plugin. Default off. |
--runner-host-override HOST | The loopback host to use when the pipeline's fluid process runs in a container and the contract says host: localhost: host.docker.internal (Docker Desktop), the bridge IP (Linux Docker), host.containers.internal (Podman) or a Service name (Kubernetes). Written as FLUID_RUNNER_HOST_OVERRIDE in the pipeline's environment. |
--list-engines | Print the acquisition and transformation engines the generator recognises, with the pip extras each needs, and exit. |
fluid generate ci contract.fluid.yaml --system tekton \
--image registry.example.com/team/python:3.12@sha256:<digest>
(since 0.33.0, forge-cli #1000) With --system gitlab, the file starts with a Run pipeline form, publishes a release packet from fluid-plan, and runs fluid diff --impact on each merge request. GitLab CI: form, release packet and impact report describes it, and Container images lists the images each system pins.
Which systems read which option
(since 0.31.0, forge-cli #942) Every system reads the options that set a default, --install-mode and --runner-host-override included (measured by generating each of the 7 systems with and without each option and comparing the files). Two are Jenkins only: --[no-]verify-strict-default and --diff-last-applied. On the other systems VERIFY_STRICT and DIFF_LAST_APPLIED are variables you set. Where a default is a variable, the file writes it as the fallback: --fluid-env-default prod makes FLUID_ENV fall back to prod when no input or repository variable sets it.
Below is the 0.30.1 measurement. As measured on 0.18.1 by generating each system with and without the option and comparing the files:
- Jenkins and Tekton render the 11-stage pipeline with parameters. Both read
--fluid-package-spec,--fluid-env-default,--apply-mode-default,--default-publish-target,--publish-stage-default,--schedule-sync-default,--scheduler-default,--scheduler-destination-defaultand--no-generate-artifacts. Both also read--install-modeand--runner-host-override. Jenkins alone reads--verify-strict-default,--publish-include-envand--diff-last-applied(each left Tekton's files byte-identical when measured on 0.18.1). Tekton with--complexity basicrenders a three-task pipeline instead; Jenkins rendered the same file for--complexity basicand--complexity enterprise(the other levels were not compared; measured on 0.18.1). - GitLab CI, GitHub Actions, Azure DevOps, Bitbucket and CircleCI render a single pipeline whose shape follows
--complexity. The options above that set a parameter default do not change those files, with three exceptions:--no-generate-artifactssets stage 3's default to off in them too,--runner-host-overridesetsFLUID_RUNNER_HOST_OVERRIDE(Jenkins and Tekton set it too), and--fluid-env-defaultsets the fallback ofFLUID_ENVin some of their commands (not CircleCI's).
Keyless cloud login
(since 0.31.0, forge-cli #905) fluid generate ci reads the contract and its overlays and wires one OIDC login per platform it binds, in the GitHub Actions and GitLab CI output. Jenkins stays on credentials binding. See Keyless cloud login for what each system gets and the trust to create on the cloud side.
The generated Jenkinsfile
(since 0.31.0, forge-cli #942) Every stage runs fluid ship "$CONTRACT" --stage N, and the header of the file says: "A change request (pull request build) stops after stage 6. A branch build applies after a person approves (APPLY_APPROVAL, an input step), one build per environment at a time (lockable-resources)." Stages 7 to 11 sit behind an Approve apply stage with an input step and a milestone, and then run under a lock named fluid-apply-<FLUID_ENV>. They run on the default branch only (main, or --default-branch), or in a job that builds no branch. APPLY_APPROVAL defaults to true: set it to false for an unattended job. A change request build has no applied-plan baseline for stage 5, because its own job never applies. The sections below describe the 0.30.1 file.
- Install. Stage 0 creates
$WORKSPACE/.fluid-venvand installsFLUID_PACKAGE_SPECinto it, because an agent whose system Python is externally managed (PEP 668) refuses a barepip install. The spec defaults to the version that generated the file, so a Jenkinsfile generated by 0.35.0 installsdata-product-forge[local]==0.35.0and does not move when a new release ships. Override it per build with theFLUID_PACKAGE_SPECparameter,FLUID_PIP_INDEX_URLandFLUID_PIP_EXTRA_INDEX_URL(TestPyPI), orFLUID_ALLOW_PRERELEASE. pip takes the highest version across every index it is given, so do not pair a test or private index with PyPI; see Operating in CI. - Plugins.
workflow-aggregatorandgit, pluscopyartifactwith--diff-last-applied. (since 0.31.0, forge-cli #942) Alsolockable-resourcesandjunit. The workspace is removed with the coredeleteDir()step, so thews-cleanupplugin is not needed. - The first build. Jenkins learns a Pipeline's parameters from its first build, so a
buildWithParametersrequest to a job that has never run is refused (observed on a Jenkins controller on 4 Oct 2026). Start the job once from the UI with no parameters. The generated file is written for that run: every parameter default is also the fallback each stage reads, so a first build, or the first build after a restart that re-seeds jobs from job-dsl or JCasC, runs what the declarations say.--fluid-env-defaultand--apply-mode-defaultset what that run does. - Stage 10.
fluid publishreceives--env "${FLUID_ENV:-dev}"by default, so the catalog gets the contract with the overlay stages 5 to 9 used. Before 0.16.3fluid publishhad no--env, and the default was off. Pass--no-publish-include-envonly when the pipeline installs a CLI older thanfluid publish --env. (since 0.31.0, forge-cli #942) The flag is removed, and ship always passes--env. GitLab CI, GitHub Actions, Azure DevOps and Bitbucket pass--env "${FLUID_ENV:-dev}"tofluid publishin their default output; the default CircleCI output has nofluid publishstep. - Parameters include
PUBLISH_TARGETS(defaultdatamesh-manager, or--default-publish-target) andAPPLY_MODE(defaultdry-run, so the first build applies nothing). - Regenerate after upgrading. A committed Jenkinsfile keeps the install, defaults and fallbacks of the version that generated it. Run
fluid generate ciagain and review the diff to pick up changes.
A Command Center holds one record per contract id, and fluid publish upserts it, so publishing one contract from two envs leaves the record of whichever published last (measured on 4 Oct 2026, with a contract published from aws and from gcp). Stage 10 is off by default; --[no-]publish-stage-default changes that default per pipeline.
fluid generate standard
Export to data product standards.
fluid generate standard contract.fluid.yaml --format odps --out ./product.odps.yaml
fluid generate standard --list
Key options:
contract--format,-f— required. Without it the command printsError: --format is required. Use --list to see available formats.and exits 1.odpsis the recommended value, not a default.--out,-o— output file. Without it the file is written underruntime/exports/. Write a relative path with a directory part (./product.odps.yaml): as of 0.18.1,--out product.odps.yamlfor theodpsandodps-bitolformats fails with[Errno 2] No such file or directory: ''.--env--list
Supported formats:
--format | What it is | Use it when |
|---|---|---|
odps | Bitol ODPS v1.0.0 — Open Data Product Standard (center-stage) | Publishing to Entropy Data / Data Mesh Manager. Rich product metadata, lineage, SLA, governance. Preserves FLUID-specific fields under an x-fluid namespace. Default file: runtime/exports/product.odps.yaml. |
odcs | Open Data Contract Standard (ODCS) v3.1.0 — the Bitol.io standard | Publishing contract-level specs (schema, quality, SLA) to Bitol-aligned tooling. Where ODPS describes a whole data product, ODCS focuses on the consumer-facing contract. |
odps-v4.1 | LF/ODPI ODPS v4.1 — Open Data Product Specification (opt-in) | ODPI-aligned catalogs that consume the Linux Foundation / Open Data Product Initiative spec. |
odps-bitol | An explicit alias of odps | Same output as odps, byte for byte, with no warning. Use it to say "Bitol" in a CI log. |
opds | A deprecated alias of odps-v4.1 | Writes the LF/ODPI v4.1 JSON and prints a deprecation warning. Use odps-v4.1. |
Examples of each format
fluid generate standard contract.fluid.yaml --format odps # FLUID -> Bitol ODPS v1.0.0 (center-stage)
fluid generate standard contract.fluid.yaml --format odcs # FLUID -> ODCS v3.1.0
fluid generate standard contract.fluid.yaml --format odps-v4.1 # FLUID -> LF/ODPI ODPS v4.1 (opt-in)
ODPS environment tuning
The ODPS exporter reads a few environment variables for output shape:
| Env var | Default | Purpose |
|---|---|---|
ODPS_INCLUDE_BUILD_INFO | true | Include build information (engine, pattern) |
ODPS_INCLUDE_EXECUTION_DETAILS | false | Include execution details (triggers, runtime) |
ODPS_TARGET_PLATFORM | generic | Platform-specific tuning (collibra, etc.) |
ODPS_VALIDATE_OUTPUT | true | Validate the emitted JSON |
ODPS_API_VERSION | v1.0.0 | Bitol ODPS apiVersion to emit. v1.1.0 (released by Bitol on 2026-09-08) opts into the RFC 0029 top-level type field (see fluid odps-bitol); v1.0.0 stays the default because the emitted apiVersion is what every existing consumer parses, so the default moves with a schema bump rather than silently. |
Formats are deterministic: identical input yields byte-identical output, so the result can be checked into version control.
Shortcut — fluid export-opds (deprecated alias)
For a one-shot file write of the LF/ODPI ODPS v4.1 JSON:
fluid export-opds CONTRACT [--env ENV] [--out PATH]
| Option | Description |
|---|---|
CONTRACT | Path to contract.fluid.yaml (positional, required) |
--env ENV | Apply an environment overlay |
--out PATH | Output file path (default: runtime/exports/product.odps-v4.1.json) |
Produces the same output as fluid generate standard --format odps-v4.1. The command is a deprecated alias and prints a deprecation warning; prefer fluid generate standard --format odps-v4.1 in new scripts. (The center-stage --format odps emits Bitol ODPS v1.0.0 to runtime/exports/product.odps.yaml instead.)
Examples
fluid generate transformation
fluid generate transformation contract.fluid.yaml -o ./dbt_project
fluid generate transformation contract.fluid.yaml -o ./dbt_project --dbt-validate
fluid generate schedule contract.fluid.yaml --scheduler airflow -o dags
fluid generate ci --system github
fluid generate standard contract.fluid.yaml --format odps
fluid generate standard contract.fluid.yaml --format odps-v4.1
Compatibility note
fluid generate-airflow still works for Airflow generation, but current docs lead with fluid generate schedule --scheduler airflow.