fluid generate artifacts
Stage 3 of the 11-stage pipeline. One call writes the catalog and execution artifacts for a contract, plus a MANIFEST.json (SHA-256 per file and a merkle root) that stage 4 verifies.
Added in 0.8.0 as a top-level subcommand of fluid generate.
Example
From a bundle, the way a pipeline runs it:
fluid bundle contract.fluid.yaml --format tgz --out runtime/bundle.tgz
fluid generate artifacts runtime/bundle.tgz --out dist/artifacts/
fluid validate-artifacts dist/artifacts/ --strict
From a contract file, as a local shortcut:
fluid generate artifacts contract.fluid.yaml --out dist/artifacts/
The command prints where it wrote and the manifest digest:
✅ Artifacts written to dist/artifacts
MANIFEST digest:
sha256:...
files: 7
The tree for a contract with two exposes, no orchestration.engine and no scheduled build:
dist/artifacts/
├── MANIFEST.json # SHA-256 per file + merkle root
├── odcs/product.odcs.<exposeId>.yaml # ODCS v3.1.0, one per exposed port
├── odps-bitol/<product-id>.odps.yaml # ODPS-Bitol v1.0.0
├── odps-bitol/<product-id>.<exposeId>.odcs.yaml # one per exposed port, next to the product file
├── opds/<product-id>.opds.json # OPDS v4.1 (LF/ODPI)
└── policy/bindings.json # compiled IAM / GRANT bindings
A contract that gets a schedule artifact adds schedule/<product-id>[__<env>]/<build-id>_dag.py, described in Scheduled builds. (since 0.33.0, forge-cli #979) A contract with a forge Spark build adds spark/<build-id>/job.json, described in Spark job configs.
Syntax
fluid generate artifacts CONTRACT [--out PATH] [--emit KEYS] [--manifest PATH] [--env ENV] [--contract-path PATH]
CONTRACT is a bundle (.tgz) or a contract YAML file. A bundle comes from fluid bundle --format tgz: generate artifacts extracts it and re-verifies its MANIFEST first. A contract file is the local-dev shortcut; the pipeline path is the bundle.
Key options
| Option | Description |
|---|---|
CONTRACT | Bundle (.tgz) or contract YAML file (positional, required). |
--out PATH | Output directory. Default dist/artifacts/. |
--emit KEYS | Comma-separated emit keys. Default: odps-bitol,odcs,opds,schedule,policies. (since 0.33.0, forge-cli #979) The default also holds spark. See the emit set. dbt is not an emit key; dbt projects come from fluid generate transformation. |
--manifest PATH | Where to write MANIFEST.json. Default <out>/MANIFEST.json. Stage 4 re-verifies it. |
--env ENV | The environment the artifacts are for. A bundle must have been built for it (fluid bundle --env ENV); a bundle built for another env is refused, because a bundle is not re-overlaid. A contract file gets the overlay applied, as fluid bundle --env ENV would. The value is also the --env every scheduled DAG passes to fluid apply. Default for the DAGs: $FLUID_ENV; if that is unset too, the DAGs carry no --env. --env '' means no --env. |
--contract-path PATH | The contract's path relative to the project directory. Scheduled DAGs append it to $FLUID_PROJECT_DIR on the Airflow worker. Default: the input's path relative to the current directory for a contract file, and contract.fluid.yaml (with a warning) for a bundle, which records no location. |
Emit set
| Key | What it emits | Notes |
|---|---|---|
odcs | ODCS v3.1.0 files under odcs/, one per exposed port | Schema vendored from bitol-io/open-data-contract-standard. |
odps-bitol | ODPS-Bitol v1.0.0 product file under odps-bitol/, with an ODCS file per exposed port beside it | Schema vendored from bitol-io/open-data-product-standard. |
opds | OPDS v4.1 (LF/ODPI) product file under opds/ (<product-id>.opds.json) | odps is accepted as a deprecated alias of opds and warns. |
schedule | Airflow DAG files under schedule/ | Emitted when orchestration.engine is set, or when a build declares a schedule trigger. (since 0.33.0, forge-cli #966) A dataset or schedule_and_dataset trigger counts too. See Scheduled builds. |
spark | spark/<build-id>/job.json for each forge Spark build | (since 0.33.0, forge-cli #979) The runner config fluid apply uploads. A contract without a Spark build writes nothing. See Spark job configs. |
policies | policy/bindings.json, the compiled IAM / GRANT bindings | (forge-cli #710, since 0.20.0) A crash inside the policy compiler fails the stage with policy_compiler_crashed (exit 1). See fluid policy compile. |
builds[].pattern (for example hybrid-reference) decides how the transformation runs and gates no emit key: a reference-only contract gets the same set as any other.
When schedule is requested and the contract has nothing to schedule, the run logs one of these events, drops the key and continues:
| Event | Cause |
|---|---|
generate_artifacts_skip_schedule_no_engine | No orchestration.engine, and no build with a schedule trigger. (since 0.34.0, forge-cli #1018) A Spark stream that declares execution.retries counts as a build with a schedule, so it does not log this event. |
generate_artifacts_skip_schedule_engine_none | orchestration.engine: none. |
generate_artifacts_skip_schedule_unreadable | The contract file cannot be read at all. |
A contract that is readable but does not load (a malformed overlay for the --env given, for example) fails the stage with generate_artifacts_failed and emit_key: schedule instead of skipping.
Scheduled builds
A build that declares a cron trigger is run on that schedule by Airflow, through fluid apply. Stage 3 writes the DAG; fluid schedule-sync (stage 11) delivers it to the scheduler.
builds:
- id: ingest_subscriptions
execution:
trigger:
type: schedule # or no type at all
schedule: "0 */4 * * *" # `cron:` is accepted too; five fields or an Airflow preset
timezone: Europe/Paris # default UTC
retries:
maxAttempts: 4 # Airflow retries = maxAttempts - 1; default 3, at most 10
fluid generate artifacts contracts/orders/contract.fluid.yaml --out dist/artifacts/ \
--env aws --contract-path contracts/orders/contract.fluid.yaml
dist/artifacts/schedule/<product-id>__aws/ingest_subscriptions_dag.py
When a DAG is generated
The schedule key writes DAGs when either holds:
orchestration.engineis set to anything exceptnone.- No
orchestration.engineis set, and a build declaresexecution.trigger.schedule(orcron) with triggertype: scheduleor no type. (since 0.33.0, forge-cli #966) Adatasetorschedule_and_datasettrigger counts too. The engine then defaults to Airflow. The file does nothing untilschedule-syncdelivers it, and stage 11 names the scheduler (airflow,mwaa,composerandastronomerrun Airflow DAGs), so the default decides the file format only. (since 0.34.0, forge-cli #1018) A forge Spark stream that declaresexecution.retriescounts as such a build too: it gets a keepalive DAG, and the engine defaults to Airflow.
Each such build gets its own DAG that runs fluid apply, unless the contract hand-declares orchestration.tasks, which keep their declared operators. (since 0.33.0, forge-cli #966) Builds that builds[].dependencies link share one DAG, with a task for each build. A build is left out when its own execution.orchestration.engine is none or not Airflow, or when its trigger is event, manual or timetable, which a DAG on a cron or on datasets cannot express. A dataset or schedule_and_dataset build gets a DAG: see Build graphs and dataset triggers. In 0.32.0 and earlier, those two were left out too, and a contract with no orchestration.engine whose only trigger was dataset logged generate_artifacts_skip_schedule_no_engine and wrote no DAG. (since 0.22.0, forge-cli #734) A continuous build, or one that runs outside forge-cli, gets no DAG and no task at all (see Continuous builds).
Layout and naming
- File:
schedule/<product-id>[__<env>]/<build-id>_dag.py. (since 0.33.0, forge-cli #966) For builds thatbuilds[].dependencieslink,<build-id>is the build that carries the trigger. One directory per product and env, so a sync to a shared DAG root deletes only that product's files (see the DAG layout on theschedule-syncpage). The__<env>suffix is present when an env is set, from--envor$FLUID_ENV. - DAG id:
<product>__<env>__<build>with an env,<product>__<build>without. - The DAG is hashed into
MANIFEST.json, and a rerun of stage 3 replaces it. - Upgrading from 0.16.7 or earlier: those versions wrote
schedule/<product-id>/with DAG id<product>__<build>. The firstfluid schedule-syncafter the upgrade has to retire the old DAG of each env, or the old and the new DAG both run and both apply the same product against the same state.
fluid generate schedule writes the same DAG without the schedule/<product-id>/ directory, into the directory given to -o (default ./dags).
What each run executes
On the Airflow worker, one BashOperator task named fluid_apply runs:
cd "$FLUID_PROJECT_DIR"
fluid apply "$FLUID_PROJECT_DIR/<contract path>" --env <env> \
--mode amend-and-build --build-id <build id> --yes
That is the apply stage 7 runs, so a cloud target keeps one OpenTofu state whether the run came from CI or from the scheduler, provided the worker has the same FLUID_STATE_BACKEND. --env is left out when the DAG was generated without an env.
(since 0.33.0, forge-cli #966) A DAG for linked builds has one such task for each build, named by the build id, and each runs fluid apply ... --build-id <build id>. See Build graphs and dataset triggers.
The DAG sets max_active_runs=1 (two applies of one build do not overlap) and catchup=False (a paused DAG does not replay missed runs). A fluid exit code of 99 fails the task instead of skipping it. The task uses the airflow.sdk DAG and schedule= of Airflow 3, with imports that fall back to Airflow 2.6 and later.
Fixed when the DAG is generated: the env, the contract path, the cron, the timezone and the retry count. Read on the worker at run time:
| Variable | Meaning |
|---|---|
FLUID_PROJECT_DIR | Required. The directory holding the product checkout, the one CI ran the pipeline from. The run fails when it is unset or the contract is not under it. |
FLUID_BIN | The fluid executable. Default fluid on PATH. |
FLUID_DAG_ENV_PASSTHROUGH | Extra variable names, space separated, to pass to fluid. |
The schedule is checked at generation time to the grammar Airflow parses it with, so a cron Airflow would refuse fails the run instead of the DAG import. This six-field cron:
generate_schedule_invalid_schedule: {'error': "build 'customer_360_pipeline': trigger schedule '0 2 * * * *' is not a five-field cron expression (minute hour day-of-month month day-of-week) or an Airflow preset such as @daily"}
fails fluid generate schedule with exit 2, and fluid generate artifacts with generate_artifacts_failed (exit 1). A six-field cron is refused because Airflow's cron parser reads the sixth field as seconds, while a Quartz cron puts seconds first.
The environment fluid sees
The DAG file holds ids, the contract path, the env name and the names of the variables the contract reads, and no secret values. fluid starts through env -i with:
PATH HOME USER LOGNAME LANG LANGUAGE LC_ALL LC_CTYPE TZ TMPDIR, the CA bundle and proxy variables, andGLUE_ROLE_ARN S3_STAGING_DIR S3_DATA_DIR GOOG_SERVICE_ACCOUNT_NAME TESTCONTAINERS_HOST_OVERRIDE.- Any variable whose name starts with
FLUID_,AWS_,GOOGLE_,GCP_,GCLOUD_,CLOUDSDK_,AZURE_,ARM_,SNOWFLAKE_,DATABRICKS_,PG,POSTGRES_,REDSHIFT_,ATHENA_,VAULT_,BAO_(since 0.31.0, forge-cli #893),DBT_,DLT_,DATAHUB_,DMM_,ODCS_,ODPS_,TF_,OPENLINEAGE_orOTEL_. - The variables the contract names through
{{ env.NAME }},${NAME}orsecretRef: env://NAME. - The names listed in
FLUID_DAG_ENV_PASSTHROUGH.
A name starting AIRFLOW does not pass, even when the contract or FLUID_DAG_ENV_PASSTHROUGH names it, so the Airflow worker's own configuration (AIRFLOW__*, AIRFLOW_CONN_*, the Fernet key) does not reach the apply. VIRTUAL_ENV is not passed either: on a worker it names Airflow's environment, and fluid's Python runner would run builds with that interpreter. A dbt project's own env_var() calls, or dlt's SOURCES__* and DESTINATION__* settings, are not on the list: name them in FLUID_DAG_ENV_PASSTHROUGH.
Spark job configs
(since 0.33.0, forge-cli #979) A forge Spark build (engine: spark with parameters.spark, in a 0.7.7 contract) runs from a job.json: the runner config that forge compiles from the contract. The spark emit key writes it for each Spark build to spark/<build id>/job.json, so the artifacts MANIFEST.json holds the hash of the config that gets deployed. A contract with no Spark build writes nothing under spark/.
fluid generate artifacts contract.fluid.yaml --out dist/artifacts/
On the contract from Spark to Iceberg on AWS, run on forge-cli main:
...
generate_artifacts_spark_apply_time_values
✅ Artifacts written to dist/artifacts
MANIFEST digest:
sha256:...
files: 6
dist/artifacts/
├── MANIFEST.json
├── odcs/product.odcs.orders.yaml
├── odps-bitol/commerce.orders-stream.odps.yaml
├── odps-bitol/commerce.orders-stream.orders.odcs.yaml
├── opds/commerce.orders-stream.opds.json
├── policy/bindings.json
└── spark/ingest-orders/job.json
job.json, trimmed:
{
"build": "ingest-orders",
"checkpoint": "s3://commerce-lake/_checkpoints/commerce.orders-stream/ingest-orders/",
"configVersion": 1,
...
"mode": "streaming",
"product": "commerce.orders-stream",
"sink": {
"catalog": "fluid",
"region": "eu-west-1",
"table": "commerce.orders",
"type": "glue",
"warehouse": "s3://commerce-lake/iceberg/commerce/orders/"
},
"source": {
"auth": "msk-iam",
"bootstrapServers": "${fluid_msk_bootstrap_brokers}",
...
"kind": "kafka",
...
"topic": "commerce.orders",
...
},
...
}
What the file is, and what it is not:
It is the deployed config.
fluid generate iacwrites the same text into theaws_s3_objectthat uploads_fluid/spark/<product>/<build>/job.json. On this contract the only difference from that object is the broker list, which OpenTofu fills in. The file is the same whethergenerate artifactsreads the contract file or a bundle made from it. With a contract file,--envapplies that overlay first; a bundle is already bound to its environment.Two values stay placeholders, because only
tofu applyknows them:<account>in the warehouse and checkpoint of an output that names no bucket (the account's shared<account>-fluid-databucket), and${fluid_msk_bootstrap_brokers}for a source named by its MSK cluster ARN. The run logsgenerate_artifacts_spark_apply_time_valuesfor each build that holds one.What the upload refuses is refused here. Two builds whose ids differ only in case or in
-and_would write one file, and a config that holds${is an unresolved interpolation that the Spark runner refuses. Either stops the stage withgenerate_artifacts_failedandemit_key: spark, as does a build the Spark compiler refuses. The first looks like this:❌ generate_artifacts_failed [ERR_GENERATE_ARTIFACTS_FAILED] error: Spark builds 'ingest-orders' and 'ingest_orders' would write one job.json (ids that differ only in case or in '-' and '_' name one file); give each its own id emit_key: sparkThe operator's upload rules are not applied. The settings that limit contract SQL and the hosts a job may name stay with
fluid apply;fluid validatereports them. Ajob.jsonwritten here can still be refused at apply.A rerun clears only files of this shape. Whatever
--emitsays, a run first removes the<out>/spark/<dir>/job.jsonfiles already there, and then the directories that leave empty. It removes no other file under<out>/spark/.
Stage 4 checks each file: fluid validate-artifacts reads it as a JSON object with an integer configVersion, a build that names its directory, and product, mode and sink.
Fragment layouts
A fragment layout is a root contract that composes its exposes and builds from files through $ref (see Contract refs). fluid generate artifacts loads the composed contract, with or without --env, so a split contract gets byte-identical artifacts to the same contract written flat:
fluid generate artifacts contract.fluid.yaml --out dist/artifacts/
In 0.24.0 and earlier the command read the root as written unless --env was given, and failed on a fragment root with FLUID expose has no usable name; generate ci also judged reference-only builds and engines from the raw root. Both are fixed in 0.25.0 (forge-cli #760).
The generated Jenkins and Tekton pipelines feed stage 3 the bundle:
fluid bundle contract.fluid.yaml --format tgz --out runtime/bundle.tgz
fluid generate artifacts runtime/bundle.tgz --out dist/artifacts/
Examples
Explicit emit set
# Catalog only (skip DAGs and bindings)
fluid generate artifacts contract.fluid.yaml --emit odcs,odps-bitol
# Execution only
fluid generate artifacts contract.fluid.yaml --emit schedule,policies
Next step
Run fluid validate-artifacts against the emitted tree to verify MANIFEST integrity and per-format schemas before shipping to catalogs or the scheduler.
fluid generate artifacts contract.fluid.yaml --out dist/artifacts/
fluid validate-artifacts dist/artifacts/ --strict