Airflow Integration
Generate Apache Airflow DAGs from a contract. fluid generate schedule writes one DAG per scheduled build; each run of the DAG executes fluid apply for that build on the Airflow worker. (since 0.33.0, forge-cli #966) Builds that builds[].dependencies link run in one DAG, one task for each build, and a build can run on a dataset trigger: see Build graphs and dataset triggers. The older fluid generate-airflow command is deprecated and is described below.
Quick start
A build becomes a DAG when it declares a cron trigger:
builds:
- id: customer_360_pipeline
execution:
trigger:
type: schedule
cron: "0 2 * * *"
timezone: Europe/Dublin
Generate the DAG into a directory named after the product id, then check what it wrote:
fluid generate schedule contract.fluid.yaml -o dags/gold.customer.analytics_360_v1
Generated 1 files (airflow scheduler):
dags/gold.customer.analytics_360_v1/customer_360_pipeline_dag.py
Tip: To regenerate after editing the contract: fluid generate schedule
fluid generate schedule renders these DAGs when the contract's orchestration.engine is airflow and the contract declares no explicit orchestration.tasks. A contract with no orchestration.engine at all but a build with a cron trigger defaults to Airflow.
(since 0.33.0, forge-cli #966) A build with a dataset or schedule_and_dataset trigger gets a DAG too, and a contract with no orchestration.engine and such a build defaults to Airflow. Builds that builds[].dependencies link run in one DAG. Build graphs and dataset triggers covers both. A build with a trigger of type event, manual or timetable still gets no DAG, because a DAG on a cron or on datasets would silently drop what those types mean. In 0.32.0 and earlier, dataset and schedule_and_dataset were left out too, and a contract with no orchestration.engine whose only trigger was dataset stopped with No scheduler engine specified. (since 0.22.0, forge-cli #734) A continuous build gets no DAG either: its platform keeps it running (see Continuous builds). --env is the overlay for the generation and the --env every scheduled run passes to fluid apply; leave it out for a contract with no overlays.
What the DAG does
The generated file says it in its own docstring (real output, trimmed):
Every run executes this on the Airflow worker, with the constants below
filled in (`--env` is left out when FLUID_ENV_NAME is empty):
cd "$FLUID_PROJECT_DIR"
fluid apply "$FLUID_PROJECT_DIR/$CONTRACT_PATH" --env "$FLUID_ENV_NAME" \
--mode amend-and-build --build-id "$BUILD_ID" --yes
Worker requirements:
- FLUID_PROJECT_DIR names the directory that holds the product checkout,
the directory CI ran the pipeline from.
- `fluid` is on PATH, or FLUID_BIN names the executable.
- The credentials the apply needs are in the worker environment. Only
these reach the fluid process:
PATH HOME USER LOGNAME LANG LANGUAGE LC_ALL LC_CTYPE TZ TMPDIR
SSL_CERT_FILE SSL_CERT_DIR REQUESTS_CA_BUNDLE CURL_CA_BUNDLE HTTP_PROXY
HTTPS_PROXY NO_PROXY http_proxy https_proxy no_proxy GLUE_ROLE_ARN
S3_STAGING_DIR S3_DATA_DIR GOOG_SERVICE_ACCOUNT_NAME
TESTCONTAINERS_HOST_OVERRIDE
every name matching
FLUID_* AWS_* GOOGLE_* GCP_* GCLOUD_* CLOUDSDK_* AZURE_* ARM_*
SNOWFLAKE_* DATABRICKS_* PG* POSTGRES_* REDSHIFT_* ATHENA_* VAULT_* BAO_*
DBT_* DLT_* DATAHUB_* DMM_* ODCS_* ODPS_* TF_* OPENLINEAGE_* OTEL_*
the variables the contract reads (CONTRACT_ENV_NAMES), and any names
listed, space separated, in FLUID_DAG_ENV_PASSTHROUGH. fluid starts with
nothing else, and never with a variable whose name starts AIRFLOW.
(since 0.31.0, forge-cli #893) BAO_* joins VAULT_* among the prefixes, so the OpenBao address, token, namespace and CA variables reach a vault:// reference's lookup. A 0.30.1 DAG passes only VAULT_*.
In practice:
Set
FLUID_PROJECT_DIRon the worker. Without it the task fails withset FLUID_PROJECT_DIR on the Airflow worker to the directory that holds the product checkout. The contract path is the path you gave the generator, relative to the working directory, or--contract-pathwhen the worker's layout differs.Put the credentials the apply needs in the worker environment. The DAG starts
fluidwith an empty environment plus the names above, so a variable that is neither on the list nor named by a{{ env.NAME }}or${NAME}placeholder in the contract, nor anenv://NAMEsecretRef, does not arrive. Add one withFLUID_DAG_ENV_PASSTHROUGH="NAME1 NAME2".A DAG uses
max_active_runs=1andcatchup=False, retries from the build'sexecution.retries.maxAttempts(attempts minus one), and sets a three-hour execution timeout.(since 0.34.0, forge-cli #1009) The task's
fluid applyruns a dbt or Python build once, whichever of these itstrigger.typeis:schedule,dataset,schedule_and_dataset,event,dependencyortimetable. The apply runs the build once. What decides when is the DAG forschedule,datasetandschedule_and_dataset, the DAG of the builds it depends on fordependency, and another scheduler or an event foreventandtimetable, which get no DAG. In 0.33.0 onlyscheduleran: a dbt or Python build of any other of these types failed withUnknown trigger typeand exit code1, so the task of adatasetorschedule_and_datasetbuild, which this page generates a DAG for, failed. Amanualbuild keeps itsiterations,continuousis refused, and atrigger.typethat is none of the names the schema knows fails withUnknown trigger typeand the listmanual, dataset, dependency, event, schedule, schedule_and_dataset, timetable. On 0.33.0 aschedule_and_datasetbuild printed this, and in 0.34.0 it runs once:Trigger: schedule_and_dataset Command: dbt build ❌ Unknown trigger type: schedule_and_dataset Supported types: manual, scheduleTrigger: schedule_and_dataset (manual apply override) Command: dbt build Iterations: 1The same product on two clouds needs a directory each (
<product>with no env,<product>__<env>with one), so each schedule sync mirrors only its own DAGs.
Build graphs and dataset triggers
(since 0.33.0, forge-cli #966) This whole section describes it. Builds that builds[].dependencies link run in one DAG, one fluid apply --build-id task per build, each after the builds it names. A build with a dataset or schedule_and_dataset trigger gets a DAG that runs when the datasets it names are updated. A task declares, as outlets, the assets of the exposes its build writes (see Outlets for the exposes that name none), so one product's DAG can start another's.
Example: ingest, transform, publish, then a second product
sales.orders_daily has three builds. Only ingest has a trigger. The other two name the build they run after:
id: sales.orders_daily
builds:
- id: ingest
engine: duckdb
execution:
trigger:
type: schedule
cron: "0 2 * * *"
timezone: Europe/Dublin
outputs: [orders_raw]
- id: transform
engine: duckdb
dependencies:
- buildId: ingest
condition: success
outputs: [orders_clean]
- id: publish
engine: duckdb
dependencies:
- buildId: transform
condition: success
outputs: [orders_daily]
exposes:
- exposeId: orders_daily
binding:
platform: aws
format: parquet
location:
bucket: acme-lake
path: gold/orders_daily/
region: eu-west-1
# orders_raw and orders_clean are S3 exposes in the same bucket
fluid generate schedule contract.fluid.yaml -o dags/sales.orders_daily
Generated 1 files (airflow scheduler):
dags/sales.orders_daily/ingest_dag.py
...
One file, named after ingest, the build that carries the trigger. Its DAG, trimmed:
with DAG(
dag_id='sales.orders_daily__ingest',
description='fluid apply sales.orders_daily --build-id each of ingest, transform, publish',
schedule=SCHEDULE,
...
) as dag:
tasks = {}
tasks['ingest'] = BashOperator(
task_id='ingest',
...
env={
"FLUID_DAG_CONTRACT": CONTRACT_PATH,
"FLUID_DAG_ENV": FLUID_ENV_NAME,
"FLUID_DAG_BUILD_ID": 'ingest',
"FLUID_DAG_CONTRACT_ENV": CONTRACT_ENV_NAMES,
},
...
outlets=[
Asset('s3://acme-lake/raw/orders'),
],
)
tasks['transform'] = BashOperator(
task_id='transform',
...
trigger_rule='all_success',
outlets=[
Asset('s3://acme-lake/silver/orders_clean'),
],
)
tasks['publish'] = BashOperator(
task_id='publish',
...
trigger_rule='all_success',
outlets=[
Asset('s3://acme-lake/gold/orders_daily'),
],
)
tasks['ingest'] >> tasks['transform']
tasks['transform'] >> tasks['publish']
A second product, finance.revenue_daily, runs when the orders_daily table is updated. Another product's expose is named by its asset URI, the one the DAG above declares in outlets:
id: finance.revenue_daily
builds:
- id: revenue
engine: duckdb
execution:
trigger:
type: dataset
datasets:
- s3://acme-lake/gold/orders_daily
outputs: [revenue_daily]
fluid generate schedule contract.fluid.yaml -o dags/finance.revenue_daily
Its DAG, trimmed:
DATASETS = [
's3://acme-lake/gold/orders_daily',
]
...
with DAG(
dag_id='finance.revenue_daily__revenue',
description='fluid apply finance.revenue_daily --build-id revenue',
schedule=[Asset(uri) for uri in DATASETS],
...
When the publish task succeeds, Airflow marks s3://acme-lake/gold/orders_daily as updated and starts the revenue DAG. The full contracts pass fluid validate, and the output above came from forge-cli main. With 0.32.0, the first contract writes only ingest_dag.py, a one-task DAG with no outlets (transform and publish have no trigger of their own, so they get no DAG), and the second stops with No scheduler engine specified.
How linked builds share a DAG
- One DAG per linked group. Builds that depend on each other, in either direction, run in one DAG. A build that nothing links and that carries a trigger keeps its own DAG.
- One trigger per DAG. The DAG runs on the one trigger its builds declare, whichever build declares it. A build with no trigger of its own (or
trigger.type: dependency) runs in the DAG of the builds it depends on. Builds of one group may repeat the same trigger. - Names. The DAG id is
<product id>__<build id>, or<product id>__<env>__<build id>with--env, and the file is<build id>_dag.py. Both are named after the build that carries the trigger (the first of them inbuilds[]when several builds of the group repeat it), andBUILD_IDin the file holds that build. In the file name, a character of the build id other than a letter, digit or_becomes_. - Task ids. Each task's
task_idis its build id, and itsFLUID_DAG_BUILD_IDis the same value, so the task runsfluid apply --build-id <build id>. A DAG with one task keeps the task idfluid_apply. - Per task. A task takes its
retriesfrom its own build'sexecution.retries.maxAttempts, and has its own execution timeout: three hours, plus the run's wait for a Spark build.
A dependency's condition becomes the task's Airflow trigger rule. A dependency with no condition is success.
condition | Trigger rule | The task starts |
|---|---|---|
success | all_success | once the builds it names succeeded |
completion | all_done | once they finished, whatever their outcome |
always | always | at once, without waiting for them |
A dependency on a build that fluid apply keeps running, such as a Spark stream, is not waited for, and the DAG's own documentation says so:
A build here depends on these, but no DAG runs them, so no task waits
for them: stream-raw (a Spark stream that fluid apply keeps running).
A contract whose only builds are such streams gets No build needs a DAG from fluid generate schedule, not No scheduler engine specified, and no file is written. (since 0.34.0, forge-cli #1018) A stream that declares execution.retries is the exception on Airflow: it gets a keepalive DAG, and a contract of such streams writes a file. See Keeping a stream running.
Dataset triggers
trigger.type | The DAG runs |
|---|---|
schedule | on trigger.cron (or trigger.schedule) |
dataset | when the datasets in trigger.datasets are updated |
schedule_and_dataset | on its cron, and when those datasets are updated |
trigger.datasets lists the datasets. datasetsOperator is all (the default: every dataset has been updated since the DAG's last run) or any (one of them). dataset takes no cron. schedule_and_dataset needs one that recurs, so @once is refused. A dataset is one of:
- An asset URI, as a string. It is checked as Airflow checks it and written in the form Airflow stores: a lower-case scheme and no trailing slash after a path (a bare
s3://bucketkeeps its slash). A URI with user info, a query, a fragment, a space or a{{ ... }}or${...}placeholder is refused. {productId, exposeId}of an expose of this contract. The generator turns it into that expose's asset URI (see Outlets). TheproductIdmust be this contract's id. For another product's expose, write the URI.
An any trigger on a cron renders like this (trimmed):
schedule=asset_or_time_schedule(
CronTriggerTimetable(SCHEDULE, timezone=TIMEZONE),
AssetAny(*(Asset(uri) for uri in DATASETS)),
),
The files load on Airflow 2.6 and later, and on Airflow 3. On Airflow 2, a datasetsOperator: any or schedule_and_dataset trigger needs 2.9 or later. The generator does not render event, manual or timetable triggers: a build with one gets no DAG.
Outlets
Each task declares the exposes its build writes as outlets, so Airflow starts a DAG scheduled on one of them when the task succeeds, in this product's DAGs or another's. A build writes the exposes in its outputs. A build with no outputs counts as writing every expose that no build lists.
An expose is named by the URI that Airflow's own providers use for the same storage, built from its binding:
binding.platform | Asset URI |
|---|---|
aws with location.bucket | s3://<bucket>/<path> (with no path, <database>/<table> under the bucket) |
gcp with location.project, dataset and table | bigquery://<project>/<dataset>/<table> |
gcp with location.bucket | gs://<bucket>/<path> |
snowflake | snowflake://<account>/<database>/<schema>/<table> |
databricks with location.host | databricks://<host>/<database>/<schema>/<table> |
postgres, pgvector | postgres://<host>:<port>/<database>/<schema>/<table> (port 5432 when none is given) |
kafka, confluent | kafka://<first bootstrap server>/<topic> |
local with an absolute location.path | file:// and the path: /data/out/x.parquet gives file:///data/out/x.parquet |
An expose gets no outlet when its URI cannot be fixed at generation: a relative local path (the project directory is somewhere else on each worker), a value filled in on the worker ({{ env.X }}, ${X}), a missing location field, or a platform not in the table. The task is generated without it.
A one-build DAG you generated before gains the Asset import and an outlets= list when its build writes an expose that has an asset URI, and nothing else. When none of them has one, the file does not change. A DAG that gains them:
try: # Airflow 3
from airflow.sdk import Asset
except ImportError: # Airflow 2.x
from airflow.datasets import Dataset as Asset
...
outlets=[
Asset('s3://lake/iceberg/curated/events_daily'),
],
Errors that stop generation
Each of these stops fluid generate schedule with generate_schedule_invalid_schedule and exit code 2, and the message says what to change:
- a cycle in
builds[].dependencies; - a dependency on the build itself, or on no build;
- two different triggers in one linked group;
- two conditions among one build's dependencies, because Airflow gives a task one trigger rule;
- a dependency on a build that another engine runs (its
execution.orchestration.enginenames another engine, ornone); - a DAG that its own output would trigger, or two or more DAGs that trigger each other through their outputs;
- a dataset Airflow would refuse, or a
{productId, exposeId}that names another product or an expose the contract does not declare.
Two messages (trimmed):
generate_schedule_invalid_schedule: {'error': "builds[].dependencies form a cycle: 'ingest' -> 'transform' -> 'publish' -> 'ingest' (each runs after the one before it), so none of them could ever start. Remove one of these dependencies"}
generate_schedule_invalid_schedule: {'error': "builds 'ingest' and 'publish' are linked by builds[].dependencies, so they run in one DAG, but their triggers differ (schedule '0 2 * * *' (Europe/Dublin) vs schedule '0 5 * * *' (UTC)). One DAG runs on one trigger: keep it on one build and remove the other's, or drop the dependency and give 'publish' a dataset trigger on what 'ingest' writes"}
Both came from forge-cli main, the first with publish made a dependency of ingest, the second with a second cron on publish.
Regenerating
With --overwrite, fluid generate schedule also removes the DAG files an earlier run wrote into the output directory for the same product and env that the contract no longer makes. A build that now runs as a task of another build's DAG would otherwise run twice. Regenerating over the three one-build DAGs that 0.32.0 wrote when each build had a cron trigger of its own prints (from forge-cli main):
Removed dags/sales.orders_daily/publish_dag.py: an earlier run wrote it for this product and env, and the contract no longer makes that DAG.
Removed dags/sales.orders_daily/transform_dag.py: an earlier run wrote it for this product and env, and the contract no longer makes that DAG.
Only a .py file that opens with the generated DAG's docstring and sets the same PRODUCT_ID and FLUID_ENV_NAME is removed.
fluid validate warns about an Airflow task's buildStepRef under orchestration: it names a transformations[] step, and no generated DAG runs one. To run a build after another, list it in builds[].dependencies.
Pushing DAGs to the scheduler
fluid generate artifacts writes the schedule artifacts under schedule/<product-id>/, and stage 11 of a generated Jenkins pipeline hands that directory to fluid schedule-sync:
fluid generate artifacts contract.fluid.yaml --out dist/artifacts
fluid schedule-sync --scheduler airflow --dags-dir dist/artifacts/schedule --destination dest --dry-run
[schedule-sync] → /usr/bin/rsync -av --delete -- .../dist/artifacts/schedule/gold.customer.analytics_360_v1/ .../dest/gold.customer.analytics_360_v1/
[schedule-sync] ✔ airflow sync complete (1 subprocess(es))
--dags-dir must contain one directory per product. With the default --delete-scope product, a loose file in --dags-dir is refused with schedule_sync_dags_dir_not_product_scoped and exit code 2, so one product's sync cannot delete another's DAGs. If you generated with fluid generate schedule, write into -o <dags-dir>/<product-id>/ as in the quick start. schedule-sync takes no contract argument: it needs --scheduler and --dags-dir, plus --destination for airflow and mwaa.
MWAA on AWS
(forge-cli #715, since 0.20.0) fluid apply does not create an Amazon MWAA environment. Create the environment yourself, in your own VPC, or use one you already run, and push the DAGs to the S3 folder it reads. Schedule the builds with builds[].execution.trigger, and leave orchestration.schedule out:
orchestration:
engine: airflow
builds:
- id: load_orders
execution:
trigger:
type: schedule
cron: "0 6 * * *"
fluid generate schedule contract.fluid.yaml -o dags/sales_orders_daily
fluid schedule-sync --scheduler mwaa --dags-dir dags --destination s3://acme-airflow-dags/dags/ --dry-run
[schedule-sync] scheduler=mwaa dags-dir=.../dags env=dev delete-scope=product dry-run=True
[schedule-sync] → /usr/local/bin/aws s3 sync .../dags/sales_orders_daily/ s3://acme-airflow-dags/dags/sales_orders_daily/ --delete
[schedule-sync] ✔ mwaa sync complete (1 subprocess(es))
Drop --dry-run to push. --destination is the S3 folder the environment reads its DAGs from, in its source bucket. Each scheduled run executes fluid apply on the MWAA worker, which needs what What the DAG does lists.
An aws contract that keeps orchestration.schedule with orchestration.engine: airflow asks fluid apply for the environment itself, and is refused with unsupported_binding, kind mwaa-environment-not-emitted, before anything is applied; see Scheduling on AWS. Before this change the apply reported success and no environment existed. The refusal's own next step names fluid generate schedule --scheduler airflow --output dags/<contract id> and fluid schedule-sync --scheduler mwaa --dags-dir dags, the product-scoped layout above, so the two commands run as printed.
fluid generate-airflow (deprecated)
(since 0.31.0, forge-cli #923) fluid generate-airflow is an alias of fluid generate schedule --scheduler airflow, and one renderer writes both. What changed is on its page. The notes below describe 0.30.1 and earlier.
fluid generate-airflow still runs, and prints a warning that points at its replacement:
fluid generate-airflow contract.yaml -o dag.py --dag-id my_pipeline --schedule "0 * * * *"
{"time": "2026-10-05T00:40:54Z", "level": "WARNING", "name": "fluid.cli", "message": "Note: 'generate-airflow' is deprecated. Use 'fluid generate schedule --scheduler airflow' instead."}
DAG written to dag.py
It renders one BashOperator task per provider action in the plan, and its flags are -o, --dag-id, --schedule, --env and --verbose. For the quickstart contract the tasks are placeholders:
provision_customer_360_master = BashOperator(
task_id='provision_customer_360_master',
bash_command="echo 'Provision customer_360_master on local'",
dag=dag
)
Notes for the DAGs it writes:
- It targets Airflow 2 APIs: the file imports
days_agoand setsschedule_interval. - Contract values that reach a
bash_commandare stripped of{and}, because Airflow rendersbash_commandas a Jinja template at run time, and then shell-quoted, so a buildscriptarrives as one token, not a shell line. - Task ids are sanitized, and two actions whose ids differ only in punctuation get
_2,_3suffixes instead of one silently replacing the other. - A dbt build runs
dbt run --select <script or build id>;--models, which dbt 2 rejects, is not used.
For a walkthrough that builds a pipeline from a contract, see the Declarative Airflow Walkthrough.
Scheduled Spark builds
(since 0.31.0, forge-cli #950) A build of engine: spark (its parameters.spark lives under builds[].properties) whose run ends on its own, a batch job or a stream with trigger.availableNow, gets the standard fluid apply DAG, like every scheduled build. Its one task, fluid_apply, runs:
fluid apply <contract> --mode amend-and-build --build-id <id> --yes
The build phase of that apply starts one run on EMR Serverless, Glue, EMR on EC2 or EMR on EKS, waits for it, and fails unless the run succeeded, so the task fails when the run does. Ctrl-C, SIGTERM or SIGHUP cancel the run. The task goes through the same hardened env -i script as every fluid apply DAG.
- Timeouts.
execution.runtime.timeout, an ISO-8601 duration such asPT2H, is how long the run may take; six hours when unset. The task's timeout is the three hours every apply DAG gets plus that wait. A value that is not a duration of at least one second is refused beforetofu applychanges anything (apply_spark_invalid). - Retries come from
execution.retries.maxAttempts, as for every scheduled build. - A streaming build gets no DAG, unless its
properties.stream.triggerisavailableNow: true. A stream runs continuously, andfluid applykeeps it running. (since 0.34.0, forge-cli #1013) A scheduler task can keep a stream running between applies: it runsfluid apply <contract> --mode reconcile --build-id <stream> --yesevery few minutes, and starts the stream again when it died, within its restart budget. See Reconcile mode. (since 0.34.0, forge-cli #1018)fluid generate schedulewrites that task, as a keepalive DAG, for a stream that declaresexecution.retries: see Keeping a stream running. A build is a stream whenparameters.spark.modesays so, or when it declaresproperties.streamand sets no mode. (since 0.33.0, forge-cli #966) A build that depends on a stream does not wait for it; see Build graphs and dataset triggers. - The worker runs the whole apply,
tofuincluded, so it needs the apply's permissions and the state backend, as for any scheduled build, plusfluidwith boto3 (data-product-forge[aws]in MWAA'srequirements.txt). The scheduler policy forge emits for a Spark product covers the run only. The operator settings (FLUID_SPARK_*) and the run's AWS actions are the ones Spark to Iceberg on AWS lists. - A build that is not Spark gets the standard DAG, as before. A contract of dbt and Spark builds gets a DAG for each scheduled build. (since 0.33.0, forge-cli #966) Builds that
builds[].dependencieslink share one DAG instead (see Build graphs and dataset triggers).
Keeping a stream running
(since 0.34.0, forge-cli #1018) A forge Spark stream that declares execution.retries gets a small DAG of its own, a keepalive. Its one task runs fluid apply --mode reconcile every few minutes, so a stream that a Spot interruption or a failed driver stopped starts again on its checkpoint, within the restart budget that execution.retries sets.
Who gets one. A Spark stream (a build with
properties.stream) that is nottrigger.availableNowand nottrigger.type: continuous, and declaresexecution.retriesas an object such as{maxAttempts: 5}, on Airflow, the default engine. It is opt-in: a stream withoutexecution.retriesgets no DAG, although--mode reconcilewould restart it too (5 restarts per rolling hour when unset). A contract whose only builds are such streams gets this DAG, and no longerNo build needs a DAG. Prefect, Dagster andfluid scaffold-composerwrite nothing for a stream, and their messages point at the Airflow keepalive (--scheduler airflow, orfluid generate schedule). A batch that depends on the stream does not wait for it: the stream is in no build graph. With noorchestration.enginein the contract, such a stream makes Airflow the default, as a build with a cron trigger does, andfluid generate artifacts(stage 3 of a pipeline) writes its DAG underschedule/.What the DAG is. A file
<build>_dag.py, with the-of the build id as_. Its DAG id is<product>[__<env>]__<build>, and its one task isfluid_keepalive. The task runs, with the sameenv -iscript and environment allowlist as everyfluid applytask:cd "$FLUID_PROJECT_DIR" fluid apply "$FLUID_PROJECT_DIR/$CONTRACT_PATH" --env "$FLUID_ENV_NAME" --mode reconcile --build-id "$BUILD_ID" --yes--envis left out whenFLUID_ENV_NAMEis empty.Schedule and limits. It runs every 5 minutes,
*/5 * * * *, or on the build'sexecution.orchestration.airflow.dagConfig.schedule, which is validated like any DAG's cron.@onceis refused, because a keepalive has to recur. It runs one run at a time (max_active_runs=1), with no catch-up, no Airflow retries and a 30-minute task timeout. The build'sexecution.retriesis the restart budget (maxAttemptsrestarts per rolling hour, 5 when unset, at most 10), and--mode reconcileapplies it. Each wait inside a run is bounded byFLUID_SPARK_WAIT_SECONDS, 900 seconds by default.No outlets. A stream updates its tables all the time, and an asset event every 5 minutes would start every DAG scheduled on them. A dataset trigger on an expose that only the stream writes is refused, as before.
What a run does. A healthy stream is a no-op. A stream that died starts again on its checkpoint. A run succeeds (exit
0) when the stream is unchanged, restarted and still running, left stopped because it was cancelled on purpose, or there is nothing to reconcile. It fails (exit1, which Airflow's alerting picks up) when the stream drifted from the contract, nothing is deployed, the job config the last apply deployed cannot be read, the restarts are spent, EMR Serverless's or EMR on EKS's own retries are spent, the EMR on EC2 cluster ended, the restarted run ends at once, or an AWS call fails (a missing permission, say). Where forge restarts (EMR on EC2, Glue and a local build), a spent budget is not final: once the oldest restart is an hour old, the next run restarts the stream, and the runs before it fail.Stopping it. A stream cancelled on purpose stays stopped:
--mode reconcileleaves a cancelled run alone.fluid applystarts it again.The worker. It needs the settings and the forge release that the CI apply has: the
FLUID_SPARK_*operator settings (a missing allowlist refuses the build, and a setting that changes the compiled config reads as drift), and the role of the product'sfluid-<product>-spark-callerpolicy, which lets it read the stream's job config and runner, and, on EMR on EC2, read the cluster and the step role. See The caller's permissions.
For a stream with execution.retries and no other build, on Airflow:
Generated 1 files (airflow scheduler):
dags/ingest_orders_dag.py
Tip: To regenerate after editing the contract: fluid generate schedule
The DAG's docstring says what a run does:
Every run reconciles one always-on stream on the Airflow worker, with the
constants below filled in (`--env` is left out when FLUID_ENV_NAME is empty):
cd "$FLUID_PROJECT_DIR"
fluid apply "$FLUID_PROJECT_DIR/$CONTRACT_PATH" --env "$FLUID_ENV_NAME" \
--mode reconcile --build-id "$BUILD_ID" --yes
It runs no tofu apply. When the stream died (a Spot interruption, a failed
driver), it starts again on its checkpoint, within the restart budget the
build's execution.retries sets, so the task itself never retries. The run
fails when the stream cannot be kept running: it drifted from the contract,
nothing is deployed, the job config the last apply deployed cannot be read,
its cluster ended, or the restarts are spent. It can also fail when this
worker's FLUID_SPARK_* settings or forge release differ from the CI apply's.
A run cancelled on purpose stays stopped until fluid apply starts it. It
runs on SCHEDULE, one run at a time, with no catch-up. The build's
execution.orchestration.airflow.dagConfig.schedule sets SCHEDULE.
With --scheduler prefect or --scheduler dagster, the same contract writes nothing:
No build needs a DAG: every build runs outside forge-cli (a Flink job, for example), as a stream that fluid apply keeps running, or continuously (trigger.type: continuous), so there is nothing to schedule. No files written.
A stream that declares execution.retries gets a keepalive DAG on Airflow: fluid generate schedule --scheduler airflow.
fluid scaffold-composer, which writes a Cloud Composer DAG, writes nothing for it either:
Every build runs outside forge-cli (a Flink job, for example), as a stream that
fluid apply keeps running, or continuously (trigger.type: continuous), so there
is nothing to schedule. No DAG written.
A stream that declares execution.retries gets a keepalive DAG from fluid
generate schedule, which restarts it when it stops.
@once as the schedule is refused:
generate_schedule_invalid_schedule: {'error': "'ingest-orders': execution.orchestration.airflow.dagConfig.schedule: a keepalive DAG reconciles the stream on a recurring cron, and @once is not one"}
These outputs are from the real fluid generate schedule and fluid scaffold-composer on main, on the example stream with execution: {retries: {maxAttempts: 5}} added; the DAG file's constants are SCHEDULE = '*/5 * * * *', RETRIES = 0 and a task timeout of 1800 seconds. No Airflow ran here: forge-cli's own tests load the DAG in a real Airflow 2 and Airflow 3 when one is named.
Cloud Composer deployment
Upload a generated DAG to the environment's bucket:
BUCKET=$(gcloud composer environments describe my-env \
--location us-central1 \
--format="get(config.dagGcsPrefix)")
gsutil cp customer_360_pipeline_dag.py $BUCKET/dags/
Create the environment with the image version your organisation supports (gcloud composer environments create my-env --location us-central1 --image-version <composer-image-version>). The worker also needs FLUID_PROJECT_DIR, the fluid CLI and the product checkout; see What the DAG does.
Local Airflow testing
Parse the DAG file before deploying it:
python3 dags/gold.customer.analytics_360_v1/customer_360_pipeline_dag.py
If the file imports, Airflow can load it. Running it needs FLUID_PROJECT_DIR and the fluid CLI on the machine that runs the Airflow worker.
See also
- Declarative Airflow Walkthrough: a build-by-build tour
- Operating in CI: stage 11 and
schedule-syncin a pipeline fluid generate: thegeneratesubcommands- GCP walkthrough: Cloud Composer, end to end