fluid scaffold-composer
Generate a Cloud Composer (Airflow) DAG from a contract.
For a DAG that runs a scheduled build through fluid apply with its env and contract path fixed in the file, use fluid generate schedule. scaffold-composer writes a validate, plan, apply chain against the GCP provider.
Syntax
fluid scaffold-composer CONTRACT [--env ENV] [--out-dir DIR]
Key options
| Option | Description |
|---|---|
CONTRACT | Path to contract.fluid.yaml (positional, required). |
--env | Overlay env to apply before reading the contract. |
--out-dir | DAGs output directory (default runtime/composer/dags). |
Examples
fluid scaffold-composer contract.fluid.yaml
fluid scaffold-composer contract.fluid.yaml --env prod
fluid scaffold-composer contract.fluid.yaml --out-dir dags/
The first command writes runtime/composer/dags/<dag_id>.py:
with DAG(
dag_id='gold_customer_analytics_360_v1',
start_date=datetime(2024,1,1),
schedule='0 2 * * *',
catchup=False,
default_args={"retries": 1},
tags=["FLUID"]
) as dag:
validate = BashOperator(
task_id="validate",
bash_command='python -m fluid_build.cli validate contract.fluid.yaml'
)
plan = BashOperator(
task_id="plan",
bash_command='python -m fluid_build.cli --provider gcp plan contract.fluid.yaml --out /tmp/plan.json'
)
apply = BashOperator(
task_id="apply",
bash_command='python -m fluid_build.cli --provider gcp apply /tmp/plan.json --yes'
)
validate >> plan >> apply
Notes
Writes one
<dag_id>.pyfile per contract, wheredag_idcomes fromid(orname) with.replaced by_.The generated DAG has three sequential
BashOperatortasks:validate,plan,apply. Theplanandapplysteps run against the GCP provider, andapplypasses--yes.The tasks call
python -m fluid_build.cliwith the contract path as you gave it toscaffold-composer, relative to the Composer worker's working directory, so the contract has to be at that path on the worker.--envselects the overlay used to read the contract when the DAG is generated; the commands inside the DAG carry no--env.Schedule is read from
builds[].execution.trigger.cronin the contract; if missing, defaults to0 2 * * *(daily at 02:00). (since 0.22.0, forge-cli #737) It is the cron of the first build a scheduler starts: a build that runs outside forge-cli (engine: flink,properties.language: flink_sql) or whose trigger iscontinuousis skipped, so a Flink job's own cron does not schedule the DAG.trigger.schedule, the keygenerate schedulereads, is not read here: a contract withschedule: "15 3 * * *"and nocronstill gets0 2 * * *.(since 0.22.0, forge-cli #737) A contract whose every build runs outside forge-cli or continuously gets no DAG unless it declares
orchestration.tasks, the testfluid generate scheduleapplies before it writes no files. The command exits0:Every build runs outside forge-cli (a Flink job, for example) or continuously (trigger.type: continuous), so there is nothing to schedule. No DAG written.(since 0.34.0, forge-cli #1018) A stream that declares
execution.retrieshas a keepalive DAG that this command does not write. The command adds a second line,A stream that declares execution.retries gets a keepalive DAG from fluid generate schedule, which restarts it when it stops., andfluid generate schedulewrites the DAG: see Keeping a stream running.A DAG the command wrote for the product before stays where it is, and the command prints its path when it finds it under the name it writes now: delete it, and remove it from Cloud Composer if you deployed it there. A file of that name that it did not write is not mentioned.
For non-Composer Airflow installs, see
fluid generate-airflow. For Airflow, Dagster and Prefect output from one command, seefluid generate schedule.