Jenkins CI/CD for FLUID Data Products

Run a data product through validation, planning, apply, verification and publication on Jenkins, with a pipeline the CLI writes for you instead of one you maintain by hand. Each stage is a fluid command, so a failed gate stops the build at the stage that found the problem.

This page covers the Jenkins side: generating the Jenkinsfile, setting up the controller and agent, credentials, running the job and what to do when a build fails. What each of the eleven stages checks, with the output each prints, is in The 11-stage pipeline. Details below come from the Jenkinsfile that CLI 0.18.1 generates.

Generate the Jenkinsfile

From the directory that holds contract.fluid.yaml:

fluid generate ci contract.fluid.yaml --system jenkins --out Jenkinsfile

[install-mode: pypi] Jenkinsfile written -> Jenkinsfile
  |- Jenkins installs FLUID_PACKAGE_SPEC into $WORKSPACE/.fluid-venv (stage 0)
  |  Override at build time via these Jenkins parameters:
  |    FLUID_PACKAGE_SPEC        = 'data-product-forge==X.Y.Z'  (another version)
  |    FLUID_PIP_INDEX_URL       = 'https://test.pypi.org/simple/'  (TestPyPI)
  |    FLUID_PIP_EXTRA_INDEX_URL = 'https://pypi.org/simple/'   (fallback)
  |    FLUID_ALLOW_PRERELEASE    = true  (pip --pre, alpha/rc releases)
  |- FLUID_PACKAGE_SPEC default: data-product-forge[local]==0.18.1
  |- Jenkins plugins required: workflow-aggregator, git

Commit the Jenkinsfile beside the contract. The file is generated once and then belongs to you: after you upgrade the CLI, regenerate it and review the diff to pick up changes. The CONTRACT parameter defaults to the path you generated from, and FLUID_PACKAGE_SPEC to the CLI version that generated the file with the extras the contract and its overlays need. A contract that has an overlays/gcp.yaml binding to BigQuery generates data-product-forge[gcp,local]==0.18.1; --fluid-package-spec overrides it.

Set up Jenkins

Stage 0 creates a virtual environment at $WORKSPACE/.fluid-venv and installs FLUID_PACKAGE_SPEC into it, because agents that follow PEP 668 refuse a bare pip install. Every later stage runs the fluid from that environment. A private package index goes in the FLUID_PIP_INDEX_URL parameter, as one mirror that also proxies PyPI. Leave FLUID_PIP_EXTRA_INDEX_URL empty, for the reason in the warning under Generate the Jenkinsfile. FLUID_ALLOW_PRERELEASE adds pip --pre.

Create the job

1. In Jenkins choose New Item, then Pipeline (or Multibranch Pipeline).
2. Under Pipeline, set Definition to Pipeline script from SCM, choose Git, enter the repository URL and set Script Path to Jenkinsfile.
3. Save, then run Build Now once.

Run that first build with the defaults. Jenkins learns a Pipeline's parameters from a run, so the first build runs with none set; the generated Jenkinsfile gives every parameter a default and every stage reads the same default as its fallback, so the first build is the dry run the defaults describe. Until a job has run once, Jenkins refuses buildWithParameters for it (observed on 4 to 5 October 2026 against a controller running a generated pipeline); after that, Build with Parameters lists them.

Give the pipeline credentials

fluid reads provider credentials from the environment of the build. The generated Jenkinsfile starts with a comment listing the variables each target reads:

Surface them to the build with Jenkins credentials. The agent's environment is the other route, and it carries the warning at the end of this section.

Jenkins credentials (recommended). Create credentials under Manage Jenkins, Credentials, then bind them in the Jenkinsfile's environment {} block. Jenkins masks a bound value in the console log. withCredentials([...]) around the steps that need a secret does the same for a narrower scope:

environment {
    GOOGLE_APPLICATION_CREDENTIALS = credentials('gcp-sa-key')
    FLUID_API_KEY = credentials('command-center-api-key')
}

For GOOGLE_APPLICATION_CREDENTIALS the credential is a Secret file, which Jenkins exposes as the path of a temporary file. Do not write a key into the Jenkinsfile or into a job parameter. A binding in the pipeline-level environment {} is visible to every stage, stage 0's pip install included. To keep it out of that stage, bind it in the environment {} of the stages that talk to the target, or use withCredentials.

Prefer short-lived credentials to stored keys where the cloud offers them: Workload Identity Federation for GCP and an OIDC role for AWS (the generated Jenkinsfile's header comment lists both as alternatives), in place of a service-account key file or an AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY pair that does not expire.

Agent environment. Set the variables on the agent: a Docker Compose environment:, a Kubernetes agent template, or Jenkins Global Node Properties. sh steps inherit them and the Jenkinsfile needs no change.

Since 0.17.0, a pipeline configured to publish to the Command Center also reports each fluid apply run there, with the same credentials; see Stage 7.

Run it

The stage toggles are RUN_STAGE_1_BUNDLE to RUN_STAGE_11_SCHEDULE_SYNC, and APPLY_MODE decides what stages 6 to 11 do. Common runs:

Parameter values reach the shell as environment variables, and the Jenkinsfile builds each command with set -- so each value stays one argument. A value such as APPLY_BUILD_ID="x --allow-data-loss" cannot add a flag.

One job per environment

FLUID_ENV is the overlay the stages pass to --env. Generate one pipeline per environment, or one pipeline you run with different FLUID_ENV values:

fluid generate ci contract.fluid.yaml --system jenkins --fluid-env-default prod --out Jenkinsfile.prod

The overlay is a file beside the contract, overlays/<env>.yaml, that patches the binding. See Per-environment overlays, One contract, two clouds and Switch clouds. A product applied from two clouds keeps two OpenTofu states. If two jobs for different clouds both publish the same product, the Command Center keeps whichever published last; see Stage 10.

Require approval before apply

The generated Jenkinsfile does not pause for approval. To add a gate, put a Jenkins input step before stage 7 and keep the edit in your repository, since regenerating the file removes it:

stage('Approve apply') {
    when { expression { params.APPLY_MODE != 'dry-run' && params.FLUID_ENV == 'prod' } }
    steps {
        input message: "Apply ${params.APPLY_MODE} to ${params.FLUID_ENV}?", submitter: 'release-managers'
    }
}

Stage 7 only applies a plan whose digests match the bundle, so what the approver reviewed in runtime/plan.html is what runs.

What the build keeps

Each stage archives its output with archiveArtifacts, so a failed build leaves the evidence:

The workspace is deleted at the end of every build, so nothing carries over except the archived files. Stages read the bundle from the same build's workspace.

Troubleshooting

These messages are the ones the generated Jenkinsfile prints.

stage 0: the CLI first on PATH is ..., not the one in $FLUID_VENV: the PATH entry of environment {} did not apply Something replaced PATH after the environment {} block, so the stages would run a different fluid from the one stage 0 installed. Remove the override, or put $WORKSPACE/.fluid-venv/bin first in it.

stage N reads runtime/bundle.tgz, which stage 1 writes: run stage 1 in the same buildRUN_STAGE_1_BUNDLE was cleared. Stages 2, 3, 5, 6, 7 and 9 read the bundle, and the workspace does not persist between builds, so stage 1 has to run in the same build.

FLUID_ENV must be a plain environment nameFLUID_ENV may contain letters, digits, ., _ and -, and may not start with . or -. Stage 0 exits 2 otherwise.

APPLY_BUILD_ID is not passed on: APPLY_MODE amend runs no build This is a notice, not an error. Only amend-and-build and replace-and-build run builds, and only they take --build-id.

Stage 5 fails on a pipeline that was green before an upgrade. Since 0.16.3 the drift gate compares live targets. See Stage 5 for the statuses and what counts as drift.

Stages 9 to 11 print stage 7 ran as a dry run ... skipped.APPLY_MODE is dry-run, the default. Nothing was applied, so there is nothing to verify, publish or schedule. Stage 8 does not skip: after a dry run it prints that the bindings are checked, not enforced, and runs policy-apply with --mode check.

Hand-written pipelines

A hand-written Jenkinsfile that calls fluid directly is the other way to do this: Universal pipeline shows one that carries no provider-specific logic. The generated pipeline adds the bundle chain, the digest checks between stages and the first-build defaults.

Related

• The 11-stage pipeline
• Operating in CI
• fluid generate ci
• Declarative Airflow