Production Troubleshooting

Symptom, diagnosis and fix for fluid pipelines in production. The error names, messages and commands on this page were checked against CLI 0.18.1. Many errors the CLI prints end with a link to this page, so the sections below are organized by the event name or class name you see in the output.

Reading an error

A failure prints in one of two shapes, and both are catalogued here and on Typed CLI Errors:

❌ bundle_env_mismatch  [ERR_BUNDLE_ENV_MISMATCH]
  bundle_env: dev
  requested_env: prod
  hint: the bundle was built for env 'dev' but this stage was asked for env 'prod'. ...

The event name (bundle_env_mismatch) is stable, and so is the ERR_ code derived from it; match on either in CI. Error codes explains each part of the output and lists the events that carry curated suggestions. The other shape is a panel titled with the error, with why, fix and doc rows, raised by build runners and providers. Look the name up in the section that matches the stage that failed.

First responder: fluid doctor

For an escalation, attach the output of fluid doctor --scope all --json.

Pipeline gates

These errors stop a generated pipeline before it changes anything.

Plan-binding rejections

Symptom: fluid apply refuses to run with a PlanBindingError before any DDL executes.

Each error carries a stable kind tag, a distinct event CI can match:

Environment, mode and plan-file errors

The federation check

See Federated upstreams.

Symptom: the apply log has apply_consumes_drift and the apply carried on.

The check for a consumes[] entry that names an upstreamWorkspace is advisory: it warns and applies anyway.

apply_consumes_drift: 1 federated consumes[] entry could not be confirmed in sync (1 drift). Applying anyway. Details: {...}

The Details JSON lists each violation with a violation_kind (drift, unreachable, unpinned, unknown-workspace or not-wired), plus counts_by_kind, drift_count and unreachable_count. A federation_gate_error line means the validator itself failed and the pins were not checked. To block a pipeline on this, match apply_consumes_drift in the apply log; the exit code does not change. The upstream fetch is bounded by FLUID_FEDERATION_TIMEOUT_SECONDS (default 30) and uses the git binary. --no-verify-federation silences the check.

Apply data-loss gate

Symptom: apply aborts and tells you to re-run with --allow-data-loss.

State and region errors

See OpenTofu state for how the key and the move work.

Symptom: fluid apply stops before tofu apply with a state or region error.

Remote OpenTofu state is keyed per contract and provider. See Environment variables for the key layout. The first apply after upgrading to a release with per-provider keys moves state from fluid/<id>/terraform.tfstate to fluid/<id>/<provider>/terraform.tfstate when the old key holds this provider's resources, prints a state move: line, and leaves the old object where it was. --dry-run never moves state, and fluid diff and fluid verify --state-drift read the old key while a move is pending.

Contract load and overlay errors

Symptom: ERR_CONTRACT_LOAD_FAILED, or ❌ Validation error: contract_load_failed. The CLI's suggestions for this code are generic (check the YAML, the encoding); read the error: line, which names the real cause.

Builds and masking

Symptom: an acquisition or embedded-SQL build failed, or refused to start.

Build refusals print as three lines under the build (what, why: ..., fix: ...) and fail the build; the typed class is in the log as embedded_sql_io_refused ... code=<Class>.

Treated columns land as strings, and fluid verify fails a masked column that landed in cleartext.

Build and run failures

Work the run-record surface under the state root (./.fluid by default; override with --state-root):

fluid runs status <product_id> --last 5             # recent runs of a product
fluid runs status <product_id> --build <build_id>   # pin a specific build
fluid runs logs <product_id> --component build      # component: build|infra|server|worker|dlq
fluid runs logs <product_id> --run-id <id> --grep ERROR --limit 200
fluid runs diff <product_id> --build <b> --run-a <baseline> --run-b <comparison>

--run-id applies to the dlq component only, and dlq returns nothing without it. --grep is a case-sensitive literal substring, not a regular expression. runs diff compares counts and states of two runs; it does not compare columns or schemas, so it cannot show that a column was added. All three verbs accept --json; see fluid runs.

Cursor rewind and replay-pending markers

Symptom: a file appears at .fluid/<product_id>/runtime/replay-pending.json.

An upstream source-aligned product's cursor moved backward (a reprocess), and the runners marked every downstream product that consumes[] it as dirty, so a downstream does not go silently stale. The marker records what happened:

{
  "upstream_product_id": "bronze.crm.customers",
  "upstream_build_id": "main_build",
  "upstream_stream": "customers",
  "old_cursor_value": "2026-04-30T00:00:00Z",
  "new_cursor_value": "2026-04-15T00:00:00Z",
  "detected_at": "2026-05-02T12:30:00Z",
  "reason": "upstream cursor rewound from '2026-04-30T00:00:00Z' to '2026-04-15T00:00:00Z'"
}

Fix: re-run the marked downstream product's build so it re-reads the rewound window, then delete the marker file. Leaving it does not affect execution, but you lose the signal for the next rewind.

Publishing to the Command Center

See The Command Center for configuration and organization selection.

Symptom: fluid publish --target fluid-command-center fails, including with --dry-run or --verify-only.

The organization comes from organization_id, else FLUID_CC_ORG_ID; with neither, a configured organization slug is matched against the Command Center's organization list; failing that, the credential's only organization. Products publish private unless the contract is classified public. For fluid apply run reports, see Environment variables.

Generated artifacts and schedules

Secrets and credentials

OpenTofu failures

LLM and copilot authentication failures

Symptom: an LLM-driven command such as fluid forge fails with a 401 or invalid-key error, typically right after a key rotation.

fluid ai status      # which provider, model and key source is configured
fluid ai test --provider anthropic --model <model>   # connectivity test, before saving anything
fluid ai setup       # re-run setup to store the rotated key

Update the key at the source fluid ai status reports, the provider environment variable (OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY) or the stored configuration, then confirm with fluid ai test. See LLM providers and backends for the resolution order.

fluid ai test is the connectivity check. As of 0.18.1, fluid ai test with no provider configured and no --provider fails with ❌ Unexpected error: name 'detect_ollama_available' is not defined and exit code 2. Name the provider, or run fluid ai setup first. fluid ai test --provider <name> --json prints a report with ok, exit_code and, on failure, an error object with code, message and suggestions.

Where logs live

On the local provider, error text in the build output, in retry log lines and in local_apply_log.jsonl is redacted: the value of every credential-named {{ env.X }} the build uses is replaced.

Retention sweep

Run state accumulates under the state root; sweep it on a schedule:

fluid retention sweep                     # sweep ./.fluid with a structured summary
fluid retention sweep --state-root /data/pipelines/.fluid --json

A replay requested past the swept horizon fails with StaleReplayError because its manifest is gone; see Typed CLI Errors. Balance the sweep cadence against how far back you replay.

See also

• Operating in CI: the pipeline these failures occur in
• Typed CLI Errors: the typed error classes, exit codes and where doc links land
• fluid doctor, fluid runs, fluid retention, fluid secrets: command references
• fluid rollback: restoring a pre-replace snapshot
• Environment variables: the FLUID_* variables