LiteLLM Backend
LiteLLM is the LLM backend for the openai, anthropic, gemini and ollama providers. It is not in the base install: install it with the litellm extra (alias ai) or standard. It was a core dependency up to 0.27.0. (since 0.28.0, forge-cli #837) The mcp-sampling provider and the local agent CLIs (claude-code, codex, cursor, kiro) do not go through it; see LLM Providers.
Where this fits
LiteLLM was a hard dependency of core data-product-forge (litellm>=1.83.14,<2 on Python 3.11 and later) up to 0.27.0. (from 0.27.0) The litellm extra, with the alias ai, names it, and standard adds mcp; in 0.27.0 the base install still contains litellm, so installing the extra changes nothing there. (since 0.28.0, forge-cli #837) litellm is not in the base install. fluid forge and fluid forge data-model with an API-key provider (--llm-provider openai|anthropic|gemini|ollama), --agent-loop and the fluid ai setup provider wizard (not --clear or --source) need pip install "data-product-forge[litellm]" (or [ai], or [standard] to add mcp); an explicit API-key --llm-provider and the fluid ai setup wizard print that line and exit 1 before doing any work when it is missing. fluid init's AI hand-off prints the line and the keyless options, then falls back to template mode and exits 0. The companion LLM Providers page covers which providers --llm-provider accepts and which environment variables to set; this page covers the routing layer. See Upgrading.
Which providers you can select
fluid forge --llm-provider accepts openai, anthropic (alias claude), gemini and ollama for the LiteLLM backend, plus the keyless providers described in LLM Providers. The backend maps each of those names to a LiteLLM model prefix and falls back to a default model when you give none:
| Provider | LiteLLM prefix | Backend fallback model |
|---|---|---|
openai | openai | gpt-4.1-mini |
anthropic, claude | anthropic | claude-haiku-4-5 |
gemini | gemini | gemini-2.5-flash |
ollama | ollama | gemma3:4b |
A forge run normally gets its model from fluid ai models, which is claude-sonnet-4-6 for Anthropic, gemini-2.5-pro for Gemini and gemma4:latest for Ollama; the fallback column applies only when nothing names a model.
The backend's routing table also maps groq, bedrock, azure, vertex and vertex_ai, mistral, cohere and github to LiteLLM prefixes. As of 0.18.1 the argument parser rejects those names for --llm-provider, and FLUID_LLM_PROVIDER=bedrock is refused with Unknown LLM provider 'bedrock', so they cannot be selected from the CLI:
fluid forge: error: argument --llm-provider: invalid choice: 'bedrock' (choose from 'openai', 'anthropic', 'claude', 'gemini', 'ollama', 'mcp-sampling', 'claude-code', 'codex', 'cursor', 'kiro')
Quickstart
(since 0.31.0, forge-cli #881) --domain is removed from fluid forge; leave it out of the first command below. mcp-sampling is also removed from the --llm-provider choices shown above.
fluid forge --domain retail # uses the configured default
fluid forge --llm-provider openai --llm-model gpt-4.1-mini
fluid forge --llm-provider ollama --llm-model gemma4:latest
LiteLLM reports the cost of each call, and Forge folds it into .fluid/agents/<run-id>/cost.json. See fluid stats for the cross-run aggregator.
Configuration
| Env var | Purpose |
|---|---|
FLUID_LLM_PROVIDER | Provider key (openai, anthropic, gemini, ollama, and the keyless providers). Used when --llm-provider is not passed. |
FLUID_LLM_MODEL | Model name. |
FLUID_LITELLM_MODEL_PREFIX | Replace the LiteLLM prefix Forge puts in front of the model name (<prefix>/<model>). |
OPENAI_API_KEY / ANTHROPIC_API_KEY / GEMINI_API_KEY | Provider keys. LiteLLM reads the same names the underlying SDKs use. |
OLLAMA_HOST | Ollama's address. See The Ollama endpoint. |
LITELLM_* | Any LiteLLM-specific variable. LiteLLM reads these directly; Forge does not filter them. |
See the environment variables index for everything else.
Cost attribution
LiteLLM reports a per-call cost, which Forge records in the per-run cost.json that fluid stats aggregates.
Where LiteLLM reports no cost for a call, the figure comes from the price table in Cost Tracking.
Capability warnings
The capability catalog at fluid_build/copilot/agents/capability_catalog.py covers the provider and model combinations Forge knows. The warnings describe the model, not the route to it.
If you point LiteLLM at a model the catalog doesn't know, the run-start banner says "model X is not in the capability catalog" and the run continues with conservative defaults. See Capability Warnings.
Caveats
- Tool-use behaviour matches LiteLLM's wrapper. Provider failures surface as the typed errors described in Typed Errors (
RateLimitError,ContextOverflowErrorand the others). - (since 0.29.0, forge-cli #850) Forge's Ollama calls go to
OLLAMA_HOSTwhatever its host. See The Ollama endpoint. (since 0.29.0, forge-cli #852) The probe for a running Ollama, model discovery and the interactive Ollama setup follow it too; through 0.28.0 they stayed onlocalhost.
See also
- LLM Providers: which providers
--llm-provideraccepts and their credentials - Capability Warnings: what the capability catalog enforces
- Cost Tracking: how cost figures land in
.fluid/agents/<run-id>/cost.json fluid stats: aggregating cost across runs- Environment variables: the
FLUID_*reference