Ollama

Running local LLMs through Ollama with Flux — setup, model strings, the parameters Ollama drops on the floor, and when not to use it.

Flux talks to a local Ollama instance through the ollama Python SDK. The integration lives in flux/tasks/ai/ollama.py and is the easiest way to develop agent workflows offline — no API key, no network call, no per-token cost.

Install Ollama

Install the Ollama runtime from ollama.com. Confirm it’s running:

ollama serve         # in one shell, leave it running
ollama pull qwen3:8b # in another, grab a model
ollama list          # verify the model is local

Then install the Python client:

pip install ollama

Or via the LLM bundle:

pip install flux-core[ai]

No ANTHROPIC_API_KEY / OPENAI_API_KEY / GEMINI_API_KEY equivalent is needed — Ollama talks plain HTTP to your local daemon.

Connection

Flux constructs AsyncClient() with no arguments. The Python ollama SDK reads OLLAMA_HOST from the environment if set, otherwise defaults to http://localhost:11434:

# Optional — point at a remote Ollama daemon
export OLLAMA_HOST="http://gpu-box.local:11434"

The Flux layer doesn’t override this. If you need TLS, custom headers, or a non-default host on a per-call basis, build a small custom provider — see Adding a new provider.

Model string

Use ollama/ as the prefix; everything after the first / becomes the Ollama model identifier (tag and all):

from flux.tasks.ai import agent

assistant = await agent(
    "You are a helpful assistant.",
    model="ollama/qwen3:8b",
)

The colon between model and tag is part of Ollama’s identifier — qwen3:8b, llama3:70b, mistral:7b-instruct. Flux’s only requirement on the model string is that it contains a / somewhere. The colon is forwarded verbatim.

Streaming

Streaming is on by default. The Ollama formatter uses the SDK’s chat(stream=True) and yields tokens through Flux’s progress() task. Streaming is disabled when response_format is set, as with every other provider.

`max_tokens` is ignored

The Ollama formatter in flux/tasks/ai/ollama.py does not pass max_tokens to the Ollama API. The parameter is accepted on agent() (shared signature) but the formatter never reads it. Model output length is governed by the model’s defaults and your num_predict setting in the Ollama Modelfile, both of which sit outside Flux’s surface.

Structured output (with a sharp edge)

Ollama supports a format="json" mode plus a system-prompt-appended schema. The Flux formatter wires it like this (ollama.py::build_messages):

if self._response_format and not self._tool_names:
    schema_json = json.dumps(self._response_format.model_json_schema())
    messages[-1]["content"] += f"\n\nRespond with JSON matching this schema:\n{schema_json}"
    call_kwargs["format"] = "json"

Note the and not self._tool_names. When tools are also configured, Ollama drops response_format — the schema is not appended and format="json" is not set. Flux logs a warning when this happens ("Ollama: response_format is ignored when tools are in use..."), so the dropped constraint is visible in the worker logs. The agent runs as a normal tool-using loop and returns whatever text the model produces, which probably doesn’t parse as JSON.

Workarounds:

For “structured output with tools” workloads, route those calls to Anthropic, OpenAI, or Gemini.
Alternatively, model the “final answer” as a tool the agent must call (return_result(...)), and parse its arguments.

When response_format is set and no tools are configured, Pydantic-level validation still runs via model_validate_json() after Ollama returns.

Reasoning effort

The Flux formatter forwards reasoning_effort to Ollama’s think parameter, preserving the level granularity (ollama.py::build_messages):

if self._reasoning_effort is not None:
    if self._reasoning_effort in ("low", "medium", "high"):
        call_kwargs["think"] = self._reasoning_effort
    else:
        call_kwargs["think"] = True

"low", "medium", and "high" are passed through as distinct levels for models that accept a graded effort. Any other truthy value falls back to the boolean think=True for models that only support on/off.

This only matters for models with built-in thinking support (qwen3:8b, deepseek-r1:*, and similar). On models without thinking, the parameter is ignored by the Ollama daemon.

Tool calling and the text-extraction fallback

Some Ollama models support structured tool calls natively; others don’t. The Ollama formatter handles both:

If the response has message.tool_calls in the structured format, those are used directly.
Otherwise, if the response content matches a known tool name in text form, Flux falls back to extract_tool_calls_from_content() to pull tool calls out of the text body.

The fallback is best-effort — model performance varies. Larger instruction-tuned models with explicit tool-calling training (e.g. qwen3:8b, llama3.1:8b-instruct) are more reliable than older or smaller variants.

Ollama also doesn’t return stable tool-call IDs, so Flux generates a call_{uuid} per call. This keeps working-memory and replay keys unique across multi-turn loops.

Production use

Ollama is the right choice for local development, testing, and CI. It’s the wrong choice for production latency-sensitive workloads:

A single Ollama daemon serializes inference. Concurrent agent calls queue.
Local GPU memory caps your model size and context length.
There’s no SLA — if ollama serve crashes, your worker can’t finish the execution.

Production deployments typically run agents against Anthropic, OpenAI, or Gemini for the managed scaling. The recommended pattern is to switch via an environment variable:

import os
from flux.tasks.ai import agent

MODEL = os.getenv("FLUX_AGENT_MODEL", "ollama/qwen3:8b")
assistant = await agent("You are a helpful assistant.", model=MODEL)

Set FLUX_AGENT_MODEL=anthropic/claude-sonnet-4-20250514 in production, leave it unset locally.