Adding a new provider

How Flux's provider system works and what it takes to plug in a new LLM — honest about the 0.56.0 limitation that there is no plugin API.

Flux ships four built-in providers — Anthropic, OpenAI, Google Gemini, and Ollama. If your workload needs a different one (Mistral’s hosted API, Cohere, an internal gateway, vLLM), you can wire it in. The path is source modification, not a plugin install.

How the provider system is wired

Each provider is two pieces:

A factory function — build_<provider>_provider(model_name, ...) returns a (task, formatter) pair.
A formatter class — subclasses flux.tasks.ai.formatter.LLMFormatter. Handles the conversion between Flux’s normalized message format and the provider’s wire format.

The agent() factory in flux/tasks/ai/agent.py parses the model string, picks a provider by name, calls the factory, and threads the result into the shared agent loop in flux/tasks/ai/agent_loop.py. Every provider sees the same system_prompt, tools, working_memory, and response_format arguments — the formatter is responsible for translating each one.

The contract lives in formatter.py:

class LLMFormatter(ABC):
    @abstractmethod
    def build_messages(system_prompt, user_content, working_memory) -> (messages, call_kwargs): ...

    @abstractmethod
    def format_assistant_message(response) -> dict: ...

    @abstractmethod
    def format_tool_results(tool_calls, results) -> list[dict]: ...

    @abstractmethod
    def format_user_message(text) -> dict: ...

    @abstractmethod
    def remove_tools_from_kwargs(call_kwargs) -> dict: ...

    @abstractmethod
    async def stream(messages, call_kwargs) -> AsyncIterator[str]: ...

    supports_reasoning_stream: bool = False
    # Optional: async def call_with_reasoning_stream(...)

Read flux/tasks/ai/openai.py first — it’s the smallest implementation and covers every method. gemini.py is the most involved because Gemini’s wire format diverges most sharply from the OpenAI-style baseline.

The four-step recipe

1. Add a provider module

Create flux/tasks/ai/<your_provider>.py. Implement the factory + formatter pair. The factory returns a (task, formatter) tuple; the task is a @task.with_options(name="...") async function that calls the provider’s SDK and returns an LLMResponse.

Keep the import of the SDK inside a try/except ImportError, the way the existing modules do:

try:
    from your_sdk import AsyncClient
except ImportError:
    AsyncClient = None  # type: ignore

def build_<your>_provider(model_name, ...):
    if AsyncClient is None:
        raise ImportError(
            "To use <your> models, install the <your_sdk> package: pip install <your_sdk>",
        )
    ...

That keeps flux-core installable without your SDK and gives users a clear error when they reach for a provider they haven’t installed.

2. Register in `agent.py`

Add a new elif provider == "<your_name>": branch to the provider dispatch in agent.py:153-318. Mirror the structure of the existing branches — import locally to keep cold-start fast, call your factory, build tool schemas, wrap in a @task.with_options(...) async function, call run_agent_loop.

Update the error message at agent.py:316-318 to include your provider in the supported list.

3. Handle streaming

Implement async def stream() on your formatter. It should yield strings — one chunk per token (or whatever granularity your SDK returns). If your provider supports thinking/reasoning streams and you want users to see those tokens through progress(), set supports_reasoning_stream = True and implement call_with_reasoning_stream(). Otherwise leave the default.

The agent loop checks supports_reasoning_stream and picks the right code path automatically.

4. Handle structured output

This is where providers diverge the most. Three patterns are in use today:

API-enforced (OpenAI, Gemini) — your build_messages adds a provider-specific config field, and the API guarantees a parseable JSON response.
Format hint (Ollama) — your formatter sets a format="json" flag and appends the schema to the last user message. Not enforced; the agent loop validates with Pydantic.
Prompt-only (Anthropic) — your formatter does nothing. The agent loop (agent_loop.py:137-139) handles schema-appending into the system prompt for you.

All three paths still trigger response_format.model_validate_json() in the agent loop, so Pydantic-level validation runs uniformly.

Two precautions if your provider supports tool calling and structured output simultaneously:

Preserve structured-output kwargs in remove_tools_from_kwargs() — the agent loop calls this to strip tools for the final “give me the answer” turn. Gemini’s implementation (gemini.py:214-234) is the reference.
Decide whether tools + response_format is supported. Ollama logs a warning and drops response_format in this case; document the behavior on your provider’s page either way.

Pull request expectations

Tests at tests/flux/tasks/ai/test_<your_provider>.py covering: build_messages with and without working memory, format_assistant_message with text / tool calls / reasoning, structured-output kwargs, streaming.
A docs page at site/src/content/docs/integrations/llm-providers/<your_provider>.mdx following the other provider pages’ structure: setup, model string, streaming, max_tokens behavior, reasoning effort, structured output, what can go wrong.
An entry in the capability matrix so the comparison stays useful.
An optional dependency declaration in pyproject.toml under [tool.poetry.dependencies] and the ai extra in [tool.poetry.extras].

Why no plugin API yet

The shape of LLMFormatter is still moving. supports_reasoning_stream was added in 0.31, the remove_tools_from_kwargs preservation rule for Gemini’s structured output landed in 0.32, and the Ollama tool-extraction fallback is younger still. Pinning the contract as a public plugin surface would lock in those decisions before the API has stabilized.

If you maintain a downstream provider out-of-tree and want the API frozen, file an issue describing the use case. A plugin entry point is on the roadmap; the gating question is which release stabilizes the formatter contract.