docs / model routing

Policy-based LLM routing

Agents don't hardcode providers or model names. They run with a capability and a runtime context, and the platform resolves the effective model from a catalog — deterministically, and auditable.

Agents don't pick models

An agent definition declares an id, a role, tools, and behavior — but no provider, no model name, not even a capability. When it runs, the runtime asks a routing resolver for the effective model, keyed by the capability it needs and the request context (team, user, agent, operation). The resolver returns a model profile; a model factory builds the concrete client.

Result: model choice is governed centrally from a catalog, decoupled from agent code — you can change models, add a faster model for a phase, or swap a provider without touching an agent.

Concepts

Concept	Meaning
capability	The technical model family the runtime needs — one of `chat`, `language`, `embedding`, `image`. The runtime sets this (chat for ReAct/Graph); the author never declares it.
operation	The pipeline phase of a multi-step agent (e.g. `routing`, `planning`, or a graph node name). The primary routing dimension.
purpose	Optional business-intent discriminator (e.g. `rag`, `chatbot`).
model profile	A named, reusable model configuration — `profile_id` + `capability` + `model` (`provider`, `name`, `settings`).

Source: fred-runtime model_routing/contracts.py (ModelCapability, ModelProfile)

The model catalog

Routing is loaded from config/models_catalog.yaml at pod startup (override with FRED_MODELS_CATALOG_FILE). The catalog is mandatory. Its keys:

common_model_settings — global defaults merged into every profile.
common_model_settings_by_capability — defaults merged per capability.
default_profile_by_capability — required: the fallback profile for each capability.
profiles — the concrete model profiles.
rules — routing overrides (empty by default).

Settings merge deterministically: common_model_settings → common_model_settings_by_capability[capability] → profile.model.settings (later wins).

version: v1
common_model_settings:
  temperature: 0.0
  max_retries: 0
default_profile_by_capability:
  chat: default.chat
profiles:
  - profile_id: default.chat
    capability: chat
    model:
      provider: openai
      name: gpt-4o-mini
  - profile_id: chat.fast
    capability: chat
    model: { provider: openai, name: gpt-4o-mini }
  - profile_id: chat.quality
    capability: chat
    model: { provider: openai, name: gpt-4o }
rules:
  - rule_id: react.routing.fast
    capability: chat
    operation: routing
    target_profile_id: chat.fast
  - rule_id: react.planning.quality
    capability: chat
    operation: planning
    target_profile_id: chat.quality

Source: fred-runtime model_routing/catalog.py · example apps/fred-agents/config/models_catalog.yaml

Providers

A profile's provider selects the client the model factory builds. Supported providers:

provider	Backend
openai	OpenAI models — and any OpenAI-compatible endpoint (e.g. Mistral, or a local mock) by setting a `base_url` in settings.
azure-openai	Azure OpenAI deployments.
azure-apim	Azure OpenAI fronted by API Management.
ollama	Locally / self-hosted models via Ollama.
vertex-ai	Google Vertex AI (Gemini).
vertex-ai-model-garden	Vertex Model Garden (Mistral / Llama / Claude).
anthropic	Anthropic Claude models.

Mistral is not a separate provider. Mistral profiles use provider: openai with a Mistral base_url. Provider credentials come from environment variables (OPENAI_API_KEY, ANTHROPIC_API_KEY, GCP ADC, …), never from the catalog.

Source: fred-core model/models.py (ModelProvider) · model/factory.py

Rules & resolution

A rule maps a context to a profile. Required fields are rule_id, capability, and target_profile_id; optional criteria are operation, purpose, agent_id, team_id, user_id. A rule with any criterion must set operation — catch-all rules are rejected (use the capability default instead).

For one selection request, the resolver:

Keeps rules whose capability equals the request's.
Keeps rules whose every defined criterion matches the context (an absent criterion is a wildcard).
Picks the winner by specificity (most defined criteria), tie-broken by declaration order.
If nothing matches, uses default_profile_by_capability[capability] — and errors if there is no default for that capability.

The decision is returned as a result carrying its source (rule or default), the profile_id, the resolved model, the matched rule_id, and the matched criteria — deterministic and testable.

Source: fred-runtime model_routing/resolver.py (ModelRoutingResolver) · contracts.py

Runtime behavior

ReAct agents resolve a model per phase: the operation is inferred from the conversation — routing after a user message, planning after a tool result — so you can route a fast model for routing and a stronger model for planning.
Graph agents can resolve per node (the operation is the node name); otherwise the model is resolved once when the graph is built.
Embeddings for ingestion are configured separately in the Knowledge Flow config, not through the chat routing path.

Source: fred-runtime react_tool_loop.py, react_model_adapter.py, graph_runtime.py

Where routing lives

Two layers with a clean split:

Decide which profile — fred-runtime's model_routing package: a pure, deterministic resolver over the loaded policy (no I/O, no model construction).
Build the client — fred-core's model factory turns the chosen profile's configuration into the concrete LangChain client.

The agent pod wires these together at startup, exposing a model-factory port to the runtime; the agent only ever asks for "a model for this operation".

Source: fred-runtime model_routing/provider.py, app/agent_app.py · fred-core model/factory.py

Governance position

The posture is policy-first: model choice is governed from the catalog, not chosen by end users at runtime — there is no end-user model picker acting as the routing authority. (The one place a user selects a model profile is the evaluation judge, a separate surface.) This keeps behavior predictable and aligned with team governance.

Model routing is the model domain of Fred's governance. Tools live in a sibling mcp_catalog.yaml loaded the same way; conversation retention and scheduling have their own policy catalogs. They are parallel per-domain catalogs rather than one monolith.

Source: fred-agents config/models_catalog.yaml, config/mcp_catalog.yaml

Policy-based LLM routing

Agents don't pick models

Concepts

The model catalog

Providers

Rules & resolution

Runtime behavior

Where routing lives

Governance position

Related documents