A wall of organized tools representing a large catalog an AI agent selects from

AI Agent Tool Selection: Dynamic Discovery for Large Catalogs

August 13, 2026 6 min read

An agent with 120 tools does not have 120 useful choices on any given request. Most schemas are irrelevant, near-identical names make the wrong action easier to pick, and the definitions eat context before any work starts. Hand-trimming the list does not last, because permissions, workflow stages and integrations keep changing.

The fix is a two-stage boundary. Application code first filters the catalog by trusted identity and policy. A retrieval step then picks a small task-relevant set and loads only those full schemas into context. Discovery decides what the model may consider. The execution gate still decides whether a proposed call may run.

Check that you need it

Tool search adds a step before every tool call, and Anthropic's advanced tool use post says as much: it helps less with a small library (under 10 tools), with compact definitions, or when every tool is used in every session. The same post reports the upside on large libraries. Loading 50-plus MCP tools up front took about 72K tokens, against roughly 8.7K total with the tool search tool, and in Anthropic's internal MCP evaluations accuracy rose from 49% to 74% on Opus 4 and from 79.5% to 88.1% on Opus 4.5. Those are vendor numbers on vendor tasks. Run your own all-tools baseline first, and if it already picks the right tool, stop there.

Frameworks reach the same conclusion from the operations side. LangChain's dynamic tool selection guide warns that too many tools overload context and increase errors, and shows runtime filtering by state, permissions, feature flags and conversation stage. A Kaiten MCP issue gives the user-side complaint: a large set of unused tools consumes agent context.

Per-request tool lists can break your prompt cache

A client-side selector that rebuilds the tool list on every request has a hidden cost. Anthropic's prompt caching docs list tool definitions as the most disruptive change: modifying names, descriptions or parameters invalidates the tools, system and messages caches. A selector that returns a different top 8 each turn pays full input price on the whole prefix each turn, which can cancel the saving you built it for.

Two ways around it. Select once per task or workflow stage and keep the set stable until the stage changes. Or use provider-side deferred loading: Anthropic states that its tool search tool does not break caching because deferred tools stay out of the initial prompt. Log cache-read tokens next to tool-token counts when you compare designs.

Keep discovery separate from execution

Treat discovery as a service with a narrow contract: trusted runtime context and a task description in, tool definitions the model may consider out. Resolve these inputs first, none of them taken from the user's or model's own claims:

authenticated user and organization
delegated or workload identity
granted roles and scopes
current workflow stage
enabled integrations and feature flags
environment and data classification

Apply these filters before semantic retrieval. Searching everything and removing forbidden results afterward leaks tool names and descriptions into logs and traces, and leaves fewer candidates than the limit you tested with.

Give each catalog entry operational metadata

Store what filtering, refresh and audit need, and send only the schema to the model:

{
  "tool_id": "crm.orders.list_open",
  "version": "2026-08-13.1",
  "namespace": "crm.orders",
  "summary": "List open orders for an authorized customer account",
  "required_scopes": ["orders.read"],
  "allowed_stages": ["investigation", "reply_draft"],
  "risk": "read",
  "schema_hash": "catalog-managed-version",
  "source_server": "crm-mcp",
  "enabled": true
}

Use a stable internal ID even when the provider shows another display name. OpenAI's Tool Search documentation recommends clearly described namespaces for deferred functions, so the model sees a namespace description before it sees any parameter schema. Keep authorization attributes out of the model-editable description. The sentence "administrators can delete orders" helps retrieval and proves nothing about the caller.

Filter first, retrieve second

def select_tools(task, runtime, catalog, limit=8):
    eligible = [
        tool for tool in catalog
        if tool.enabled
        and tool.environment == runtime.environment
        and tool.required_scopes <= runtime.scopes
        and runtime.stage in tool.allowed_stages
        and tool.source_server in runtime.approved_servers
    ]

    ranked = semantic_rank(task, eligible)
    candidates = ranked[:limit]

    if retrieval_confidence(candidates) < runtime.minimum_confidence:
        return deterministic_fallback(task, eligible)

    return load_complete_schemas(candidates)

Build the retrieval text from the namespace, summary and example requests. Keep secrets, tenant names and live permission data out of it. Pick the candidate limit from measured recall, because a correct tool that never enters the set cannot be recovered by prompting. Skip semantic ranking for tools that should always be present, such as an approval or stop function, and for small read-only tools whose use is unambiguous.

Nearest is not relevant

Semantic retrieval always returns something. If the top result is simply the closest of a bad set, the model gets a plausible wrong tool, and the numbers show how often. The RAG-MCP paper reports tool selection accuracy of 43.13 percent with retrieval against a 13.62 percent baseline without it, which cuts prompt tokens by more than half but still leaves most selections wrong in that benchmark (RAG-MCP). Put an admission gate between retrieval and the model. A candidate must clear a cosine-similarity threshold set by its risk class: lowest for reads, higher for writes, highest for high-risk actions. When two write candidates clear the threshold within a small margin, return ambiguous so the agent asks which one is meant, instead of guessing between two side effects. "No action" is a normal outcome. When retrieval is down, admit only tools whose trusted aliases match the request and never fall back to exposing everything. The thresholds you start with are defaults from a first version, not tuned numbers, so label them that way.

Another approach keeps the catalog in the prompt in compressed form: each tool's name and a first-sentence summary of about 100 characters, with a rule that the agent must call a schema-lookup tool for the full parameter schema before using a tool whose arguments it is unsure of. Both designs share one rule. A fallback must not restore the full catalog, because that brings back the original problem exactly when the selector is least reliable.

Refresh catalogs without stale authority

The MCP tools specification defines paginated tools/list and a notifications/tools/list_changed event for servers that declare the capability. Use them to update metadata, and do not let a notification widen access on its own. Fetch every page into a snapshot, validate names and input schemas, apply your trust policy and allowlist, diff the additions, removals and schema changes, rebuild affected retrieval records, and publish atomically under a new catalog version. If refresh fails midway, keep the last validated snapshot. Authorization should read current policy, never a cached retrieval document, so a removed tool stops being callable even while its old embedding is still indexed.

Store the catalog version and the selected tool IDs in every trace. A failed run cannot be reproduced after schemas change without them.

Test selection as its own system

A good final answer does not prove discovery worked. Label 30 to 50 real tasks with the acceptable tools, and include similar tools, multi-tool tasks, disabled integrations, renamed schemas and requests for forbidden capabilities. Compare the selector with the all-tools baseline on candidate recall, false selections, whether denied tools appear in names or traces, tool-definition tokens, cache-read tokens, latency, call validity and task completion. Run the same request as two users with different scopes and check that their eligible catalogs differ.

Roll out in shadow mode: generate the dynamic set while the current path makes the real decision, and compare. Enable one low-risk namespace first and keep the typed execution gate behind it.

References

Anthropic advanced tool use for the token and accuracy figures, the small-library caveat and the caching statement.
Anthropic prompt caching documentation for the cache-invalidation rule on tool definition changes.
OpenAI Tool Search for deferred loading, namespaces, hosted search and client-executed discovery.
LangChain dynamic tool selection for runtime filtering by state, permissions, feature flags and stage.
MCP tools specification for paginated tool discovery, schemas and catalog-change notifications.
Kaiten MCP issue 123 for a practitioner report of unused MCP tools consuming context.
RAG-MCP: Mitigating Prompt Bloat in LLM Tool Selection via Retrieval-Augmented Generation: 43.13 percent tool selection accuracy with retrieval against 13.62 percent without, and prompt tokens cut by over 50 percent.

About Fire In Belly: Independent senior engineering from Tallinn, Estonia. We design and build AI workflow automation with the policy-aware discovery, deferred schema loading, and selection evaluation described above, at published fixed prices. Schedule a call to discuss your next project.