LLM Provider Migration: Moving Production AI Workflows

Moving an AI workflow to another model or provider breaks things that a ten-prompt smoke test never exercises. Some of the worst breaks happen inside one vendor, between two model versions. This post covers what changes, how to test it, and a rollout that can be reversed.

Same-vendor upgrades break loudly

The usual warning is that an OpenAI-compatible endpoint is not behavior-compatible. That is true: Anthropic's OpenAI SDK compatibility guide says its layer is mainly for testing and comparison, and Google's Gemini compatibility documentation lists limitations and parameters it ignores. A less discussed case is a version bump on the same platform. Anthropic's Claude Sonnet 5.5 migration guide lists settings that now return a 400 error, including sampling parameters (temperature, top_p, top_k), thinking budgets, a prefilled assistant turn, and a forced tool_choice of type any or tool. Code that worked on the previous model fails at the first request.

Record the contract in one module per model family. Send none of the parameters the new model rejects, omit thinking where it is not supported, set the effort level explicitly because thinking tokens bill as output, and replace a forced tool with auto plus a system-prompt instruction and a check that the response contains the expected tool. Key every capability check off the model family substring and not a full ID, so the base ID and each regional inference-profile prefix on a cloud platform behave the same. Keep a source link next to each rule, and do not scatter this knowledge across request builders.

The tokenizer changes your bill and your limits

The same guide says Sonnet 5.5 uses Sonnet 5's tokenizer, and against Sonnet 4.6, Sonnet 4.5 and Haiku 4.5 the same text produces about 30 percent more tokens, depending on content. Three things follow. Cost per request rises by up to that much even at an unchanged price per token, and coming from Haiku 4.5 the price per token is higher too. A max_tokens or context budget sized for the old model truncates structured output sooner. And any token estimator you wrote is now wrong. Recount a sample of real prompts with the provider's token-counting endpoint before you compare prices, and rerun the comparison per task, not per token.

Valid output is not equal output

Even a dated model name does not stay still. Chen, Zaharia and Zou ran the March and June 2023 versions of GPT-4 and GPT-3.5 on the same tasks and found large behavior changes. GPT-4 identified prime versus composite numbers with 84 percent accuracy in March and 51 percent in June, and the authors attribute part of the drop to weaker chain-of-thought behavior and a general decline in instruction-following (How Is ChatGPT's Behavior Changing over Time?). The responses still looked well formed. So when you migrate, or when the vendor updates a model behind your alias, compare task outcomes on a fixed evaluation set rather than checking that the output parses.

The migration test therefore needs outcome metrics next to validity metrics: tool calls and model calls per task, wall-clock time, how often the run hits the iteration cap, and whether the final business result matches the baseline. A candidate that passes schema validation and takes twice the calls costs twice as much and times out sooner.

Define what the workflow owns

Write down the promises that must survive: one classification per accepted ticket, results that pass a local schema, no tool call changing state before policy checks pass, and no action run twice after a timeout. Store conversations in an internal envelope with the tool name, a local call ID, an arguments object that is always present (even when empty), and the schema version. Keep raw provider payloads for diagnostics only. One Agno issue reports a zero-argument Anthropic tool call persisted without its empty arguments object, which then failed when the history was replayed through OpenAI. The OpenAI function-calling guide separates tool definitions, model tool calls and tool outputs, and your stored record should keep them separate too.

{
  "conversation_id": "case-1842",
  "turn_id": "turn-7",
  "message": {
    "role": "assistant",
    "tool_requests": [
      { "call_id": "call-3", "tool_name": "route_ticket",
        "arguments": {}, "schema_version": "2" }
    ]
  },
  "execution": { "provider": "candidate", "adapter_version": "4", "side_effect_mode": "dry_run" }
}

Adapters, with capability checks that fail

Write one adapter per provider that translates requests, normalizes responses, validates capabilities and classifies errors. Keep business decisions out of it. A lowest-common-denominator wrapper tends to remove the features that made the workflow reliable, so keep a portable core and declare optional extensions. If a required field has no equivalent, fail at startup or on the request with a clear error. A silently dropped field produces plausible output that breaks the contract. LangChain's model interface gives a common way to call many providers and notes that supported parameters still vary by model and provider.

Test, shadow, canary, roll back

1. Build a corpus from redacted production traces: routine cases, long histories, refusals, past tool errors, and the oldest stored conversation format.
2. Run deterministic checks first: local schema validation, known tool names, required arguments, every stop reason mapped to an internal value, unknown content blocks rejected.
3. Replay stored conversations through the old adapter and out through the candidate. This is where history-format bugs surface.
4. Run tools as dry-run implementations that return realistic results, including errors, and log the proposed action next to the baseline's.
5. Mirror live traffic to the candidate with its actions discarded. Then move a small, identifiable cohort, keeping one provider per conversation unless cross-provider replay has passed.
6. Set hard stops before the canary: any unauthorized or duplicated tool action, a rise in schema failures, unmapped stop reasons, conversations that cannot resume, or cost and latency beyond budget.

Rollback should change routing, not require a redeploy. Keep the old adapter, credentials and prompt version available until in-flight conversations finish and the rollback window closes. Keep the golden corpus in CI afterwards, since portability drifts whenever an API adds a field.

References

• Claude Sonnet 5.5 migration guide
• Anthropic OpenAI SDK compatibility
• Google Gemini OpenAI compatibility
• OpenAI function calling
• LangChain models
• Agno issue 8971
• Chen, Zaharia and Zou, How Is ChatGPT's Behavior Changing over Time?: GPT-4 prime-number accuracy fell from 84 to 51 percent between March and June 2023.

---