AI Workflow Versioning: Upgrade Without Breaking In-Flight Runs

Most advice on versioning long-running AI workflows assumes you will keep old runs alive and make new code compatible with them. The cheaper question comes first: how long does a run really need to live? This post covers the choice between pinning, patching, migrating and expiring, plus two failure modes teams meet in practice: rolling deploys where old and new code share a database, and runs killed mid-side-effect by a deploy.

Shorten the run before you version it

Every in-flight run is a compatibility promise. A run that waits three weeks for an approval needs a migration plan. One that waits 30 minutes needs a deploy window. Temporal's Worker Versioning documentation frames it the same way: a pinned workflow is guaranteed to finish on one deployment version, it recommends pinning for short workflows that complete before the next deploy, and it says that if all your workflows will stay short-running you can default to pinned. For agent or chatbot workflows that last weeks, it recommends pinning plus upgrading at Continue-as-New.

The short route works for human waits. Store a pending write approval with its arguments encrypted, expire it after 30 minutes, and let a sweep mark it expired and blank the stored payload. Delete settled rows after seven days. Because nothing waits longer than half an hour, a deploy never has to resume an approval created by code from last month, and the state schema for approvals never needs a multi-version reader. If your waits are long because nobody designed an expiry, add one and a "request again" path before building migration machinery. Where a wait really must be long, store the request as data (what to do, to which object, under which policy version) and not as a frozen graph position, so a new build can re-derive the next step. Remember too that a rollout is not instantaneous. AWS documents that Step Functions state machine updates are eventually consistent, so executions started immediately after an update might still use the previous definition and role (AWS Step Functions, read consistency). Plan for two definitions being live during every deploy.

Rolling deploys mean two versions share one database

Even pure request/response services have a window where old tasks and new tasks run side by side, and the old ones keep writing. A new column the old code does not know about, with no default, breaks every insert from the old release. Give a new column a database-level default, because a task still on the previous release inserts rows without it during a deploy. Use the same discipline for everything a durable run touches. Expand first (add the column or table with a default, deploy readers that accept both shapes), then switch writers, then contract (remove the old shape) once no old worker can run. A migration that renames a column in one step breaks the old tasks the moment it applies.

A related habit is to keep a job's payload to pointers and arguments only, never secrets, and to have handlers reload their configuration fresh when the job runs. That keeps the stored shape small and stable across builds. It also means a queued job runs under whatever configuration is current when it fires, which is the right default for credentials and a deliberate decision for anything that changes business behavior.

A deploy can kill a run mid-side-effect

The compatibility checklists cover replay and state shape. They tend to skip the run that was halfway through a tool call when the container stopped. Two rules apply to any durable workflow. First, a run that has sat in the running state past a timeout is not retried automatically. A sweep marks it failed with the message that the outcome is uncertain after an interruption and that it was not retried, and the conversation resumes with that result. Retrying an email send or payment of unknown status risks doing it twice, and a clearly reported "unknown" is cheaper. Second, nothing in a conversation should be silently dropped. A deploy that stops a turn midway can otherwise swallow the next message without a word. Set a durable resume request when a turn is interrupted, and let a background poller answer the message after the old claim goes stale.

Classify the change, then pick the treatment

Change	Main risk	Default treatment
Logging, metrics or internal refactor	Replay behavior changes by accident	Replay old histories, then deploy
New optional state field with a default	Older checkpoints omit it	Read both shapes, write the new one
Renamed or retyped state field	Old checkpoints fail the new schema	Explicit state migration
Added, removed or reordered branch	Old history no longer matches code	Pin old runs or add a version gate
Tool argument or result contract change	A resumed run calls an incompatible tool	Support both contracts or migrate at a safe boundary
Prompt or model policy change	Mixed behavior inside one decision	Pin the decision stage and record the policy version
Business rule change	Old cases may need another legal path	Get a product decision before migrating

Temporal's safe deployment guidance explains the replay constraint: workflow code must stay deterministic against recorded history, so incompatible changes need pinning or an explicit patch. An auto-upgrading workflow, per the same documentation, moves to new code as you roll out, so it needs patching to stay replay-safe. Microsoft's Durable Functions versioning guidance says the same about breaking changes and running instances. Record these on every durable run, stored with the run and not only in logs: workflow version, state schema version, prompt bundle version, tool contract version and worker build.

Migrate state as a pure function

A state migration should map one known schema to the next with no model call, no clock and no mutable external lookup, so the same checkpoint always produces the same result. Apply it with a conditional update on the stored version, keep the original checkpoint until the rollback window closes, and quarantine a checkpoint that fails migration instead of letting retries hammer it. Check the schema you migrate to, too. A version row can be present while required tables are missing, so setup looks successful and reads fail later. Verify the objects themselves at startup, not only the version number. Persistence concepts are described in the LangGraph persistence documentation.

Release sequence

1. Inventory active runs by workflow version, state schema, status, age and worker build.
2. Ship readers that accept the oldest supported state before any writer emits a new shape.
3. Run the migration against copies of real checkpoints, and replay sampled histories (by workflow type and branch, not just the most recent) against the candidate.
4. Start the new build, route a small share of new starts to it, and keep old runs on their declared path.
5. Compare completion, replay, migration, tool and approval outcomes by version, then drain, migrate or terminate old runs.
6. Remove old code only after a check shows no history, checkpoint or active run needs it.

Traffic rollback is fast. State rollback may be impossible once new code has written new shapes, so keep migrations monotonic: new code reads both shapes, only new code writes the new one. Test the failures too: an unknown schema version must quarantine the run, and a missing checkpoint table must stop startup.

References

• Temporal Worker Versioning
• Temporal safe deployments
• Microsoft Durable Functions versioning
• LangGraph persistence
• AWS Step Functions: state machines, read consistency: updates are eventually consistent and new executions may briefly use the previous definition.

---