AI Agent Concurrency Control: Prevent Lost Updates
Lost updates in an AI workflow come from time. A model call takes seconds, a database write takes milliseconds, and during those seconds a person, a webhook or another agent changes the same record. This post covers conditional writes and conflict policies, then three points that are easy to miss: a lease does not protect you from a slow model call, some work is better serialized than versioned, and a retry on conflict costs another model call.
Version the write, not the read
The failure is ordinary. An agent reads version 12, a manager approves the case, and the agent saves a proposal based on version 12. Nothing crashes, and the approval is gone. The fix is a conditional write at the boundary that owns the record. Every state-changing command carries a stable command ID, the expected version, a narrow patch or domain command (not a whole stale object), and the actor and evidence for the audit trail. The store applies it only if the current version equals the expected one. DynamoDB's optimistic locking works this way, and for HTTP services If-Match carries the entity tag the client read. A separate "check the version" query followed by an unconditional update still has a race, so put the comparison in the update predicate.
Keep the model call outside any transaction. Read the minimum facts and the version, call the model, validate its proposed command, recheck authorization in application code, then attempt the conditional write with the original version and record either the commit or the conflict. An idempotency key answers whether you already applied this command. The version answers whether the state behind the decision has changed. Production workflows need both.
A lease is not enough when the call is slow
Teams often prevent concurrent runs with a lock or lease and assume that settles it. It does not, because the holder can be slow. Martin Kleppmann's essay on distributed locking gives the case: a client takes the lease with token 33, pauses, the lease expires, a second client gets token 34 and writes, and then the first client wakes and writes anyway. His fix is a fencing token that the storage layer checks, rejecting any write whose token is lower than one it already accepted. An LLM call is a long pause by design: a rate-limit retry or a slow response can outlast any lease you set.
A durable job queue in Postgres can apply this. Workers claim jobs with SELECT ... FOR UPDATE SKIP LOCKED, each claim takes a lease (say 60 seconds) and increments a fencing token, and completing or failing a job is a compare-and-set on the lease owner and the token. A worker whose lease expired and whose job was reclaimed then gets a "lost lease" result when it tries to commit and cannot overwrite the new owner. Read the PostgreSQL documentation before you rely on SKIP LOCKED for more than the claim. It says skipping locked rows provides an inconsistent view of the data, so it is "not suitable for general purpose work" and fits queue-like tables with multiple consumers (PostgreSQL SELECT). Keep the claim query that narrow. Give the lease a heartbeat if runs are long, and make the final write fenced anyway.
Serialize conversations, version records
Optimistic versioning fits shared records that are rarely contended. A chat thread is different: messages arrive in bursts, and two simultaneous turns on one thread would both answer the same message. Let one turn run per thread at a time, taken by a compare-and-set on the session status. A message that arrives while a turn is running waits for the claim. If that turn already fetched the thread after the message landed, drop the new message as already answered, because a second turn would only repeat the first. If the wait runs out, let a background poller pick the message up when the running turn releases.
Stale claims need the same care. A turn whose process died would hold the thread forever, so a conditional update should free any claim older than a set window, say ten minutes. If turns take 5 to 60 seconds, a live turn is never reclaimed, and the reclaim is a single UPDATE ... WHERE status = 'running' AND updated_at < cutoff, never a read followed by a write. A deploy that stops a turn midway would otherwise drop the next message without a word, which is the failure the poller exists to prevent. Pick the reclaim window from the longest real run, not the average.
A conflict costs a model call
On a version conflict, never retry the same stale write. Load the current record and choose one of four outcomes. Retry from current facts when the changed field affects the decision, with a cap on attempts and elapsed time so a hot record does not burn model calls on proposals that keep going stale. Merge only under a deterministic domain rule, such as appending items with unique identities. Reject when the new state makes the command pointless, for example a reply drafted for a case that a manager already closed. Escalate when both intentions are valid and cannot be combined, such as an operator changing a refund amount while an agent prepares payment. Last writer wins treats timing as authority, which is wrong for approvals, revocations, assignments and money.
Approvals race with proposals
Bind an approval to the proposal hash or record version the reviewer saw. If the agent changes the recipient, amount or target afterwards, require a new approval. Let revocations and suspensions fail closed: a stale approved action must not commit after authority was withdrawn. Our guide on binding human approval to exact agent actions covers the mechanics. Workflow checkpoints are shared state too. Key them with a version or lease generation, and test concurrent writes, because one LangGraph issue reports silent data loss from concurrent checkpoint writes. That is one practitioner's report, not a verdict on the framework. For parallel graph branches, LangGraph documents an error for concurrent writes to one state key without a reducer, and Temporal documents that update and signal handlers can run concurrently.
Version checks cover one aggregate. If a decision depends on several rows or a predicate such as "no active job exists for this account", use a unique constraint, a short row lock, or a serializable transaction, and be ready to retry the whole transaction, as the PostgreSQL isolation documentation describes. Keep those transactions short and free of model calls.
Test the order, not the load
A load test may never produce the ordering that matters. Use barriers. Start the agent and an operator from version 20, pause both after they prepare commands, commit the operator's change, then let the agent commit with version 20 and assert it gets a conflict and that no field from version 21 disappears. Use real threads on separate database connections, since a row lock only means something between transactions that really commit. The same barrier test works for a per-account limit: two requests race, the first is held between its limit check and its insert, and the second must wait, recount and be refused. Then cover an approval racing a proposal change, two parallel tool results, a revocation racing a write, duplicate deliveries, and a hot record that forces several conflicts in a row and must stop at its budget.
References
- Martin Kleppmann: How to do distributed locking
- AWS DynamoDB optimistic locking
- HTTP Semantics: If-Match
- PostgreSQL transaction isolation
- LangGraph: INVALID_CONCURRENT_GRAPH_UPDATE
- LangGraph issue 8115
- Temporal message passing
- PostgreSQL: SELECT, the locking clause: skipping locked rows gives an inconsistent view and is not suitable for general purpose work.