Deterministic AI Workflow Design: Fallbacks for Probabilistic Systems

An LLM step in a workflow will sometimes return malformed JSON, a wrong label, or a confident answer to the wrong question. The useful design question is what the surrounding code does next. This guide covers the split between the model and deterministic code, how to validate output, why a fallback can be worse than a failure, and how to measure whether the fallback path has quietly become the main path.

Let the model classify and let code decide

The most reliable pattern is to limit the model to a structured judgment and to keep every consequential decision in deterministic code. The tau-bench benchmark shows why. Even function-calling agents such as gpt-4o succeed on fewer than 50 percent of its tasks, and on the retail domain the pass^8 figure, the chance of solving the same task correctly on eight repeated runs, is below 25 percent (Yao et al.). Let the model classify the message. A separate policy layer then chooses between answering, asking a clarifying question, acting, handing off to a human, or declining. Priority ordering, thresholds, ambiguity handling, and the filter on allowed action keys live in that code, so each decision can be unit-tested and inspected. Write the classifier call so that every failure path ends in a safe tools-off answer and never in an exception.

A second layer sits in front of tool use. When candidate actions come from semantic retrieval, add an admission gate that compares each candidate's similarity with a threshold that depends on its risk class, with write actions held to a stricter threshold than reads. Mark two write candidates that score within a margin of each other as ambiguous instead of picking one. When retrieval is down, do not fall back to exposing every tool. Admit only candidates that match exact aliases from trusted tool policy. Treat "no action" as a normal result.

The lesson from both layers: the safe default when the model cannot be trusted is to do less, not to do something similar by other means.

Validate structure, and know what a schema guarantee covers

Never pass raw model text to downstream systems. Parse it and validate it against a schema.

import json
from pydantic import BaseModel, ValidationError

class TicketTriage(BaseModel):
    category: str
    urgency: int  # 1-4
    summary: str

def parse_triage(response_text: str) -> TicketTriage | None:
    try:
        return TicketTriage(**json.loads(response_text))
    except (json.JSONDecodeError, ValidationError) as err:
        record_validation_failure(response_text, err)
        return None

Providers now offer schema-constrained output, which removes most malformed JSON. It does not remove the need for this code. Anthropic's structured outputs documentation says a refusal (stop_reason: "refusal") comes back as a normal 200 response and may not match your schema, and that a max_tokens stop can leave the JSON incomplete. It also lists schema features that are rejected with a 400 error, such as numeric and string-length constraints. So check the stop reason before parsing, and keep range and business-rule checks in your own code. Schema-valid output can still be wrong. A ticket with urgency: 1 parses perfectly and may be an emergency.

Do not route on the model's own confidence without measuring it

A common design asks the model for a confidence number and sends low-confidence cases to a human. Whether that number means anything depends on how you ask. A study of verbalized confidence scores found that their reliability depends strongly on the prompting method, and that some methods give much better-calibrated scores than others. Treat the score as a hypothesis. Take 100 labelled cases, bucket them by reported confidence, and check the actual accuracy in each bucket before choosing a threshold. If the buckets are flat, use a different signal: schema validity, agreement between two independent calls, or a deterministic check against the source data.

Fallbacks that do something different are the dangerous ones

There are two kinds of fallback, and they have opposite risk profiles.

• Fail to a person or a safe stop. The ticket goes to the default human queue, the action is not taken, the user gets an honest "I could not do that." The cost is latency and effort, and the failure is visible.
• Fail to a substitute. A keyword rule classifies the ticket, a smaller model answers, a cached result is returned. The user gets an answer that looks the same, produced by a weaker mechanism, and nobody notices.

Prefer the first kind for anything that writes data, spends money, or tells a customer something. Use the second kind only for low-stakes steps, and label the output so downstream systems and reviewers can see which path produced it. The same applies to timeouts. A cheap classifier that routes a message to a smaller or larger model can treat its own timeout as "complex" and send the message down the heavier path. That substitution is safe for correctness and costs latency, so it is worth counting.

Retry with new information or do not retry

Resending the identical prompt after a validation failure often reproduces the failure, and it doubles your cost and latency. If you retry, change something: include the validation error in the prompt, lower the temperature, or shrink the input. Allow at most one such retry, then take the fail-safe path. For why retries must also fit inside a deadline, see our guide to LLM timeout budgets.

Measure the fallback rate, because a rising rate is the alert

A fallback that works hides the failure behind it. Record every fallback with the input, the raw model output, the validation error, and the path taken. Then track the fallback rate per workflow as a first-class metric. A sudden rise usually means one of three things: the provider changed a model version, your input distribution shifted, or a prompt edit broke the output format. If most requests end up on the fallback path, the primary path is broken and the fallback is only masking it.

Test this directly. Inject malformed output, a refusal, a truncated response, and a timeout into each LLM step, and assert on three things: the workflow reached its fail-safe state, no downstream write happened, and the fallback counter incremented.

Check that the fallback keeps the same permissions

The simpler path is where access checks get forgotten. If the AI path enforces per-user document access, the keyword-search fallback must apply the same filter. An attacker who can make the primary path fail on purpose, for example with oversized or malformed input, is probing exactly this gap. List the transitions from the model path to the fallback path and confirm each one carries the caller's identity and scope.

When not to add a model step at all

If a rules check covers 80 percent of your inputs reliably, put it first and send only the ambiguous remainder to the model. That order lowers cost and latency and shrinks the share of traffic exposed to model failure. If a step has a precise specification and a checkable answer, such as a date format or a status mapping, write it as code. Each model step you remove is a fallback you never need to design.

Next steps

Pick the workflow where a malformed or wrong LLM response does the most damage. Check that structured output is validated before anything downstream sees it, that the failure path stops instead of substituting, and that a fallback counter exists. Then run the four injected failures above against it.

References

1. Yao et al., tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. Reports gpt-4o-class agents succeeding on under 50 percent of tasks with pass^8 below 25 percent in the retail domain.
2. Anthropic, Structured outputs. Documents refusal and max_tokens cases where output may not match the schema, and unsupported schema constraints.
3. Yang et al., On Verbalized Confidence Scores for LLMs. Reports that reliability of verbalized confidence depends strongly on how the model is asked.

---