Sub-second triage: tiering LLMs for latency-critical paths

Some checks can take their time. If a model is summarizing a document or drafting a reply, a second or two of latency is invisible: the user is reading, thinking, waiting anyway. But some checks sit on a hot path where every message has to pass through them first, and where being slow is itself a failure. A safety gate that decides “is this the dangerous case?” on every single turn is the canonical example: it runs before anything else happens, it runs constantly, and if it’s slow it makes everything slow.

The instinct is to reach for your best model and let it judge. It will judge well. It will also cost you a few hundred milliseconds to a couple of seconds per call, on every turn, forever. And it bills you for a large model’s tokens to answer what is usually a trivial question. This post is about the shape that gets you out of that trade: a cascade of differently-sized models, a resilience layer so the cascade fails safe instead of failing open, and a fallback that’s deliberately biased toward caution.

How much latency can an LLM safety check afford?

Start by being honest about the budget. A check on the hot path isn’t allowed to “usually” be fast. It has to be fast at the tail, because the tail is what users actually feel. This is the point Dean and Barroso make in The Tail at Scale: when a request fans out across components, the slow ones dominate the experience, so you have to design for p99, not the mean. If your gate’s median is 200 ms but its p99 is three seconds, a meaningful slice of every conversation stalls for three seconds before anything else can happen.

So the budget is a distribution, not a number. “Under 500 ms” means under 500 ms for nearly all requests, with a known, bounded tail. That’s exactly the framing the Google SRE book’s chapter on SLOs pushes you toward: pick the percentile that reflects what users care about, set a target, and measure against it.

I finally stopped drawing that curve from a synthetic harness and measured it: 800 real calls to the OpenAI API from my machine in Brazil, through the same Mastra agent stack and structured-output path the clinic assistant uses, one patient message per call, on the current models (gpt-5.4-nano for the fast tier, gpt-5.4-mini for the large one). API latency drifts with provider load and your region, so treat the absolute numbers as a snapshot. The shape is the point, and it was not the shape I expected.

I had assumed the fast tier would sit well to the left of the large one, clearing the budget while the big model lagged. It doesn’t. The two curves are almost on top of each other: gpt-5.4-nano runs a 1.1-second median, gpt-5.4-mini runs 1.6, and their distributions overlap through most of the range. The small model is barely quicker, and the only real separation is out in the tail (p99 of 3.0 s against 6.0 s). The cascade, the third curve, is actually the slowest of the three through the middle of the distribution, because an escalated turn runs the detector and then the reasoning model in sequence. Tiering did almost nothing for latency.

That surprised me because it used to be the whole point. A generation ago the small model was not just faster, it could be dramatically slower in the wrong hands: the GPT-5.0 reasoning models would spend five seconds “thinking” before answering a yes/no, so a mis-set nano was the slowest thing on the hot path. On the current gpt-5.4 line that trap is gone. I swept the reasoning-effort dial across its whole range and the median moved about 10% on the nano and 50% on the mini. The knob that used to blow a latency budget barely registers now.

So if the two tiers run at the same speed, why split them at all? The honest answer is that the reasons are not on this chart. They are cost, and what the cheap model gets wrong. (And a note on the word in the title: nobody here clears 500 ms, and the fast tier does not clear even one second at the median. With a 700-token prompt and structured JSON output from Brazil, sub-second is a target you engineer toward, not one these models hand you.)

Right-sizing the model per job

The move is to stop asking one model to do two jobs. The hot-path check is mostly a classification: is this the dangerous case, yes or no? That’s a job a small model does well, and since it costs a fraction of the large model while running at the same speed, the small tier is close to free on the common path. The harder job (reasoning about an ambiguous case, picking an action, weighing severity) goes to a larger model, but only when it’s actually needed.

This is the LLM cascade idea from the literature. FrugalGPT (Chen, Zaharia, Zou) shows you can match a top model’s quality at a fraction of the cost by sending each query to the cheapest model that can handle it and escalating only when necessary; RouteLLM frames the same thing as a learned router between a weak and a strong model. You don’t need a learned router to start: a confidence threshold gets you most of the way.

The fast tier is a stateless detector: a tiny model, a tight prompt, no conversational memory or semantic recall, because recall is latency you can’t afford on every turn. In a Mastra agent that looks like this (note the pruned memory config, all in service of speed):

export const fastDetector = new Agent({
  name: 'fastDetector',
  model: openai('gpt-5.4-nano'), // small and fast for a yes/no; see the latency note below
  // Minimal memory: recent turns only, no semantic recall (recall is
  // latency we can't pay on every message).
  memory: new Memory({
    storage: memoryStorage,
    options: {
      lastMessages: 5,
      semanticRecall: false,   // disabled for speed
      workingMemory: { enabled: false },
    },
  }),
  instructions: `
You are a detection agent. Your only job is to decide whether a
message describes the critical case. Return JSON only.
Be conservative: when in doubt, flag it.
Respond as fast as possible.`,
});

The reasoning tier is a different agent on a larger model, and it runs behind the detector: the cheap check happens on every turn, the expensive reasoning only for the cases that earn it. That is where the cost bends:

At current pricing, the fast tier is about four times cheaper per thousand requests than running the large model on every turn: $0.21 against $0.88. The cascade lands in the middle at $0.58, because on my emergency-heavy test set roughly 40% of messages escalated. Real clinic traffic is mostly routine scheduling, so its escalation rate would be lower and the cascade would sit closer to the fast tier. Cost is the cascade’s clearest win, not latency: an escalated turn pays both tiers in sequence, so the cascade only beats a large-model-every-turn design on the majority of turns that never escalate.

What the cheap tier misses

Cost and latency are the easy things to measure. For a safety gate the number that matters is what the fast tier gets wrong. Across the emergency messages in my set, gpt-5.4-nano caught 93% of them. Good, but a gate that waves through one dangerous case in fourteen is not good enough, and the misses were the tell. It flagged every message with an obvious keyword (dor no peito, desmaiou, convulsão) and missed the two emergencies with no keyword at all: “tomei um remédio e agora estou com o corpo todo inchado” (anaphylaxis) and “dor de barriga muito forte do lado direito, não aguento ficar em pé” (a possible appendicitis). Catching those takes reasoning, not keyword-matching, which is exactly the job I moved off the hot path.

This is why the cascade is a safety mechanism and not only a cost one. On the misses, the detector was not confident: it returned isEmergency: false with a confidence around 0.35 to 0.6, under the escalation threshold. So the cascade sends those uncertain cases up to the reasoning tier, which catches them, and measured recall climbs from 93% to 100%. It only works because the fast tier is honestly unsure when it is wrong. A model that is confidently wrong defeats it: an earlier candidate, gpt-4.1-nano, returned false at 0.8 confidence on that same anaphylaxis message, sailed past the threshold, and pulled cascade recall down to 94%. “Wrong but unsure” turns out to be a property worth testing a detector for.

Failing safe

A cascade adds moving parts, and moving parts fail. Your fast tier’s API has a bad minute; rate limits kick in; a deploy on the provider’s side adds a second of latency to every call. If your gate is synchronous on the hot path, a flaky upstream doesn’t just slow you down, it can take the whole conversation flow with it. Two patterns keep that contained.

The first is the circuit breaker, popularized by Michael Nygard in Release It! and written up in Martin Fowler’s CircuitBreaker article. The idea: wrap the risky call, count failures, and once they cross a threshold stop calling for a cooldown: fail fast instead of piling requests onto a service that’s already drowning. After the cooldown it lets a trial request through to see if the service recovered.

async execute<T>(fn: () => Promise<T>): Promise<T> {
  if (this.state === CircuitState.OPEN) {
    if (this.nextAttemptTime && new Date() < this.nextAttemptTime) {
      throw new Error(`Circuit breaker [${this.name}] is OPEN`);
    }
    this.state = CircuitState.HALF_OPEN; // time to test the service again
  }

  try {
    const result = await fn();
    if (this.state === CircuitState.HALF_OPEN) this.reset(); // recovered
    return result;
  } catch (error) {
    this.recordFailure();
    if (this.state === CircuitState.HALF_OPEN) this.trip(); // still broken
    throw error;
  }
}

The second is retry with exponential backoff for transient failures (a dropped connection, a 429, a 5xx) so a single blip doesn’t surface as a hard error. The two compose: retry handles the one-off hiccup, the breaker handles the sustained outage, and the breaker only counts a failure once retries are exhausted.

const delay = Math.min(
  opts.initialDelayMs * Math.pow(opts.exponentialBase, attempt - 1),
  opts.maxDelayMs,
);
await new Promise((resolve) => setTimeout(resolve, delay));

But resilience plumbing only decides whether you get an answer. The more important question is what you return when you don’t. Here the safety framing changes the default. For a gate whose whole purpose is catching the dangerous case, the two ways to be wrong are not equal. A false positive (flagging a benign message as dangerous) is cheap: at worst it routes someone to a more careful path. A false negative (waving the dangerous case through) is the failure you built the gate to prevent. So the fallback is deliberately biased: when detection fails, assume the dangerous case.

structuredOutput: {
  schema: detectionSchema,
  errorStrategy: 'fallback',
  fallbackValue: {
    isCritical: true,        // conservative: assume the dangerous case on error
    confidence: 0.5,
    reasoning: 'Detection failed; escalating out of caution.',
  },
},

That one field encodes the whole approach. Fast on the common path, cheap on the common path, and when anything goes wrong it errs toward the expensive-but-safe outcome rather than the cheap-but-dangerous one.

Measuring the trade

None of this is worth much if you can’t see it. A cascade is a bet: that the fast tier is accurate enough, and escalates rarely enough, to be worth the extra moving parts. My 40% escalation rate came from a test set stuffed with emergencies; on live traffic I need to watch the real number, because if it drifts up I am quietly paying for the large model on more and more turns. The only way to know is per-tier instrumentation: latency, token usage, and cost, tagged by which model handled the request.

The cheap version is to record it on every call. Capture a start time, measure the duration, and emit the model, the token counts, and a computed cost:

await trackAgentMetrics({
  agentName: 'fastDetector',
  model: 'gpt-5.4-nano',
  promptTokens,
  completionTokens,
  totalTokens,
  responseTimeMs: durationMs,
  costUsd: calculateLLMCost('gpt-5.4-nano', promptTokens, completionTokens),
  success: true,
});

For something you can graph and alert on, expose the same numbers as Prometheus metrics (latency as a Histogram so you can compute real p95/p99 quantiles rather than averages, tokens and cost as counters, all labelled by tier) and put a Grafana dashboard on top. In Node that’s a few lines with prom-client. Once it’s wired up, the questions that decide whether the cascade is healthy become observable: What’s the p99 of the fast tier? What fraction of requests escalate? Is the escalation rate creeping up, meaning the fast tier is losing confidence and you’re slowly paying for the large model on more traffic? Those are the numbers that tell you when to retune the threshold, swap a model tier, or tighten a prompt.

Where I used this

I built exactly this shape into a real-time messaging assistant of my own: a tiny stateless model on the hot path to catch the one case that had to be caught immediately, a larger model behind it for the cases that needed real reasoning, wrapped in a circuit breaker and a retry layer, with a fallback hardwired to escalate when anything failed. The domain specifics don’t matter; the structure is the point, and it generalizes to any check that has to be fast, cheap, and safe at the same time.

The durable lessons are the ones that travel across implementations. Treat the latency budget as a distribution and design for the tail. Measure before you assume the tiering buys you speed: on current models the cheap and expensive tiers can run neck and neck, so you are really tiering for cost and for safety, not milliseconds. Escalate only when the cheap tier is unsure, and remember that a cascade rescues the dangerous case only when the fast tier is honestly uncertain about it. Wrap the hot path in a breaker so a flaky upstream fails fast instead of cascading. Bias the fallback toward the safe error. And measure every tier, because a cascade is only as good as your ability to see when it stops paying off.

References and further reading

• FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. Chen, Zaharia & Zou; the LLM-cascade idea (cheap model first, escalate when needed)
• RouteLLM: Learning to Route LLMs with Preference Data. Routing between a weak and a strong model with a learned router
• The Tail at Scale. Dean & Barroso (CACM, 2013); why you design for p99, not the mean
• Google SRE book: Service Level Objectives. Choosing latency percentiles and targets
• CircuitBreaker. Martin Fowler; the pattern Michael Nygard popularized in Release It!
• Prometheus: histograms and summaries. Measuring latency quantiles · Grafana panels & visualizations · prom-client (Node.js)
• Mastra. The TypeScript agent framework used in the code snippets
• Artificial Analysis: GPT-5 nano vs GPT-4.1 nano. Time-to-first-token and reasoning behaviour; why a reasoning model can be the slow one
• OpenAI API pricing. Current per-token rates used for the cost figures (a July 2026 snapshot; latency numbers are from the same period and will drift)