The floor was the call

A WhatsApp order bot. Every message a shop sends, "maggi 500 ke 4", "2 aur atta", "bas itna hi, confirm", goes through one LLM call that returns the intent and the cart. One production log said the whole reply took 6.4 seconds, and 5.2 of them were that call.

The obvious fix that wasn't

The prompt asked for the whole cart back on every message, every line with the buyer's original words. A five-line cart got re-typed in full just to say "confirm". So the plan was obvious: ask only for the changes.

{"intent": "ORDER", "ops": [
  {"op": "set", "line": 1, "item": "atta 10kg", "qty": 2},
  {"op": "remove", "line": 3}
]}

I benchmarked it before shipping, with real API calls. The old prompt answered "confirm" in about 2.9 seconds. The delta prompt got that answer down to 21 output tokens, and it still took about two.

answering "confirm", same modelwhole cart back      ████████████████████████████  ~2.9s
changes only (21 tok) ███████████████████           ~2.0s
                      └── the floor: what a near-empty answer still costs

The faster model I didn't ship

Same delta prompt, smaller models, same 13 cases: Hinglish, Devanagari, "maggi 500 ke 4" where 500 is the size and 4 the quantity, removals, "2 more", "same as last time", an order with a question in it, order status, view, clear.

per message, delta prompt, 13 casesthe model we had   2.2–3.2s   13/13
"mini" tier        0.9–1.6s   13/13  costs more per call
"nano" tier        0.9–1.2s   wrong  read "maggi 6 kar do" as a new item "maggi 6"

Mini looked like the answer. Then I checked the price: per call it cost more than the model we already had. Ours was newer, cheaper and smarter. It just had a higher floor.

Paying more on every message to save under a second is a bad trade for a bot that runs on every message every shop sends. So I kept the model and cut everything sitting on top of the floor.

Cut what sits on top of the floor

Let the prompt prefix cache. The classifier's instructions are identical on every call; only the cart and the message change. Fixed part first, variable part last, and a stable cache key on the request, so the provider can reuse that prefix instead of reading it fresh each time.

Stop sending context the message doesn't need. The prompt included the buyer's last order so "same as last time" could work. While I was testing the mini model, it kept using it: "tata salt 5, and what's the oil rate?" came back with the last order's oil line edited into the cart.

The fix wasn't a stronger instruction. It was to stop sending the last order unless the message mentions it:

const MENTIONS_LAST_ORDER = /\b(last|same|repeat|previous|again|pichl\w*|wahi|pehle|dobara)\b/i;

const input = [
  `CURRENT_CART: ${JSON.stringify(cart)}`,
  MENTIONS_LAST_ORDER.test(text) ? `LAST_ORDER: ${JSON.stringify(lastOrder)}` : null,
  `MESSAGE: ${text}`,
].filter(Boolean).join('\n');

I kept that on the bigger model too. Less input is cheaper, and it gives any model less to misuse.

Keep the short answers. The delta format stayed. It didn't move the floor, but it's fewer tokens out, and it made the next part possible.

Let code apply the changes

The model returns operations; plain code applies them. A set on a line that doesn't exist becomes an add. A remove of one is ignored. Removes are applied last, so the line numbers the model saw stay valid while the others land. Everything downstream still gets a whole cart, so nothing else had to change.

Same model, a second or so faster

On the same 13 cases: all correct, ~1.2–2.4 seconds per message against ~3 before. Same model, no extra cost per call. Bench numbers again, not production percentiles.

What I'd take from it:

• Time an LLM call with a trivial answer before you optimise its output. If that's already slow, the floor is the call itself.
• A faster model is a cost decision, not just a speed one. Price it per call, times every message, before you switch.
• Smaller models follow context more literally. Every extra block in the prompt is something they might use. Send it only when the message needs it, whatever model you're on.
• Let the model describe changes and let code apply them. Fewer tokens out, and code can check line numbers.