Reduce turn latency: cap recent events in prompt and nudge concise model reasoning #22

New Issue

2026-10-11T22:54:58+02:00

kamilkrzywda commented

2026-10-11 22:54:58 +02:00

Problem

Production LLM turns are slow — recent [llm] chat ok log lines show turns taking 39s, 39s, 170s, 223s (all finishReason: stop, so not 502s, just slow). The driver is the reasoning model (qwen3.8-27b) producing a long chain-of-thought in the reasoning field; generation time scales with CoT length.

reasoning_effort is already at low (the floor that still keeps thinking enabled) and LLM_MAX_TOKENS is 4000 (the model finishes within budget, so lowering it risks re-introducing the 502 budget-exhaustion issue). The controllable levers are prompt size and CoT length.

Change

Cap recentEvents in the turn prompt from the last 10 to the last 5 events (buildUserPrompt, server/engine.ts) — shrinks the input and the material the model has to reason over.
Add a "keep your reasoning brief" HARD CONSTRAINT to the system prompt (buildSystemPrompt) — nudges the reasoning model toward a shorter CoT before emitting the JSON. Reasoning stays enabled (not disabled); only its length is limited.

Both are prompt-level only: the deterministic stub, the TurnResult contract, max_tokens, and reasoning_effort are unchanged.

Expected effect

Lower average/tail turn latency and reduced risk of the 4000-token output budget being exhausted before the JSON answer (the original 502 failure mode).

Note

Follow-up identified from production logs (no prior issue). Version bump v0.8.0 → v0.9.0.

## Problem Production LLM turns are slow — recent `[llm] chat ok` log lines show turns taking **39s, 39s, 170s, 223s** (all `finishReason: stop`, so not 502s, just slow). The driver is the reasoning model (`qwen3.8-27b`) producing a long chain-of-thought in the `reasoning` field; generation time scales with CoT length. `reasoning_effort` is already at `low` (the floor that still keeps thinking enabled) and `LLM_MAX_TOKENS` is 4000 (the model finishes within budget, so lowering it risks re-introducing the 502 budget-exhaustion issue). The controllable levers are prompt size and CoT length. ## Change 1. **Cap `recentEvents` in the turn prompt** from the last **10** to the last **5** events (`buildUserPrompt`, `server/engine.ts`) — shrinks the input and the material the model has to reason over. 2. **Add a "keep your reasoning brief" HARD CONSTRAINT** to the system prompt (`buildSystemPrompt`) — nudges the reasoning model toward a shorter CoT before emitting the JSON. Reasoning stays enabled (not disabled); only its length is limited. Both are prompt-level only: the deterministic stub, the `TurnResult` contract, `max_tokens`, and `reasoning_effort` are unchanged. ## Expected effect Lower average/tail turn latency and reduced risk of the 4000-token output budget being exhausted before the JSON answer (the original 502 failure mode). ## Note Follow-up identified from production logs (no prior issue). Version bump v0.8.0 → v0.9.0.

kamilkrzywda referenced this issue from a commit

2026-10-11 23:01:28 +02:00

Reduce turn latency: cap recent events (10→5) + concise-reasoning constraint (v0.9.0)

kamilkrzywda referenced a pull request that will close this issue

2026-10-11 23:01:44 +02:00

Reduce turn latency: cap recent events (10→5) + concise-reasoning constraint (v0.9.0) #23

kamilkrzywda closed this issue

2026-10-11 23:03:53 +02:00

Sign in to join this conversation.