reliability: low reasoning_effort, retry-with-backoff, LLM call logging

- send top-level reasoning_effort: "low" (LLM_REASONING_EFFORT) to cut the qwen3.8-27b CoT ~85%, so content is no longer truncated to null -> the ~50% 502s
- new env: LLM_MAX_TOKENS (default 4000, per-call opts.maxTokens still overrides), LLM_REASONING_EFFORT (default "low", off/empty omits the field)
- chat() retries up to 3 attempts: timeout/abort, 5xx, network errors (ECONNRESET/ECONNREFUSED/socket closed), null/empty content; backoff 2s->3s (2000*1.5^(n-1), +/-20% jitter); 4xx throws immediately
- logging: '[llm] chat ok' {ms, finishReason} on success; '[llm] chat failed' {attempts, lastStatus, message, totalMs} on final failure
- tests: 12 in llm.test.ts (retry success/failure, no 4xx retry, null-content retry, logging spies); AGENTS.md env table + gotchas updated
This commit is contained in:
2026-10-11 13:46:18 +00:00
parent 612de2fc40
commit 3297658c81
3 changed files with 395 additions and 63 deletions
+17 -5
View File
@@ -54,7 +54,7 @@ history/
├── server/
│ ├── index.ts # Express: /api/*, /sse/:id, static+SPA; reads PORT/LLM_*/DATA_DIR
│ ├── engine.ts # pure: buildPrompt / parse+validate / applyTurnResult / runTurn / advisor
│ ├── llm.ts # LlmClient (injectable transport) + env config
│ ├── llm.ts # LlmClient (injectable transport, retry+backoff, logging) + env config
│ ├── stub.ts # deterministic LLM transport (tests/e2e) + ADVISOR_MARKER
│ ├── map.ts # GameState → RegionMap (ISO_A3 → owner + stable color)
│ ├── presets.ts # load preset JSON files from disk
@@ -137,19 +137,31 @@ history/
| `LLM_API_BASE` | `https://ai.kkhost.pl/v1` | OpenAI-compatible **vLLM** base URL (no API key required) |
| `LLM_MODEL` | `qwen3.8-27b` | vLLM **reasoning** model — CoT in `reasoning`, final answer in `content` |
| `LLM_TIMEOUT_MS` | `180000` | the 27B reasoning model can be slow |
| `LLM_MAX_TOKENS` | `4000` | max **output** tokens per request; with `reasoning_effort:low` a realistic turn uses ~800–1100, so 4000 is ~4× headroom and stays under the 90s proxy wall |
| `LLM_REASONING_EFFORT` | `low` | sent as top-level `reasoning_effort`; cuts the model's CoT by ~85% (the 502 fix). Set to empty/`off` to not send the field at all |
| `LLM_STUB` | _(unset = live)_ | `1` → deterministic stub transport (no model needed) |
> **Reasoning model:** `qwen3.8-27b` emits its chain-of-thought in the response's `reasoning`
> field and the final JSON answer in `content`. `server/llm.ts` therefore always sends a generous
> `max_tokens` (`DEFAULT_MAX_TOKENS = 4000`) so the model finishes reasoning before writing
> `content` (a too-small budget returns a null `content`), and reads only `content`, ignoring
> `reasoning`. No auth header is sent — the endpoint requires no key.
> field and the final JSON answer in `content`. `server/llm.ts` therefore sends top-level
> `reasoning_effort: "low"` (`LLM_REASONING_EFFORT`) to cut the CoT by ~85% — small enough to
> fit the `max_tokens` budget (`LLM_MAX_TOKENS`, default 4000) — and reads only `content`,
> ignoring `reasoning`. `chat()` retries transient failures (timeout/abort, 5xx, network
> errors, null/empty `content`) up to 3 attempts with 2s→3s backoff and logs every call
> (`[llm] chat ok` / `[llm] chat failed`). No auth header is sent — the endpoint requires no key.
| `PORT` | `4321` | single server port |
| `DATA_DIR` | `./data` | where `history.db` lives |
| `CLIENT_DIST` | `./dist` | built client to serve (auto-found) |
## Gotchas
- **502s were the reasoning model exhausting its output budget.** The original client sent only
`max_tokens: 4000`; `qwen3.8-27b`'s chain-of-thought alone often exceeded 4000 tokens, so
`content` came back `null`, which the client treated as fatal → HTTP 502 (~half of turns),
with no retry and no logging. Fixed by sending `reasoning_effort: "low"` (cuts the CoT ~85%),
retrying transient failures (timeout/abort, 5xx, network errors, null/empty `content`) up to
3 attempts with 2s→3s backoff, and logging every LLM call. A realistic turn now takes
~11–16s; the effective ceiling is the upstream OpenResty ~90s timeout, **not** our 180s
`LLM_TIMEOUT_MS`.
- **Live LLM is a runtime dependency.** The live model is `qwen3.8-27b` on
`https://ai.kkhost.pl/v1` (a vLLM **reasoning** model, no API key). If that endpoint is down,
turns return 502 (and the advisor degrades to a canned list). Set `LLM_STUB=1` for a fully