OpenCode/MeshBot
2
0
Fork 0
Code Issues Pull Requests Actions Activity

Watchdog tracks decoded frames, not raw bytes; buffer-overflow backstop (v0.21.2)
CI / build + unit (pull_request) Failing after 39s
Details

Browse Source 
Second production zombie slipped past v0.21.1's watchdog: text+telemetry RX
dead for >1 h while TX worked and the watchdog never fired. Root cause proven
at runtime via /proc/net/snmp — InSegs delta = 2 segments per 30 s (OutSegs 0):
the bridge peer keeps dribbling small NON-frame bytes (~every 15 s, keepalive
noise or a desynced stream) and v0.21.1 updated lastPacketAt on EVERY socket
chunk, so 'no traffic for MESH_IDLE_TIMEOUT_MS' was never true even though zero
valid Meshtastic frames decoded.

- The 'data' handler no longer updates lastPacketAt per chunk; it is updated
  only when extractFrames yields >=1 valid packet (the reset in the 'connect'
  handler is unchanged), so a stream that decodes nothing trips the watchdog
  after MESH_IDLE_TIMEOUT_MS even while raw bytes keep arriving.
- Buffer-overflow backstop: if the undecoded frame buffer exceeds
  MESH_MAX_FRAME_BUFFER (new env, default 256000) — e.g. one data event of a
  desynced stream that never forms frames — log ERROR and destroy the socket;
  the existing close -> scheduleReconnect loop self-heals it. The undecoded
  buffer is also reset on every (re)connect so leftover garbage from a dead
  session can't re-trigger the backstop or corrupt the first frames of a
  healthy new stream.

Tests: 169 -> 171 — raw non-frame bytes do NOT keep the watchdog alive
(garbage-only stream forces disconnect + reconnect, today's incident shape),
and an undecoded chunk over MESH_MAX_FRAME_BUFFER triggers destroy + a fresh
session that decodes normally (cap env-overridden to 100 since Node delivers
socket data in ~64 KB events, far below the 256_000 default).
This commit is contained in:
kamilkrzywda
2026-09-26 08:50:27 +00:00
parent 306756c63b
commit 33a88f3d83
5 changed files with 100 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+1 -1
View File
@@ -12,7 +12,7 @@
## Verification
1. **Local checks** — `npm run build` (nest build) and `npm test` (vitest; no e2e suite yet). Both must pass before a PR is opened.
2. **Docker** — `docker compose build`, then after `docker compose up -d`: `curl -s -H "Host: bot.dev.kkhost.pl" http://localhost/api/status` returns JSON (`{ connected, myNodeNum, host, port }`). `"connected":false` is normal while the bridge/device is down — the service retries every 5 s. An idle watchdog also force-reconnects an ESTABLISHED-but-silent connection (zombie bridge: outbound works, inbound dead) when no packet arrives from the bridge for `MESH_IDLE_TIMEOUT_MS` ms (default 600000 = 10 min; checked every 30 s), and the socket has TCP keepalive enabled (30 s).
2. **Docker** — `docker compose build`, then after `docker compose up -d`: `curl -s -H "Host: bot.dev.kkhost.pl" http://localhost/api/status` returns JSON (`{ connected, myNodeNum, host, port }`). `"connected":false` is normal while the bridge/device is down — the service retries every 5 s. An idle watchdog also force-reconnects an ESTABLISHED-but-silent connection (zombie bridge: outbound works, inbound dead) when no DECODED FRAME arrives from the bridge for `MESH_IDLE_TIMEOUT_MS` ms (default 600000 = 10 min; checked every 30 s — raw non-frame bytes do NOT count), an undecoded frame buffer over `MESH_MAX_FRAME_BUFFER` bytes (default 256000) forces a reconnect as a stream-desync backstop, and the socket has TCP keepalive enabled (30 s).
## API