Watchdog: track decoded frames, not raw bytes (v0.21.2) #45

2026-09-26T10:53:22+02:00

kamilkrzywda commented

2026-09-26 10:53:22 +02:00

Second zombie incident today exposed a flaw in v0.21.1's watchdog: it updated lastPacketAt on every inbound socket chunk, but the bridge peer keeps sending small NON-frame bytes (~every 15 s — proven via /proc/net/snmp: InSegs +2/30 s while zero frames decoded). Raw noise kept resetting the clock forever, so "no traffic for MESH_IDLE_TIMEOUT_MS" was never true even though text+telemetry RX had been dead for over an hour.

lastPacketAt now updates only when extractFrames yields ≥1 valid packet (plus unchanged reset on connect) — raw byte streams can no longer keep a zombie connection alive
Buffer-overflow backstop: undecoded buffer beyond MESH_MAX_FRAME_BUFFER (new env, default 256000) → ERROR + socket.destroy() → existing reconnect loop; caps growth from non-frame streams and self-heals desync
Undecoded buffer reset on (re)connect so stale garbage can't corrupt a fresh stream

Tests 169 → 171: garbage-only stream past timeout now forces reconnect (today's incident shape); valid frames still reset the clock; overflow triggers destroy. Build clean, oxlint clean. Lockfile untouched.

Second zombie incident today exposed a flaw in v0.21.1's watchdog: it updated lastPacketAt on every inbound socket chunk, but the bridge peer keeps sending small NON-frame bytes (~every 15 s — proven via /proc/net/snmp: InSegs +2/30 s while zero frames decoded). Raw noise kept resetting the clock forever, so "no traffic for MESH_IDLE_TIMEOUT_MS" was never true even though text+telemetry RX had been dead for over an hour. - lastPacketAt now updates only when extractFrames yields ≥1 valid packet (plus unchanged reset on connect) — raw byte streams can no longer keep a zombie connection alive - Buffer-overflow backstop: undecoded buffer beyond MESH_MAX_FRAME_BUFFER (new env, default 256000) → ERROR + socket.destroy() → existing reconnect loop; caps growth from non-frame streams and self-heals desync - Undecoded buffer reset on (re)connect so stale garbage can't corrupt a fresh stream Tests 169 → 171: garbage-only stream past timeout now forces reconnect (today's incident shape); valid frames still reset the clock; overflow triggers destroy. Build clean, oxlint clean. Lockfile untouched.

kamilkrzywda added 1 commit 2026-09-26 10:53:22 +02:00

Watchdog tracks decoded frames, not raw bytes; buffer-overflow backstop (v0.21.2)

CI / build + unit (pull_request) Failing after 39s

Details

33a88f3d83

Second production zombie slipped past v0.21.1's watchdog: text+telemetry RX
dead for >1 h while TX worked and the watchdog never fired. Root cause proven
at runtime via /proc/net/snmp — InSegs delta = 2 segments per 30 s (OutSegs 0):
the bridge peer keeps dribbling small NON-frame bytes (~every 15 s, keepalive
noise or a desynced stream) and v0.21.1 updated lastPacketAt on EVERY socket
chunk, so 'no traffic for MESH_IDLE_TIMEOUT_MS' was never true even though zero
valid Meshtastic frames decoded.

- The 'data' handler no longer updates lastPacketAt per chunk; it is updated
  only when extractFrames yields >=1 valid packet (the reset in the 'connect'
  handler is unchanged), so a stream that decodes nothing trips the watchdog
  after MESH_IDLE_TIMEOUT_MS even while raw bytes keep arriving.
- Buffer-overflow backstop: if the undecoded frame buffer exceeds
  MESH_MAX_FRAME_BUFFER (new env, default 256000) — e.g. one data event of a
  desynced stream that never forms frames — log ERROR and destroy the socket;
  the existing close -> scheduleReconnect loop self-heals it. The undecoded
  buffer is also reset on every (re)connect so leftover garbage from a dead
  session can't re-trigger the backstop or corrupt the first frames of a
  healthy new stream.

Tests: 169 -> 171 — raw non-frame bytes do NOT keep the watchdog alive
(garbage-only stream forces disconnect + reconnect, today's incident shape),
and an undecoded chunk over MESH_MAX_FRAME_BUFFER triggers destroy + a fresh
session that decodes normally (cap env-overridden to 100 since Node delivers
socket data in ~64 KB events, far below the 256_000 default).

kamilkrzywda merged commit 77459adc13 into main

2026-09-26 10:53:28 +02:00

kamilkrzywda deleted branch fix/watchdog-frame-tracking

2026-09-26 10:53:29 +02:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: OpenCode/MeshBot#45