Second production zombie slipped past v0.21.1's watchdog: text+telemetry RX
dead for >1 h while TX worked and the watchdog never fired. Root cause proven
at runtime via /proc/net/snmp — InSegs delta = 2 segments per 30 s (OutSegs 0):
the bridge peer keeps dribbling small NON-frame bytes (~every 15 s, keepalive
noise or a desynced stream) and v0.21.1 updated lastPacketAt on EVERY socket
chunk, so 'no traffic for MESH_IDLE_TIMEOUT_MS' was never true even though zero
valid Meshtastic frames decoded.
- The 'data' handler no longer updates lastPacketAt per chunk; it is updated
only when extractFrames yields >=1 valid packet (the reset in the 'connect'
handler is unchanged), so a stream that decodes nothing trips the watchdog
after MESH_IDLE_TIMEOUT_MS even while raw bytes keep arriving.
- Buffer-overflow backstop: if the undecoded frame buffer exceeds
MESH_MAX_FRAME_BUFFER (new env, default 256000) — e.g. one data event of a
desynced stream that never forms frames — log ERROR and destroy the socket;
the existing close -> scheduleReconnect loop self-heals it. The undecoded
buffer is also reset on every (re)connect so leftover garbage from a dead
session can't re-trigger the backstop or corrupt the first frames of a
healthy new stream.
Tests: 169 -> 171 — raw non-frame bytes do NOT keep the watchdog alive
(garbage-only stream forces disconnect + reconnect, today's incident shape),
and an undecoded chunk over MESH_MAX_FRAME_BUFFER triggers destroy + a fresh
session that decodes normally (cap env-overridden to 100 since Node delivers
socket data in ~64 KB events, far below the 256_000 default).
Production incident: the bridge TCP connection went zombie — outbound sends
kept working for ~10 h while inbound packets stopped entirely, and
/api/status stayed connected:true because nothing detects inbound silence on
an established connection.
- Track lastPacketAt: updated on every chunk received from the bridge in the
lowest-level socket 'data' path (any frame kind, decodable or not) and reset
when a connection successfully starts, so a fresh connection can't
self-trigger.
- Idle watchdog: a 30 s interval checks connected AND now - lastPacketAt >
MESH_IDLE_TIMEOUT_MS (new env, default 600000 = 10 min) — logs WARN with the
idle duration and destroys the socket so the EXISTING connect/retry loop
re-establishes it with a fresh sync. No changes to connect-error retry or
backoff; this only adds detection for established-but-silent connections.
- TCP keepalive: socket.setKeepAlive(true, 30_000) on the bridge socket —
cheap insurance against dead peers at the TCP level.
Tests (fake-bridge based): silence past an overridden timeout forces a
disconnect + reconnect; inbound traffic keeps resetting the idle clock so no
false trigger; the default timeout does not fire in a test window; and
setKeepAlive(true, 30_000) is asserted on the socket. checkIdle() is public so
tests drive a tick without waiting for the real interval (like flushHistory()).
battery_level was only populated during the initial sync; NodeInfo
frames streamed as NODEINFO_APP (port 4) now refresh deviceMetrics in
the node database so /api/nodes and downstream consumers see current
values between syncs. decodeNodeInfo + PORT_NODEINFO_APP added to the
codec, one branch in the existing packet-handling path.
Send want_config_id on connect to trigger initial sync (#6)
Firmware only streams myInfo/channels/configComplete after receiving a
ToRadio with want_config_id — the service previously wrote nothing on
connect, so /api/channels stayed [] and myNodeNum null. v0.1.4.
kamilkrzywda