Watchdog tracks decoded frames, not raw bytes; buffer-overflow backstop (v0.21.2)
CI / build + unit (pull_request) Failing after 39s

Second production zombie slipped past v0.21.1's watchdog: text+telemetry RX
dead for >1 h while TX worked and the watchdog never fired. Root cause proven
at runtime via /proc/net/snmp — InSegs delta = 2 segments per 30 s (OutSegs 0):
the bridge peer keeps dribbling small NON-frame bytes (~every 15 s, keepalive
noise or a desynced stream) and v0.21.1 updated lastPacketAt on EVERY socket
chunk, so 'no traffic for MESH_IDLE_TIMEOUT_MS' was never true even though zero
valid Meshtastic frames decoded.

- The 'data' handler no longer updates lastPacketAt per chunk; it is updated
  only when extractFrames yields >=1 valid packet (the reset in the 'connect'
  handler is unchanged), so a stream that decodes nothing trips the watchdog
  after MESH_IDLE_TIMEOUT_MS even while raw bytes keep arriving.
- Buffer-overflow backstop: if the undecoded frame buffer exceeds
  MESH_MAX_FRAME_BUFFER (new env, default 256000) — e.g. one data event of a
  desynced stream that never forms frames — log ERROR and destroy the socket;
  the existing close -> scheduleReconnect loop self-heals it. The undecoded
  buffer is also reset on every (re)connect so leftover garbage from a dead
  session can't re-trigger the backstop or corrupt the first frames of a
  healthy new stream.

Tests: 169 -> 171 — raw non-frame bytes do NOT keep the watchdog alive
(garbage-only stream forces disconnect + reconnect, today's incident shape),
and an undecoded chunk over MESH_MAX_FRAME_BUFFER triggers destroy + a fresh
session that decodes normally (cap env-overridden to 100 since Node delivers
socket data in ~64 KB events, far below the 256_000 default).
This commit is contained in:
2026-09-26 08:50:27 +00:00
parent 306756c63b
commit 33a88f3d83
5 changed files with 100 additions and 13 deletions
+63
View File
@@ -280,6 +280,69 @@ describe('MeshService idle watchdog', () => {
}
});
it('raw non-frame bytes do NOT keep the watchdog alive — a garbage-only stream forces a reconnect', async () => {
const bridge = await startBridge();
process.env.MESH_HOST = '127.0.0.1';
process.env.MESH_PORT = String(bridge.port);
process.env.MESH_RECONNECT_MS = '50'; // fast retry so the re-establishment is observable
process.env.MESH_IDLE_TIMEOUT_MS = '100'; // override — the default (600000) would never fire in a test
const svc = new MeshService();
try {
svc.onModuleInit();
await waitFor(
() => bridge.sockets.length === 1 && svc.status().connected,
);
// today's incident: the peer keeps dribbling small non-frame bytes (keepalive noise / desynced stream) — raw chunks must not reset the idle clock
const garbage = Buffer.from('zz-no-frames-here'); // pure ASCII — never forms a 0x94 0xc3 frame header
for (let i = 0; i < 6; i++) {
await sleep(50); // ~300 ms wall time, well past the timeout
bridge.sockets[0].write(garbage);
}
svc.checkIdle();
await waitFor(() => bridge.sockets.length >= 2 && svc.status().connected);
} finally {
delete process.env.MESH_RECONNECT_MS;
delete process.env.MESH_IDLE_TIMEOUT_MS;
svc.onModuleDestroy();
bridge.close();
}
});
it('forces a reconnect when the undecoded frame buffer exceeds MESH_MAX_FRAME_BUFFER', async () => {
const bridge = await startBridge();
process.env.MESH_HOST = '127.0.0.1';
process.env.MESH_PORT = String(bridge.port);
process.env.MESH_RECONNECT_MS = '50'; // fast retry so the re-establishment is observable
process.env.MESH_MAX_FRAME_BUFFER = '100'; // tiny cap — Node delivers socket data in ~64 KB events, far below the 256_000 default
const svc = new MeshService();
try {
svc.onModuleInit();
await waitFor(
() => bridge.sockets.length === 1 && svc.status().connected,
);
// non-frame bytes (0x42 is not part of any frame header) — a desynced stream that never forms frames; the first ~64 KB data event alone exceeds the cap
bridge.sockets[0].write(Buffer.alloc(300_000, 0x42));
// overflow forced a reconnect — fresh session
await waitFor(() => bridge.sockets.length >= 2 && svc.status().connected);
// the new socket starts with a clean buffer and decodes valid frames normally (the myInfo frame is well under the cap)
const fr = create(P.Mesh.FromRadioSchema, {
payloadVariant: { case: 'myInfo', value: { myNodeNum: 777 } },
});
bridge.sockets[1].write(frame(toBinary(P.Mesh.FromRadioSchema, fr)));
await waitFor(() => svc.status().myNodeNum === 777);
} finally {
delete process.env.MESH_RECONNECT_MS;
delete process.env.MESH_MAX_FRAME_BUFFER;
svc.onModuleDestroy();
bridge.close();
}
});
it('inbound traffic keeps resetting the idle clock — no false trigger while packets arrive', async () => {
const bridge = await startBridge();
process.env.MESH_HOST = '127.0.0.1';