Watchdog tracks decoded frames, not raw bytes; buffer-overflow backstop (v0.21.2)
CI / build + unit (pull_request) Failing after 39s
CI / build + unit (pull_request) Failing after 39s
Second production zombie slipped past v0.21.1's watchdog: text+telemetry RX dead for >1 h while TX worked and the watchdog never fired. Root cause proven at runtime via /proc/net/snmp — InSegs delta = 2 segments per 30 s (OutSegs 0): the bridge peer keeps dribbling small NON-frame bytes (~every 15 s, keepalive noise or a desynced stream) and v0.21.1 updated lastPacketAt on EVERY socket chunk, so 'no traffic for MESH_IDLE_TIMEOUT_MS' was never true even though zero valid Meshtastic frames decoded. - The 'data' handler no longer updates lastPacketAt per chunk; it is updated only when extractFrames yields >=1 valid packet (the reset in the 'connect' handler is unchanged), so a stream that decodes nothing trips the watchdog after MESH_IDLE_TIMEOUT_MS even while raw bytes keep arriving. - Buffer-overflow backstop: if the undecoded frame buffer exceeds MESH_MAX_FRAME_BUFFER (new env, default 256000) — e.g. one data event of a desynced stream that never forms frames — log ERROR and destroy the socket; the existing close -> scheduleReconnect loop self-heals it. The undecoded buffer is also reset on every (re)connect so leftover garbage from a dead session can't re-trigger the backstop or corrupt the first frames of a healthy new stream. Tests: 169 -> 171 — raw non-frame bytes do NOT keep the watchdog alive (garbage-only stream forces disconnect + reconnect, today's incident shape), and an undecoded chunk over MESH_MAX_FRAME_BUFFER triggers destroy + a fresh session that decodes normally (cap env-overridden to 100 since Node delivers socket data in ~64 KB events, far below the 256_000 default).
This commit is contained in:
@@ -280,6 +280,69 @@ describe('MeshService idle watchdog', () => {
|
||||
}
|
||||
});
|
||||
|
||||
it('raw non-frame bytes do NOT keep the watchdog alive — a garbage-only stream forces a reconnect', async () => {
|
||||
const bridge = await startBridge();
|
||||
process.env.MESH_HOST = '127.0.0.1';
|
||||
process.env.MESH_PORT = String(bridge.port);
|
||||
process.env.MESH_RECONNECT_MS = '50'; // fast retry so the re-establishment is observable
|
||||
process.env.MESH_IDLE_TIMEOUT_MS = '100'; // override — the default (600000) would never fire in a test
|
||||
const svc = new MeshService();
|
||||
try {
|
||||
svc.onModuleInit();
|
||||
await waitFor(
|
||||
() => bridge.sockets.length === 1 && svc.status().connected,
|
||||
);
|
||||
|
||||
// today's incident: the peer keeps dribbling small non-frame bytes (keepalive noise / desynced stream) — raw chunks must not reset the idle clock
|
||||
const garbage = Buffer.from('zz-no-frames-here'); // pure ASCII — never forms a 0x94 0xc3 frame header
|
||||
for (let i = 0; i < 6; i++) {
|
||||
await sleep(50); // ~300 ms wall time, well past the timeout
|
||||
bridge.sockets[0].write(garbage);
|
||||
}
|
||||
|
||||
svc.checkIdle();
|
||||
await waitFor(() => bridge.sockets.length >= 2 && svc.status().connected);
|
||||
} finally {
|
||||
delete process.env.MESH_RECONNECT_MS;
|
||||
delete process.env.MESH_IDLE_TIMEOUT_MS;
|
||||
svc.onModuleDestroy();
|
||||
bridge.close();
|
||||
}
|
||||
});
|
||||
|
||||
it('forces a reconnect when the undecoded frame buffer exceeds MESH_MAX_FRAME_BUFFER', async () => {
|
||||
const bridge = await startBridge();
|
||||
process.env.MESH_HOST = '127.0.0.1';
|
||||
process.env.MESH_PORT = String(bridge.port);
|
||||
process.env.MESH_RECONNECT_MS = '50'; // fast retry so the re-establishment is observable
|
||||
process.env.MESH_MAX_FRAME_BUFFER = '100'; // tiny cap — Node delivers socket data in ~64 KB events, far below the 256_000 default
|
||||
const svc = new MeshService();
|
||||
try {
|
||||
svc.onModuleInit();
|
||||
await waitFor(
|
||||
() => bridge.sockets.length === 1 && svc.status().connected,
|
||||
);
|
||||
|
||||
// non-frame bytes (0x42 is not part of any frame header) — a desynced stream that never forms frames; the first ~64 KB data event alone exceeds the cap
|
||||
bridge.sockets[0].write(Buffer.alloc(300_000, 0x42));
|
||||
|
||||
// overflow forced a reconnect — fresh session
|
||||
await waitFor(() => bridge.sockets.length >= 2 && svc.status().connected);
|
||||
|
||||
// the new socket starts with a clean buffer and decodes valid frames normally (the myInfo frame is well under the cap)
|
||||
const fr = create(P.Mesh.FromRadioSchema, {
|
||||
payloadVariant: { case: 'myInfo', value: { myNodeNum: 777 } },
|
||||
});
|
||||
bridge.sockets[1].write(frame(toBinary(P.Mesh.FromRadioSchema, fr)));
|
||||
await waitFor(() => svc.status().myNodeNum === 777);
|
||||
} finally {
|
||||
delete process.env.MESH_RECONNECT_MS;
|
||||
delete process.env.MESH_MAX_FRAME_BUFFER;
|
||||
svc.onModuleDestroy();
|
||||
bridge.close();
|
||||
}
|
||||
});
|
||||
|
||||
it('inbound traffic keeps resetting the idle clock — no false trigger while packets arrive', async () => {
|
||||
const bridge = await startBridge();
|
||||
process.env.MESH_HOST = '127.0.0.1';
|
||||
|
||||
Reference in New Issue
Block a user