Bridge idle watchdog + forced reconnect (v0.21.1) #44

2026-09-26T08:53:00+02:00

kamilkrzywda commented

2026-09-26 08:53:00 +02:00

Production incident: bridge TCP connection went zombie — outbound sends worked for ~10 h while inbound packets stopped entirely; /api/status stayed connected:true because nothing detects inbound silence.

lastPacketAt updated on every inbound socket chunk (lowest-level receive path, any frame kind) + reset on successful connect
Idle watchdog: 30 s interval; if connected and silent for MESH_IDLE_TIMEOUT_MS (new env, default 600000 = 10 min) → WARN log + socket.destroy() → existing close/reconnect loop re-establishes with fresh sync
TCP keepalive on the bridge socket: setKeepAlive(true, 30_000)
No changes to connect-error retry/backoff

Tests 165 → 169 (forced reconnect on silence, no false trigger on inbound drip, default-timeout non-firing, keepalive asserted). Build clean, oxlint clean. Lockfile untouched.

Production incident: bridge TCP connection went zombie — outbound sends worked for ~10 h while inbound packets stopped entirely; /api/status stayed connected:true because nothing detects inbound silence. - `lastPacketAt` updated on every inbound socket chunk (lowest-level receive path, any frame kind) + reset on successful connect - Idle watchdog: 30 s interval; if connected and silent for MESH_IDLE_TIMEOUT_MS (new env, default 600000 = 10 min) → WARN log + socket.destroy() → existing close/reconnect loop re-establishes with fresh sync - TCP keepalive on the bridge socket: setKeepAlive(true, 30_000) - No changes to connect-error retry/backoff Tests 165 → 169 (forced reconnect on silence, no false trigger on inbound drip, default-timeout non-firing, keepalive asserted). Build clean, oxlint clean. Lockfile untouched.

kamilkrzywda added 1 commit 2026-09-26 08:53:00 +02:00

Force reconnect when the bridge connection goes silent (v0.21.1)

CI / build + unit (pull_request) Successful in 1m8s

Details

af1c77fd54

Production incident: the bridge TCP connection went zombie — outbound sends
kept working for ~10 h while inbound packets stopped entirely, and
/api/status stayed connected:true because nothing detects inbound silence on
an established connection.

- Track lastPacketAt: updated on every chunk received from the bridge in the
  lowest-level socket 'data' path (any frame kind, decodable or not) and reset
  when a connection successfully starts, so a fresh connection can't
  self-trigger.
- Idle watchdog: a 30 s interval checks connected AND now - lastPacketAt >
  MESH_IDLE_TIMEOUT_MS (new env, default 600000 = 10 min) — logs WARN with the
  idle duration and destroys the socket so the EXISTING connect/retry loop
  re-establishes it with a fresh sync. No changes to connect-error retry or
  backoff; this only adds detection for established-but-silent connections.
- TCP keepalive: socket.setKeepAlive(true, 30_000) on the bridge socket —
  cheap insurance against dead peers at the TCP level.

Tests (fake-bridge based): silence past an overridden timeout forces a
disconnect + reconnect; inbound traffic keeps resetting the idle clock so no
false trigger; the default timeout does not fire in a test window; and
setKeepAlive(true, 30_000) is asserted on the socket. checkIdle() is public so
tests drive a tick without waiting for the real interval (like flushHistory()).

kamilkrzywda merged commit 306756c63b into main

2026-09-26 08:53:06 +02:00

kamilkrzywda deleted branch fix/bridge-idle-reconnect

2026-09-26 08:53:07 +02:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: OpenCode/MeshBot#44