The v0.21.2 frame-based watchdog works in production (it fired today: 'no
decoded bridge frames for 628474 ms — forcing reconnect', and the reconnect
delivered queued packets), but the bridge/radio side at 192.168.1.4 keeps
killing inbound RX within ~20 min of a fresh sync, repeatedly. Two tuning
changes requested by the owner:
- MESH_IDLE_TIMEOUT_MS default lowered from 600000 (10 min) to 60000 (1 min).
- New lazy periodic autoreconnect: MESH_RECONNECT_INTERVAL_MS (new env,
default 300000 = 5 min, read at construction like the others). An interval
started in onModuleInit unconditionally logs and destroys the socket so the
existing close -> scheduleReconnect loop re-establishes with a fresh sync on
schedule; cleared in onModuleDestroy. When not connected at tick time,
destroy is a no-op and the retry loop owns recovery — no special-casing.
Tests: 171 -> 172 — periodic reconnect observed end-to-end with an overridden
short interval (fake bridge disconnect + re-establish); existing watchdog tests
pass unchanged, comments updated to the new 60 s default.
Second production zombie slipped past v0.21.1's watchdog: text+telemetry RX
dead for >1 h while TX worked and the watchdog never fired. Root cause proven
at runtime via /proc/net/snmp — InSegs delta = 2 segments per 30 s (OutSegs 0):
the bridge peer keeps dribbling small NON-frame bytes (~every 15 s, keepalive
noise or a desynced stream) and v0.21.1 updated lastPacketAt on EVERY socket
chunk, so 'no traffic for MESH_IDLE_TIMEOUT_MS' was never true even though zero
valid Meshtastic frames decoded.
- The 'data' handler no longer updates lastPacketAt per chunk; it is updated
only when extractFrames yields >=1 valid packet (the reset in the 'connect'
handler is unchanged), so a stream that decodes nothing trips the watchdog
after MESH_IDLE_TIMEOUT_MS even while raw bytes keep arriving.
- Buffer-overflow backstop: if the undecoded frame buffer exceeds
MESH_MAX_FRAME_BUFFER (new env, default 256000) — e.g. one data event of a
desynced stream that never forms frames — log ERROR and destroy the socket;
the existing close -> scheduleReconnect loop self-heals it. The undecoded
buffer is also reset on every (re)connect so leftover garbage from a dead
session can't re-trigger the backstop or corrupt the first frames of a
healthy new stream.
Tests: 169 -> 171 — raw non-frame bytes do NOT keep the watchdog alive
(garbage-only stream forces disconnect + reconnect, today's incident shape),
and an undecoded chunk over MESH_MAX_FRAME_BUFFER triggers destroy + a fresh
session that decodes normally (cap env-overridden to 100 since Node delivers
socket data in ~64 KB events, far below the 256_000 default).
Production incident: the bridge TCP connection went zombie — outbound sends
kept working for ~10 h while inbound packets stopped entirely, and
/api/status stayed connected:true because nothing detects inbound silence on
an established connection.
- Track lastPacketAt: updated on every chunk received from the bridge in the
lowest-level socket 'data' path (any frame kind, decodable or not) and reset
when a connection successfully starts, so a fresh connection can't
self-trigger.
- Idle watchdog: a 30 s interval checks connected AND now - lastPacketAt >
MESH_IDLE_TIMEOUT_MS (new env, default 600000 = 10 min) — logs WARN with the
idle duration and destroys the socket so the EXISTING connect/retry loop
re-establishes it with a fresh sync. No changes to connect-error retry or
backoff; this only adds detection for established-but-silent connections.
- TCP keepalive: socket.setKeepAlive(true, 30_000) on the bridge socket —
cheap insurance against dead peers at the TCP level.
Tests (fake-bridge based): silence past an overridden timeout forces a
disconnect + reconnect; inbound traffic keeps resetting the idle clock so no
false trigger; the default timeout does not fire in a test window; and
setKeepAlive(true, 30_000) is asserted on the socket. checkIdle() is public so
tests drive a tick without waiting for the real interval (like flushHistory()).
The single-message summary was too small (owner request). The summarizer
prompt now asks for exactly 3-5 lines, one item per line, in the format
'<short header>: <one or two short sentences>' with a <=20-char header,
no intro/outro/numbering. maxTokens stays 2500 (reasoning-token headroom).
The reply is parsed into items: drop empty lines and lines without a ':'
header separator, cap at 5; fewer than 3 valid lines sends what we got
(no failure/retry), zero valid lines sends nothing. Each item goes out as
its own mesh message via respond(id, line) - all threaded to the trigger
packet id on its channel. Per-item backstop fitOneLine() hard-cuts any
line >185 chars at a word boundary near 180 so every item fits one packet;
the old whole-reply fitOnePacket() cut is gone.
CommandService sends array replies line by line through the same routing
as single replies (respond per line, or legacy sendText fallback per line
when there is no packet id). LLM failure path unchanged: single
'summary failed' reply.
Replies >~190 chars fragment into fire-and-forget packets recipients
often never receive; old prompt produced ~326-334 char replies.
Prompt now asks for at most 3 short bullet lines, each under 45 chars,
bullets only (verified live: exactly 3 compliant bullets, ~110 chars).
fitOnePacket() backstop cuts any over-length reply at complete-bullet
boundaries so delivery is one packet even if the model ignores the prompt.
maxTokens 200 -> 2500 on the summary chat(): empirically verified that
this server's reasoning tokens count against max_tokens (cap 150 ->
completion 150 of which 105 were reasoning; cap 500 fully consumed by
reasoning, zero visible content) and thinking is uncontrollable
server-side (/no_think, chat_template_kwargs.enable_thinking,
thinking_budget all ignored), observed 32..1776 tokens for this task.
2500 covers every observed run + ~80 content tokens, worst case ~42 s
under the 60 s LLM timeout.
#rss subcommands on the command channel: help / add <url> <name> /
remove <name> / tags / sources / get <name|#tag>. Sources are stored in
a new rss_sources table in the existing MESH_DB_PATH file (second
node:sqlite connection, case-insensitive names); feeds parsed with
fast-xml-parser (RSS 2.0 + Atom); AI tags and headline summaries via
injected LlmService.chat() with bounded maxTokens (50/200).
CommandService gains prefix registration so one handler owns the whole
#rss family while #ping stays an exact match; wired in CommandModule's
factory — still no constructor-level cycle, no forwardRef. One dead feed
does not sink a tag summary.
LlmService wraps the local OpenAI-compatible server (default
http://192.168.1.70:1111/v1, no auth) with native fetch only — no new
dependencies. listModels() -> data[].id; chat(messages, opts?) ->
choices[0].message.content (non-2xx throws with status + body snippet);
summarize(text) = chat() with a concise-summary system prompt.
Env: LLM_API_BASE / LLM_MODEL (qwen3.8-27b) / LLM_TIMEOUT_MS (60000,
AbortSignal.timeout). Plain feature module wired into app.module.ts;
docs synced (AGENTS.md, README env list).
Event-driven CommandService dispatches exact case-sensitive #command
matches on COMMAND_CHANNEL (default 2) through a single-registration-point
registry; first command #ping -> #pong!. Replies ride COMMAND_CHANNEL:
directed DM back to the sender when present, else broadcast. Failing
handler/send only logs a warn.
DI: CommandModule's factory injects MeshService and attaches itself via
MeshService.attachCommands() — a constructor-level cycle leaves one side
null in NestJS 12's injector (verified), so no forwardRef is used.
Tests 106 -> 121; docs synced (AGENTS.md, MESHTASTIC.md, README).
Owner request. BatteryAlertService reads new env BATTERY_ALERT_CHANNEL
(default 2) at construction and passes it through MeshService.sendText,
which gains an optional trailing channel argument defaulting to the
pinned MESH_SEND_CHANNEL — every other sender is untouched. Threshold,
cooldown, merged one-message-per-cycle format and @mentions unchanged.
Owner follow-up: 'if multiple ppl shows at once — merge messages to one'.
check() now collects every qualifying node (at/below threshold and not
within its per-node cooldown) and sends AT MOST ONE broadcast per cycle
containing all of them, recording the alert timestamp for each after
sending; nothing is sent when none qualify. Format: '⚠️ battery low:
@!<hex> (<name>) <X>%[, @!<hex> (<name>) <Y>%]' — single-node case uses
the same one-entry template. Cooldown semantics unchanged.
BatteryAlertModule + BatteryAlertService wired into app.module like
the weather module. Every BATTERY_CHECK_INTERVAL_MS (default 60 s) the
service scans the node state backing GET /api/nodes and broadcasts a
single-packet '⚠️ @!<8-hex> battery low: <X>% (<name>)' alert for every
node at or below BATTERY_ALERT_THRESHOLD percent (default 20, <= on
purpose) not alerted within BATTERY_ALERT_COOLDOWN_MS (default 6 h,
in-memory map — resets on restart). Sends ride the pinned send channel
via MeshService.sendText like every other bot send.
battery_level was only populated during the initial sync; NodeInfo
frames streamed as NODEINFO_APP (port 4) now refresh deviceMetrics in
the node database so /api/nodes and downstream consumers see current
values between syncs. decodeNodeInfo + PORT_NODEINFO_APP added to the
codec, one branch in the existing packet-handling path.
kamilkrzywda