Description
I’m seeing Golioth skip server-to-device messages on connected devices whose sessions are otherwise healthy. More than half of LightDB State observe notifications are never delivered, and RPCs time out with 499 at a similar rate. The writes are accepted, the session stays up, and keepalives are answered the whole time. I first saw it on our own cellular hardware, but I’ve now reproduced it with the SDK’s Linux port on a laptop. So it doesn’t look like our firmware, hardware or network.
Steps to Reproduce
SDK: Golioth Firmware SDK v0.21.1
Network: home Wi-Fi.
Keepalive: the default 9 s.
Test: the device observes a path in lightDB state that contains a rev counter, and a script writes a new rev to it through the REST API. We tried random intervals first, then a fixed 1-minute interval.
Logging: libcoap debug logging is on, so every DTLS record the device receives is logged before the SDK sees it.
Expected Behavior
When a new rev value is set in light db state, the devices are notified of a change in LightDB state and the new rev value is sent to the device.
Actual Behavior
All on one DTLS session lasting about 2 h, with no reconnects:
Keepalives: 665 of 665 answered, within about 250 ms.
Notifications: 49 writes accepted, 21 actually notified.
Timing: delivered notifications typically arrive within 0.1 s of the write.
The 28 that went missing: nothing came in at all, neither a notification nor a CoAP retransmission, then or later. The device stays on the old value until it reconnects and re-observes.
Misses come in runs: the longest was 7 in a row (over 10 minutes).
Test: Increment connectivity/desired/rev at 1-minute intervals. 10 of 21 notified
Timestamps (UTC, Oct 30 2026)
Rev Write accepted Notified
33 01:16:24.407 Yes
34 01:17:23.921 No
35 01:18:23.963 No
36 01:19:23.868 Yes
37 01:20:23.879 Yes
38 01:21:23.925 No
39 01:22:23.904 No
40 01:23:23.925 Yes
41 01:24:23.907 Yes
42 01:25:23.876 No
43 01:26:23.995 Yes
44 01:27:23.902 Yes
45 01:28:23.872 No
46 01:29:24.039 No
47 01:30:23.858 No
48 01:31:23.981 Yes
49 01:32:23.985 Yes
50 01:33:24.013 Yes
51 01:34:23.948 No
52 01:35:23.888 Yes
53 01:36:23.940 No
Same thing on our hardware
Our hub is an nRF52840 with an nRF9160 over PPP, on cellular. It runs the Zephyr port with DTLS Connection ID on and a 30 s keepalive.
Impact
Typically when a path in LightDB state is updated the rev is incremented and the new data in that path is sent immediately to our device. This issue means we can no longer rely on being notified by changes in the LightDB state and must instead poll, which increases energy consumption.
Environment
Two independent setups, both on Golioth Firmware SDK v0.21.1, both showing the same loss:
- Golioth Firmware SDK v0.21.1, Linux port (libcoap v4.3.5-rc2 + OpenSSL 3.6.2), built on macOS 26.6 (Apple Silicon). The port’s eventfd/POSIX-timer calls were swapped for pipe/pthread equivalents; the SDK and libcoap code are unchanged.
- Home Wi-Fi behind a consumer NAT router
- Default 9 s keepalive, no DTLS Connection ID (OpenSSL has none)
- PSK auth; observes LightDB State “connectivity/desired”; registers RPCs
- rev written via the REST API, at random intervals and then every 1 minute
- Production hardware
- Custom nRF52840 board, with an nRF9160 running Nordic’s Serial LTE Modem (SLM) over PPP/CMUX
- nRF Connect SDK v3.1.1 (Zephyr 4.1.99), Golioth Zephyr port
- DTLS Connection ID on; 30 s keepalive, 75 s receive timeout (also tested at 9, 20 and 50 s, and with CID off)
- Cellular, two New Zealand carriers (One NZ and Spark)
- Two seperate Golioth projects
No pipelines are involved. The observed paths are plain LightDB State.
Logs and Console Output
- Linux port, libcoap debug log (times NZDT = UTC+13). Every DTLS record received is logged before the SDK sees it.
Delivered notification (rev 21, write accepted 00:33:56.022 UTC):
13:33:56.035 DTLS: netif: recv 97 bytes
v:1 t:CON c:2.05 i:a947 {432190c8fb7dbedf} [ Content-Format:application/json ] :: ‘{“rev”:21,“schema”:1,“value”:{“offline_s”:0}}’
13:33:56.037 DTLS: netif: sent 41 bytes
v:1 t:ACK c:0.00 i:a947 {} [ ]
Missed notification (rev 22, write accepted 00:35:07.613 UTC). Only keepalives in and out, and no notification or retransmission afterwards either:
13:35:08.165 DTLS: netif: sent 49 bytes
v:1 t:CON c:DELETE i:2692 {0000000000000000} [ ]
13:35:08.400 DTLS: netif: recv 70 bytes
v:1 t:ACK c:4.04 i:2692 […] :: ‘404 page not found’
13:35:17.410 DTLS: netif: sent 49 bytes
v:1 t:CON c:DELETE i:2693 {0000000000000000} [ ]
13:35:17.638 DTLS: netif: recv 70 bytes
v:1 t:ACK c:4.04 i:2693 […] :: ‘404 page not found’
… the same every 9 s until rev 29 arrives at 00:46:31 UTC. Revs 22-28 never arrive.
Session totals: 49 writes, 21 notifications received; 665/665 keepalives answered; 0 reconnects.
Peer for every record: 34.135.90.112:5684.
- Production hardware (nRF52840 RTT log; [uptime], packets logged before DTLS by a Zephyr net packet filter)
Delivered (connectivity rev 14):
[16:34:09.196] link_dbg: rx 125 B from 34.135.90.112
[16:34:09.196] link_dbg: push: connectivity/desired
[16:34:09.197] app_state: connectivity rev 14: offline_s 0
[16:34:09.821] app_state: connectivity/reported rev 14 acknowledged
Missed (connectivity rev 13, written 21:09:56 UTC). Only keepalive exchanges:
[16:31:06.124] link_dbg: 21:09:47 UTC: cloud up, session 1, 0 drop(s)
[16:31:11.346] link_dbg: tx 89 B to 34.135.90.112
[16:31:12.246] link_dbg: rx 89 B from 34.135.90.112
[16:31:36.125] link_dbg: 21:10:17 UTC: cloud up, session 1, 0 drop(s)
[16:31:41.346] link_dbg: tx 89 B to 34.135.90.112
[16:31:42.231] link_dbg: rx 89 B from 34.135.90.112
… nothing else inbound until rev 14 at 21:12:50 UTC.
RPCs from the console to this device return 499 about 35-45% of the time. The device’s packet log shows the RPC never arrived.
Full logs available on request.