System suddenly sends 20logs/second and consumes 2.1GB of data

[hasheddan]

@john.stuewe

No. I’m saying that the Golioth web interface only allows viewing 25 pages of 10 logs which is the most recent 250 logs. Golioth is not rate limiting and 250 of these logs are were replaced and overwritten every 12 seconds which used 2GB of data before we noticed and turned off cellular to that remote device.

Thank you for this clarification.

@john.stuewe

I’m not sure how much data Golioth is consuming because the scale on the left side of the graphic is cut off. There are too many digits to display which can’t be a good thing. (See picture above.)

Thank you for this feedback. We will ensure to fix the scaling of the chart such that the quantity on the y-axis is not truncated. In the meantime, as evidenced by your screenshot, you can see the exact usage by hovering / clicking on the bar(s) for the date in question. For example, I can see that your usage on April 6th was 330,512 KB of logs (device_logs) and 230 KB of Light DB Stream (data_lightdb_stream), putting your cumulative usage over 300,000 KB (300 MB), which is the top tick on the y-axis (with each tick representing 50,000 KB). We will also update the usage chart to ensure that the top y-axis tick represents the “next” increment above your peak usage to make this more clear.

@john.stuewe

If it was rate limiting it would limit the cellular data but it would still consume a lot. I turned off any logs I saw using data but I never saw this one in development so it wasn’t turned off. If any module in Zephyr or Golioth can do this then it’s not safe to go to sleep with a system running and it isn’t safe to leave the Golioth logging system enabled in production which defeats the whole purpose of having logging in the first place.

If the logs were the only data being sent to Golioth (or the vast majority of the data), you may have been falling just under rate limiting at 20 req/s. There is additional data that is transmitted over your cellular network when establishing a connection with Golioth and delivering logs, but in the case of logs, usage is only recorded for the payload content. This, along with data that is not counted against Golioth usage (i.e. device management operations), could leading to the difference between your Golioth usage (~0.33 GB) and cellular network usage (2.1 GB).

Regarding the error you are seeing in the logs, this looks similar to SDK 0.10.0 sends >15million logs/day in error condition, though the frequency of invalid access on sock 0 logs is much higher and there are no logs referencing the unknown CoAP response, which was an indicator of experiencing rate limiting. You mentioned that you are using SDK v0.11.1, which should have the relevant bug fix. However, I am curious if your logging configuration between SDK 0.10.0 sends >15million logs/day in error condition and this ticket has changed, as the logs representing rate limiting could be getting filtered out now. Would you mind sharing your logging configuration?