Settings not recognized, not matching up

Golioth Cloud Help
1

I have an interesting situation that I found. (I hesitate to call it a bug yet.)

We’re in the middle of a big refactor of a project, and I’m bringing up Golioth and its libraries into an existing Golioth project. I’m finding that the errors on the settings page for a device don’t match what I would expect from the project.

I’m using an ESP32S3 with IDF v5.2. I have currently implemented the Loop Delay and have tested that changing it on the console successfully updates it on the MCU.

However, here is what I’m seeing on the Console’s Devices > (this device) > Settings tab:

image
image1962×1092 141 KB

(Pardon the redactions).

  • Green is exists in the Device Settings, but doesn’t exist in the MCU code. This should be flagged as Key Not Recognized.
  • Yellow is exists in the Device Settings, but doesn’t exist in the MCU code. This is correctly flagged.
  • Blue is doesn’t exist in either Device Settings or MCU code. Not sure how this is getting flagged at all. It’s an old setting we’ve deprecated and deleted from the Project-wide Device Settings.

I’m assuming the Blue is due to it being set at a Device or Blueprint level, but not at the Project level. And it’s still being pulled in to be displayed here.

Green is baffling me though. Why aren’t these being tagged as key not found?

2

Thanks for reporting this @sethkazarians! I noticed in your image that it looks like your reported error message is from 08/09/2023 which indicates to me that the device is not reporting settings status. That would explain why you are seeing a setting that no longer exists (blue – existed at last report but no longer does) as well as some that are not mentioned in the reported errors (green – did not exist at time of last report). Are you seeing any errors in your firmware logs that would indicate an inability to deliver a settings status update? Also are you able to share which version of the Golioth Firmware SDK you are using?

3

@sethkazarians it is likely that your errors response is exceeding the max response buffer limit. To test this you could increase the buffer limit or implement more of the settings on the MCU side, which should reduce the size of the status report payload by minimizing the number of errors. However, we should handle this case more gracefully in the firmware SDK, likely by sending a payload that indicates that there were too many errors to enumerate. We’ll track making that improvement.

4

Yeah, I noticed that 8/9/23 date as well. I’d be curious to see what causes that to update. (The physical device and the PSK were reused, so that’s probably the last time this device was booted with the old firmware.)

I’m not seeing any settings related issues on the firmware logs. It’s getting the loop delay in a normal manner, then proceeding as if everything is ok.

I’m using SDK version 0.10.0.

I’ll try adding the other settings and see if that starts clearing it up.

5

I’m looking at other devices in the same project, and they aren’t showing the blue setting at all. They are showing expected error list. (We still haven’t finalized the list of settings, and have several versions of firmware floating about.)

This feels like the firmware is not triggering an update within Golioth. What code block should I be looking for that will force an update and/or generate an error locally?

6

Just further debugging: I created a new device with a new PSK, and programmed the device with it. The settings status is now: Device did not report settings status yet.
This confirms that the Blue setting, is just a hold-over from before.

But that also begs the question: what actually does the setting updating?

I’m implementing all the settings, and did a status check mid-way through and noticed a weird behavior. The implemented settings are getting their values as expected, but the log is showing them being set multiple times. I’m counting roughly 8 instances of the loop delay being set on boot up. (This might be an artifact of another weird behavior where if I set one setting, they ALL update, and the hiccups are causing just enough delay for them to update in waves.)

7

The code I’m using is based on the golioth_basics() function.

    struct golioth_settings *settings = golioth_settings_init(client);


    golioth_settings_register_int(settings, "LOOP_DELAY_S", on_loop_delay_setting, NULL);
8

I’ve implemented all the settings. The status is still not being updated.

Here is the log output I’m getting. There is a new error message now popping up, but the settings are getting set correctly.

I (1888) golioth_mbox: Mbox created, bufsize: 2352, num_items: 20, item_size: 112
I (1908) {Project name}: Waiting for connection to Golioth...
I (1918) golioth_coap_client_libcoap: Start CoAP session with host: coaps://coap.golioth.io
I (1918) wifi:<ba-add>idx:1 (ifx:0, da:b3:aa:aa:aa:aa), tid:0, ssn:2, winSize:64
I (1918) golioth_coap_client_libcoap: Session PSK-ID: {redacted}
I (1948) libcoap: Setting PSK key

I (1948) golioth_coap_client_libcoap: Entering CoAP I/O loop
I (2548) {Project specific message}
W (2548) {Project specific message}
I (2638) golioth_coap_client_libcoap: Golioth CoAP client connected
I (2638) {Project Name}: Golioth client connected
I (2638) golioth_fw_update: Current firmware version: main - 1.2.5
E (2648) golioth_coap_client: Failed to enqueue request, queue full
E (2658) golioth_coap_client: Failed to enqueue request, queue full
E (2668) golioth_coap_client: Failed to enqueue request, queue full
E (2668) golioth_coap_client: Failed to enqueue request, queue full
W (2678) golioth_coap_client: Failed to enqueue request, queue full
I (2678) {Project Name}: Hello, Golioth! #0
I (3558) {Project specific message}
W (3558) {Project specific message}
I (3558) golioth_fw_update: Waiting to receive OTA manifest
9

@hasheddan
I found one of the issues. We had added a 17th setting, which goes over the max of 16. Where is this max coming from? I’m fairly certain we’re going to need more than 16.

I’m still trying to figure out what triggers the settings status to update. I was in a state Golioth Console was saying “Code : 1. Key not recognized” for one last setting, but I was successfully changing the setting that wasn’t being recognized and the device was responding correctly.

After about 10 minutes, it finally updated the status to be green.

Also, the cacophony of settings updates is still happening on boot. (The issue where the setting callback is being called several times for each setting. This didn’t used to happen before with roughly the same number of settings.)

10

@sethkazarians thank you for providing more information! I have included responses to a number of your questions below, but please feel free to follow up on any as necessary.

I found one of the issues. We had added a 17th setting, which goes over the max of 16. Where is this max coming from? I’m fairly certain we’re going to need more than 16.

Max settings is defined by CONFIG_GOLIOTH_MAX_NUM_SETTINGS, which is set to 16 by default.

Yeah, I noticed that 8/9/23 date as well. I’d be curious to see what causes that to update. (The physical device and the PSK were reused, so that’s probably the last time this device was booted with the old firmware.)

This date is updated when a successful report is delivered from the device. If the report was exceeding the buffer or max settings count it is possible that no valid update was received.

I’m looking at other devices in the same project, and they aren’t showing the blue setting at all. They are showing expected error list. (We still haven’t finalized the list of settings, and have several versions of firmware floating about.)

If they have reported more recently, then they likely did not include the legacy setting (blue setting) in their report, which is why it would not show up in the console.

I’m still trying to figure out what triggers the settings status to update. I was in a state Golioth Console was saying “Code : 1. Key not recognized” for one last setting, but I was successfully changing the setting that wasn’t being recognized and the device was responding correctly.

When settings are received the on_settings callback is invoked, which will decode received settings, then report status back to Golioth. I expect that exceeding the buffer (due to number of errors) or exceeding the maximum settings count.

Also, the cacophony of settings updates is still happening on boot. (The issue where the setting callback is being called several times for each setting. This didn’t used to happen before with roughly the same number of settings.)

We’ll look into this and follow up.

11

Is there a reason that number is 16? As in, is there a limitation on the backend which dictates that? Or can I make it any number I want? (If the latter, then why does that max exist at all?)

That makes sense.

Understood, but is there a delay between the on_settings callback being called, and the Golioth console getting the status? It was 30-ish minutes since I had the right settings code (correct names, amount <16) and the status going green. It was also several minutes between me verifying that the final problematic setting was functioning and the status going green.

12

This is not a backend limitation, so I believe the limit here is to allow you to guard against accidentally flooding your device with settings. However, I’ll follow up on this and ensure this isn’t representative of some fundamental limitation.

There should not be any delay if the settings update is delivered successfully. However, you may need to click the Refresh button on the settings page in order to get the latest report. I am assuming you were instinctively doing that already. After the ~30+ minutes was the correct status shown with a timestamp that was 30 minutes old, or was it more recent?

13

@sethkazarians following up here – memory for registered settings is statically allocated, which is why the number of settings is configurable at compile time. This should not impact the ability to process settings beyond that maximum (e.g. report not registered), though you may hit the aforementioned GOLIOTH_SETTINGS_MAX_RESPONSE_LEN.