How to implement LightDB Stream with CBOR

Golioth Guides
1

The Golioth LightDB Stream is purpose-built for collecting time-based data from devices in the field. Every time data is received, the server records it along with a timestamp. This is perfect for collecting sensor data as it creates a historic record of all readings.

Send LightDB Stream payload from Device

In this example, we want to send the following payload, which must be prepared as an object. The data will be formatted as a serialized CBOR object. This should be done on the device using a library of your choice. The zcbor library is already available as a submodule dependency of the Golioth Firmware SDK with Zephyr RTOS.

{
  "sensor": {
    "active": true,
    "data": {
      "calib": 42,
      "reading": 1.337
    },
    "name": "Golioth"
  }
}

Here is an example of using zcbor in canonical mode (ZCBOR_CANONICAL or in Zephyr: CONFIG_ZCBOR_CANONICAL) to encode a CBOR payload.

#include <zcbor_encode.h>

uint8_t buff[128] = {0};
bool my_bool = true;
double my_float = 1.337;
int my_int = 42;
char my_str[] = "Golioth";

// Macro for declaring and initializing an encoding state
ZCBOR_STATE_E(state, 8, buff, sizeof(buff), 0);
// Start map header encoding
zcbor_map_start_encode(state, 3);
// Encode string
zcbor_tstr_put_lit(state, "name");
zcbor_tstr_put_term(state, my_str, 10);
// Encode bool
zcbor_tstr_put_lit(state, "active");
zcbor_bool_put(state, my_bool);
zcbor_tstr_put_lit(state, "data");
// Start map header encoding
zcbor_map_start_encode(state, 2);
// Encode float64
zcbor_tstr_put_lit(state, "reading");
zcbor_float64_put(state, my_float);
// Encode int32
zcbor_tstr_put_lit(state, "calib");
zcbor_int32_put(state, my_int);
// Close both maps
zcbor_map_end_encode(state, 2);
zcbor_map_end_encode(state, 3);

size_t cbor_payload_len = state->payload - buff;

As you can see, the payload has no sensor key. That is because we will tell Golioth to set the "sensor" path in the function call. Specifying a path allows for custom routing of streaming data to a destination using Pipelines. For more information, please see the Pipelines section of the Golioth Docs.

Time-series (Stream) data can be sent to Golioth in two different ways:

Synchronous

The simplest way to send Stream data to Golioth is using a synchronous function call. This blocking function will send data to the given path and wait for a response (or error) from the server. If a response is not received within the given timeout (1 second in the example below), a GOLIOTH_ERR_TIMEOUT error code will be returned.

err = golioth_stream_set_sync(client,
                              "sensor",
                              GOLIOTH_CONTENT_TYPE_CBOR,
                              cbor_buf,
                              sizeof(cbor_buf),
                              1);

Asynchronous

The asynchronous functions are a non-blocking approach to sending Stream data to Golioth. When the task is completed, an optional callback function may be run to process the result of the async operation.

enum obj_type {
    JSON_ASYNC,
    CBOR_ASYNC
};

err = golioth_stream_set_async(client,
                               "sensor",
                               GOLIOTH_CONTENT_TYPE_CBOR,
                               cbor_buf,
                               sizeof(cbor_buf),
                               async_error_handler,
                               (void *)CBOR_ASYNC);

Callback Function

Callback functions must follow the type specified by the function registers them. In this case, a golioth_set_cb_fn type callback function must be defined:

static void obj_async_handler(golioth_client_t client,
                              const golioth_response_t* response,
                              const char* path,
                              void* arg) {
    if (response->status != GOLIOTH_OK) {
        // The set operation failed.
        return;
    }

    // The set operation was successful!
    switch((enum obj_type)arg) {
        case JSON_ASYNC:
            printf("JSON async set successful!\n");
            break;
        case CBOR_ASYNC:
            printf("CBOR async set successful!\n");
            break;
        default:
            printf("Async set successful, but arg value unrecognized\n");
    }
    return;
}

The response is passed to the callback function as a golioth_response_t struct that includes status. It is recommended that callbacks test for the GOLIOTH_OK status which indicates a successful operation.

Multiple async functions may use the same callback. In this example, the arg value determines which function led to this callback, so a different message may be printed.

Pipelines

Head over to the Golioth Console to view the data routing for your project by selecting Pipelines from the left sidebar of the Golioth console. New projects will have one Pipeline called LightDB Stream CBOR.

Click on the name of the pipeline to open the edit window.

filter:
  path: "*"
  content_type: application/cbor
steps:
  - name: step-0
    transformer:
      type: cbor-to-json
      version: v1
    destination:
      type: lightdb-stream
      version: v1

Here you can see that the pipeline is targeting CBOR data on all paths (*). It
applies one step that transforms the data from CBOR to JSON, then routes it to
the Golioth LightDB stream.

View LightDB Stream in Device Summary

To view the LightDB Stream data for a device:

  1. Click the Devices option in the left sidebar of the Golioth
    Console
  2. Click on the device Name in the resulting list
  3. Click on the LightDB Stream tab

LightDB-1
LightDB-12366×832 64.1 KB

LightDB Stream page overview

On this page you can view the following information:

  • timestamp: Time/Date at which the data was received (timestamps can also be set by adding them as a member of the dataset)
  • data: The data received, displayed in JSON format. This will be “folded” by default
  • Time selector: historic data can be access by choosing time/date from the time selection box
  • Refresh tools: both manual and automatic refresh buttons are available for updating the data being displayed

You may also view all LightDB Stream data for you fleet by clicking the Monitor option in the left sidebar and selecting LightDB Stream from the list that unfolds.

Exercise: add metadata to Pipelines

Let’s make a small change to the pipeline to try it out. Let’s tell Golioth to add metadata to the stream. This is quite useful when routing to your own database outside of Golioth.

  1. Change the current step-0 to step-1.
  2. Add a new step-0 that uses cbor-to-json as the transformer but don’t add a destination to this step. (The next step will automatically be used as the destination.)
  3. Update the transformer in step-1 to use the inject-metadata transformer.
  4. Save your pipeline.

Go back to the LightDB Stream view for your device and you will now see a device_id, project_id, and timestamp alongside each newly received payload.

LightDB-2
LightDB-22362×894 128 KB

Click to reveal solution 
filter:
  path: "*"
  content_type: application/cbor
steps:
  - name: step-0
    transformer:
      type: cbor-to-json
      version: v1
  - name: step-1
    transformer:
      type: inject-metadata
      version: v1
    destination:
      type: lightdb-stream
      version: v1