[Question] Possible bug: Transformation script creates mixed / inconsistent telemetry data

[Chabrol]

Component

• Rule Engine

Description A raw CAN-data stream is sent to a thingsboard instance via a thingsboard-gateway. The data is encoded as a string, containing hex values. In the rule chain this data is converted / amended by an transformation script, which adds elements to the telemetry data. The data rate is fairly high with roughtly 50 packets per second. Now I encounter sporadic value anomalies. It looks like the false value is contained in another telemetry packet, which is send at about the same time (~100ms later). I verified and compared the data at the source and the hex encoded telemetry string element (queried via REST get telemetry data). They match, but in case the anomaly occur, the converted value doesn't match the data in the raw data (The script converts a part of the hex string to a ieee-754 float value). How can this mixup between different telemetry messages happen? It feels like the input parameters to the transform function can change, while the script is still running.

Environment

• OS: Ubuntu 18.04.5 LTS
• ThingsBoard: 3.1.1-1

[ashvayka]

The JS function for the conversion is executed in the same context. So, if you write something like this:

data = 1;

It will update a global variable that will be shared between messages. However, if you write:

var data = 1;

it is a local variable. Please check your code for such errors

[Chabrol]

@ashvayka Every variable I use in the script is a local one defined with var. Except the functions input parameters of course. Is there a possibility that a helper function defined in the transformation function is not safe regarding execution context? Here the function for reference:

function hex2float(hex) {
    var v = new DataView(new ArrayBuffer(4));
    v.setUint32(0, parseInt(hex,16));
    return parseFloat(v.getFloat32(0)).toFixed(3);
}

Thank you for your support

[Karabiy]

please, try to set the sequential by originator queue, so that the issue should not persist. Let us know in case if those persists you can use the next schema for those

https://thingsboard.io/docs/user-guide/rule-engine-2-0/overview/#rule-engine-queue so that you can set the checkpoint rule node before the script processing and the acknowledge rule node afters, in case if you are using the ThingsBoard 2.5+ verison

[Chabrol]

@Karabiy thank you for your reply. The forced sequential execution seems to prevents the mixed data. But nevertheless it shouldn't occur without it too, as long as there is no use of global variables in the javascript code. At least this proves, that we really encountered a parallelism related issue.

[Karabiy]

@Chabrol please, share the full script of yours, as those requires to have a deep investigation to be taken the usage of the sequential queue may be considered as a workaround

[Chabrol]

@Karabiy I have to relativize my previous message. I removed the sequential start and stop again, and the mixing has not reoccured. With another thingsboard instance, in this case a community edition, I encountered the converse effect: I was not able to workaround this effect by adding the sync blocks. I suspect there are multiple things that affect this issue (data timing, system thread timing, system load, ...)

Here is the used script. The input data is an ASCII string representation of binary data in hex with an appended "/" to prevent javascript from interpreting hex numbers without letters as decimal integers. For example "0480061301423C0000/". I get sporadic results where the portion of the float payload (via hex2float) does not match the value in the raw data. The wrong values can be found in following messages (in one analyzed case from the third following CAN packet).

var data = msg.data;

var retmsg = {data: data};

var can_id = parseInt(data.substr(0, 4), 16);
var data_len = parseInt(data.substr(4, 2), 16); // can payload bytes
var cmd = null;
var cmd_id = null;

if (data_len >= 1) {
    cmd = parseInt(data.substr(6, 2), 16);
} else {
    cmd = null;
}

if (data_len >= 2) {
    cmd_idx = parseInt(data.substr(8, 2), 16);
} else {
    cmd_idx = null;
}

if (can_id == 0x410) {
    if (cmd == 0x10 && cmd_idx == 0x0F && data_len == 6) {
        retmsg.batteryCurrent = hex2float(data.substr(10, 8));
    } else if (cmd == 0x13 && cmd_idx == 0x01 && data_len == 6) {
        retmsg.soc = hex2float(data.substr(10, 8));
    }  else if (cmd == 0x11 && data_len == 2) {
        retmsg.status = parseInt(data.substr(8, 2), 16);
    }  else if (cmd == 0x12 && data_len == 5) {
        retmsg.error = parseInt(data.substr(8, 8), 16);
    }
} else if (can_id >= 0x480 && can_id <= 0x4FF) {
    pb = can_id - 0x480;
    if (cmd == 0x21 && cmd_idx >= 0 && cmd_idx <= 7 && data_len == 6) {
        retmsg["pb" + pb + "u" + cmd_idx] = hex2float(data.substr(10, 8));
    } else if (cmd == 0x20 && cmd_idx >= 0 && cmd_idx <= 7 && data_len == 6) {
        retmsg["pb" + pb + "i" + cmd_idx] = hex2float(data.substr(10, 8));
    } else if (cmd == 0x12 && data_len == 5) {
        retmsg["pb" + pb + "status"] = hex2float(data.substr(8, 8));
    } else if (cmd == 0x1F && cmd_idx >= 0 && cmd_idx <= 7 && data_len == 6) {
        retmsg["pb" + pb + "q" + cmd_idx] = hex2float(data.substr(10, 8));
    }
}

return {msg: retmsg, metadata: metadata, msgType: msgType};


function hex2float(hex) {
    var v = new DataView(new ArrayBuffer(4));
    v.setUint32(0, parseInt(hex,16));
    return parseFloat(v.getFloat32(0)).toFixed(3);
}