Improve support for Cumulocity inventory with dynamic fragments

[didier-wenzek]

With the current support for Cumulocity inventory, a device owner can configure the custom fragments sent on start to Cumulocity. However, the true need is for an agent developer to be able to build an agent that extracts from the device the custom fragments to be sent to Cumulocity on change. It would even be better to have this feature ready to be used by any cloud.

“An agent developer should be able to implement an agent that builds a dynamic summary view of the device from fragments collected from sub-components and child-devices and sent to the cloud on change“.

A solution might be

• The dynamic view of the device is materialized by a JSON document.
• This JSON document is build from by a hierarchy of JSON documents, the fragments.
  • The paths to all these fragments form the root of the JSON device view.
  • The contents of these fragments form the leaf of the JSON device view.
• These fragments can be updated independently via MQTT using a topic that reflects the fragment path.
• Thin-edge listen to all the fragment updates and aggregates them into a whole JSON that is the current device view.
  • So an agent developer can package along thin-edge components that publish fragments for the device status or metadata.
• Any change to the whole JSON is published over MQTT.
  • A cloud mapper can then listen for such view changes and propagates them to the cloud.

Alternatives

• Can the idea of fragments be used for other file formats than JSON?
• Can a fragment be directly updated via the filesystem?
• How to persist the fragments? Use the filesystem or MQTT retain messages?
• How to publish the whole JSON document? Over HTTP? MQTT? The file system?

Additional context

• See this discussion

[reubenmiller]

This part might actually relate more to the other discussion ticket #1199, however I will leave it here for now as it provides some background into the proposal.

Background

Inventory data is typically used to answer the following questions about a device (or sub-devices)

• What is installed on the device?
• What is the operating state of the device?

The answer to these questions control how an actor interfaces with the device (select the appropriate software for the current operating system) and how the device is expected to respond (e.g. how quickly is the download likely to be when installing software).

Inventory data can include some of the following data (but not limited to):

• Customer information
• Information related to the installation of the device
  • IP Address or other connectivity related data such as mobile connection technology (e.g. LTE or 3G)
  • Firmware name/version
  • Operating System information (version (Ubuntu/Debian/CentOS), kernel version etc.)
  • Installed software packages

Interface

I'm not sure that a file based approach is best here as it requires the agent developer to write a piece of software to execute the logic which will write the inventory files/fragments to disk.

It seems there are currently two agent scenarios:

• Scenario 1: SMART agent MQTT interface
  • Agent can register inventory data asynchronously with thin-edge via the MQTT interface
• Scenario 2: no-agent (just thin-edge)
  • Inventory data is provided by the execution of scripts/binaries (called "inventory scripts")
  • The name of the inventory script determines the fragment name and the standard output (key/value) determines the contents of the fragment

Inventory data triggers

Due to the nature of the inventory data, it is generally slow to change. The real usefulness of the inventory data is the ability to hook it into existing functionality of the thin-edge, so that it is only updated when required (e.g. when any side-effects have occurred).

The following triggers could be used to execute the inventory scripts:

• On startup
• Periodic (every ~24 hours)
• After any side-effect such as firmware or software updates

Inventory data - via simple script interface

The example shows the inventory data being provided via a simple script. The scripts are stored in a known location. The name of the script determines which fragment will be updated, and the contents of the fragment are provided by the key/value standard output.

Below shows two inventory scripts which update the mobile data information, and the device operating system info.

file: inventory_mobileData.sh

#/bin/bash
echo "connectedTechnology=LTE"

file: inventory_device_OS.sh

#/bin/bash
OS_VERSION=$( cat /etc/os-release | grep "VERSION_ID" | tr -d '"' | cut -d= -f2 )

# output values to stdout in the format of key=value
echo "version=$OS_VERSION"
echo "arch=$(uname -m)"

Output: inventory data

The above two scripts would produce the following inventory json data.

{
    "device_OS": {
        "version": "10",
        "arch": "x86_64"
    },
    "mobileData": {
        "connectedTechnology": "LTE"
    }
}

References

The inventory script approach is used by products which share a similar space as thin-edge.

• Mender Inventory Data

[didier-wenzek]

@reubenmiller

Background

Definitely helpful ... and a good start for a reference guide ;-)

Interface

I don't fully grasp the difference you introduce between a SMART agent and no-agent (just thin-edge).

• Even in what you call the just thin-edge case, one needs to have extra components provided by some developers.
• Both approaches can be combined and are not exclusive.

Inventory data triggers

This is a key point raised by your comment. With the initial proposal of fragments sent over MQTT, the question of when a fragment needs to be updated is delegated by thin-edge to the components added by the agent developer. With this proposal, thin-edge becomes responsible of triggering the updates after rules given by the device admin.

We introduced similar rules to trigger scripts on operation requests. This has not really been welcome due to the pain to install a plethora of scripts and configuration files with no remote support. However, the alternative to let the user trigger things, is not so convenient too. We need to understand this feedback a bit further.

[reubenmiller]

I don't fully grasp the difference you introduce between a SMART agent and no-agent (just thin-edge).

Even in what you call the just thin-edge case, one needs to have extra components provided by some developers. Both approaches can be combined and are not exclusive.

The main difference between what I have called a SMART agent and the no-agent (just thin-edge) is that a SMART agent involves writing a process, where as the no-agent scenario just involves writing a few (stateless) scripts. The SMART agent is a bigger beast as it would likely involve writing a new service which is additional overhead.

In my experience these "inventory scripts" are baked into the Operating System image which is applied on the device, so there is less thought needed to distribute these scripts, as they are updated when the Operation System is updated (i.e. via mender A/B updates). However any scripts or components can be just as easily distributed via software updates (e.g. Debian APT packages). The distribution of scripts or components would have to be handled anyway regardless, either via an agent or individual software updates, it is more just a packaging question.

However I think a key point which is missing is that the inventory scripts are to describe the device, so to have them logically grouped with the device rather than the agent makes more sense. This always you to install the same agent on multiple devices where the inventory scripts maybe different.

This is a key point raised by your comment. With the initial proposal of fragments sent over MQTT, the question of when a fragment needs to be updated is delegated by thin-edge to the components added by the agent developer. With this proposal, thin-edge becomes responsible of triggering the updates after rules given by the device admin.

We introduced similar rules to trigger scripts on operation requests. This has not really been welcome due to the pain to install a plethora of scripts and configuration files with no remote support. However, the alternative to let the user trigger things, is not so convenient too. We need to understand this feedback a bit further.

I think have script based handlers for operations is a bit trickier as operations are much more complex as they require both parsing of the input parameters and the generation of some output (depending on the operation), so it is very hard to create a uniform output format. Inventory scripts should be stateless and require no input. This also makes inventory scripts really easy to debug as the user on the device can execute them locally to check the output, and the output parsing is just simple key/value pairs.

[reubenmiller]

Resolved by https://github.com/thin-edge/thin-edge.io/issues/2279. System tests have been added to cover the new topic:

• thin-edge_device_telemetry.robot#L133

[reubenmiller]

QA has thoroughly checked the feature and here are the results:

• [x] Test for ticket exists in the test suite.
• [x] QA has tested the bug and could not reproduce it anymore.