Add gNMI Extension field parsing support
gNMI allows for the use of an extension field in each top-level message of the gNMI RPCs: https://github.com/openconfig/reference/blob/master/rpc/gnmi/gnmi-extensions.md Given this is an arbitrary Protobyte payload, the default gNMI protobufs can't decode the payload contained within the field. This PR adds the necessary configuration options to load in an arbitrary Protobuf file per extension identifier, with a message-name to lookup the message type.
In this change:
- A new function
DecodeExtensionis added. This function usesprotoreflectto dynamically marshal arbitrary protoBytes into JSON. The loaded JSON is then put back in the Extension message as bytes (mainting type) - The
Targettype has anExtensionProtoMapadded, allowing for the lookup of Extension IDs to loaded-in protobufs - The required changes to the
TargetConfigtype to support loading in the new configuration - Modified
collector.goto output the gNMI message after inlining the decoded protoBytes - Loading in the protobufs was added to
app/target.go:parseExtensionProtos. This uses theParserprovided byprotoreflect/desc/protoparse - Added functionality to
event.goto insert the K/Vs provided by the Extension as Tags. Given we come from JSON, all numbers are float64, so the only 2 types supported currently arestringandfloat64 - Minor helper function to turn the arbitrary JSON into an arbitrary map.
This has been tested with a device emiting an extension field:
[gnmic] target "edge01_test01": gNMI Subscribe Response: &{
SubscriptionName:port_stats
SubscriptionConfig:{"name":"port_stats","paths":["/interfaces/interface/"],"mode":"STREAM","stream-mode":"SAMPLE","encoding":"JSON","sample-interval":15000000000,"heartbeat-interval":15000000000,"outputs":["prom-scrape-output"]}
Response:
update:{timestamp:1723653363502452302 prefix:{elem:{name:"interfaces"} elem:{name:"interface" key:{key:"name" value:"et-1/0/3"}}}
update:{path:{elem:{name:"state"} elem:{name:"hardware-port"}} val:{json_val:"\"FPC1:PIC0:PORT3\""}}
update:{path:{elem:{name:"state"} elem:{name:"transceiver"}} val:{json_val:"\"FPC1:PIC0:PORT3:Xcvr0\""}}}
extension:{registered_ext:{id:1
msg:"{\"systemId\":\"edge01_test01\",\"componentId\":65535,\"sensorName\":\"sensor_1005_2_1\",\"subscribedPath\":\"/interfaces/interface/\",\"streamedPath\":\"/interfaces/interface/\",\"component\":\"chassisd\",\"sequenceNumber\":\"770002\",\"payloadGetTimestamp\":\"1723653363502\",\"streamCreationTimestamp\":\"1723653361858\",\"exportTimestamp\":\"1723653363504\",\"streamId\":\"PERIODIC\"}"}}}
Which is then properly rendered to a Prometheus metric:
gnmi_interfaces_interface_state_hardware_port{component="chassisd",componentId="65535",hardware_port="FPC1:PIC0:PORT3",interface_name="et-1/0/3",metric_source="edge01_test01",subscription_name="port_stats",systemId="edge01_test01"} 1
Note that some label-drop rules have been added to remove the spurious labels to avoid a cardinality explosion.
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).
View this failed invocation of the CLA check for more information.
For the most up to date status, view the checks section at the bottom of the pull request.
This is a proposed solution for #503
Updated the commit to fix the type assertion issue as well as the variable assignment issue.
Anything else needed here @karimra ?
While this works for that junos extension, I'm not sure it will for any other extension. The data in an extension is not always meant to be added as tags, we should find a way to pass it on to the output and use processors to select some keys to set as tags. The other thing I'm not entirely convinced by is that these proto files are set by target. If you have 100s of targets of the same type (same extensions), we will have to parse the proto files 100s of times. This is fine for now, but it should be improved/fixed in a future PR.
Hello awesome people. Any movement on this. It would great to be able to do some processing of the attributes from extensions in for example starlark - so we could filter/markup/combine etc as needed.
Very interested in this also. Any update ?