gnmic icon indicating copy to clipboard operation
gnmic copied to clipboard

Add gNMI Extension field parsing support

Open Ichabond opened this issue 1 year ago • 7 comments

gNMI allows for the use of an extension field in each top-level message of the gNMI RPCs: https://github.com/openconfig/reference/blob/master/rpc/gnmi/gnmi-extensions.md Given this is an arbitrary Protobyte payload, the default gNMI protobufs can't decode the payload contained within the field. This PR adds the necessary configuration options to load in an arbitrary Protobuf file per extension identifier, with a message-name to lookup the message type.

In this change:

  1. A new function DecodeExtension is added. This function uses protoreflect to dynamically marshal arbitrary protoBytes into JSON. The loaded JSON is then put back in the Extension message as bytes (mainting type)
  2. The Target type has an ExtensionProtoMap added, allowing for the lookup of Extension IDs to loaded-in protobufs
  3. The required changes to the TargetConfig type to support loading in the new configuration
  4. Modified collector.go to output the gNMI message after inlining the decoded protoBytes
  5. Loading in the protobufs was added to app/target.go: parseExtensionProtos. This uses the Parser provided by protoreflect/desc/protoparse
  6. Added functionality to event.go to insert the K/Vs provided by the Extension as Tags. Given we come from JSON, all numbers are float64, so the only 2 types supported currently are string and float64
  7. Minor helper function to turn the arbitrary JSON into an arbitrary map.

This has been tested with a device emiting an extension field:

[gnmic] target "edge01_test01": gNMI Subscribe Response: &{
SubscriptionName:port_stats
SubscriptionConfig:{"name":"port_stats","paths":["/interfaces/interface/"],"mode":"STREAM","stream-mode":"SAMPLE","encoding":"JSON","sample-interval":15000000000,"heartbeat-interval":15000000000,"outputs":["prom-scrape-output"]}
Response:
update:{timestamp:1723653363502452302  prefix:{elem:{name:"interfaces"}  elem:{name:"interface"  key:{key:"name"  value:"et-1/0/3"}}}
update:{path:{elem:{name:"state"}  elem:{name:"hardware-port"}}  val:{json_val:"\"FPC1:PIC0:PORT3\""}}
update:{path:{elem:{name:"state"}  elem:{name:"transceiver"}}  val:{json_val:"\"FPC1:PIC0:PORT3:Xcvr0\""}}}
extension:{registered_ext:{id:1
    msg:"{\"systemId\":\"edge01_test01\",\"componentId\":65535,\"sensorName\":\"sensor_1005_2_1\",\"subscribedPath\":\"/interfaces/interface/\",\"streamedPath\":\"/interfaces/interface/\",\"component\":\"chassisd\",\"sequenceNumber\":\"770002\",\"payloadGetTimestamp\":\"1723653363502\",\"streamCreationTimestamp\":\"1723653361858\",\"exportTimestamp\":\"1723653363504\",\"streamId\":\"PERIODIC\"}"}}}

Which is then properly rendered to a Prometheus metric:

gnmi_interfaces_interface_state_hardware_port{component="chassisd",componentId="65535",hardware_port="FPC1:PIC0:PORT3",interface_name="et-1/0/3",metric_source="edge01_test01",subscription_name="port_stats",systemId="edge01_test01"} 1

Note that some label-drop rules have been added to remove the spurious labels to avoid a cardinality explosion.

Ichabond avatar Aug 14 '24 16:08 Ichabond

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

google-cla[bot] avatar Aug 14 '24 16:08 google-cla[bot]

This is a proposed solution for #503

Ichabond avatar Aug 14 '24 16:08 Ichabond

Updated the commit to fix the type assertion issue as well as the variable assignment issue.

Ichabond avatar Aug 14 '24 18:08 Ichabond

Anything else needed here @karimra ?

Ichabond avatar Aug 15 '24 08:08 Ichabond

While this works for that junos extension, I'm not sure it will for any other extension. The data in an extension is not always meant to be added as tags, we should find a way to pass it on to the output and use processors to select some keys to set as tags. The other thing I'm not entirely convinced by is that these proto files are set by target. If you have 100s of targets of the same type (same extensions), we will have to parse the proto files 100s of times. This is fine for now, but it should be improved/fixed in a future PR.

karimra avatar Aug 15 '24 22:08 karimra

Hello awesome people. Any movement on this. It would great to be able to do some processing of the attributes from extensions in for example starlark - so we could filter/markup/combine etc as needed.

peejaychilds avatar Mar 31 '25 07:03 peejaychilds

Very interested in this also. Any update ?

triggerhappy17 avatar Oct 18 '25 00:10 triggerhappy17