frr icon indicating copy to clipboard operation
frr copied to clipboard

zebra: flood of trace commits

Open raja-rajasekar opened this issue 2 months ago • 2 comments

check individual commits, but the idea is to cover most of the important places in Zebra where we need traces

co-authored by Nvidia FRR team

Signed-off-by: Rajasekar Raja [email protected]

raja-rajasekar avatar Dec 09 '25 15:12 raja-rajasekar

hmm - "check individual commits" isn't really very useful -the commit has a bare log. could you add an outline of the areas where you're adding traces, at the very least?

mjstapp avatar Dec 09 '25 21:12 mjstapp

hmm - "check individual commits" isn't really very useful -the commit has a bare log. could you add an outline of the areas where you're adding traces, at the very least?

still a WIP, will update when i have all commits ready before i move the PR to ready

raja-rajasekar avatar Dec 09 '25 21:12 raja-rajasekar

ci:rerun

raja-rajasekar avatar Dec 15 '25 16:12 raja-rajasekar

Commit Intf/RIB

ip_prefix_send_to_client

frr_zebra:rib_uninstall_kernel_route {'prefix': 'fe80::/64', 'nhe_id': 47, 'nhe_flags': 515, 'dplane_status': 0}
frr_zebra:if_dplane_ifp_handling {'interface_name': 'r1-eth0-macvlan', 'ifindex': 10, 'location': 'RTM_NEWLINK UPD: Intf has gone Down-2'}
frr_zebra:if_dplane_result {'oper': 'DPLANE_OP_INTF_INSTALL', 'interface_name': 'r1-eth0-macvlan', 'ifindex': 10, 'dplane_result': 'ZEBRA_DPLANE_REQUEST_QUEUED', 'ns_id': 0}
frr_zebra:if_dplane_ifp_handling_new {'interface_name': 'r1-eth0-macvlan', 'ifindex': 10, 'vrf_id': 0, 'zif_type': 5, 'zif_slave_type': 0, 'master_ifindex': 0, 'flags': 4099, 'location': 'RTM_NEWLINK UPD'}
frr_zebra:rib_process_subq_dequeue {'qindex': 2}

frr_zebra:zread_route_add {'api_flag': 65, 'api_msg': 3, 'api_safi': 1, 'nhg_id': 0, 'prefix': '10.100.100.101/32', 'vrf_id': 0, 'nexthops': '10.0.0.44 if 0 (I)'}
frr_zebra:zread_route_del {'api_flag': 65, 'api_msg': 3, 'api_safi': 1, 'prefix': '10.100.100.101/32', 'table_id': 254}

frr_zebra:if_protodown {'interface_name': 'r1-eth0-macvlan', 'ifindex': 10, 'protodown': 1, 'old_bitfield': 0, 'new_bitfield': 16, 'location': 'Intf Update Protodown'}
frr_zebra:if_upd_ctx_dplane_result {'oper': 'DPLANE_OP_INTF_UPDATE', 'interface_name': 'r1-eth0-macvlan', 'ifindex': 10, 'down': 1, 'pd_reason_val': 1, 'location': 'Zebra Inf Upd Success'}

frr_zebra:if_vrf_change {'ifindex': 11, 'vrf_name': 'RED', 'tableid': 0, 'location': 'DPLANE_OP_INTF_DELETE'}
frr_zebra:if_ip_addr_add_del {'ifname': 'r1-eth3', 'address': 'fe80::909c:5fff:fe04:c7d2', 'location': 'RTM_NEWADDR IPv6'}

frr_zebra:if_add_del_update {'vrfid': 11, 'interface_name': 'RED', 'ifindex': 11, 'ifstatus': 13, 'location': 'Interface Index Add'}
frr_zebra:if_dplane_ifp_handling_vrf_change {'interface_name': 'r1-eth0', 'ifindex': 2, 'old_vrf_id': 0, 'vrf_id': 11}

frr_zebra:zsend_redistribute_route {'cmd': 'ZEBRA_REDISTRIBUTE_ROUTE_ADD', 'client_proto': 10, 'api_type': 31, 'vrfid': 0, 'prefix_len': 32, 'nexthops': '172.31.1.10 if 3 (A)'}

frr_zebra:get_iflink_speed {'ifname': 'dummy1', 'error': 19, 'strerr': 'No such device', 'location': 'IOCTL failure to read interface'}

Commit NHG/VTEP

nhg_ctx_process_new_nhe\|zebra_nhg_id_counter_wrapped\|zebra_nhg_intf_lkup_failed\|zread_nhg_del

frr_zebra:zebra_nhg_free_nhe_refcount {'nhe_id': 1501, 'nhe_flags': 7, 'ref_cnt': 0}
frr_zebra:zebra_nhg_uninstall_kernel {'nhe_id': 1502, 'nhe_flags': 3, 'dplane_status': 0}
frr_zebra:zebra_nhg_dep {'nhe_id': 1515, 'dep_id': 1514}
frr_zebra:zebra_nhg_install_kernel {'nhe_id': 1515, 'nhe_flags': 1, 'location': 'Queuing NH ADD'}
frr_zebra:zebra_nhg_dplane_result {'op': 'DPLANE_OP_NH_INSTALL', 'nhe_id': 1515, 'status': 'ZEBRA_DPLANE_REQUEST_SUCCESS'}


frr_zebra:zebra_interface_nhg_reinstall {'ifp': 'r1-eth0', 'ifindex': 2, 'nhe_id': 69, 'nhe_flags': 3, 'location': 'Interface dependent NHE'}
frr_zebra:zebra_nhg_nhe2grp_internal_failure {'depend_id': 241}
frr_zebra:zread_nhg_add {'id': 181818177, 'proto': 'sharp', 'vrf_id': 0, 'if_index': 4, 'type': 3, 'nexthops': '1.1.1.2 if 4 (I) 1.1.1.3 if 3 (I) 1.1.1.4 if 6 (I)'}
frr_zebra:if_down_nhg_dependents {'ifp': 'r1-eth1', 'ifindex': 3, 'nhe_id': 12, 'nhe_flags': 3}

frr_zebra:dplane_vtep_add_del {'ifp': 'vxlan0', 'ifindex': 6, 'vni': 101, 'ip_addr': '10.30.30.30', 'location': 'VTEP ADD'}
frr_zebra:dplane_vtep_add_del {'ifp': 'vxlan48', 'ifindex': 7, 'vni': 1000112, 'ip_addr': 'fd00:0:20::2', 'location': 'VTEP ADD'}
frr_zebra:dplane_vtep_add_del {'ifp': 'vxlan0', 'ifindex': 6, 'vni': 4000, 'ip_addr': '10.27.7.0', 'location': 'VTEP DELETE'}

Commit SRv6

frr_zebra:get_srv6_sid {'ctx_str': 'End.DT46 vrf_id 3 (Vrf10)', 'sid_value': 'fcbb:bbbb:1:1:fe00::', 'locator_name': 'LOC2'}
frr_zebra:get_srv6_sid_explicit {'ctx_str': 'End.DT46 vrf_id 3 (Vrf10)', 'sid_value': 'fcbb:bbbb:1:1:fe00::', 'location': 'Allocated explicit SRv6 SID function'}
frr_zebra:srv6_manager_get_sid_internal {'ctx_str': 'End.DT46 vrf_id 3 (Vrf10)', 'sid_value': 'fcbb:bbbb:1:1:fe00::', 'locator_name': 'LOC2', 'ret': 1, 'location': 'Got new SRv6 SID'}
frr_zebra:srv6_manager_release_sid_internal {'ctx_str': 'End.DT46 vrf_id 3 (Vrf10)', 'locator_name': 'LOC2'}
frr_zebra:release_srv6_sid {'sid_value': 'fcbb:bbbb:1:1:fe00::', 'ctx_str': 'End.DT46 vrf_id 3 (Vrf10)', 'proto': 4, 'instance': 0, 'client_list_count': 1}
frr_zebra:release_srv6_sid_func_explicit {'block_prefix': 'fcbb:bbbb:1::', 'sid_func': 65024}

Commit Netlink

netlink_nexthop_change_err\|netlink_nexthop_msg_encode_err\|netlink_send_msg\|netlink_vrf_change

frr_zebra:netlink_route_multipath_msg_encode {'family': 'AF_INET', 'pfx': '0.0.0.0', 'pfxlen': 0, 'cmd': 24, 'nhg_id': 362, 'nexthops': '1.1.1.2 if 4 (A) 1.1.1.3 if 3 (I) 1.1.1.4 if 6 (A)'}
frr_zebra:netlink_msg_err {'msg_type': 'RTM_NEWADDR', 'data': 0, 'location': 'Invalid/tentative addr'}

frr_zebra:netlink_neigh_update_msg_encode {'mac': '2e:70:e6:b4:50:b5', 'ip': '192:168:100::21', 'nhg': 0, 'flags': 22, 'state': 2, 'family': 'bridge', 'type': 0, 'op': 19}
frr_zebra:netlink_macfdb_change {'nl_msg_type': 'DELNEIGH', 'ndm_ifindex': 7, 'ndm_state': 2, 'ndm_flags': 18, 'nhg': 0, 'vni': 0, 'mac': '1a:b2:fc:15:36:89', 'vtep_ip': '192:168:100::41'}

frr_zebra:netlink_ipneigh_change {'msg_type': 28, 'ndm_family': 10, 'ifindex': 2, 'interface_name': 'r2-eth0', 'vrf_id': 0, 'mac': '52:c5:67:b9:a0:48', 'ip': 'fe80::50c5:67ff:feb9:a048'}
frr_zebra:netlink_intf_err {'ifname': 'vxlan-102', 'ifindex': 7, 'location': 'Cannot find VNI for VID and IF for vlan state update'}
frr_zebra:netlink_nexthop_msg_encode {'nh_index': 2, 'nh_vrfid': 0, 'nhg_id': 39}

Commit Vxlan

zebra_vxlan_sg_send\|zebra_vxlan_sg_new\|zebra_vxlan_sg_del

frr_zebra:zebra_vxlan_remote_vtep_del {'vtep_ip': '10.27.0.0', 'vni': 4000, 'client_proto': 'bgp'}
frr_zebra:zebra_vxlan_remote_macip_del {'vni': 101, 'mac': 'd2:3b:c2:a8:03:bf', 'ip': '10.10.1.55', 'vtep_ip': '10.10.10.10', 'ip_len': 4}
frr_zebra:zebra_vxlan_remote_vtep_add {'vtep_ip': '10.27.0.0', 'vni': 4000, 'flood_control': 1}

frr_zebra:vxlan_vni_state_change {'id': 1, 'vni': 101, 'state': 8, 'zif_name': 'vxlan0'}
frr_zebra:zebra_vxlan_advertise_gw_macip {'advertise': 0, 'vni': 101, 'curr_advertise_gw_macip': 0}

frr_zebra:zebra_vxlan_remote_macip_add {'vni': 101, 'mac': '44:20:00:ff:ff:01', 'ip': 'fe80::4620:ff:feff:ff01', 'vtep_ip': '10.30.30.30', 'flags': 0, 'esi': '00:00:00:00:00:00:00:00:00:00'}
frr_zebra:if_br_vxlan_upd {'interface_name': 'vxlan999', 'ifindex': 8, 'access_vlan_id': 999}

frr_zebra:zebra_vxlan_handle_vni_transition {'vni': 75300, 'location': 'Adding L2-VNI - transition from L3-VNI'}
frr_zebra:zebra_vxlan_handle_vni_transition {'vni': 75100, 'location': 'Del L2-VNI - transition to L3-VNI'}

Commit EVPN

l3vni_remote_rmac_update\|l3vni_remote_vtep_nh_upd\|remote_nh_add_rmac_change\|intf_in_different_ns

frr_zebra:building_vni_table {'interface_type': 'vni', 'interface': 'vx-1000'}
frr_zebra:zevpn_build_l2vni_hash {'vni': 1000, 'interface': 'vx-1000', 'ifindex': 8, 'vtep_ip': '192.168.100.15'}

frr_zebra:zevpn_build_vni_hash {'vni': 999, 'interface': 'vxlan999', 'ifindex': 8, 'location': 'Create l3vni hash'}
frr_zebra:zevpn_build_l3vni_hash {'vni': 999, 'svi_interface': 'vlan999', 'mac_vlan_interface': 'NIL'}
frr_zebra:send_l3vni_oper_to_client {'vrf_id': 6, 'vni': 999, 'location': 'l3vni oper up'}

frr_zebra:evpn_dplane_remote_rmac_add {'rmac': '00:00:00:00:00:00', 'vtep_ip': '10.30.30.30', 'vni': 4000, 'vlan_id': 400, 'vxlan_if': 6}
frr_zebra:evpn_dplane_remote_nh_add {'rmac': '44:20:00:ff:ff:01', 'nh_ip': '10.30.30.30', 'vrf_id': 9, 'ifindex': 10}
frr_zebra:l3vni_remote_rmac {'location': 'Add', 'vni': 100, 'vtep_ip': '10.30.30.30', 'rmac': '44:20:00:ff:ff:01'}
frr_zebra:evpn_dplane_remote_nh_del {'rmac': '00:00:00:00:00:00', 'nh_ip': '::ffff:a1e:1e1e', 'ifindex': 6}
frr_zebra:evpn_dplane_remote_rmac_del {'rmac': '00:00:00:00:00:00', 'vtep_ip': '10.30.30.30', 'vni': 4000, 'vlan_id': 400, 'vxlan_if': 6}

raja-rajasekar avatar Dec 15 '25 18:12 raja-rajasekar

Rajasekar is going to remove the controversial ones and reraise the resulting PR sans the two commits.

donaldsharp avatar Dec 16 '25 16:12 donaldsharp

This pull request has conflicts, please resolve those before we can evaluate the pull request.

github-actions[bot] avatar Dec 19 '25 09:12 github-actions[bot]

Rebased to master

raja-rajasekar avatar Dec 23 '25 16:12 raja-rajasekar

you've got a "do not merge" label - do you want to remove that now and proceed with the PR?

mjstapp avatar Dec 23 '25 17:12 mjstapp

you've got a "do not merge" label - do you want to remove that now and proceed with the PR?

I don’t know how to remove it. Russ added it in last meeting when I said I will have to rework and rebase.. now that it is done, we can remove it

raja-rajasekar avatar Dec 23 '25 17:12 raja-rajasekar

Removing the label, so we can merge

mjstapp avatar Dec 23 '25 17:12 mjstapp