FRRouting
vpn routes are not leaked to rib/ip tables of customer vrf in frr 8.2.2
- FRR VERSION
- 8.2.2
- OPERATING SYSTEM VERSION
- fedora 32
- KERNEL VERSION Linux 72cbea2a6371 5.11.22-100.fc32.x86_64 #1 SMP Wed May 19 18:58:25 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Topology: router1 (loopback - 1.1.1.1/32) AS 10 vrf default ----------- vrf red AS20 -router2 vrf default--------------- vrf default AS 30 router3 vrf red------------ AS 40(loopback 5.5.5.5/32)router4
Description of bug
Router 2 and Router 3 are exchanging vpn routes.Router 2 gets the vpn route 5.5.5.5/32 from Router 3 and Router 3 gets the vpn route 1.1.1.1/32 from Router2.Both the routes are shown as vpn routes in the respective routers. But these routes are not leaked into the ip tables of that customer vrf.
372b055aeaac# show bgp ipv4 vpn BGP table version is 2, local router ID is 0.0.0.2, vrf id 0 Default local pref 100, local AS 20 Status codes: s suppressed, d damped, h history, * valid, > best, = multipath, i internal, r RIB-failure, S Stale, R Removed Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self Origin codes: i - IGP, e - EGP, ? - incomplete RPKI validation codes: V valid, I invalid, N Not found Network Next Hop Metric LocPrf Weight Path Route Distinguisher: 22:22 *> 1.1.1.1/32 10.12.0.1@5< 0 0 10 i UN=10.12.0.1 EC{22:22} label=200 type=bgp, subtype=5 *> 5.5.5.5/32 10.24.0.4 0 0 40 50 i UN=10.24.0.4 EC{22:22} label=200 type=bgp, subtype=0 Displayed 2 routes and 2 total paths 372b055aeaac# show ip bgp vrf red BGP table version is 1, local router ID is 0.0.0.2, vrf id 5 Default local pref 100, local AS 20 Status codes: s suppressed, d damped, h history, * valid, > best, = multipath, i internal, r RIB-failure, S Stale, R Removed Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self Origin codes: i - IGP, e - EGP, ? - incomplete RPKI validation codes: V valid, I invalid, N Not found Network Next Hop Metric LocPrf Weight Path *> 1.1.1.1/32 10.12.0.1 0 0 10 i 5.5.5.5/32 10.24.0.4@0< 0 0 40 50 i Displayed 2 routes and 2 total paths
-->
- [ ] Did you check if this is a duplicate issue? It does not look like duplicate issue
- [ ] Did you test it on the latest FRRouting/frr master branch?
- [ ] No
To Reproduce
Create 4 routers with mpls l3vpn config (Attached the config from 4 routers in the bug report)
CE1-------------PE1-----------PE2-----------------CE2
CE1 advertises 1.1.1.1/32 configured on its loopback to PE1 in vrf red CE2 advertises 5.5.5.5/32 configured on its loopback to PE2 in vrf red PE1 advertises 1.1.1.1/32 as vpn route to PE2 PE2 advertises 5.5.5.5/32 as vpn route to PE1
Expected behavior PE1 shows 5.5.5.5/32 as valid vpn route but it is not marked as valid bgp route and not installed in RIB/FIB. PE2 shows 1.1.1.1/32 as valid vpn route but it is not marked as valid bgp route and not installed in RIB/FIB.
Expected behaviour PE1 should install 5.5.5.5/32 and PE2 should install 1.1.1.1/32 in the RIB/FIB
Note: When "rt vpn import RTLIST" is executed again, the routes are installed in RIB/FIB
What difference I see between 7.5.1 and 8.x is, there were some fixes related to vrf leak, in which nh_valid is being checked and in 7.5.1, for leaking a roue from vpn to vrf it was simple nh installed check and with 8.2.2, it is checked with if nh is valid with a mpls label.So this flow was working in 7.5.1
https://github.com/FRRouting/frr/pull/7936/commits/4053e9520adebc6a56b8cb940e70f07cdb2d8ef0 (commit which added changes to nh validation for mpls l3vpn)
But with 8.2.2, I see nh is not valid, the bgp route is also not marked as valid and not installed in RIB/FIB.
But when the "rt vpn import ..." command is issued again, leak_update again gets executed and somehow makes the route installed. Could not exactly get how this is happening.
Logs during the issue:
Route not leaked to vrf logs: (Issue state)
2022/08/19 15:37:51.653 BGP: [M59KS-A3ZXZ] bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from 10.24.0.4 in vrf default 2022/08/19 15:37:51.653 BGP: [M59KS-A3ZXZ] bgp_update_receive: rcvd End-of-RIB for IPv4 VPN from 10.24.0.4 in vrf default 2022/08/19 15:37:51.653 BGP: [M59KS-A3ZXZ] bgp_update_receive: rcvd End-of-RIB for IPv6 Unicast from 10.24.0.4 in vrf default 2022/08/19 15:37:55.435 BGP: [S82AC-6YAC8] vpn_leak_zebra_vrf_label_update: vrf VRF red: afi IPv4: setting label -1 for vrf id 5 2022/08/19 15:37:55.499 BGP: [S82AC-6YAC8] vpn_leak_zebra_vrf_label_update: vrf VRF red: afi IPv4: setting label -1 for vrf id 5 2022/08/19 15:37:55.607 BGP: [S82AC-6YAC8] vpn_leak_zebra_vrf_label_update: vrf VRF red: afi IPv4: setting label 200 for vrf id 5 2022/08/19 15:37:55.607 BGP: [J47J0-K06GG] Found existing bnc 10.12.0.1/32(VRF red) flags 0x3 ifindex 0 #paths 1 peer 0x7feb43df5010 2022/08/19 15:37:55.607 BGP: [J9M43-9VGVR] vpn_leak_to_vrf_update: start (path_vpn=0x296d730) 2022/08/19 15:37:55.607 BGP: [XPBCM-TXSK8] vpn_leak_to_vrf_update_onevrf: skipping: import not set 2022/08/19 15:37:56.355 BGP: [J9M43-9VGVR] vpn_leak_to_vrf_update: start (path_vpn=0x296bd30) 2022/08/19 15:37:56.355 BGP: [Z20F0-JF945] vpn_leak_to_vrf_update_onevrf: updating 5.5.5.5/32 to vrf VRF red 2022/08/19 15:37:56.355 BGP: [GAK9R-YBKNZ] vpn_leak_to_vrf_update_onevrf: pfx 5.5.5.5/32: num_labels 1 2022/08/19 15:37:56.355 BGP: [RD1WY-YE9EC] leak_update: entry: leak-to=VRF red, p=5.5.5.5/32, type=9, sub_type=0 2022/08/19 15:37:56.355 BGP: [J47J0-K06GG] Found existing bnc 10.24.0.4/32(VRF default) flags 0xb ifindex 0 #paths 0 peer 0x7feb43c9f010 2022/08/19 15:37:56.355 BGP: [Q0G6E-5ECA6] leak_update: nexthop is not valid (in vrf VRF default) 2022/08/19 15:37:56.355 BGP: [Z3E24-0WZCT] leak_update: ->VRF red: 5.5.5.5/32: Added new route 2022/08/19 15:37:56.355 BGP: [XPBCM-TXSK8] vpn_leak_to_vrf_update_onevrf: skipping: import not set 2022/08/19 15:37:56.405 BGP: [QGT2T-9Z0AR] bgp_process_main_one: p=5.5.5.5/32i(VRF default) afi=IPv4, safi=vpn start 2022/08/19 15:37:56.405 BGP: [KQ9MH-8EWT4] 5.5.5.5/32(VRF default): path 10.24.0.4 is the initial bestpath 2022/08/19 15:37:56.405 BGP: [N6CTF-2RSKS] 5.5.5.5/32(VRF default): After path selection, newbest is path 10.24.0.4 oldbest was NONE 2022/08/19 15:37:56.405 BGP: [ZF63D-VT50R] 5.5.5.5/32(VRF default): path 10.24.0.4 is the bestpath, add to the multipath list 2022/08/19 15:37:56.405 BGP: [R1JR3-6FXH0] 5.5.5.5/32(VRF default): starting mpath update, newbest 10.24.0.4 num candidates 1 old-mpath-count 0 old-cum-bw u0 2022/08/19 15:37:56.405 BGP: [HTQ4X-T9P2N] 5.5.5.5/32(VRF default): comparing candidate 10.24.0.4 with existing mpath NONE 2022/08/19 15:37:56.405 BGP: [RP0W2-GRH3C] 5.5.5.5/32(VRF default): New mpath count (incl newbest) 1 mpath-change NO all_paths_lb 0 cum_bw u0 2022/08/19 15:37:56.405 BGP: [GVV1N-MH1P0] bgp_process_main_one: p=5.5.5.5/32(VRF default) afi=IPv4, safi=vpn, old_select=0x0, new_select=0x296bd30 2022/08/19 15:37:56.405 BGP: [RR6EG-07TGC] bgp_process_main_one: setting SELECTED flag 2022/08/19 15:37:56.405 BGP: [QGT2T-9Z0AR] bgp_process_main_one: p=5.5.5.5/32i(VRF red) afi=IPv4, safi=unicast start 2022/08/19 15:37:56.405 BGP: [QZMK5-093W0] bgp_best_selection: pi 0x2967f80 in holddown 2022/08/19 15:37:56.405 BGP: [N6CTF-2RSKS] 5.5.5.5/32(VRF red): After path selection, newbest is NONE oldbest was NONE 2022/08/19 15:37:56.405 BGP: [R1JR3-6FXH0] 5.5.5.5/32(VRF red): starting mpath update, newbest NONE num candidates 0 old-mpath-count 0 old-cum-bw u0 2022/08/19 15:37:56.405 BGP: [GVV1N-MH1P0] bgp_process_main_one: p=5.5.5.5/32(VRF red) afi=IPv4, safi=unicast, old_select=0x0, new_select=0x0
Routes leaked from vpn to vrf after executing the "rt vpn import 22:22" again (non issue state)
2022/08/19 16:06:13.097 BGP: [YC3EK-NZYE3] vpn_leak_to_vrf_withdraw_all: entry 2022/08/19 16:06:13.097 BGP: [Z20F0-JF945] vpn_leak_to_vrf_update_onevrf: updating 5.5.5.5/32 to vrf VRF red 2022/08/19 16:06:13.097 BGP: [GAK9R-YBKNZ] vpn_leak_to_vrf_update_onevrf: pfx 5.5.5.5/32: num_labels 1 2022/08/19 16:06:13.097 BGP: [RD1WY-YE9EC] leak_update: entry: leak-to=VRF red, p=5.5.5.5/32, type=9, sub_type=0 2022/08/19 16:06:13.097 BGP: [J47J0-K06GG] Found existing bnc 10.24.0.4/32(VRF default) flags 0xb ifindex 0 #paths 1 peer 0x7feb43c9f010 2022/08/19 16:06:13.097 BGP: [Q0G6E-5ECA6] leak_update: nexthop is not valid (in vrf VRF default) 2022/08/19 16:06:13.097 BGP: [JRFCJ-R9BZC] leak_update: ->VRF red: 5.5.5.5/32 Found route, changed attr 2022/08/19 16:06:13.147 BGP: [QGT2T-9Z0AR] bgp_process_main_one: p=5.5.5.5/32i(VRF red) afi=IPv4, safi=unicast start 2022/08/19 16:06:13.147 BGP: [KQ9MH-8EWT4] 5.5.5.5/32(VRF red): path 10.24.0.4 is the initial bestpath 2022/08/19 16:06:13.147 BGP: [N6CTF-2RSKS] 5.5.5.5/32(VRF red): After path selection, newbest is path Static announcement oldbest was NONE 2022/08/19 16:06:13.147 BGP: [ZF63D-VT50R] 5.5.5.5/32(VRF red): path Static announcement is the bestpath, add to the multipath list 2022/08/19 16:06:13.147 BGP: [R1JR3-6FXH0] 5.5.5.5/32(VRF red): starting mpath update, newbest Static announcement num candidates 1 old-mpath-count 0 old-cum-bw u0 2022/08/19 16:06:13.147 BGP: [HTQ4X-T9P2N] 5.5.5.5/32(VRF red): comparing candidate Static announcement with existing mpath NONE 2022/08/19 16:06:13.147 BGP: [RP0W2-GRH3C] 5.5.5.5/32(VRF red): New mpath count (incl newbest) 1 mpath-change NO all_paths_lb 0 cum_bw u0 2022/08/19 16:06:13.147 BGP: [GVV1N-MH1P0] bgp_process_main_one: p=5.5.5.5/32(VRF red) afi=IPv4, safi=unicast, old_select=0x0, new_select=0x2967f80 2022/08/19 16:06:13.147 BGP: [RR6EG-07TGC] bgp_process_main_one: setting SELECTED flag 2022/08/19 16:06:13.147 BGP: [X11WM-VBA5R] bgp_zebra_announce: p=5.5.5.5/32, bgp_is_valid_label: 2 2022/08/19 16:06:13.147 BGP: [V15FP-4CPVK] Tx route add VRF 5 5.5.5.5/32 metric 0 tag 0 count 1 nhg 0 2022/08/19 16:06:13.147 BGP: [JQXM8-V0CKB] nhop [1]: 10.24.0.4 if 0 VRF 0 wt 0 label 200 2022/08/19 16:06:13.147 BGP: [JMVGR-VQ1M1] bgp_zebra_announce: 5.5.5.5/32: announcing to zebra (recursion set)
can someone help me on this ? Is this a bug, some notification missing during the flow, and later reconfiguration is taking care of this ?
@pjdruddy @taspelund @donaldsharp Do you see any issues in this flow ? Can you help me this ?
configs:
router 1:
d12d9ba44f39# show running-config Building configuration...
Current configuration: ! frr version 8.2.2 frr defaults traditional hostname d12d9ba44f39 log file /var/log/frr/frr.log log timestamp precision 3 ! password frrpass enable password frrpass ! interface loopback0 ip address 1.1.1.1/32 exit ! interface sw1p0 ip address 10.12.0.1/24 exit ! router bgp 10 bgp router-id 0.0.0.1 no bgp ebgp-requires-policy no bgp suppress-duplicates no bgp default ipv4-unicast no bgp network import-check neighbor 10.12.0.2 remote-as 20 neighbor 10.12.0.2 solo neighbor 10.12.0.2 disable-connected-check neighbor 10.12.0.2 update-source 10.12.0.1 ! address-family ipv4 unicast network 1.1.1.1/32 neighbor 10.12.0.2 activate neighbor 10.12.0.2 filter-list ebgp-export-neigh-10.12.0.2-v4 out exit-address-family ! address-family ipv6 unicast neighbor 10.12.0.2 activate neighbor 10.12.0.2 filter-list ebgp-export-neigh-10.12.0.2-v6 out exit-address-family exit ! bgp as-path access-list ebgp-export-neigh-10.12.0.2-v4 seq 5 permit ^$ bgp as-path access-list ebgp-export-neigh-10.12.0.2-v6 seq 5 permit ^$ ! end d12d9ba44f39#
router 2:
7281b050a6f6# show running-config Building configuration...
Current configuration: ! frr version 8.2.2 frr defaults traditional hostname 7281b050a6f6 log file /var/log/frr/frr.log log timestamp precision 3 ! password frrpass enable password frrpass ! interface sw1p1 ip address 10.24.0.2/24 exit ! interface sw1p0 ip address 10.12.0.2/24 exit ! router bgp 20 vrf red bgp router-id 0.0.0.2 no bgp ebgp-requires-policy no bgp suppress-duplicates no bgp default ipv4-unicast no bgp network import-check neighbor 10.12.0.1 remote-as 10 neighbor 10.12.0.1 solo neighbor 10.12.0.1 disable-connected-check neighbor 10.12.0.1 update-source 10.12.0.2 ! address-family ipv4 unicast neighbor 10.12.0.1 activate label vpn export 200 rd vpn export 22:22 rt vpn both 22:22 export vpn import vpn exit-address-family ! address-family ipv6 unicast neighbor 10.12.0.1 activate exit-address-family exit ! router bgp 20 bgp router-id 0.0.0.2 no bgp ebgp-requires-policy no bgp suppress-duplicates no bgp default ipv4-unicast no bgp network import-check neighbor 10.24.0.4 remote-as 40 neighbor 10.24.0.4 solo neighbor 10.24.0.4 disable-connected-check neighbor 10.24.0.4 update-source 10.24.0.2 ! address-family ipv4 unicast neighbor 10.24.0.4 activate exit-address-family ! address-family ipv4 vpn neighbor 10.24.0.4 activate exit-address-family ! address-family ipv6 unicast neighbor 10.24.0.4 activate exit-address-family exit ! end
router 3:
3a68f27b7cb4# show running-config Building configuration...
Current configuration: ! frr version 8.2.2 frr defaults traditional hostname 3a68f27b7cb4 log file /var/log/frr/frr.log log timestamp precision 3 ! password frrpass enable password frrpass ! interface sw1p0 ip address 10.24.0.4/24 exit ! interface sw1p2 ip address 10.45.0.4/24 exit ! router bgp 40 vrf red bgp router-id 0.0.0.4 no bgp ebgp-requires-policy no bgp suppress-duplicates no bgp default ipv4-unicast no bgp network import-check neighbor 10.45.0.5 remote-as 50 neighbor 10.45.0.5 solo neighbor 10.45.0.5 disable-connected-check neighbor 10.45.0.5 update-source 10.45.0.4 ! address-family ipv4 unicast neighbor 10.45.0.5 activate label vpn export 200 rd vpn export 22:22 rt vpn both 22:22 export vpn import vpn exit-address-family ! address-family ipv6 unicast neighbor 10.45.0.5 activate exit-address-family exit ! router bgp 40 bgp router-id 0.0.0.4 no bgp ebgp-requires-policy no bgp suppress-duplicates no bgp default ipv4-unicast no bgp network import-check neighbor 10.24.0.2 remote-as 20 neighbor 10.24.0.2 solo neighbor 10.24.0.2 disable-connected-check neighbor 10.24.0.2 update-source 10.24.0.4 ! address-family ipv4 unicast neighbor 10.24.0.2 activate exit-address-family ! address-family ipv4 vpn neighbor 10.24.0.2 activate exit-address-family ! address-family ipv6 unicast neighbor 10.24.0.2 activate exit-address-family exit ! end 3a68f27b7cb4#
router 4:
43ada21eb563# show running-config Building configuration...
Current configuration: ! frr version 8.2.2 frr defaults traditional hostname 43ada21eb563 log file /var/log/frr/frr.log log timestamp precision 3 ! password frrpass enable password frrpass ! interface loopback0 ip address 5.5.5.5/32 exit ! interface sw1p0 ip address 10.45.0.5/24 exit ! router bgp 50 bgp router-id 0.0.0.5 no bgp ebgp-requires-policy no bgp suppress-duplicates no bgp default ipv4-unicast no bgp network import-check neighbor 10.45.0.4 remote-as 40 neighbor 10.45.0.4 solo neighbor 10.45.0.4 disable-connected-check neighbor 10.45.0.4 update-source 10.45.0.5 ! address-family ipv4 unicast network 5.5.5.5/32 neighbor 10.45.0.4 activate neighbor 10.45.0.4 filter-list ebgp-export-neigh-10.45.0.4-v4 out exit-address-family ! address-family ipv6 unicast neighbor 10.45.0.4 activate neighbor 10.45.0.4 filter-list ebgp-export-neigh-10.45.0.4-v6 out exit-address-family exit ! bgp as-path access-list ebgp-export-neigh-10.45.0.4-v4 seq 5 permit ^$ bgp as-path access-list ebgp-export-neigh-10.45.0.4-v6 seq 5 permit ^$ ! end 43ada21eb563#
BGP outputs
router 2:
7281b050a6f6# show ip bgp vrf red BGP table version is 1, local router ID is 0.0.0.2, vrf id 5 Default local pref 100, local AS 20 Status codes: s suppressed, d damped, h history, * valid, > best, = multipath, i internal, r RIB-failure, S Stale, R Removed Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self Origin codes: i - IGP, e - EGP, ? - incomplete RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path *> 1.1.1.1/32 10.12.0.1 0 0 10 i 5.5.5.5/32 10.24.0.4@0< 0 0 40 50 i
Displayed 2 routes and 2 total paths 7281b050a6f6# show ip bgp No BGP prefixes displayed, 0 exist 7281b050a6f6# show ip route vrf red Codes: K - kernel route, C - connected, S - static, O - OSPF, B - BGP, T - Table, v - VNC, V - VNC-Direct, f - OpenFabric, > - selected route, * - FIB route, q - queued, r - rejected, b - backup t - trapped, o - offload failure
VRF red: B>* 1.1.1.1/32 [20/0] via 10.12.0.1, sw1p0, weight 1, 00:04:13 C>* 10.12.0.0/24 is directly connected, sw1p0, 00:04:16 7281b050a6f6# show ip route Codes: K - kernel route, C - connected, S - static, O - OSPF, B - BGP, T - Table, v - VNC, V - VNC-Direct, f - OpenFabric, > - selected route, * - FIB route, q - queued, r - rejected, b - backup t - trapped, o - offload failure
C>* 10.24.0.0/24 is directly connected, sw1p1, 00:04:19
7281b050a6f6# show bgp ipv4 vpn BGP table version is 2, local router ID is 0.0.0.2, vrf id 0 Default local pref 100, local AS 20 Status codes: s suppressed, d damped, h history, * valid, > best, = multipath, i internal, r RIB-failure, S Stale, R Removed Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self Origin codes: i - IGP, e - EGP, ? - incomplete RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path Route Distinguisher: 22:22 *> 1.1.1.1/32 10.12.0.1@5< 0 0 10 i UN=10.12.0.1 EC{22:22} label=200 type=bgp, subtype=5 *> 5.5.5.5/32 10.24.0.4 0 0 40 50 i UN=10.24.0.4 EC{22:22} label=200 type=bgp, subtype=0
Displayed 2 routes and 2 total paths
router 3:
3a68f27b7cb4# show ip bgp vrf red BGP table version is 1, local router ID is 0.0.0.4, vrf id 6 Default local pref 100, local AS 40 Status codes: s suppressed, d damped, h history, * valid, > best, = multipath, i internal, r RIB-failure, S Stale, R Removed Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self Origin codes: i - IGP, e - EGP, ? - incomplete RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path 1.1.1.1/32 10.24.0.2@0< 0 0 20 10 i *> 5.5.5.5/32 10.45.0.5 0 0 50 i
Displayed 2 routes and 2 total paths 3a68f27b7cb4# show ip bgp No BGP prefixes displayed, 0 exist 3a68f27b7cb4# show ip route vrf red Codes: K - kernel route, C - connected, S - static, O - OSPF, B - BGP, T - Table, v - VNC, V - VNC-Direct, f - OpenFabric, > - selected route, * - FIB route, q - queued, r - rejected, b - backup t - trapped, o - offload failure
VRF red: B>* 5.5.5.5/32 [20/0] via 10.45.0.5, sw1p2, weight 1, 00:05:26 C>* 10.45.0.0/24 is directly connected, sw1p2, 00:05:30 3a68f27b7cb4# show ip route Codes: K - kernel route, C - connected, S - static, O - OSPF, B - BGP, T - Table, v - VNC, V - VNC-Direct, f - OpenFabric, > - selected route, * - FIB route, q - queued, r - rejected, b - backup t - trapped, o - offload failure
C>* 10.24.0.0/24 is directly connected, sw1p0, 00:05:34
3a68f27b7cb4# show bgp ipv4 vpn BGP table version is 2, local router ID is 0.0.0.4, vrf id 0 Default local pref 100, local AS 40 Status codes: s suppressed, d damped, h history, * valid, > best, = multipath, i internal, r RIB-failure, S Stale, R Removed Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self Origin codes: i - IGP, e - EGP, ? - incomplete RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path Route Distinguisher: 22:22 *> 1.1.1.1/32 10.24.0.2 0 0 20 10 i UN=10.24.0.2 EC{22:22} label=200 type=bgp, subtype=0 *> 5.5.5.5/32 10.45.0.5@6< 0 0 50 i UN=10.45.0.5 EC{22:22} label=200 type=bgp, subtype=5
Displayed 2 routes and 2 total paths 3a68f27b7cb4#
Is this related to specifically 8.2.2 or even 8.0, because you said 7.5.1 - OK, 8.x - NOT OK.
I'm unable to replicate your case:
pe2# sh ip bgp vrf RED 172.16.255.1/32
BGP routing table entry for 172.16.255.1/32, version 2
Paths: (1 available, best #1, vrf RED)
Advertised to non peer-group peers:
192.168.2.1
65000
192.168.2.1 from 192.168.2.1 (192.168.2.1)
Origin incomplete, localpref 50, valid, external, best (First path received)
Last update: Mon Aug 22 13:06:06 2022
pe2# con
pe2(config)# router bgp 65001 vrf RED
pe2(config-router)# do sh run | include rt vpn import
pe2(config-router)# address-family ipv4
pe2(config-router-af)# rt vpn import 192.168.2.2:2 192.168.1.2:2
pe2(config-router-af)# do sh run | include rt vpn import
rt vpn import 192.168.2.2:2 192.168.1.2:2
pe2(config-router-af)# do sh ip bgp vrf RED 172.16.255.1/32
BGP routing table entry for 172.16.255.1/32, version 6
Paths: (2 available, best #1, vrf RED)
Advertised to non peer-group peers:
192.168.2.1
Imported from 192.168.1.2:2:172.16.255.1/32
65000
10.10.10.10 (metric 10) from 0.0.0.0 (192.168.2.2) vrf default(0) announce-nh-self
Origin incomplete, metric 0, localpref 100, valid, sourced, local, best (Local Pref)
Extended Community: RT:192.168.1.2:2
Remote label: 1111
Last update: Mon Aug 22 13:07:03 2022
65000
192.168.2.1 from 192.168.2.1 (192.168.2.1)
Origin incomplete, localpref 50, valid, external
Last update: Mon Aug 22 13:06:06 2022
[no] rt vpn import works fine for me. Do you run ldpd?
@ton31337 Thanks for taking a look at this.
I checked in 8.2.2, But I guess this part of code has many fixes after 7.5.1, so I guess the behavior I see must be seen starting from 8.x.
I am not running ldpd.
Can you also paste your configs on the PE routers?
My configs on PE router:
config connected to CE router
router bgp 20 vrf red bgp router-id 0.0.0.2 no bgp ebgp-requires-policy no bgp suppress-duplicates no bgp default ipv4-unicast no bgp network import-check neighbor 10.12.0.1 remote-as 10 neighbor 10.12.0.1 solo neighbor 10.12.0.1 disable-connected-check neighbor 10.12.0.1 update-source 10.12.0.2 ! address-family ipv4 unicast neighbor 10.12.0.1 activate label vpn export 200 rd vpn export 22:22 rt vpn both 22:22 export vpn import vpn exit-address-family exit !
Config connected to another PE router
router bgp 20 bgp router-id 0.0.0.2 no bgp ebgp-requires-policy no bgp suppress-duplicates no bgp default ipv4-unicast no bgp network import-check neighbor 10.24.0.4 remote-as 40 neighbor 10.24.0.4 solo neighbor 10.24.0.4 disable-connected-check neighbor 10.24.0.4 update-source 10.24.0.2 ! address-family ipv4 unicast neighbor 10.24.0.4 activate exit-address-family ! address-family ipv4 vpn neighbor 10.24.0.4 activate exit-address-family ! address-family ipv6 unicast neighbor 10.24.0.4 activate exit-address-family exit
Any difference in configuration between what we both are trying which is changing the flow ?
Can you show show ip bgp vrf red 1.1.1.1/32? And again, enable debug bgp nht, debug bgp updates and show the logs.
Any difference in configuration between what we both are trying which is changing the flow?
I used MPLS/LDP. The problem is that your next hop is not labeled, and is probably rejected (inaccessible).
31a3dcbeb7ae# show ip bgp vrf red 1.1.1.1/32 BGP routing table entry for 1.1.1.1/32, version 1 Paths: (1 available, best #1, vrf red) Not advertised to any peer 10 10.12.0.1 from 10.12.0.1 (0.0.0.1) Origin IGP, metric 0, valid, external, best (First path received) Last update: Thu Aug 25 14:09:52 2022
Logs:
2022/08/25 14:09:51.084 BGP: [QMD8C-A32H7] Allocated bnc 10.12.0.1/32(0)(VRF red) peer 0x7ff6f01dd010 2022/08/25 14:09:51.084 BGP: [W8SS9-SX6XY] VRF red(5): Rcvd NH update 10.12.0.0/24(0) - metric 0/0 #nhops 1/0 flags Reg 2022/08/25 14:09:51.084 BGP: [HJ3XJ-8EHF0] nhop via if 2 (0 labels) 2022/08/25 14:09:51.084 BGP: [V660C-H2QQR] NH update for 10.12.0.1/32(0)(VRF red) - flags Valid Reg chgflags Changed - evaluate paths 2022/08/25 14:09:51.084 BGP: [G0V2R-JTRQK] parse nexthop update(10.12.0.0/24(0)(VRF red)): bnc info not found for import check 2022/08/25 14:09:51.085 BGP: [J47J0-K06GG] Found existing bnc 10.12.0.1/32(VRF red) flags 0x3 ifindex 0 #paths 0 peer 0x0 2022/08/25 14:09:51.955 BGP: [QMD8C-A32H7] Allocated bnc 10.24.0.4/32(0)(VRF default) peer 0x7ff6f0333010 2022/08/25 14:09:51.955 BGP: [W8SS9-SX6XY] VRF default(0): Rcvd NH update 10.24.0.0/24(0) - metric 0/0 #nhops 1/0 flags Reg 2022/08/25 14:09:51.955 BGP: [HJ3XJ-8EHF0] nhop via if 3 (0 labels) 2022/08/25 14:09:51.955 BGP: [V660C-H2QQR] NH update for 10.24.0.4/32(0)(VRF default) - flags Valid Reg chgflags Changed - evaluate paths 2022/08/25 14:09:51.955 BGP: [MZNH7-XJV7F] evaluate_paths: Updating peer (10.24.0.4(VRF default)) status with NHT nexthops 1 2022/08/25 14:09:51.955 BGP: [G0V2R-JTRQK] parse nexthop update(10.24.0.0/24(0)(VRF default)): bnc info not found for import check 2022/08/25 14:09:51.955 BGP: [J47J0-K06GG] Found existing bnc 10.24.0.4/32(VRF default) flags 0xb ifindex 0 #paths 0 peer 0x7ff6f0333010 2022/08/25 14:09:52.185 BGP: [K9G3A-CJ5QQ] u1:s1 announcing routes upon coalesce timer expiry(1100 ms) 2022/08/25 14:09:52.185 BGP: [K9G3A-CJ5QQ] u2:s2 announcing routes upon coalesce timer expiry(1100 ms) 2022/08/25 14:09:52.185 BGP: [P9VY3-5B6NX] 10.12.0.1 rcvd UPDATE wlen 0 attrlen 0 alen 0 2022/08/25 14:09:52.185 BGP: [G3V0W-FSWRC] bgp_best_path_select_defer: processing route for IPv4 Unicast : cnt 0 2022/08/25 14:09:52.185 BGP: [M59KS-A3ZXZ] bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from 10.12.0.1 in vrf red 2022/08/25 14:09:52.185 BGP: [P9VY3-5B6NX] 10.12.0.1 rcvd UPDATE wlen 0 attrlen 6 alen 0 2022/08/25 14:09:52.185 BGP: [G3V0W-FSWRC] bgp_best_path_select_defer: processing route for IPv6 Unicast : cnt 0 2022/08/25 14:09:52.185 BGP: [M59KS-A3ZXZ] bgp_update_receive: rcvd End-of-RIB for IPv6 Unicast from 10.12.0.1 in vrf red 2022/08/25 14:09:52.878 BGP: [GYHDK-S2BXG] peer 10.24.0.4, EOR_required 0 2022/08/25 14:09:52.878 BGP: [GYHDK-S2BXG] peer 10.24.0.4, EOR_required 0 2022/08/25 14:09:53.055 BGP: [Z2KFR-ZGJC8] 10.12.0.1 rcvd UPDATE w/ attr: nexthop 10.12.0.1, origin i, metric 0, path 10 2022/08/25 14:09:53.055 BGP: [P9VY3-5B6NX] 10.12.0.1 rcvd UPDATE wlen 0 attrlen 28 alen 5 2022/08/25 14:09:53.055 BGP: [W0R38-AAD36] 10.12.0.1 rcvd 1.1.1.1/32 IPv4 unicast 2022/08/25 14:09:53.055 BGP: [J47J0-K06GG] Found existing bnc 10.12.0.1/32(VRF red) flags 0x3 ifindex 0 #paths 0 peer 0x7ff6f0489010 2022/08/25 14:09:53.105 BGP: [K423X-ETGCQ] group_announce_route_walkcb: afi=IPv4, safi=unicast, p=1.1.1.1/32 2022/08/25 14:09:53.105 BGP: [T5JFA-13199] subgroup_process_announce_selected: p=1.1.1.1/32, selected=0x290eca0 2022/08/25 14:09:53.189 BGP: [WJC21-WPGZF] u5:s1 announcing routes to 10.12.0.1, combined into 1 peers 2022/08/25 14:09:53.275 BGP: [WJC21-WPGZF] u6:s2 announcing routes to 10.12.0.1, combined into 1 peers 2022/08/25 14:09:54.878 BGP: [J47J0-K06GG] Found existing bnc 10.24.0.4/32(VRF default) flags 0xb ifindex 0 #paths 0 peer 0x7ff6f0333010 2022/08/25 14:09:54.879 BGP: [J47J0-K06GG] Found existing bnc 10.24.0.4/32(VRF default) flags 0xb ifindex 0 #paths 0 peer 0x7ff6f0333010 2022/08/25 14:09:54.879 BGP: [J47J0-K06GG] Found existing bnc 10.24.0.4/32(VRF default) flags 0xb ifindex 0 #paths 0 peer 0x7ff6f0333010 2022/08/25 14:09:55.980 BGP: [K9G3A-CJ5QQ] u7:s5 announcing routes upon coalesce timer expiry(1100 ms) 2022/08/25 14:09:55.980 BGP: [K9G3A-CJ5QQ] u9:s7 announcing routes upon coalesce timer expiry(1100 ms) 2022/08/25 14:09:55.980 BGP: [K9G3A-CJ5QQ] u8:s6 announcing routes upon coalesce timer expiry(1100 ms) 2022/08/25 14:09:56.031 BGP: [P9VY3-5B6NX] 10.24.0.4 rcvd UPDATE wlen 0 attrlen 0 alen 0 2022/08/25 14:09:56.031 BGP: [G3V0W-FSWRC] bgp_best_path_select_defer: processing route for IPv4 Unicast : cnt 0 2022/08/25 14:09:56.031 BGP: [M59KS-A3ZXZ] bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from 10.24.0.4 in vrf default 2022/08/25 14:09:56.031 BGP: [P9VY3-5B6NX] 10.24.0.4 rcvd UPDATE wlen 0 attrlen 6 alen 0 2022/08/25 14:09:56.031 BGP: [G3V0W-FSWRC] bgp_best_path_select_defer: processing route for IPv4 VPN : cnt 0 2022/08/25 14:09:56.031 BGP: [M59KS-A3ZXZ] bgp_update_receive: rcvd End-of-RIB for IPv4 VPN from 10.24.0.4 in vrf default 2022/08/25 14:09:56.031 BGP: [P9VY3-5B6NX] 10.24.0.4 rcvd UPDATE wlen 0 attrlen 6 alen 0 2022/08/25 14:09:56.031 BGP: [G3V0W-FSWRC] bgp_best_path_select_defer: processing route for IPv6 Unicast : cnt 0 2022/08/25 14:09:56.031 BGP: [M59KS-A3ZXZ] bgp_update_receive: rcvd End-of-RIB for IPv6 Unicast from 10.24.0.4 in vrf default 2022/08/25 14:09:59.451 BGP: [J47J0-K06GG] Found existing bnc 10.12.0.1/32(VRF red) flags 0x3 ifindex 0 #paths 1 peer 0x7ff6f0489010 2022/08/25 14:09:59.501 BGP: [K423X-ETGCQ] group_announce_route_walkcb: afi=IPv4, safi=vpn, p=1.1.1.1/32 2022/08/25 14:09:59.501 BGP: [T5JFA-13199] subgroup_process_announce_selected: p=1.1.1.1/32, selected=0x2913bc0 2022/08/25 14:09:59.501 BGP: [TN0HX-6G1RR] u8:s6 send UPDATE w/ attr: nexthop 0.0.0.0, metric 0, extcommunity RT:22:22, originator 0.0.0.2, path 10 2022/08/25 14:09:59.501 BGP: [H06SA-0JAPR] u8:s6 send MP_REACH for afi/safi IPv4/vpn 2022/08/25 14:09:59.501 BGP: [HVRWP-5R9NQ] u8:s6 send UPDATE RD 22:22 1.1.1.1/32 label 200 IPv4 vpn 2022/08/25 14:09:59.501 BGP: [WEV7K-2GAQ5] u8:s6 send UPDATE len 96 (max message len: 65535) numpfx 1 2022/08/25 14:09:59.501 BGP: [MBFVT-8GSC6] u8:s6 10.24.0.4 send UPDATE w/ nexthop 10.24.0.2 and RD 2022/08/25 14:10:00.004 BGP: [Z2KFR-ZGJC8] 10.24.0.4 rcvd UPDATE w/ attr: , origin i, metric 0, extcommunity RT:22:22, path 40 50 2022/08/25 14:10:00.004 BGP: [P9VY3-5B6NX] 10.24.0.4 rcvd UPDATE wlen 0 attrlen 73 alen 0 2022/08/25 14:10:00.004 BGP: [W0R38-AAD36] 10.24.0.4 rcvd RD 22:22 5.5.5.5/32 label 200 IPv4 vpn 2022/08/25 14:10:00.004 BGP: [J47J0-K06GG] Found existing bnc 10.24.0.4/32(VRF default) flags 0xb ifindex 0 #paths 0 peer 0x7ff6f0333010 2022/08/25 14:10:00.054 BGP: [K423X-ETGCQ] group_announce_route_walkcb: afi=IPv4, safi=vpn, p=5.5.5.5/32 2022/08/25 14:10:00.054 BGP: [T5JFA-13199] subgroup_process_announce_selected: p=5.5.5.5/32, selected=0x2957dc0 2022/08/25 14:10:00.054 BGP: [K423X-ETGCQ] group_announce_route_walkcb: afi=IPv4, safi=unicast, p=5.5.5.5/32
@ton31337 Attached the logs as requested.
Yes you are right, the nexthop should be labeled which is check added in later releases before leaking vpn routes to vrf RIBs, but here nexthop does not become labelled with the configs I have, but if "rt vpn import RTLIST" if executed again, nexthop with label is created.
current nexthop on interface connected on PE1 to PE2 router : (no labelled nexthop): 31a3dcbeb7ae# show interface sw1p1 nexthop-group Interface sw1p1: ID: 13 (zebra) RefCnt: 2 Uptime: 00:05:44 VRF: default Valid, Installed Interface Index: 3 is directly connected, sw1p1 (vrf default) ID: 29 (zebra) RefCnt: 2 Uptime: 00:05:23 VRF: default Valid, Installed Interface Index: 3 is directly connected, sw1p1 (vrf default)
I'm able to replicate this with stable/8.2 (stable/8.3 too), but can't replicate it with the master branch. Are you able to test with master?
We use released branches generally. What is the master branch ? How to use it ?
Is there a fix in master that is solving the issue ? is there a work around or any change in configuration that could fix it ?
I went through the commits in the master branch of bgpd and zebra, not very sure on what would have fixed the issue. Do you see anything that would have fixed this ?
In master, we do not allow using unlabeled next-hops. I'm a bit confused, what are you trying to do at first? Is this a lab or do you use this setup in production with 7.5.1 and upgraded to 8.x and stopped working? I'm confused also how NH becomes labeled after rt vpn import .... Anyone with L3VPN better knowledge can comment here? @pguibert6WIND @louis-6wind
we use 7.5.1 in production and L3VPN is working fine and we want to upgrade to frr 8.x, which I am trying locally and observed mpls l3vpn failing. So our decision to upgrade 8.x in production will be dependent on if L3VPN works or not (atleast some workaround)
I guess when rt vpn import ... is executed again, nh is still invalid, but somehow the route is getting installed in ip table and labelled nexthop is also created after that.Below is the flow when "rt vpn import ..." command is executed again
Routes leaked from vpn to vrf after executing the "rt vpn import 22:22" again (non issue state)
2022/08/19 16:06:13.097 BGP: [YC3EK-NZYE3] vpn_leak_to_vrf_withdraw_all: entry 2022/08/19 16:06:13.097 BGP: [Z20F0-JF945] vpn_leak_to_vrf_update_onevrf: updating 5.5.5.5/32 to vrf VRF red 2022/08/19 16:06:13.097 BGP: [GAK9R-YBKNZ] vpn_leak_to_vrf_update_onevrf: pfx 5.5.5.5/32: num_labels 1 2022/08/19 16:06:13.097 BGP: [RD1WY-YE9EC] leak_update: entry: leak-to=VRF red, p=5.5.5.5/32, type=9, sub_type=0 2022/08/19 16:06:13.097 BGP: [J47J0-K06GG] Found existing bnc 10.24.0.4/32(VRF default) flags 0xb ifindex 0 #paths 1 peer 0x7feb43c9f010 2022/08/19 16:06:13.097 BGP: [Q0G6E-5ECA6] leak_update: nexthop is not valid (in vrf VRF default) 2022/08/19 16:06:13.097 BGP: [JRFCJ-R9BZC] leak_update: ->VRF red: 5.5.5.5/32 Found route, changed attr 2022/08/19 16:06:13.147 BGP: [QGT2T-9Z0AR] bgp_process_main_one: p=5.5.5.5/32i(VRF red) afi=IPv4, safi=unicast start 2022/08/19 16:06:13.147 BGP: [KQ9MH-8EWT4] 5.5.5.5/32(VRF red): path 10.24.0.4 is the initial bestpath 2022/08/19 16:06:13.147 BGP: [N6CTF-2RSKS] 5.5.5.5/32(VRF red): After path selection, newbest is path Static announcement oldbest was NONE 2022/08/19 16:06:13.147 BGP: [ZF63D-VT50R] 5.5.5.5/32(VRF red): path Static announcement is the bestpath, add to the multipath list 2022/08/19 16:06:13.147 BGP: [R1JR3-6FXH0] 5.5.5.5/32(VRF red): starting mpath update, newbest Static announcement num candidates 1 old-mpath-count 0 old-cum-bw u0 2022/08/19 16:06:13.147 BGP: [HTQ4X-T9P2N] 5.5.5.5/32(VRF red): comparing candidate Static announcement with existing mpath NONE 2022/08/19 16:06:13.147 BGP: [RP0W2-GRH3C] 5.5.5.5/32(VRF red): New mpath count (incl newbest) 1 mpath-change NO all_paths_lb 0 cum_bw u0 2022/08/19 16:06:13.147 BGP: [GVV1N-MH1P0] bgp_process_main_one: p=5.5.5.5/32(VRF red) afi=IPv4, safi=unicast, old_select=0x0, new_select=0x2967f80 2022/08/19 16:06:13.147 BGP: [RR6EG-07TGC] bgp_process_main_one: setting SELECTED flag 2022/08/19 16:06:13.147 BGP: [X11WM-VBA5R] bgp_zebra_announce: p=5.5.5.5/32, bgp_is_valid_label: 2 2022/08/19 16:06:13.147 BGP: [V15FP-4CPVK] Tx route add VRF 5 5.5.5.5/32 metric 0 tag 0 count 1 nhg 0 2022/08/19 16:06:13.147 BGP: [JQXM8-V0CKB] nhop [1]: 10.24.0.4 if 0 VRF 0 wt 0 label 200 2022/08/19 16:06:13.147 BGP: [JMVGR-VQ1M1] bgp_zebra_announce: 5.5.5.5/32: announcing to zebra (recursion set)
When you say you are able to reproduce the issue in 8.x and not able to reproduce the issue in master, it means some commit fixed the issue after 8.3 release? Can we wait till next frr release, so that L3VPN works (vpn routes will be leaked to ip vrf table) ?
and where is the guarantee that the package with this bug did not get into debian?
my seventh version still crashes when:
sh ip ospf interface
from debian 11
When you say you are able to reproduce the issue in 8.x and not able to reproduce the issue in master, it means some commit fixed the issue after 8.3 release? Can we wait till next frr release, so that L3VPN works (vpn routes will be leaked to ip vrf table) ?
Route leaking is not working well in current releases. There are some fixes from https://github.com/FRRouting/frr/pull/11127. Could you test with those patches ?
@rera1712 do you care actually about a static imported routes (network x.x.x.x)?
https://github.com/FRRouting/frr/pull/11127 fixes are not just about static imported routes (network x.x.x.x)
Unfortunately, I have no time to look at the issue in the following days.
When you say you are able to reproduce the issue in 8.x and not able to reproduce the issue in master, it means some commit fixed the issue after 8.3 release? Can we wait till next frr release, so that L3VPN works (vpn routes will be leaked to ip vrf table) ?
Route leaking is not working well in current releases. There are some fixes from #11127. Could you test with those patches ?
@louis-6wind Generally we use released versions to test, will check how to integrate the patch with our code and test. I hope then L3VPN issues should be fixed in next releases correct ? In that case can we wait until next release ?
@rera1712 do you care actually about a static imported routes (network x.x.x.x)?
@ton31337 Understood your question, yes the fixes Louis pointed out, mentioned about importing the routes which are mentioned in network statement, but the issue I see is related to routes present in vpn table imported from PE routers and not leaked to vrf ip table
@louis-6wind Went through the patch again #11127, I doubt it will fix the problem I mentioned in this thread.
@louis-6wind Are there any workarounds for this issue in general ?
@pguibert6WIND @louis-6wind do you have any idea how unlabeled next-hop becomes labeled when entering rt vpn import ... twice? (in versions <= 8.2)
@ton31337 @pguibert6WIND I see similar issue reported by others https://github.com/FRRouting/frr/issues/9251 and I see some recent changes merged in master for fixing this. https://github.com/FRRouting/frr/commit/340ed5f9e20871bd8ae391cb06432138e1fa11b7
There was also a fix mentioned as https://github.com/FRRouting/frr/pull/10319/files
Can you please confirm if above fixes will be available in next release ? may be 8.3.2 ?
I tested in 8.3.1 release and issue still exists.
I see the issue is fixed with the patch https://github.com/FRRouting/frr/commit/340ed5f9e20871bd8ae391cb06432138e1fa11b7
@ton31337 Yes I tried that patch and it works, Will it be available in next release ? (8.3.2 ?)