Dataplane not working for BGP in Kubernetes cluster

[danielcra]

Description

Contiv 1.1.7 as CNI plugin in Kubernetes v1.10.2 Contiv in routing mode with VLAN dataplane. Peering Contiv with external BGP speaker; all routes mutually exchanged. However, from within a pod I cannot ping external IP addresses (external to the k8s cluster). No ICMP packets seem to be leaving the compute host. Flow table in OVS seems wrong.

Expected Behavior

Ping should work.

Observed Behavior

Problem seems to be with the OpenFlow rules that point to OF port 0. Zero is an invalid port number AFAIK and does not exist in the dataplane (printout below). Packets get dropped. See action "output:0" below in flow entry " table=7, n_packets=106, n_bytes=10388".

sudo ovs-ofctl -OOpenFlow13 dump-flows contivVlanBridge | cut -d, -f3-
OFPST_FLOW reply (OF1.3) (xid=0x2):
 table=0, n_packets=0, n_bytes=0, priority=300,arp,dl_src=00:00:11:11:11:11 actions=goto_table:7
 table=0, n_packets=0, n_bytes=0, priority=102,udp,in_port=1,tp_dst=53 actions=goto_table:1
 table=0, n_packets=0, n_bytes=0, priority=101,udp,dl_vlan=4093,dl_src=02:02:00:00:00:00/ff:ff:00:00:00:00,tp_dst=53 actions=pop_vlan,goto_table:1
 table=0, n_packets=0, n_bytes=0, priority=100,dl_src=01:00:00:00:00:00/01:00:00:00:00:00 actions=drop
 table=0, n_packets=0, n_bytes=0, priority=100,udp,dl_src=02:02:00:00:00:00/ff:ff:00:00:00:00,tp_dst=53 actions=CONTROLLER:65535
 table=0, n_packets=4, n_bytes=168, priority=100,arp actions=CONTROLLER:65535
 table=0, n_packets=703, n_bytes=46654, priority=1 actions=goto_table:1
 table=1, n_packets=570, n_bytes=34210, priority=100,in_port=1 actions=goto_table:6
 table=1, n_packets=14, n_bytes=1068, priority=10,in_port=2 actions=write_metadata:0x100000000/0xff00000000,goto_table:3
 table=1, n_packets=118, n_bytes=11316, priority=10,in_port=3 actions=write_metadata:0x100000000/0xff00000000,goto_table:3
 table=1, n_packets=1, n_bytes=60, priority=1 actions=drop
 table=3, n_packets=0, n_bytes=0, priority=10,tcp,nw_dst=10.96.0.1 actions=CONTROLLER:65535
 table=3, n_packets=0, n_bytes=0, priority=10,udp,nw_dst=10.96.0.10 actions=CONTROLLER:65535
 table=3, n_packets=0, n_bytes=0, priority=10,tcp,nw_dst=10.96.0.10 actions=CONTROLLER:65535
 table=3, n_packets=132, n_bytes=12384, priority=1 actions=goto_table:4
 table=4, n_packets=0, n_bytes=0, priority=100,ip,metadata=0x100000000/0xff00000000,nw_dst=20.1.8.2 actions=write_metadata:0/0xfffe,goto_table:5
 table=4, n_packets=132, n_bytes=12384, priority=1 actions=goto_table:5
 table=5, n_packets=132, n_bytes=12384, priority=1 actions=goto_table:6
 table=6, n_packets=702, n_bytes=46594, priority=1 actions=goto_table:7
 table=7, n_packets=0, n_bytes=0, priority=103,ip,dl_vlan=1,nw_dst=10.8.0.2 actions=set_field:00:00:11:11:11:11->eth_src,set_field:7a:0c:2d:e9:86:27->eth_dst,pop_vlan,output:2
 table=7, n_packets=0, n_bytes=0, priority=103,ip,dl_vlan=8,nw_dst=20.1.8.2 actions=set_field:00:00:11:11:11:11->eth_src,set_field:02:02:14:01:08:02->eth_dst,pop_vlan,output:3
 table=7, n_packets=0, n_bytes=0, priority=102,ip,nw_dst=10.8.0.2 actions=set_field:00:00:11:11:11:11->eth_src,set_field:7a:0c:2d:e9:86:27->eth_dst,output:2
 table=7, n_packets=106, n_bytes=10388, priority=101,ip,nw_dst=10.0.1.2 actions=set_field:34:07:fb:37:57:0e->eth_src,set_field:00:00:00:00:00:00->eth_dst,output:0
 table=7, n_packets=0, n_bytes=0, priority=101,ip,nw_dst=10.0.1.3 actions=set_field:34:07:fb:37:57:0e->eth_src,set_field:00:00:00:00:00:00->eth_dst,output:0
 table=7, n_packets=0, n_bytes=0, priority=102,ip,nw_dst=20.1.8.2 actions=set_field:00:00:11:11:11:11->eth_src,set_field:02:02:14:01:08:02->eth_dst,output:3
 table=7, n_packets=596, n_bytes=36206, priority=1 actions=drop

sudo ovs-ofctl show contivVlanBridge
OFPT_FEATURES_REPLY (xid=0x2): dpid:00003407fb37570e
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst
 1(eth0): addr:24:17:fb:37:23:0e
     config:     0
     state:      0
     current:    10GB-FD AUTO_NEG
     advertised: 1GB-FD 10GB-FD FIBER AUTO_NEG AUTO_PAUSE
     supported:  1GB-FD 10GB-FD FIBER AUTO_NEG AUTO_PAUSE
     speed: 10000 Mbps now, 10000 Mbps max
 2(inb01): addr:2a:17:2d:e9:36:27
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 3(vvport1): addr:4a:f2:8b:94:5e:a3
     config:     0
     state:      0
     current:    10GB-FD COPPER
     speed: 10000 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

Steps to Reproduce (for bugs)

1. sudo kubeadm init
2. sudo ./install/k8s/install.sh -w routing -v eth0
3. netctl net create -t default --encap="vlan" --pkt-tag 8 --subnet=20.1.8.0/24 -g 20.1.8.1 mynet
4. kubectl create -f busybox.yaml (this yaml binding to mynet)
5. kubectl exec -ti busybox -- sh
6. ping 10.0.1.2

Your Environment

netctl version
Client Version:
Version: 1.1.7
GitCommit: dc27f2c
BuildTime: 11-14-2017.17-45-23.UTC

Server Version:
Version: 1.1.7
GitCommit: dc27f2c
BuildTime: 11-14-2017.17-45-23.UTC

kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:22:21Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:10:24Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

cat /etc/issue
Debian GNU/Linux 9 \n \l

Question: is this supposed to work or has it never been implemented and tested?