Tuesday, October 6, 2015

Multiple external networks with a single L3 agent testing on RDO Liberty per Lars Kellogg-Stedman

UPDATE 11/25/2015
  My apology  to Lars Kellog-Stedman, posting this I didn't know about
  http://blog.oddbit.com/2015/08/13/provider-external-networks-details/
END UPDATE

Following bellow is supposed to test in multi node environment
Multiple external networks with a single L3 agent by Lars Kellogg-Stedman

However, current post contains an attempt to analyze and understand how traffic to/from external network flows through br-int when provider external networks has been involved

I was also hit by  Bug  neutron-openvswitch-agent is crashing with "invalid literal for int() with base 10" error
and patch https://review.openstack.org/#/c/225001/   was also applied

Basic 3 VM node setup was done per https://www.linux.com/community/blogs/133-general-linux/854587-rdo-liberty-beta-set-up-for-three-vm-nodes-controllernetworkcompute-ml2aovsavxlan-on-centos71/

Nested KVM was enable for all VM hosting RDO Liberty nodes.

Create to two Libvirt sub-nets external3,external4 on KVM Virtualization Host (F22)

[root@fedora22wksr ~]# cat external3.xml
<network>
   <name>external3</name>
   <uuid>d0e9964b-f95d-40c2-b749-b609aed52cf2</uuid>
   <forward mode='nat'>
     <nat>
       <port start='1024' end='65535'/>
     </nat>
   </forward>
   <bridge name='virbr6' stp='on' delay='0' />
   <mac address='52:54:00:60:f8:6d'/>
   <ip address='10.3.0.1' netmask='255.255.255.0'>
     <dhcp>
       <range start='10.3.0.1' end='10.3.0.254' />
     </dhcp>
   </ip>
</network>
[root@fedora22wksr ~]# cat external4.xml
<network>
   <name>external4</name>
   <uuid>d0e9964b-f97d-40c2-b749-b609aed52cf2</uuid>
   <forward mode='nat'>
     <nat>
       <port start='1024' end='65535'/>
     </nat>
   </forward>
   <bridge name='virbr7' stp='on' delay='0' />
   <mac address='52:54:00:60:f8:6d'/>
   <ip address='10.4.0.1' netmask='255.255.255.0'>
     <dhcp>
       <range start='10.4.0.1' end='10.4.0.254' />
     </dhcp>
   </ip>
</network>

Shutdown VM hosting Network Node and add two VNICs eth3 belongs
external3 , eth4 belongs  external4
Startup VM and create corresponding files ifcfg-eth3,ifcfg-eth4 with static
IP addresses.

# service network restart

or reboot Nerwork Node.

*************************
On Network Node
*************************
# ovs-vsctl add-br br-eth3
# ovs-vsctl add-port br-eth3 eth3
# ovs-vsctl add-br br-eth4
# ovs-vsctl add-port br-eth4 eth4

******************************
Update l3_agent.ini file
******************************
external_network_bridge =
gateway_external_network_id =

***********************************************************************
Update /etc/neutron/plugins/ml2/openvswitch_agent.ini
***********************************************************************
[ovs]
network_vlan_ranges = physnet3,physnet4
bridge_mappings = physnet3:br-eth3,physnet4:br-eth4

Then copy  /etc/neutron/plugins/ml2/openvswitch_agent.ini
to /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini

************************************************************************
SSH to Controller 192.169.142.127 and update ml2_conf.ini
************************************************************************
[ml2]
type_drivers = local,flat,gre,vxlan

[ml2_type_flat]
flat_networks = *

# openstack-service restart on Controller

**********************************************************
Get back to VM hosting Network Node
**********************************************************
# openstack-service restart neutron
# systemctl | grep neutron

[root@ip-192-169-142-147 ~]# systemctl| grep neutron
neutron-dhcp-agent.service                                                          loaded active running   OpenStack Neutron DHCP Agent
neutron-l3-agent.service                                                            loaded active running   OpenStack Neutron Layer 3 Agent
neutron-metadata-agent.service                                                      loaded active running   OpenStack Neutron Metadata Agent
neutron-openvswitch-agent.service                                                   loaded active running   OpenStack Neutron Open vSwitch Agent
neutron-ovs-cleanup.service                                                         loaded active exited    OpenStack Neutron Open vSwitch Cleanup Utility

****************************************
External networks creation
****************************************
# source keystonerc_admin
# neutron net-create external3 -- --router:external  \
  --provider:network_type=flat \
  --provider:physical_network=physnet3

# neutron net-create external4 -- --router:external  \
  --provider:network_type=flat \
  --provider:physical_network=physnet4

# neutron subnet-create --disable-dhcp external3 10.3.0.0/24
# neutron subnet-create --disable-dhcp external4 10.4.0.0/24
*************************************************
Then login as demo and create
*************************************************
RouterExt3 with gateway to external3
RouterExt4 with gateway to external4

Then create private networks demo-network4,demo_network5
Attach first to RouterExt4 , second to RouterExt3


 
    
 
 On Network Node
 
[root@ip-192-169-142-147 ~(keystone_admin)]# neutron router-list| grep Ext
| 1e9bad93-2d5d-43fc-aed0-fc3745fe4d10 | RouterExt3 | {"network_id": "fffafde8-c6eb-4b20-b26d-63944300a6bf", "enable_snat": true, "external_fixed_ips": [{"subnet_id": "a038ae4d-9ea7-466e-bf4e-fab65981151c", "ip_address": "10.3.0.2"}]}     | False       | False |
| f47a87d9-c789-47a8-bdb1-8117990c49be | RouterExt4 | {"network_id": "2130df5b-5483-4cb8-a6b6-2a32eb7d882a", "enable_snat": true, "external_fixed_ips": [{"subnet_id": "eea125fc-6236-4570-9d3e-f4489671d2bb", "ip_address": "10.4.0.2"}]}     | False       | False |
[root@ip-192-169-142-147 ~(keystone_admin)]# ip netns exec qrouter-1e9bad93-2d5d-43fc-aed0-fc3745fe4d10 ifconfig
lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 0  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

qg-615baaa8-a6: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.3.0.2  netmask 255.255.255.0  broadcast 10.3.0.255
        inet6 fe80::f816:3eff:fea7:98be  prefixlen 64  scopeid 0x20<link>
        ether fa:16:3e:a7:98:be  txqueuelen 0  (Ethernet)
        RX packets 810478  bytes 1101227298 (1.0 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 449476  bytes 34585959 (32.9 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

qr-45110e77-5b: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 160.0.0.1  netmask 255.255.255.0  broadcast 160.0.0.255
        inet6 fe80::f816:3eff:fe55:5c68  prefixlen 64  scopeid 0x20<link>
        ether fa:16:3e:55:5c:68  txqueuelen 0  (Ethernet)
        RX packets 449433  bytes 34589519 (32.9 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 810390  bytes 1101224102 (1.0 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@ip-192-169-142-147 ~(keystone_admin)]# ip netns exec qrouter-f47a87d9-c789-47a8-bdb1-8117990c49be  ifconfig
lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 0  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

qg-54aa0373-dd: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.4.0.2  netmask 255.255.255.0  broadcast 10.4.0.255
        inet6 fe80::f816:3eff:fe2e:35ee  prefixlen 64  scopeid 0x20<link>
        ether fa:16:3e:2e:35:ee  txqueuelen 0  (Ethernet)
        RX packets 802750  bytes 1088213425 (1.0 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 447723  bytes 34699912 (33.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

qr-a99aa111-1d: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 150.0.0.1  netmask 255.255.255.0  broadcast 150.0.0.255
        inet6 fe80::f816:3eff:fe9b:3a9a  prefixlen 64  scopeid 0x20<link>
        ether fa:16:3e:9b:3a:9a  txqueuelen 0  (Ethernet)
        RX packets 448277  bytes 34759884 (33.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 802834  bytes 1088249558 (1.0 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
 
***************************************************************************************************
Notice that qg-xxxxxxx interfaces from both qrouter-namespaces are attached to br-int
While using provider external networks,traffic to/from external network flows through br-int. 
br-int and br-eth3  will be connected using veth pair int-br-eth3 and phy-br-eth3. 
br-int and br-eth4 will be connected using  veth pair int-br-eth4 and phy-br-eth4. 
This will be automatically created by neutron-openvswitch-agent 
based on the bridge_mappings configured earlier. 
***************************************************************************************************
[root@ip-192-169-142-147 ~(keystone_admin)]# ovs-vsctl show
38e920e3-da61-4a1b-876a-052a49d777a2
    Bridge br-tun
        fail_mode: secure
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "vxlan-0a000089"
            Interface "vxlan-0a000089"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="10.0.0.147", out_key=flow, remote_ip="10.0.0.137"}
        Port br-tun
            Interface br-tun
                type: internal
    Bridge "br-eth4"
        Port "br-eth4"
            Interface "br-eth4"
                type: internal
        Port "phy-br-eth4"
            Interface "phy-br-eth4"
                type: patch
                options: {peer="int-br-eth4"}
        Port "eth4"
            Interface "eth4"
    Bridge br-int
        fail_mode: secure
        Port "tap709fbf6f-ab"
            tag: 13
            Interface "tap709fbf6f-ab"
                type: internal
        Port br-int
            Interface br-int
                type: internal
        Port "qr-a99aa111-1d"
            tag: 13
            Interface "qr-a99aa111-1d"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "qg-54aa0373-dd"
            tag: 14
            Interface "qg-54aa0373-dd"
                type: internal
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
        Port "int-br-eth3"
            Interface "int-br-eth3"
                type: patch
                options: {peer="phy-br-eth3"}
        Port "qg-615baaa8-a6"
            tag: 15
            Interface "qg-615baaa8-a6"
                type: internal
        Port "tap06adaf37-d4"
            tag: 17
            Interface "tap06adaf37-d4"
                type: internal
        Port "qr-45110e77-5b"
            tag: 17
            Interface "qr-45110e77-5b"
                type: internal
        Port "int-br-eth4"
            Interface "int-br-eth4"
                type: patch
                options: {peer="phy-br-eth4"}
    Bridge br-ex
        Port "eth2"
            Interface "eth2"
        Port phy-br-ex
            Interface phy-br-ex
                type: patch
                options: {peer=int-br-ex}
        Port br-ex
            Interface br-ex
                type: internal
    Bridge "br-eth3"
        Port "eth3"
            Interface "eth3"
        Port "phy-br-eth3"
            Interface "phy-br-eth3"
                type: patch
                options: {peer="int-br-eth3"}
        Port "br-eth3"
            Interface "br-eth3"
                type: internal
    ovs_version: "2.3.1" 
 
     
    
 
 

No comments:

Post a Comment