UPDATE 11/25/2015
My apology to Lars Kellog-Stedman, posting this I didn't know about
http://blog.oddbit.com/2015/08/13/provider-external-networks-details/
END UPDATE
Following bellow is supposed to test in multi node environment
Multiple external networks with a single L3 agent by Lars Kellogg-Stedman
However, current post contains an attempt to analyze and understand how traffic to/from external network flows through br-int when provider external networks has been involved
I was also hit by Bug neutron-openvswitch-agent is crashing with "invalid literal for int() with base 10" error
and patch https://review.openstack.org/#/c/225001/ was also applied
Basic 3 VM node setup was done per https://www.linux.com/community/blogs/133-general-linux/854587-rdo-liberty-beta-set-up-for-three-vm-nodes-controllernetworkcompute-ml2aovsavxlan-on-centos71/
Nested KVM was enable for all VM hosting RDO Liberty nodes.
Create to two Libvirt sub-nets external3,external4 on KVM Virtualization Host (F22)
[root@fedora22wksr ~]# cat external3.xml
<network>
<name>external3</name>
<uuid>d0e9964b-f95d-40c2-b749-b609aed52cf2</uuid>
<forward mode='nat'>
<nat>
<port start='1024' end='65535'/>
</nat>
</forward>
<bridge name='virbr6' stp='on' delay='0' />
<mac address='52:54:00:60:f8:6d'/>
<ip address='10.3.0.1' netmask='255.255.255.0'>
<dhcp>
<range start='10.3.0.1' end='10.3.0.254' />
</dhcp>
</ip>
</network>
[root@fedora22wksr ~]# cat external4.xml
<network>
<name>external4</name>
<uuid>d0e9964b-f97d-40c2-b749-b609aed52cf2</uuid>
<forward mode='nat'>
<nat>
<port start='1024' end='65535'/>
</nat>
</forward>
<bridge name='virbr7' stp='on' delay='0' />
<mac address='52:54:00:60:f8:6d'/>
<ip address='10.4.0.1' netmask='255.255.255.0'>
<dhcp>
<range start='10.4.0.1' end='10.4.0.254' />
</dhcp>
</ip>
</network>
Shutdown VM hosting Network Node and add two VNICs eth3 belongs
external3 , eth4 belongs external4
Startup VM and create corresponding files ifcfg-eth3,ifcfg-eth4 with static
IP addresses.
# service network restart
or reboot Nerwork Node.
*************************
On Network Node
*************************
# ovs-vsctl add-br br-eth3
# ovs-vsctl add-port br-eth3 eth3
# ovs-vsctl add-br br-eth4
# ovs-vsctl add-port br-eth4 eth4
******************************
Update l3_agent.ini file
******************************
external_network_bridge =
***********************************************************************
Update /etc/neutron/plugins/ml2/openvswitch_agent.ini
***********************************************************************
[ovs]
network_vlan_ranges = physnet3,physnet4
bridge_mappings = physnet3:br-eth3,physnet4:br-eth4
Then copy /etc/neutron/plugins/ml2/openvswitch_agent.ini
to /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini
************************************************************************
SSH to Controller 192.169.142.127 and update ml2_conf.ini
************************************************************************
[ml2]
type_drivers = local,flat,gre,vxlan
[ml2_type_flat]
flat_networks = *
# openstack-service restart on Controller
**********************************************************
Get back to VM hosting Network Node
**********************************************************
# openstack-service restart neutron
# systemctl | grep neutron
[root@ip-192-169-142-147 ~]# systemctl| grep neutron
neutron-dhcp-agent.service loaded active running OpenStack Neutron DHCP Agent
neutron-l3-agent.service loaded active running OpenStack Neutron Layer 3 Agent
neutron-metadata-agent.service loaded active running OpenStack Neutron Metadata Agent
neutron-openvswitch-agent.service loaded active running OpenStack Neutron Open vSwitch Agent
neutron-ovs-cleanup.service loaded active exited OpenStack Neutron Open vSwitch Cleanup Utility
****************************************
External networks creation
****************************************
# source keystonerc_admin
# neutron net-create external3 -- --router:external \
--provider:network_type=flat \
--provider:physical_network=physnet3
# neutron net-create external4 -- --router:external \
--provider:network_type=flat \
--provider:physical_network=physnet4
# neutron subnet-create --disable-dhcp external3 10.3.0.0/24
# neutron subnet-create --disable-dhcp external4 10.4.0.0/24
*************************************************
Then login as demo and create
*************************************************
RouterExt3 with gateway to external3
RouterExt4 with gateway to external4
Then create private networks demo-network4,demo_network5
Attach first to RouterExt4 , second to RouterExt3
My apology to Lars Kellog-Stedman, posting this I didn't know about
http://blog.oddbit.com/2015/08/13/provider-external-networks-details/
END UPDATE
Following bellow is supposed to test in multi node environment
Multiple external networks with a single L3 agent by Lars Kellogg-Stedman
However, current post contains an attempt to analyze and understand how traffic to/from external network flows through br-int when provider external networks has been involved
I was also hit by Bug neutron-openvswitch-agent is crashing with "invalid literal for int() with base 10" error
and patch https://review.openstack.org/#/c/225001/ was also applied
Basic 3 VM node setup was done per https://www.linux.com/community/blogs/133-general-linux/854587-rdo-liberty-beta-set-up-for-three-vm-nodes-controllernetworkcompute-ml2aovsavxlan-on-centos71/
Nested KVM was enable for all VM hosting RDO Liberty nodes.
Create to two Libvirt sub-nets external3,external4 on KVM Virtualization Host (F22)
[root@fedora22wksr ~]# cat external3.xml
<network>
<name>external3</name>
<uuid>d0e9964b-f95d-40c2-b749-b609aed52cf2</uuid>
<forward mode='nat'>
<nat>
<port start='1024' end='65535'/>
</nat>
</forward>
<bridge name='virbr6' stp='on' delay='0' />
<mac address='52:54:00:60:f8:6d'/>
<ip address='10.3.0.1' netmask='255.255.255.0'>
<dhcp>
<range start='10.3.0.1' end='10.3.0.254' />
</dhcp>
</ip>
</network>
[root@fedora22wksr ~]# cat external4.xml
<network>
<name>external4</name>
<uuid>d0e9964b-f97d-40c2-b749-b609aed52cf2</uuid>
<forward mode='nat'>
<nat>
<port start='1024' end='65535'/>
</nat>
</forward>
<bridge name='virbr7' stp='on' delay='0' />
<mac address='52:54:00:60:f8:6d'/>
<ip address='10.4.0.1' netmask='255.255.255.0'>
<dhcp>
<range start='10.4.0.1' end='10.4.0.254' />
</dhcp>
</ip>
</network>
Shutdown VM hosting Network Node and add two VNICs eth3 belongs
external3 , eth4 belongs external4
Startup VM and create corresponding files ifcfg-eth3,ifcfg-eth4 with static
IP addresses.
# service network restart
or reboot Nerwork Node.
*************************
On Network Node
*************************
# ovs-vsctl add-br br-eth3
# ovs-vsctl add-port br-eth3 eth3
# ovs-vsctl add-br br-eth4
# ovs-vsctl add-port br-eth4 eth4
******************************
Update l3_agent.ini file
******************************
external_network_bridge =
gateway_external_network_id =***********************************************************************
Update /etc/neutron/plugins/ml2/openvswitch_agent.ini
***********************************************************************
[ovs]
network_vlan_ranges = physnet3,physnet4
bridge_mappings = physnet3:br-eth3,physnet4:br-eth4
Then copy /etc/neutron/plugins/ml2/openvswitch_agent.ini
to /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini
************************************************************************
SSH to Controller 192.169.142.127 and update ml2_conf.ini
************************************************************************
[ml2]
type_drivers = local,flat,gre,vxlan
[ml2_type_flat]
flat_networks = *
# openstack-service restart on Controller
**********************************************************
Get back to VM hosting Network Node
**********************************************************
# openstack-service restart neutron
# systemctl | grep neutron
[root@ip-192-169-142-147 ~]# systemctl| grep neutron
neutron-dhcp-agent.service loaded active running OpenStack Neutron DHCP Agent
neutron-l3-agent.service loaded active running OpenStack Neutron Layer 3 Agent
neutron-metadata-agent.service loaded active running OpenStack Neutron Metadata Agent
neutron-openvswitch-agent.service loaded active running OpenStack Neutron Open vSwitch Agent
neutron-ovs-cleanup.service loaded active exited OpenStack Neutron Open vSwitch Cleanup Utility
****************************************
External networks creation
****************************************
# source keystonerc_admin
# neutron net-create external3 -- --router:external \
--provider:network_type=flat \
--provider:physical_network=physnet3
# neutron net-create external4 -- --router:external \
--provider:network_type=flat \
--provider:physical_network=physnet4
# neutron subnet-create --disable-dhcp external3 10.3.0.0/24
# neutron subnet-create --disable-dhcp external4 10.4.0.0/24
*************************************************
Then login as demo and create
*************************************************
RouterExt3 with gateway to external3
RouterExt4 with gateway to external4
Then create private networks demo-network4,demo_network5
Attach first to RouterExt4 , second to RouterExt3
On Network Node [root@ip-192-169-142-147 ~(keystone_admin)]# neutron router-list| grep Ext
| 1e9bad93-2d5d-43fc-aed0-fc3745fe4d10 | RouterExt3 | {"network_id": "fffafde8-c6eb-4b20-b26d-63944300a6bf", "enable_snat": true, "external_fixed_ips": [{"subnet_id": "a038ae4d-9ea7-466e-bf4e-fab65981151c", "ip_address": "10.3.0.2"}]} | False | False |
| f47a87d9-c789-47a8-bdb1-8117990c49be | RouterExt4 | {"network_id": "2130df5b-5483-4cb8-a6b6-2a32eb7d882a", "enable_snat": true, "external_fixed_ips": [{"subnet_id": "eea125fc-6236-4570-9d3e-f4489671d2bb", "ip_address": "10.4.0.2"}]} | False | False |
[root@ip-192-169-142-147 ~(keystone_admin)]# ip netns exec qrouter-1e9bad93-2d5d-43fc-aed0-fc3745fe4d10 ifconfig
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 0 (Local Loopback)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qg-615baaa8-a6: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.3.0.2 netmask 255.255.255.0 broadcast 10.3.0.255
inet6 fe80::f816:3eff:fea7:98be prefixlen 64 scopeid 0x20<link>
ether fa:16:3e:a7:98:be txqueuelen 0 (Ethernet)
RX packets 810478 bytes 1101227298 (1.0 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 449476 bytes 34585959 (32.9 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qr-45110e77-5b: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 160.0.0.1 netmask 255.255.255.0 broadcast 160.0.0.255
inet6 fe80::f816:3eff:fe55:5c68 prefixlen 64 scopeid 0x20<link>
ether fa:16:3e:55:5c:68 txqueuelen 0 (Ethernet)
RX packets 449433 bytes 34589519 (32.9 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 810390 bytes 1101224102 (1.0 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@ip-192-169-142-147 ~(keystone_admin)]# ip netns exec qrouter-f47a87d9-c789-47a8-bdb1-8117990c49be ifconfig
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 0 (Local Loopback)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qg-54aa0373-dd: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.4.0.2 netmask 255.255.255.0 broadcast 10.4.0.255
inet6 fe80::f816:3eff:fe2e:35ee prefixlen 64 scopeid 0x20<link>
ether fa:16:3e:2e:35:ee txqueuelen 0 (Ethernet)
RX packets 802750 bytes 1088213425 (1.0 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 447723 bytes 34699912 (33.0 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qr-a99aa111-1d: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 150.0.0.1 netmask 255.255.255.0 broadcast 150.0.0.255
inet6 fe80::f816:3eff:fe9b:3a9a prefixlen 64 scopeid 0x20<link>
ether fa:16:3e:9b:3a:9a txqueuelen 0 (Ethernet)
RX packets 448277 bytes 34759884 (33.1 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 802834 bytes 1088249558 (1.0 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 ***************************************************************************************************Notice that qg-xxxxxxx interfaces from both qrouter-namespaces are attached to br-intWhile using provider external networks,traffic to/from external network flows through br-int.
br-int and br-eth3 will be connected using veth pair int-br-eth3 and phy-br-eth3.
br-int and br-eth4 will be connected using veth pair int-br-eth4 and phy-br-eth4.
This will be automatically created by neutron-openvswitch-agent
based on the bridge_mappings configured earlier.
***************************************************************************************************[root@ip-192-169-142-147 ~(keystone_admin)]# ovs-vsctl show
38e920e3-da61-4a1b-876a-052a49d777a2
Bridge br-tun
fail_mode: secure
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Port "vxlan-0a000089"
Interface "vxlan-0a000089"
type: vxlan
options: {df_default="true", in_key=flow, local_ip="10.0.0.147", out_key=flow, remote_ip="10.0.0.137"}
Port br-tun
Interface br-tun
type: internal
Bridge "br-eth4"
Port "br-eth4"
Interface "br-eth4"
type: internal
Port "phy-br-eth4"
Interface "phy-br-eth4"
type: patch
options: {peer="int-br-eth4"}
Port "eth4"
Interface "eth4"
Bridge br-int
fail_mode: secure
Port "tap709fbf6f-ab"
tag: 13
Interface "tap709fbf6f-ab"
type: internal
Port br-int
Interface br-int
type: internal
Port "qr-a99aa111-1d"
tag: 13
Interface "qr-a99aa111-1d"
type: internal
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
Port "qg-54aa0373-dd"
tag: 14
Interface "qg-54aa0373-dd"
type: internal
Port int-br-ex
Interface int-br-ex
type: patch
options: {peer=phy-br-ex}
Port "int-br-eth3"
Interface "int-br-eth3"
type: patch
options: {peer="phy-br-eth3"}
Port "qg-615baaa8-a6"
tag: 15
Interface "qg-615baaa8-a6"
type: internal
Port "tap06adaf37-d4"
tag: 17
Interface "tap06adaf37-d4"
type: internal
Port "qr-45110e77-5b"
tag: 17
Interface "qr-45110e77-5b"
type: internal
Port "int-br-eth4"
Interface "int-br-eth4"
type: patch
options: {peer="phy-br-eth4"}
Bridge br-ex
Port "eth2"
Interface "eth2"
Port phy-br-ex
Interface phy-br-ex
type: patch
options: {peer=int-br-ex}
Port br-ex
Interface br-ex
type: internal
Bridge "br-eth3"
Port "eth3"
Interface "eth3"
Port "phy-br-eth3"
Interface "phy-br-eth3"
type: patch
options: {peer="int-br-eth3"}
Port "br-eth3"
Interface "br-eth3"
type: internal
ovs_version: "2.3.1" 
No comments:
Post a Comment