Friday, December 11, 2015

High Availability cloud VMs (Neutron && VRRP) on RDO Liberty

 It is actually an update for Neutron on RDO Liberty of original blog entry
 http://blog.aaronorosen.com/implementing-high-availability-instances-with-neutron-using-vrrp/
  I only attempted to make post understandable for people with no knowledge
how to access cloud VMs having just private IPs. How work commands
`ip netns` , `ip netns exec qdhcp-namespace ssh -i oskeyvip.pem ubuntu@private-ip` ?
 Highlighted Neutron Commands which are not commonly known and give  an option to create floating IP working as VIP and providing High Available pair  of Ubuntu 14.04 cloud instances.
   The core idea belongs to Aaron Rosen, published in his post for  Openstack Havana Release and I don't have any intend to steal it. I just believe that Neutron Power deserves a bit more attention from  people still doing legacy ( e.g. Nova ) networking.

Create private network to launch a couple of Ubuntu Trusty VMs

[root@ip-192-169-142-54 ~(keystone_demo)]# neutron net-create vrrp-net
Created a new network:
+-----------------+--------------------------------------+
| Field           | Value                                |
+-----------------+--------------------------------------+
| admin_state_up  | True                                 |
| id              | b526aca8-e8b0-4d39-a7d4-4d4e0ebfe5ed |
| mtu             | 0                                    |
| name            | vrrp-net                             |
| router:external | False                                |
| shared          | False                                |
| status          | ACTIVE                               |
| subnets         |                                      |
| tenant_id       | df2302c143a84de9b6849ef75cc4368c     |
+-----------------+--------------------------------------+

[root@ip-192-169-142-54 ~(keystone_demo)]# neutron subnet-create  --name vrrp-subnet --allocation-pool start=10.0.0.2,end=10.0.0.200 vrrp-net 10.0.0.0/24 --dns_nameservers list=true 83.221.202.254
Created a new subnet:
+-------------------+--------------------------------------------+
| Field             | Value                                      |
+-------------------+--------------------------------------------+
| allocation_pools  | {"start": "10.0.0.2", "end": "10.0.0.200"} |
| cidr              | 10.0.0.0/24                                |
| dns_nameservers   | 83.221.202.254                             |
| enable_dhcp       | True                                       |
| gateway_ip        | 10.0.0.1                                   |
| host_routes       |                                            |
| id                | 8742e4d1-849e-4f83-8357-0996b93d7ec8       |
| ip_version        | 4                                          |
| ipv6_address_mode |                                            |
| ipv6_ra_mode      |                                            |
| name              | vrrp-subnet                                |
| network_id        | b526aca8-e8b0-4d39-a7d4-4d4e0ebfe5ed       |
| subnetpool_id     |                                            |
| tenant_id         | df2302c143a84de9b6849ef75cc4368c           |
+-------------------+--------------------------------------------+

************************************************************************
Create port on vrrp-net with IP  which is out of allocation pool
************************************************************************

[root@ip-192-169-142-54 ~(keystone_demo)]# neutron port-create --fixed-ip ip_address=10.0.0.201 --security-group default vrrp-net
Created a new port:

+-----------------------+--------------------------------------------------------------------------------------------------------+
| Field                 | Value                                                                                                  |
+-----------------------+--------------------------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                                   |
| allowed_address_pairs |                                                                                                        |
| binding:vnic_type     | normal                                                                                                 |
| device_id             |                                                                                                        |
| device_owner          |                                                                                                        |
| dns_assignment        | {"hostname": "host-10-0-0-201", "ip_address": "10.0.0.201", "fqdn": "host-10-0-0-201.openstacklocal."} |
| dns_name              |                                                                                                        |
| fixed_ips             | {"subnet_id": "8742e4d1-849e-4f83-8357-0996b93d7ec8", "ip_address": "10.0.0.201"}                      |
| id                    | 678f042b-dc2f-4426-b1b0-0d941ab21d5b                                                                   |
| mac_address           | fa:16:3e:73:ad:ef                                                                                      |
| name                  |                                                                                                        |
| network_id            | b526aca8-e8b0-4d39-a7d4-4d4e0ebfe5ed                                                                   |
| security_groups       | 39bc2297-5fc4-426a-b266-43e3a86a03f9                                                                   |
| status                | DOWN                                                                                                   |
| tenant_id             | df2302c143a84de9b6849ef75cc4368c                                                                       |
+-----------------------+--------------------------------------------------------------------------------------------------------+

**********************************************
Associate FIP with port has been created
**********************************************

[root@ip-192-169-142-54 ~(keystone_demo)]# neutron floatingip-create --port-id=678f042b-dc2f-4426-b1b0-0d941ab21d5b public
Created a new floatingip:

+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| fixed_ip_address    | 10.0.0.201                           |
| floating_ip_address | 192.169.142.151                      |
| floating_network_id | e5f7d2f3-f924-4158-a111-9dfa2f116e34 |
| id                  | 81e7cd2c-f073-4805-ae4c-06d54db8e52d |
| port_id             | 678f042b-dc2f-4426-b1b0-0d941ab21d5b |
| router_id           | 15aaee00-223f-4bf9-b7e0-a1ff4f97c20e |
| status              | DOWN                                 |
| tenant_id           | df2302c143a84de9b6849ef75cc4368c     |
+---------------------+--------------------------------------+

[root@ip-192-169-142-54 ~(keystone_demo)]# neutron net-list
+--------------------------------------+----------+-------------------------------------------------------+
| id                                   | name     | subnets                                               |
+--------------------------------------+----------+-------------------------------------------------------+
| e5f7d2f3-f924-4158-a111-9dfa2f116e34 | public   | afb7d629-1685-4c3b-a4e3-1bcebeef2844 192.169.142.0/24 |
| b526aca8-e8b0-4d39-a7d4-4d4e0ebfe5ed | vrrp-net | 8742e4d1-849e-4f83-8357-0996b93d7ec8 10.0.0.0/24      |
+--------------------------------------+----------+-------------------------------------------------------+

****************************************************
Detect ports corresponding VMs been launched
****************************************************

[root@ip-192-169-142-54 ~(keystone_demo)]# neutron port-list -- --network_id=b526aca8-e8b0-4d39-a7d4-4d4e0ebfe5ed
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                         |
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------+
| 08f28bb2-8abe-4ea0-bd5b-566ef9881bf3 |      | fa:16:3e:38:01:1b | {"subnet_id": "8742e4d1-849e-4f83-8357-0996b93d7ec8", "ip_address": "10.0.0.5"}   |
| 09d3dee4-2ee6-4d2a-b7c9-034f31991606 |      | fa:16:3e:8e:93:e9 | {"subnet_id": "8742e4d1-849e-4f83-8357-0996b93d7ec8", "ip_address": "10.0.0.2"}   |
| 5d3a69c8-2e88-481c-a3aa-df923db6d624 |      | fa:16:3e:13:a4:c5 | {"subnet_id": "8742e4d1-849e-4f83-8357-0996b93d7ec8", "ip_address": "10.0.0.4"}   |
| 678f042b-dc2f-4426-b1b0-0d941ab21d5b |      | fa:16:3e:73:ad:ef | {"subnet_id": "8742e4d1-849e-4f83-8357-0996b93d7ec8", "ip_address": "10.0.0.201"} |
| dce31b48-620c-4265-ab2a-13017f6ed97c |      | fa:16:3e:6a:52:38 | {"subnet_id": "8742e4d1-849e-4f83-8357-0996b93d7ec8", "ip_address": "10.0.0.1"}   |
+--------------------------------------+------+------------------

-+-----------------------------------------------------------------------------------+
[root@ip-192-169-142-54 ~(keystone_demo)]# nova list
+--------------------------------------+-------------+--------+------------+-------------+-------------------+
| ID                                   | Name        | Status | Task State | Power State | Networks          |
+--------------------------------------+-------------+--------+------------+-------------+-------------------+
| 88efc361-0b0e-487a-8634-07b9782af9bd | UbuntuSRV01 | ACTIVE | -          | Running     | vrrp-net=10.0.0.4 |
| 8f4e4c2e-6049-4451-94c2-1990ee4072ea | UbuntuSRV02 | ACTIVE | -          | Running     | vrrp-net=10.0.0.5 |
+--------------------------------------+-------------+--------+------------+-------------+-------------------+

******************************************************************************
Update status of both ports using "Allowed_address_pairs" feature
******************************************************************************

[root@ip-192-169-142-54 ~(keystone_demo)]# neutron port-update 08f28bb2-8abe-4ea0-bd5b-566ef9881bf3  --allowed_address_pairs list=true type=dict ip_address=10.0.0.201
Updated port: 08f28bb2-8abe-4ea0-bd5b-566ef9881bf3


[root@ip-192-169-142-54 ~(keystone_demo)]# neutron port-update 5d3a69c8-2e88-481c-a3aa-df923db6d624  --allowed_address_pairs list=true type=dict ip_address=10.0.0.201
Updated port: 5d3a69c8-2e88-481c-a3aa-df923db6d624


***************************************************************
Now make sure that commands above succeeded
***************************************************************

[root@ip-192-169-142-54 ~(keystone_demo)]# neutron port-show 08f28bb2-8abe-4ea0-bd5b-566ef9881bf3
+-----------------------+--------------------------------------------------------------------------------------------------+
| Field                 | Value                                                                                            |
+-----------------------+--------------------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                             |
| allowed_address_pairs | {"ip_address": "10.0.0.201", "mac_address": "fa:16:3e:38:01:1b"}                                 |
| binding:vnic_type     | normal                                                                                           |
| device_id             | 8f4e4c2e-6049-4451-94c2-1990ee4072ea                                                             |
| device_owner          | compute:nova                                                                                     |
| dns_assignment        | {"hostname": "host-10-0-0-5", "ip_address": "10.0.0.5", "fqdn": "host-10-0-0-5.openstacklocal."} |
| dns_name              |                                                                                                  |
| extra_dhcp_opts       |                                                                                                  |
| fixed_ips             | {"subnet_id": "8742e4d1-849e-4f83-8357-0996b93d7ec8", "ip_address": "10.0.0.5"}                  |
| id                    | 08f28bb2-8abe-4ea0-bd5b-566ef9881bf3                                                             |
| mac_address           | fa:16:3e:38:01:1b                                                                                |
| name                  |                                                                                                  |
| network_id            | b526aca8-e8b0-4d39-a7d4-4d4e0ebfe5ed                                                             |
| security_groups       | 39bc2297-5fc4-426a-b266-43e3a86a03f9                                                             |
| status                | ACTIVE                                                                                           |
| tenant_id             | df2302c143a84de9b6849ef75cc4368c                                                                 |
+-----------------------+--------------------------------------------------------------------------------------------------+

[root@ip-192-169-142-54 ~(keystone_demo)]# neutron port-show 5d3a69c8-2e88-481c-a3aa-df923db6d624
+-----------------------+--------------------------------------------------------------------------------------------------+
| Field                 | Value                                                                                            |
+-----------------------+-------------------------------------------------------------------
| admin_state_up        | True                                                                                             |
| allowed_address_pairs | {"ip_address": "10.0.0.201", "mac_address": "fa:16:3e:13:a4:c5"}                                 |
| binding:vnic_type     | normal                                                                                           |
| device_id             | 88efc361-0b0e-487a-8634-07b9782af9bd                                                             |
| device_owner          | compute:nova                                                                                     |
| dns_assignment        | {"hostname": "host-10-0-0-4", "ip_address": "10.0.0.4", "fqdn": "host-10-0-0-4.openstacklocal."} |
| dns_name              |                                                                                                  |
| extra_dhcp_opts       |                                                                                                  |
| fixed_ips             | {"subnet_id": "8742e4d1-849e-4f83-8357-0996b93d7ec8", "ip_address": "10.0.0.4"}                  |
| id                    | 5d3a69c8-2e88-481c-a3aa-df923db6d624                                                             |
| mac_address           | fa:16:3e:13:a4:c5                                                                                |
| name                  |                                                                                                  |
| network_id            | b526aca8-e8b0-4d39-a7d4-4d4e0ebfe5ed                                                             |
| security_groups       | 39bc2297-5fc4-426a-b266-43e3a86a03f9                                                             |
| status                | ACTIVE                                                                                           |
| tenant_id             | df2302c143a84de9b6849ef75cc4368c                                                                 |
+-----------------------+--------------------------------------------------------------------------------------------------+


[root@ip-192-169-142-54 ~(keystone_demo)]# neutron port-show 08f28bb2-8abe-4ea0-bd5b-566ef9881bf3
+-----------------------+--------------------------------------------------------------------------------------------------+
| Field                 | Value                                                                                            |
+-----------------------+--------------------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                             |
| allowed_address_pairs | {"ip_address": "10.0.0.201", "mac_address": "fa:16:3e:38:01:1b"}                                 |
| binding:vnic_type     | normal                                                                                           |
| device_id             | 8f4e4c2e-6049-4451-94c2-1990ee4072ea                                                             |
| device_owner          | compute:nova                                                                                     |
| dns_assignment        | {"hostname": "host-10-0-0-5", "ip_address": "10.0.0.5", "fqdn": "host-10-0-0-5.openstacklocal."} |
| dns_name              |                                                                                                  |
| extra_dhcp_opts       |                                                                                                  |
| fixed_ips             | {"subnet_id": "8742e4d1-849e-4f83-8357-0996b93d7ec8", "ip_address": "10.0.0.5"}                  |
| id                    | 08f28bb2-8abe-4ea0-bd5b-566ef9881bf3                                                             |
| mac_address           | fa:16:3e:38:01:1b                                                                                |
| name                  |                                                                                                  |
| network_id            | b526aca8-e8b0-4d39-a7d4-4d4e0ebfe5ed                                                             |
| security_groups       | 39bc2297-5fc4-426a-b266-43e3a86a03f9                                                             |
| status                | ACTIVE                                                                                           |
| tenant_id             | df2302c143a84de9b6849ef75cc4368c                                                                 |
+-----------------------+--------------------------------------------------------------------------------------------------+

****************************************************************************
At this point we need log into ubuntu VMs without floating IPs, proceed as
follows. First detect qrouter and qdhcp namespaces names been created on
the system
*****************************************************************************
[root@ip-192-169-142-54 ~(keystone_demo)]# ip netns
qrouter-15aaee00-223f-4bf9-b7e0-a1ff4f97c20e
qdhcp-b526aca8-e8b0-4d39-a7d4-4d4e0ebfe5ed

*****************************************************************************
Now using command bellow and ssh-keypair created along with VM log into
each one of instances to configure services keepalived and apache
******************************************************************************

[root@ip-192-169-142-54 ~(keystone_demo)]# ip netns exec qdhcp-b526aca8-e8b0-4d39-a7d4-4d4e0ebfe5ed ssh -i oskeyvip.pem ubuntu@10.0.0.4
The authenticity of host '10.0.0.4 (10.0.0.4)' can't be established.
ECDSA key fingerprint is b2:03:72:69:9e:d2:0b:2c:7c:43:47:90:21:42:af:b6.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.0.0.4' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 14.04.3 LTS (GNU/Linux 3.13.0-68-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

  System information as of Fri Dec 11 09:38:05 UTC 2015

  System load: 0.6               Memory usage: 2%   Processes:       53
  Usage of /:  57.1% of 1.32GB   Swap usage:   0%   Users logged in: 0

  Graph this data and manage this system at:
    https://landscape.canonical.com/

  Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud

0 packages can be updated.
0 updates are security updates.



The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

ubuntu@ubuntusrv01:~$ sudo su -
sudo: unable to resolve host ubuntusrv01
root@ubuntusrv01:~# apt-get install keepalived

$vi  /etc/keepalived/keepalived.conf
vrrp_instance vrrp_group_1 {
 state MASTER
 interface eth0
 virtual_router_id 1
 priority 100
 authentication {
  auth_type PASS
  auth_pass password
 }
 virtual_ipaddress {
  10.0.0.201/24 brd 10.0.0.255 dev eth0
 }
}
:wq

# service keepalived start

root@ubuntusrv01:~# apt-get install apache2 -y
root@ubuntusrv01:~# echo "UbuntuSRV01 is up" > /var/www/html/index.html
root@ubuntusrv01:~# service apache2 restart
 * Restarting web server apache2                                                                 AH00557: apache2: apr_sockaddr_info_get() failed for ubuntusrv01
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
                                                                                          [ OK ]

[root@ip-192-169-142-54 ~(keystone_demo)]# ip netns exec qdhcp-b526aca8-e8b0-4d39-a7d4-4d4e0ebfe5ed ssh -i oskeyvip.pem ubuntu@10.0.0.5

Welcome to Ubuntu 14.04.3 LTS (GNU/Linux 3.13.0-68-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

  System information as of Fri Dec 11 09:38:05 UTC 2015

  System load: 0.6               Memory usage: 2%   Processes:       53
  Usage of /:  57.1% of 1.32GB   Swap usage:   0%   Users logged in: 0

  Graph this data and manage this system at:
    https://landscape.canonical.com/

  Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud

0 packages can be updated.
0 updates are security updates.



The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

ubuntu@ubuntusrv01:~$ sudo su -
sudo: unable to resolve host ubuntusrv02
root@ubuntusrv01:~# apt-get install keepalived

$ vi  /etc/keepalived/keepalived.conf
vrrp_instance vrrp_group_1 {
 state BACKUP
 interface eth0
 virtual_router_id 1
 priority 50
 authentication {
  auth_type PASS
  auth_pass password
 }
 virtual_ipaddress {
  10.0.0.201/24 brd 10.0.0.255 dev eth0
 }
}
:wq

root@ubuntusrv02:~# apt-get install apache2 -y
root@ubuntusrv02:~# echo "UbuntuSRV02 is up" > /var/www/html/index.html
root@ubuntusrv02:~# service apache2 restart
 * Restarting web server apache2                                                                 AH00557: apache2: apr_sockaddr_info_get() failed for ubuntusrv01
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message

*******************************************************************************
Restart keepalived to load the configuration change on both nodes:
*******************************************************************************

# service keepalived restart

******************************************************************************
Snapshots bellow demonstrates as FIP 192.169.142.151  works as VIP providing
HA for VMs 10.0.0.4 and 10.0.0.5 (private IPs)
*******************************************************************************

  

No comments:

Post a Comment