High Availability cloud VMs (Neutron && VRRP) on RDO Liberty

 It is actually an update for Neutron on RDO Liberty of original blog entry
 http://blog.aaronorosen.com/implementing-high-availability-instances-with-neutron-using-vrrp/
  I only attempted to make post understandable for people with no knowledge
how to access cloud VMs having just private IPs. How work commands
`ip netns` , `ip netns exec qdhcp-namespace ssh -i oskeyvip.pem ubuntu@private-ip` ?
 Highlighted Neutron Commands which are not commonly known and give  an option to create floating IP working as VIP and providing High Available pair  of Ubuntu 14.04 cloud instances.
   The core idea belongs to Aaron Rosen, published in his post for  Openstack Havana Release and I don't have any intend to steal it. I just believe that Neutron Power deserves a bit more attention from  people still doing legacy ( e.g. Nova ) networking.

Create private network to launch a couple of Ubuntu Trusty VMs

[root@ip-192-169-142-54 ~(keystone_demo)]# neutron net-create vrrp-net
Created a new network:
+-----------------+--------------------------------------+
| Field           | Value                                |
+-----------------+--------------------------------------+
| admin_state_up  | True                                 |
| id              | b526aca8-e8b0-4d39-a7d4-4d4e0ebfe5ed |
| mtu             | 0                                    |
| name            | vrrp-net                             |
| router:external | False                                |
| shared          | False                                |
| status          | ACTIVE                               |
| subnets         |                                      |
| tenant_id       | df2302c143a84de9b6849ef75cc4368c     |
+-----------------+--------------------------------------+

[root@ip-192-169-142-54 ~(keystone_demo)]# neutron subnet-create  --name vrrp-subnet --allocation-pool start=10.0.0.2,end=10.0.0.200 vrrp-net 10.0.0.0/24 --dns_nameservers list=true 83.221.202.254
Created a new subnet:
+-------------------+--------------------------------------------+
| Field             | Value                                      |
+-------------------+--------------------------------------------+
| allocation_pools  | {"start": "10.0.0.2", "end": "10.0.0.200"} |
| cidr              | 10.0.0.0/24                                |
| dns_nameservers   | 83.221.202.254                             |
| enable_dhcp       | True                                       |
| gateway_ip        | 10.0.0.1                                   |
| host_routes       |                                            |
| id                | 8742e4d1-849e-4f83-8357-0996b93d7ec8       |
| ip_version        | 4                                          |
| ipv6_address_mode |                                            |
| ipv6_ra_mode      |                                            |
| name              | vrrp-subnet                                |
| network_id        | b526aca8-e8b0-4d39-a7d4-4d4e0ebfe5ed       |
| subnetpool_id     |                                            |
| tenant_id         | df2302c143a84de9b6849ef75cc4368c           |
+-------------------+--------------------------------------------+

************************************************************************
Create port on vrrp-net with IP  which is out of allocation pool
************************************************************************

[root@ip-192-169-142-54 ~(keystone_demo)]# neutron port-create --fixed-ip ip_address=10.0.0.201 --security-group default vrrp-net
Created a new port:
+-----------------------+--------------------------------------------------------------------------------------------------------+
| Field                 | Value                                                                                                  |
+-----------------------+--------------------------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                                   |
| allowed_address_pairs |                                                                                                        |
| binding:vnic_type     | normal                                                                                                 |
| device_id             |                                                                                                        |
| device_owner          |                                                                                                        |
| dns_assignment        | {"hostname": "host-10-0-0-201", "ip_address": "10.0.0.201", "fqdn": "host-10-0-0-201.openstacklocal."} |
| dns_name              |                                                                                                        |
| fixed_ips             | {"subnet_id": "8742e4d1-849e-4f83-8357-0996b93d7ec8", "ip_address": "10.0.0.201"}                      |
| id                    | 678f042b-dc2f-4426-b1b0-0d941ab21d5b                                                                   |
| mac_address           | fa:16:3e:73:ad:ef                                                                                      |
| name                  |                                                                                                        |
| network_id            | b526aca8-e8b0-4d39-a7d4-4d4e0ebfe5ed                                                                   |
| security_groups       | 39bc2297-5fc4-426a-b266-43e3a86a03f9                                                                   |
| status                | DOWN                                                                                                   |
| tenant_id             | df2302c143a84de9b6849ef75cc4368c                                                                       |
+-----------------------+--------------------------------------------------------------------------------------------------------+

**********************************************
Associate FIP with port has been created
**********************************************

[root@ip-192-169-142-54 ~(keystone_demo)]# neutron floatingip-create --port-id=678f042b-dc2f-4426-b1b0-0d941ab21d5b public
Created a new floatingip:
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| fixed_ip_address    | 10.0.0.201                           |
| floating_ip_address | 192.169.142.151                      |
| floating_network_id | e5f7d2f3-f924-4158-a111-9dfa2f116e34 |
| id                  | 81e7cd2c-f073-4805-ae4c-06d54db8e52d |
| port_id             | 678f042b-dc2f-4426-b1b0-0d941ab21d5b |
| router_id           | 15aaee00-223f-4bf9-b7e0-a1ff4f97c20e |
| status              | DOWN                                 |
| tenant_id           | df2302c143a84de9b6849ef75cc4368c     |
+---------------------+--------------------------------------+

[root@ip-192-169-142-54 ~(keystone_demo)]# neutron net-list
+--------------------------------------+----------+-------------------------------------------------------+
| id                                   | name     | subnets                                               |
+--------------------------------------+----------+-------------------------------------------------------+
| e5f7d2f3-f924-4158-a111-9dfa2f116e34 | public   | afb7d629-1685-4c3b-a4e3-1bcebeef2844 192.169.142.0/24 |
| b526aca8-e8b0-4d39-a7d4-4d4e0ebfe5ed | vrrp-net | 8742e4d1-849e-4f83-8357-0996b93d7ec8 10.0.0.0/24      |
+--------------------------------------+----------+-------------------------------------------------------+

****************************************************
Detect ports corresponding VMs been launched
****************************************************

[root@ip-192-169-142-54 ~(keystone_demo)]# neutron port-list -- --network_id=b526aca8-e8b0-4d39-a7d4-4d4e0ebfe5ed
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                         |
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------+
| 08f28bb2-8abe-4ea0-bd5b-566ef9881bf3 |      | fa:16:3e:38:01:1b | {"subnet_id": "8742e4d1-849e-4f83-8357-0996b93d7ec8", "ip_address": "10.0.0.5"}   |
| 09d3dee4-2ee6-4d2a-b7c9-034f31991606 |      | fa:16:3e:8e:93:e9 | {"subnet_id": "8742e4d1-849e-4f83-8357-0996b93d7ec8", "ip_address": "10.0.0.2"}   |
| 5d3a69c8-2e88-481c-a3aa-df923db6d624 |      | fa:16:3e:13:a4:c5 | {"subnet_id": "8742e4d1-849e-4f83-8357-0996b93d7ec8", "ip_address": "10.0.0.4"}   |
| 678f042b-dc2f-4426-b1b0-0d941ab21d5b |      | fa:16:3e:73:ad:ef | {"subnet_id": "8742e4d1-849e-4f83-8357-0996b93d7ec8", "ip_address": "10.0.0.201"} |
| dce31b48-620c-4265-ab2a-13017f6ed97c |      | fa:16:3e:6a:52:38 | {"subnet_id": "8742e4d1-849e-4f83-8357-0996b93d7ec8", "ip_address": "10.0.0.1"}   |
+--------------------------------------+------+------------------

-+-----------------------------------------------------------------------------------+
[root@ip-192-169-142-54 ~(keystone_demo)]# nova list
+--------------------------------------+-------------+--------+------------+-------------+-------------------+
| ID                                   | Name        | Status | Task State | Power State | Networks          |
+--------------------------------------+-------------+--------+------------+-------------+-------------------+
| 88efc361-0b0e-487a-8634-07b9782af9bd | UbuntuSRV01 | ACTIVE | -          | Running     | vrrp-net=10.0.0.4 |
| 8f4e4c2e-6049-4451-94c2-1990ee4072ea | UbuntuSRV02 | ACTIVE | -          | Running     | vrrp-net=10.0.0.5 |
+--------------------------------------+-------------+--------+------------+-------------+-------------------+

******************************************************************************
Update status of both ports using "Allowed_address_pairs" feature
******************************************************************************

[root@ip-192-169-142-54 ~(keystone_demo)]# neutron port-update 08f28bb2-8abe-4ea0-bd5b-566ef9881bf3  --allowed_address_pairs list=true type=dict ip_address=10.0.0.201
Updated port: 08f28bb2-8abe-4ea0-bd5b-566ef9881bf3

[root@ip-192-169-142-54 ~(keystone_demo)]# neutron port-update 5d3a69c8-2e88-481c-a3aa-df923db6d624  --allowed_address_pairs list=true type=dict ip_address=10.0.0.201
Updated port: 5d3a69c8-2e88-481c-a3aa-df923db6d624

***************************************************************
Now make sure that commands above succeeded
***************************************************************

[root@ip-192-169-142-54 ~(keystone_demo)]# neutron port-show 08f28bb2-8abe-4ea0-bd5b-566ef9881bf3
+-----------------------+--------------------------------------------------------------------------------------------------+
| Field                 | Value                                                                                            |
+-----------------------+--------------------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                             |
| allowed_address_pairs | {"ip_address": "10.0.0.201", "mac_address": "fa:16:3e:38:01:1b"}                                 |
| binding:vnic_type     | normal                                                                                           |
| device_id             | 8f4e4c2e-6049-4451-94c2-1990ee4072ea                                                             |
| device_owner          | compute:nova                                                                                     |
| dns_assignment        | {"hostname": "host-10-0-0-5", "ip_address": "10.0.0.5", "fqdn": "host-10-0-0-5.openstacklocal."} |
| dns_name              |                                                                                                  |
| extra_dhcp_opts       |                                                                                                  |
| fixed_ips             | {"subnet_id": "8742e4d1-849e-4f83-8357-0996b93d7ec8", "ip_address": "10.0.0.5"}                  |
| id                    | 08f28bb2-8abe-4ea0-bd5b-566ef9881bf3                                                             |
| mac_address           | fa:16:3e:38:01:1b                                                                                |
| name                  |                                                                                                  |
| network_id            | b526aca8-e8b0-4d39-a7d4-4d4e0ebfe5ed                                                             |
| security_groups       | 39bc2297-5fc4-426a-b266-43e3a86a03f9                                                             |
| status                | ACTIVE                                                                                           |
| tenant_id             | df2302c143a84de9b6849ef75cc4368c                                                                 |
+-----------------------+--------------------------------------------------------------------------------------------------+

[root@ip-192-169-142-54 ~(keystone_demo)]# neutron port-show 5d3a69c8-2e88-481c-a3aa-df923db6d624
+-----------------------+--------------------------------------------------------------------------------------------------+
| Field                 | Value                                                                                            |
+-----------------------+-------------------------------------------------------------------
| admin_state_up        | True                                                                                             |
| allowed_address_pairs | {"ip_address": "10.0.0.201", "mac_address": "fa:16:3e:13:a4:c5"}                                 |
| binding:vnic_type     | normal                                                                                           |
| device_id             | 88efc361-0b0e-487a-8634-07b9782af9bd                                                             |
| device_owner          | compute:nova                                                                                     |
| dns_assignment        | {"hostname": "host-10-0-0-4", "ip_address": "10.0.0.4", "fqdn": "host-10-0-0-4.openstacklocal."} |
| dns_name              |                                                                                                  |
| extra_dhcp_opts       |                                                                                                  |
| fixed_ips             | {"subnet_id": "8742e4d1-849e-4f83-8357-0996b93d7ec8", "ip_address": "10.0.0.4"}                  |
| id                    | 5d3a69c8-2e88-481c-a3aa-df923db6d624                                                             |
| mac_address           | fa:16:3e:13:a4:c5                                                                                |
| name                  |                                                                                                  |
| network_id            | b526aca8-e8b0-4d39-a7d4-4d4e0ebfe5ed                                                             |
| security_groups       | 39bc2297-5fc4-426a-b266-43e3a86a03f9                                                             |
| status                | ACTIVE                                                                                           |
| tenant_id             | df2302c143a84de9b6849ef75cc4368c                                                                 |
+-----------------------+--------------------------------------------------------------------------------------------------+


[root@ip-192-169-142-54 ~(keystone_demo)]# neutron port-show 08f28bb2-8abe-4ea0-bd5b-566ef9881bf3
+-----------------------+--------------------------------------------------------------------------------------------------+
| Field                 | Value                                                                                            |
+-----------------------+--------------------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                             |
| allowed_address_pairs | {"ip_address": "10.0.0.201", "mac_address": "fa:16:3e:38:01:1b"}                                 |
| binding:vnic_type     | normal                                                                                           |
| device_id             | 8f4e4c2e-6049-4451-94c2-1990ee4072ea                                                             |
| device_owner          | compute:nova                                                                                     |
| dns_assignment        | {"hostname": "host-10-0-0-5", "ip_address": "10.0.0.5", "fqdn": "host-10-0-0-5.openstacklocal."} |
| dns_name              |                                                                                                  |
| extra_dhcp_opts       |                                                                                                  |
| fixed_ips             | {"subnet_id": "8742e4d1-849e-4f83-8357-0996b93d7ec8", "ip_address": "10.0.0.5"}                  |
| id                    | 08f28bb2-8abe-4ea0-bd5b-566ef9881bf3                                                             |
| mac_address           | fa:16:3e:38:01:1b                                                                                |
| name                  |                                                                                                  |
| network_id            | b526aca8-e8b0-4d39-a7d4-4d4e0ebfe5ed                                                             |
| security_groups       | 39bc2297-5fc4-426a-b266-43e3a86a03f9                                                             |
| status                | ACTIVE                                                                                           |
| tenant_id             | df2302c143a84de9b6849ef75cc4368c                                                                 |
+-----------------------+--------------------------------------------------------------------------------------------------+

****************************************************************************
At this point we need log into ubuntu VMs without floating IPs, proceed as
follows. First detect qrouter and qdhcp namespaces names been created on
the system
*****************************************************************************
[root@ip-192-169-142-54 ~(keystone_demo)]# ip netns
qrouter-15aaee00-223f-4bf9-b7e0-a1ff4f97c20e
qdhcp-b526aca8-e8b0-4d39-a7d4-4d4e0ebfe5ed

*****************************************************************************
Now using command bellow and ssh-keypair created along with VM log into
each one of instances to configure services keepalived and apache
******************************************************************************

[root@ip-192-169-142-54 ~(keystone_demo)]# ip netns exec qdhcp-b526aca8-e8b0-4d39-a7d4-4d4e0ebfe5ed ssh -i oskeyvip.pem ubuntu@10.0.0.4
The authenticity of host '10.0.0.4 (10.0.0.4)' can't be established.
ECDSA key fingerprint is b2:03:72:69:9e:d2:0b:2c:7c:43:47:90:21:42:af:b6.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.0.0.4' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 14.04.3 LTS (GNU/Linux 3.13.0-68-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

  System information as of Fri Dec 11 09:38:05 UTC 2015

  System load: 0.6               Memory usage: 2%   Processes:       53
  Usage of /:  57.1% of 1.32GB   Swap usage:   0%   Users logged in: 0

  Graph this data and manage this system at:
    https://landscape.canonical.com/

  Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud

0 packages can be updated.
0 updates are security updates.



The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

ubuntu@ubuntusrv01:~$ sudo su -
sudo: unable to resolve host ubuntusrv01
root@ubuntusrv01:~# apt-get install keepalived

$vi  /etc/keepalived/keepalived.conf
vrrp_instance vrrp_group_1 {
 state MASTER
 interface eth0
 virtual_router_id 1
 priority 100
 authentication {
  auth_type PASS
  auth_pass password
 }
 virtual_ipaddress {
  10.0.0.201/24 brd 10.0.0.255 dev eth0
 }
}
:wq

# service keepalived start

root@ubuntusrv01:~# apt-get install apache2 -y
root@ubuntusrv01:~# echo "UbuntuSRV01 is up" > /var/www/html/index.html
root@ubuntusrv01:~# service apache2 restart
 * Restarting web server apache2                                                                 AH00557: apache2: apr_sockaddr_info_get() failed for ubuntusrv01
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
                                                                                          [ OK ]

[root@ip-192-169-142-54 ~(keystone_demo)]# ip netns exec qdhcp-b526aca8-e8b0-4d39-a7d4-4d4e0ebfe5ed ssh -i oskeyvip.pem ubuntu@10.0.0.5

Welcome to Ubuntu 14.04.3 LTS (GNU/Linux 3.13.0-68-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

  System information as of Fri Dec 11 09:38:05 UTC 2015

  System load: 0.6               Memory usage: 2%   Processes:       53
  Usage of /:  57.1% of 1.32GB   Swap usage:   0%   Users logged in: 0

  Graph this data and manage this system at:
    https://landscape.canonical.com/

  Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud

0 packages can be updated.
0 updates are security updates.



The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

ubuntu@ubuntusrv01:~$ sudo su -
sudo: unable to resolve host ubuntusrv02
root@ubuntusrv01:~# apt-get install keepalived

$ vi  /etc/keepalived/keepalived.conf
vrrp_instance vrrp_group_1 {
 state BACKUP
 interface eth0
 virtual_router_id 1
 priority 50
 authentication {
  auth_type PASS
  auth_pass password
 }
 virtual_ipaddress {
  10.0.0.201/24 brd 10.0.0.255 dev eth0
 }
}
:wq

root@ubuntusrv02:~# apt-get install apache2 -y
root@ubuntusrv02:~# echo "UbuntuSRV02 is up" > /var/www/html/index.html
root@ubuntusrv02:~# service apache2 restart
 * Restarting web server apache2                                                                 AH00557: apache2: apr_sockaddr_info_get() failed for ubuntusrv01
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message

*******************************************************************************
Restart keepalived to load the configuration change on both nodes:
*******************************************************************************

# service keepalived restart

******************************************************************************
Snapshots bellow demonstrates as FIP 192.169.142.151  works as VIP providing
HA for VMs 10.0.0.4 and 10.0.0.5 (private IPs)
*******************************************************************************

  

DVR_SNAT && DVR on RDO Mitaka M1 (CentOS 7.1)

********************************************************
Setup Delorean Repos for Mitaka M1 on CentOS 7.1
********************************************************

yum -y install yum-plugin-priorities
cd /etc/yum.repos.d/
# for Centos 7 and RHEL 7
wget http://trunk.rdoproject.org/centos7/delorean-deps.repo
wget http://trunk.rdoproject.org/centos7/current-passed-ci/delorean.repo

yum -y install openstack-packstack 

********************************************************************
Before running packstack on RDO Mitaka M1 be aware of
********************************************************************

1.  https://bugzilla.redhat.com/show_bug.cgi?id=1288179
2.  https://bugzilla.redhat.com/show_bug.cgi?id=1285314

[root@ip-192-169-142-127 ~(keystone_admin)]# cat answer3Node.txt
[general]
CONFIG_SSH_KEY=/root/.ssh/id_rsa.pub
CONFIG_DEFAULT_PASSWORD=
CONFIG_MARIADB_INSTALL=y
CONFIG_GLANCE_INSTALL=y
CONFIG_CINDER_INSTALL=y
CONFIG_NOVA_INSTALL=y
CONFIG_NEUTRON_INSTALL=y
CONFIG_HORIZON_INSTALL=y
CONFIG_SWIFT_INSTALL=y
# For now, just to avoid headaches
CONFIG_CEILOMETER_INSTALL=n
CONFIG_HEAT_INSTALL=n
CONFIG_CLIENT_INSTALL=y
CONFIG_NTP_SERVERS=
CONFIG_NAGIOS_INSTALL=y
EXCLUDE_SERVERS=
CONFIG_DEBUG_MODE=n
CONFIG_CONTROLLER_HOST=192.169.142.127
CONFIG_COMPUTE_HOSTS=192.169.142.137,192.169.142.147
CONFIG_NETWORK_HOSTS=192.169.142.127
CONFIG_VMWARE_BACKEND=n
CONFIG_UNSUPPORTED=n
CONFIG_VCENTER_HOST=
CONFIG_VCENTER_USER=
CONFIG_VCENTER_PASSWORD=
CONFIG_VCENTER_CLUSTER_NAME=
CONFIG_STORAGE_HOST=192.169.142.127
CONFIG_USE_EPEL=y
CONFIG_REPO=
CONFIG_RH_USER=
CONFIG_SATELLITE_URL=
CONFIG_RH_PW=
CONFIG_RH_OPTIONAL=y
CONFIG_RH_PROXY=
CONFIG_RH_PROXY_PORT=
CONFIG_RH_PROXY_USER=
CONFIG_RH_PROXY_PW=
CONFIG_SATELLITE_USER=
CONFIG_SATELLITE_PW=
CONFIG_SATELLITE_AKEY=
CONFIG_SATELLITE_CACERT=
CONFIG_SATELLITE_PROFILE=
CONFIG_SATELLITE_FLAGS=
CONFIG_SATELLITE_PROXY=
CONFIG_SATELLITE_PROXY_USER=
CONFIG_SATELLITE_PROXY_PW=
CONFIG_AMQP_BACKEND=rabbitmq
CONFIG_AMQP_HOST=192.169.142.127
CONFIG_AMQP_ENABLE_SSL=n
CONFIG_AMQP_ENABLE_AUTH=n
CONFIG_AMQP_NSS_CERTDB_PW=PW_PLACEHOLDER
CONFIG_AMQP_SSL_PORT=5671
CONFIG_AMQP_SSL_CERT_FILE=/etc/pki/tls/certs/amqp_selfcert.pem
CONFIG_AMQP_SSL_KEY_FILE=/etc/pki/tls/private/amqp_selfkey.pem
CONFIG_AMQP_SSL_SELF_SIGNED=y
CONFIG_AMQP_AUTH_USER=amqp_user
CONFIG_AMQP_AUTH_PASSWORD=PW_PLACEHOLDER
CONFIG_MARIADB_HOST=192.169.142.127
CONFIG_MARIADB_USER=root
CONFIG_MARIADB_PW=7207ae344ed04957
CONFIG_KEYSTONE_DB_PW=abcae16b785245c3
CONFIG_KEYSTONE_REGION=RegionOne
CONFIG_KEYSTONE_ADMIN_TOKEN=3ad2de159f9649afb0c342ba57e637d9
CONFIG_KEYSTONE_ADMIN_PW=7049f834927e4468
CONFIG_KEYSTONE_DEMO_PW=bf737b785cfa4398
CONFIG_KEYSTONE_TOKEN_FORMAT=UUID
CONFIG_KEYSTONE_SERVICE_NAME=httpd
CONFIG_GLANCE_DB_PW=41264fc52ffd4fe8
CONFIG_GLANCE_KS_PW=f6a9398960534797
CONFIG_GLANCE_BACKEND=file
CONFIG_CINDER_DB_PW=5ac08c6d09ba4b69
CONFIG_CINDER_KS_PW=c8cb1ecb8c2b4f6f
CONFIG_CINDER_BACKEND=lvm
CONFIG_CINDER_VOLUMES_CREATE=y
CONFIG_CINDER_VOLUMES_SIZE=5G
CONFIG_CINDER_GLUSTER_MOUNTS=
CONFIG_CINDER_NFS_MOUNTS=
CONFIG_CINDER_NETAPP_LOGIN=
CONFIG_CINDER_NETAPP_PASSWORD=
CONFIG_CINDER_NETAPP_HOSTNAME=
CONFIG_CINDER_NETAPP_SERVER_PORT=80
CONFIG_CINDER_NETAPP_STORAGE_FAMILY=ontap_cluster
CONFIG_CINDER_NETAPP_TRANSPORT_TYPE=http
CONFIG_CINDER_NETAPP_STORAGE_PROTOCOL=nfs
CONFIG_CINDER_NETAPP_SIZE_MULTIPLIER=1.0
CONFIG_CINDER_NETAPP_EXPIRY_THRES_MINUTES=720
CONFIG_CINDER_NETAPP_THRES_AVL_SIZE_PERC_START=20
CONFIG_CINDER_NETAPP_THRES_AVL_SIZE_PERC_STOP=60
CONFIG_CINDER_NETAPP_NFS_SHARES_CONFIG=
CONFIG_CINDER_NETAPP_VOLUME_LIST=
CONFIG_CINDER_NETAPP_VFILER=
CONFIG_CINDER_NETAPP_VSERVER=
CONFIG_CINDER_NETAPP_CONTROLLER_IPS=
CONFIG_CINDER_NETAPP_SA_PASSWORD=
CONFIG_CINDER_NETAPP_WEBSERVICE_PATH=/devmgr/v2
CONFIG_CINDER_NETAPP_STORAGE_POOLS=
CONFIG_NOVA_DB_PW=1e1b5aeeeaf342a8
CONFIG_NOVA_KS_PW=d9583177a2444f06
CONFIG_NOVA_SCHED_CPU_ALLOC_RATIO=16.0
CONFIG_NOVA_SCHED_RAM_ALLOC_RATIO=1.5
CONFIG_NOVA_COMPUTE_MIGRATE_PROTOCOL=tcp
CONFIG_NOVA_COMPUTE_PRIVIF=eth1
CONFIG_NOVA_NETWORK_MANAGER=nova.network.manager.FlatDHCPManager
CONFIG_NOVA_NETWORK_PUBIF=eth0
CONFIG_NOVA_NETWORK_PRIVIF=eth1
CONFIG_NOVA_NETWORK_FIXEDRANGE=192.168.32.0/22
CONFIG_NOVA_NETWORK_FLOATRANGE=10.3.4.0/22
CONFIG_NOVA_NETWORK_DEFAULTFLOATINGPOOL=nova
CONFIG_NOVA_NETWORK_AUTOASSIGNFLOATINGIP=n
CONFIG_NOVA_NETWORK_VLAN_START=100
CONFIG_NOVA_NETWORK_NUMBER=1
CONFIG_NOVA_NETWORK_SIZE=255
CONFIG_NEUTRON_KS_PW=808e36e154bd4cee
CONFIG_NEUTRON_DB_PW=0e2b927a21b44737
CONFIG_NEUTRON_L3_EXT_BRIDGE=br-ex
CONFIG_NEUTRON_L2_PLUGIN=ml2
CONFIG_NEUTRON_METADATA_PW=a965cd23ed2f4502
CONFIG_LBAAS_INSTALL=n
CONFIG_NEUTRON_METERING_AGENT_INSTALL=n
CONFIG_NEUTRON_FWAAS=n
CONFIG_NEUTRON_ML2_TYPE_DRIVERS=vxlan
CONFIG_NEUTRON_ML2_TENANT_NETWORK_TYPES=vxlan
CONFIG_NEUTRON_ML2_MECHANISM_DRIVERS=openvswitch
CONFIG_NEUTRON_ML2_FLAT_NETWORKS=*
CONFIG_NEUTRON_ML2_VLAN_RANGES=
CONFIG_NEUTRON_ML2_TUNNEL_ID_RANGES=1001:2000
CONFIG_NEUTRON_ML2_VXLAN_GROUP=239.1.1.2
CONFIG_NEUTRON_ML2_VNI_RANGES=1001:2000
CONFIG_NEUTRON_L2_AGENT=openvswitch
CONFIG_NEUTRON_LB_TENANT_NETWORK_TYPE=local
CONFIG_NEUTRON_LB_VLAN_RANGES=
CONFIG_NEUTRON_LB_INTERFACE_MAPPINGS=
CONFIG_NEUTRON_OVS_TENANT_NETWORK_TYPE=vxlan
CONFIG_NEUTRON_OVS_VLAN_RANGES=
CONFIG_NEUTRON_OVS_BRIDGE_MAPPINGS=physnet1:br-ex
CONFIG_NEUTRON_OVS_BRIDGE_IFACES=
CONFIG_NEUTRON_OVS_TUNNEL_RANGES=1001:2000
CONFIG_NEUTRON_OVS_TUNNEL_IF=eth1
CONFIG_NEUTRON_OVS_VXLAN_UDP_PORT=4789
CONFIG_HORIZON_SSL=n
CONFIG_SSL_CERT=
CONFIG_SSL_KEY=
CONFIG_SSL_CACHAIN=
CONFIG_SWIFT_KS_PW=8f75bfd461234c30
CONFIG_SWIFT_STORAGES=
CONFIG_SWIFT_STORAGE_ZONES=1
CONFIG_SWIFT_STORAGE_REPLICAS=1
CONFIG_SWIFT_STORAGE_FSTYPE=ext4
CONFIG_SWIFT_HASH=a60aacbedde7429a
CONFIG_SWIFT_STORAGE_SIZE=2G
CONFIG_PROVISION_DEMO=y
CONFIG_PROVISION_TEMPEST=n
CONFIG_PROVISION_TEMPEST_USER=
CONFIG_PROVISION_TEMPEST_USER_PW=44faa4ebc3da4459
CONFIG_PROVISION_DEMO_FLOATRANGE=172.24.4.224/28
CONFIG_PROVISION_TEMPEST_REPO_URI=https://github.com/openstack/tempest.git
CONFIG_PROVISION_TEMPEST_REPO_REVISION=master
CONFIG_PROVISION_ALL_IN_ONE_OVS_BRIDGE=n
CONFIG_HEAT_DB_PW=PW_PLACEHOLDER
CONFIG_HEAT_AUTH_ENC_KEY=fc3fb7fee61e46b0
CONFIG_HEAT_KS_PW=PW_PLACEHOLDER
CONFIG_HEAT_CLOUDWATCH_INSTALL=n
CONFIG_HEAT_USING_TRUSTS=y
CONFIG_HEAT_CFN_INSTALL=n
CONFIG_HEAT_DOMAIN=heat
CONFIG_HEAT_DOMAIN_ADMIN=heat_admin
CONFIG_HEAT_DOMAIN_PASSWORD=PW_PLACEHOLDER
CONFIG_CEILOMETER_SECRET=19ae0e7430174349
CONFIG_CEILOMETER_KS_PW=337b08d4b3a44753
CONFIG_MONGODB_HOST=192.169.142.127
CONFIG_NAGIOS_PW=02f168ee8edd44e4

********************
Then follow :-
********************
RDO Liberty DVR Neutron workflow on CentOS 7.1
http://dbaxps.blogspot.ru/2015/10/rdo-liberty-rc-dvr-deployment.html

[root@ip-192-169-142-127 ~(keystone_admin)]# nova-manage --version
No handlers could be found for logger "oslo_config.cfg"
13.0.0
[root@ip-192-169-142-127 ~(keystone_admin)]#  neutron l3-agent-list-hosting-router RouterDMS
+--------------------------------------+----------------------------------------+----------------+-------+----------+
| id                                   | host                                   | admin_state_up | alive | ha_state |
+--------------------------------------+----------------------------------------+----------------+-------+----------+
| 0e5f8de4-bae4-4b92-872c-b4a692ffca2b | ip-192-169-142-147.ip.secureserver.net | True           | :-)   |          |
| a08b319b-ce27-4b7c-8f72-e530c148ab70 | ip-192-169-142-127.ip.secureserver.net | True           | :-)   |          |
| ebab6768-cd2f-4d09-a1ba-a6e3aa6b4751 | ip-192-169-142-137.ip.secureserver.net | True           | :-)   |          |
+--------------------------------------+----------------------------------------+----------------+-------+----------+

[root@ip-192-169-142-127 ~(keystone_admin)]# neutron agent-show 0e5f8de4-bae4-4b92-872c-b4a692ffca2b
+---------------------+-------------------------------------------------------------------------------+
| Field               | Value                                                                         |
+---------------------+-------------------------------------------------------------------------------+
| admin_state_up      | True                                                                          |
| agent_type          | L3 agent                                                                      |
| alive               | True                                                                          |
| binary              | neutron-l3-agent                                                              |
| configurations      | {                                                                             |
|                     |      "router_id": "",                                                         |
|                     |      "agent_mode": "dvr",                                                     |
|                     |      "gateway_external_network_id": "",                                       |
|                     |      "handle_internal_only_routers": true,                                    |
|                     |      "use_namespaces": true,                                                  |
|                     |      "routers": 1,                                                            |
|                     |      "interfaces": 1,                                                         |
|                     |      "floating_ips": 1,                                                       |
|                     |      "interface_driver": "neutron.agent.linux.interface.OVSInterfaceDriver",  |
|                     |      "log_agent_heartbeats": false,                                           |
|                     |      "external_network_bridge": "br-ex",                                      |
|                     |      "ex_gw_ports": 1                                                         |
|                     | }                                                                             |
| created_at          | 2015-12-08 09:44:37                                                           |
| description         |                                                                               |
| heartbeat_timestamp | 2015-12-08 20:07:16                                                           |
| host                | ip-192-169-142-147.ip.secureserver.net                                        |
| id                  | 0e5f8de4-bae4-4b92-872c-b4a692ffca2b                                          |
| started_at          | 2015-12-08 12:11:16                                                           |
| topic               | l3_agent                                                                      |
+---------------------+-------------------------------------------------------------------------------+

[root@ip-192-169-142-127 ~(keystone_admin)]# nova service-list
+----+------------------+----------------------------------------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary           | Host                                   | Zone     | Status  | State | Updated_at                 | Disabled Reason |
+----+------------------+----------------------------------------+----------+---------+-------+----------------------------+-----------------+
| 1  | nova-consoleauth | ip-192-169-142-127.ip.secureserver.net | internal | enabled | up    | 2015-12-08T18:44:41.000000 | -               |
| 2  | nova-scheduler   | ip-192-169-142-127.ip.secureserver.net | internal | enabled | up    | 2015-12-08T18:44:41.000000 | -               |
| 3  | nova-conductor   | ip-192-169-142-127.ip.secureserver.net | internal | enabled | up    | 2015-12-08T18:44:41.000000 | -               |
| 4  | nova-cert        | ip-192-169-142-127.ip.secureserver.net | internal | enabled | up    | 2015-12-08T18:44:41.000000 | -               |
| 5  | nova-compute     | ip-192-169-142-147.ip.secureserver.net | nova     | enabled | up    | 2015-12-08T18:44:36.000000 | -               |
| 6  | nova-compute     | ip-192-169-142-137.ip.secureserver.net | nova     | enabled | up    | 2015-12-08T18:44:42.000000 | -               |
+----+------------------+----------------------------------------+----------+---------+-------+----------------------------+-----------------+

[root@ip-192-169-142-127 ~(keystone_admin)]#  neutron l3-agent-list-hosting-router RouterDMS
+--------------------------------------+----------------------------------------+----------------+-------+----------+
| id                                   | host                                   | admin_state_up | alive | ha_state |
+--------------------------------------+----------------------------------------+----------------+-------+----------+
| 0e5f8de4-bae4-4b92-872c-b4a692ffca2b | ip-192-169-142-147.ip.secureserver.net | True           | :-)   |          |
| a08b319b-ce27-4b7c-8f72-e530c148ab70 | ip-192-169-142-127.ip.secureserver.net | True           | :-)   |          |
| ebab6768-cd2f-4d09-a1ba-a6e3aa6b4751 | ip-192-169-142-137.ip.secureserver.net | True           | :-)   |          |
+--------------------------------------+----------------------------------------+----------------+-------+----------+

[root@ip-192-169-142-127 ~(keystone_admin)]# neutron agent-list
+--------------------------------------+--------------------+----------------------------------------+-------+----------------+---------------------------+
| id                                   | agent_type         | host                                   | alive | admin_state_up | binary                    |
+--------------------------------------+--------------------+----------------------------------------+-------+----------------+---------------------------+
| 0e5f8de4-bae4-4b92-872c-b4a692ffca2b | L3 agent           | ip-192-169-142-147.ip.secureserver.net | :-)   | True           | neutron-l3-agent          |
| 18f3207c-9cde-441c-bfda-26e35b393b5b | Open vSwitch agent | ip-192-169-142-147.ip.secureserver.net | :-)   | True           | neutron-openvswitch-agent |
| 1d22e3ae-9f27-4cf2-aff7-9febb6745a28 | Open vSwitch agent | ip-192-169-142-127.ip.secureserver.net | :-)   | True           | neutron-openvswitch-agent |
| 39dd0e7b-9359-4ea8-890d-bfe2ebda95b8 | Open vSwitch agent | ip-192-169-142-137.ip.secureserver.net | :-)   | True           | neutron-openvswitch-agent |
| 3fd7c6d3-f2bd-4612-9f88-4e7c01ceb90d | DHCP agent         | ip-192-169-142-127.ip.secureserver.net | :-)   | True           | neutron-dhcp-agent        |
| 55f597f5-0b7a-414a-aae2-1098b79fdec4 | Metadata agent     | ip-192-169-142-137.ip.secureserver.net | :-)   | True           | neutron-metadata-agent    |
| a08b319b-ce27-4b7c-8f72-e530c148ab70 | L3 agent           | ip-192-169-142-127.ip.secureserver.net | :-)   | True           | neutron-l3-agent          |
| d8bed085-b87c-4d7d-be8d-503b9ce60eda | Metadata agent     | ip-192-169-142-127.ip.secureserver.net | :-)   | True           | neutron-metadata-agent    |
| ddf1906c-09bd-4415-ab62-4a8d290f7fe9 | Metadata agent     | ip-192-169-142-147.ip.secureserver.net | :-)   | True           | neutron-metadata-agent    |
| ebab6768-cd2f-4d09-a1ba-a6e3aa6b4751 | L3 agent           | ip-192-169-142-137.ip.secureserver.net | :-)   | True           | neutron-l3-agent          |
+--------------------------------------+--------------------+----------------------------------------+-------+----------------+---------------------------+

[root@ip-192-169-142-127 ~(keystone_admin)]# neutron router-port-list RouterDMS
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                              |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
| 48ec616f-1c9c-4654-a5bc-0626765801bf |      | fa:16:3e:4b:05:09 | {"subnet_id": "8d8210c4-9900-4245-a57f-cdcaa2e8f916", "ip_address": "50.0.0.11"}       |
| 7d7f60e7-f385-4eb2-ada9-d44e85a05200 |      | fa:16:3e:28:0d:19 | {"subnet_id": "8d8210c4-9900-4245-a57f-cdcaa2e8f916", "ip_address": "50.0.0.1"}        |
| 835367f9-8a98-4a44-9a51-0f07ad2b7e82 |      | fa:16:3e:58:a4:83 | {"subnet_id": "c79f4a19-ff8e-40e3-8868-dd6b23723ea4", "ip_address": "192.169.142.150"} |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+
[root@ip-192-169-142-127 ~(keystone_admin)]#  neutron router-show RouterDMS
+-----------------------+------------------------===------------------------------------------------+
| Field                 | Value                                                                       |
+-----------------------+--------------------------------------------------------------------------------+
| admin_state_up    | True                                                                    |
| distributed           | True                                                                     |
| external_gateway_info | {"network_id": "252498c4-c7d8-4748-9583-587a52eb8b94", "enable_snat": true, "external_fixed_ips": [{"subnet_id": "c79f4a19-ff8e-40e3-8868-dd6b23723ea4", "ip_address": "192.169.142.150"}]} |
| ha                    | False                                                                         |
| id                     | ad7c9612-408e-4dcb-ac16-32ed916f28b3               |
| name               | RouterDMS                                                               |
| routes              |                                                                                  |
| status              | ACTIVE                                                                     |
| tenant_id         | fd5942f812284d0c99ec25485cc3b297                    |   
+-----------------------+---------------------------------------------------------------------------------+



  

  
*******************************
DVR_SNAT Section 
*******************************
Controller :-

  

 
  
OVS Flows

[root@ip-192-169-142-127 ~(keystone_admin)]# ovs-ofctl show br-int | grep "sg-"
 8(sg-48ec616f-1c): addr:00:00:00:00:00:00

[root@ip-192-169-142-127 ~(keystone_admin)]# ovs-ofctl dump-flows  br-int| grep "output:8"
 cookie=0x976c449145b02ae6, duration=4239.096s, table=1, n_packets=1427417, n_bytes=94772642, idle_age=0, priority=4,dl_vlan=1,dl_dst=fa:16:3e:4b:05:09 actions=strip_vlan,mod_dl_src:fa:16:3e:28:0d:19,output:8
[root@ip-192-169-142-127 ~(keystone_admin)]# ovs-ofctl dump-flows  br-int| grep "output:8"
 cookie=0x976c449145b02ae6, duration=4241.129s, table=1, n_packets=1429159, n_bytes=94888334, idle_age=0, priority=4,dl_vlan=1,dl_dst=fa:16:3e:4b:05:09 actions=strip_vlan,mod_dl_src:fa:16:3e:28:0d:19,output:8
[root@ip-192-169-142-127 ~(keystone_admin)]# ovs-ofctl dump-flows  br-int| grep "output:8"
 cookie=0x976c449145b02ae6, duration=4245.441s, table=1, n_packets=1432026, n_bytes=95078792, idle_age=0, priority=4,dl_vlan=1,dl_dst=fa:16:3e:4b:05:09 actions=strip_vlan,mod_dl_src:fa:16:3e:28:0d:19,output:8
[root@ip-192-169-142-127 ~(keystone_admin)]# ovs-ofctl dump-flows  br-int| grep "output:8"
 cookie=0x976c449145b02ae6, duration=4249.608s, table=1, n_packets=1434642, n_bytes=95252072, idle_age=0, priority=4,dl_vlan=1,dl_dst=fa:16:3e:4b:05:09 actions=strip_vlan,mod_dl_src:fa:16:3e:28:0d:19,output:8
[root@ip-192-169-142-127 ~(keystone_admin)]# ovs-ofctl dump-flows  br-int| grep "output:8"
 cookie=0x976c449145b02ae6, duration=4256.953s, table=1, n_packets=1439185, n_bytes=95553634, idle_age=0, priority=4,dl_vlan=1,dl_dst=fa:16:3e:4b:05:09 actions=strip_vlan,mod_dl_src:fa:16:3e:28:0d:19,output:8
[root@ip-192-169-142-127 ~(keystone_admin)]# ovs-ofctl dump-flows  br-int| grep "output:8"
 cookie=0x976c449145b02ae6, duration=4262.048s, table=1, n_packets=1442924, n_bytes=95800408, idle_age=0, priority=4,dl_vlan=1,dl_dst=fa:16:3e:4b:05:09 actions=strip_vlan,mod_dl_src:fa:16:3e:28:0d:19,