Saturday, June 4, 2016

Triple0 QuickStart HA Setup on Intel Core i7-4790 Desktop

This posting follows up Deploying OpenStack on just one hosted server   but is focused on utilizing i7 4790/4770  CPUs  with inexpensive boards like ASUS Z97-P having 32 GB RAM.  As remote workstation C2D E8400 with 8 GB RAM has been used . OS installed on both PCs was CentOS 7.2 (Release 1604). Mentioned  environment allows in about couple of hours to obtain stable working configuration based on ha.yml template :-

######################
# Template code
######################
# Deploy an HA openstack environment.
#
# This will require (6144 * 4) == approx. 24GB for the overcloud
# nodes, plus another 8GB for the undercloud, for a total of around
# 32GB.
control_memory: 6144
compute_memory: 6144

undercloud_memory: 8192

# Giving the undercloud additional CPUs can greatly improve heat's
# performance (and result in a shorter deploy time).
undercloud_vcpu: 4

# Create three controller nodes and one compute node.
overcloud_nodes:
  - name: control_0
    flavor: control
  - name: control_1
    flavor: control
  - name: control_2
    flavor: control

  - name: compute_0
    flavor: compute
  - name: compute_1
    flavor: compute

# We don't need introspection in a virtual environment (because we are
# creating all the "hardware" we really know the necessary
# information).
introspect: false

# Tell tripleo about our environment.
network_isolation: true
extra_args: >-
  --control-scale 3 --compute-scale 2 --neutron-network-type vxlan
  --neutron-tunnel-types vxlan
  -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml
  --ntp-server pool.ntp.org
deploy_timeout: 75
tempest: false
pingtest: true


**********************************************************************************
Fist fix bugs mentioned in link above on the Server's  Desktop
and run  `yum groupinstall "Virtualization Host"`
**********************************************************************************

Then start on workstation :-

$ git clone https://github.com/openstack/tripleo-quickstart
$ cd tripleo-quickstart
$ sudo bash quickstart.sh --install-deps
$ sudo yum -y  install redhat-rpm-config


$export VIRTHOST=192.168.1.75 #put your own IP here
$ ssh-keygen
$ssh-copy-id root@$VIRTHOST
$ssh root@$VIRTHOST uname -a # no root login prompt

Then run under tripleo-quickstart

$ bash quickstart.sh --config ./config/general_config/ha.yml  $VIRTHOST
 
If everything went well you will be brought back to command prompt
and see following output

##################################
Virtual Environment Setup Complete
##################################


Access the undercloud by:

 ssh -F /home/boris/.quickstart/ssh.config.ansible undercloud 
 
There are scripts in the home directory to continue the deploy:

 undercloud-install.sh will run the undercloud install
 undercloud-post-install.sh will perform all pre-deploy steps
 overcloud-deploy.sh will deploy the overcloud
 overcloud-deploy-post.sh will do any post-deploy configuration
 overcloud-validate.sh will run post-deploy validation

During overcloud deployment open remote ssh session to server and run top
You will see that memory allocation matches your  ha.yml
When done : su - stack on Server

[root@ServerCentOS72 ~]# su - stack
Last login: Sun Jun  5 09:08:16 MSK 2016 from 192.168.1.54 on pts/0
[stack@ServerCentOS72 ~]$ virsh list
 Id    Name                           State
----------------------------------------------------
 2     undercloud                     running
 8     control_1                      running
 9     compute_0                      running
 10    compute_1                      running
 11    control_0                      running
 12    control_2                      running


[stack@ServerCentOS72 ~]$ virsh dumpxml undercloud | grep vcpu
 <vcpu placement='static'>4</vcpu>
[stack@ServerCentOS72 ~]$ virsh dumpxml undercloud | grep memory
  <memory unit='KiB'>8388608</memory>
[stack@ServerCentOS72 ~]$ virsh dumpxml control_0  | grep memory
  <memory unit='KiB'>6291456</memory>
[stack@ServerCentOS72 ~]$ virsh dumpxml control_0  | grep cpu
  <vcpu placement='static'>1</vcpu>
  <cpu mode='host-passthrough'/>
[stack@ServerCentOS72 ~]$ virsh dumpxml compute_0  | grep cpu
  <vcpu placement='static'>1</vcpu>
  <cpu mode='host-passthrough'/>
[stack@ServerCentOS72 ~]$ virsh dumpxml compute_0  | grep memory
  <memory unit='KiB'>6291456</memory>
 
***************************************
Up on completion of last script run :-
***************************************
[stack@undercloud ~]$ . stackrc
[stack@undercloud ~]$ heat stack-list
+--------------------------------------+------------+-----------------+---------------------+--------------+
| id                                   | stack_name | stack_status    | creation_time       | updated_time |
+--------------------------------------+------------+-----------------+---------------------+--------------+
| 4474e9c7-74a0-4ba6-9cb4-889127a31c1f | overcloud  | CREATE_COMPLETE | 2016-06-05T06:27:44 | None         |
+--------------------------------------+------------+-----------------+---------------------+--------------+

[stack@undercloud ~]$ nova list
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                    | Status | Task State | Power State | Networks            |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| b1d93ac8-3e00-4f5c-8d02-73f0853c7297 | overcloud-controller-0  | ACTIVE | -          | Running     | ctlplane=192.0.2.11 |
| 69c560b7-7fcd-442e-8360-3c85f686b19f | overcloud-controller-1  | ACTIVE | -          | Running     | ctlplane=192.0.2.9  |
| ce73a87b-e49d-4b4b-9d45-b4297c39dbaf | overcloud-controller-2  | ACTIVE | -          | Running     | ctlplane=192.0.2.12 |
| 4cb7cb86-9f72-4654-ae83-34bd1e1576e7 | overcloud-novacompute-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.10 |
| 391d13b4-ed9b-45d4-96a4-f97b07d53088 | overcloud-novacompute-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.8  |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+


[stack@undercloud ~]$ ssh heat-admin@192.0.2.11
Last login: Sun Jun  5 08:34:21 2016 from gateway
[heat-admin@overcloud-controller-0 ~]$ sudo su -
Last login: Sun Jun  5 08:34:26 UTC 2016 on pts/0

[root@overcloud-controller-0 ~]# . keystonerc_admin
[root@overcloud-controller-0 ~]# neutron net-list
+---------------------------------+---------------------------------+-----------------------------------+
| id                              | name                            | subnets                           |
+---------------------------------+---------------------------------+-----------------------------------+
| 548abd13-dd34-4b8a-             | public                          | 8887421f-                         |
| b3b2-ccebc87ebffa               |                                 | eb7b-4639-b7be-066873038434       |
|                                 |                                 | 192.0.2.0/24                      |
| 7fe9187a-d46c-4e03-9e1f-        | HA network tenant 773c2a0603eb4 | 967d823a-                         |
| ae4ab56ca12f                    | 72eb1ca050a69f8ddfb             | 0fb0-4e80-99b0-977dfcead5b9       |
|                                 |                                 | 169.254.192.0/18                  |
| 2530d88f-5b03-4d8c-beab-        | demo_network                    | 83c1fd7f-3ee6-49f8-bdcb-          |
| ce176394aa8f                    |                                 | 132771a875d5 50.0.0.0/24          |
+---------------------------------+---------------------------------+-----------------------------------+ 

[root@overcloud-controller-0 ~]# pcs status
Cluster name: tripleo_cluster
Last updated: Sun Jun  5 07:20:59 2016        Last change: Sun Jun  5 07:10:31 2016 by root via cibadmin on overcloud-controller-0
Stack: corosync
Current DC: overcloud-controller-2 (version 1.1.13-10.el7_2.2-44eb2dd) - partition with quorum
3 nodes and 123 resources configured

Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

Full list of resources:

 ip-192.0.2.6    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-0
 Clone Set: haproxy-clone [haproxy]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 ip-192.0.2.7    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-1
 Master/Slave Set: galera-master [galera]
     Masters: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: memcached-clone [memcached]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: rabbitmq-clone [rabbitmq]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-core-clone [openstack-core]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Master/Slave Set: redis-master [redis]
     Masters: [ overcloud-controller-1 ]
     Slaves: [ overcloud-controller-0 overcloud-controller-2 ]
 Clone Set: mongod-clone [mongod]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-aodh-evaluator-clone [openstack-aodh-evaluator]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-scheduler-clone [openstack-nova-scheduler]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-l3-agent-clone [neutron-l3-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-netns-cleanup-clone [neutron-netns-cleanup]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-ovs-cleanup-clone [neutron-ovs-cleanup]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 openstack-cinder-volume    (systemd:openstack-cinder-volume):    Started overcloud-controller-2
 Clone Set: openstack-heat-engine-clone [openstack-heat-engine]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-api-clone [openstack-ceilometer-api]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-aodh-listener-clone [openstack-aodh-listener]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-metadata-agent-clone [neutron-metadata-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-gnocchi-metricd-clone [openstack-gnocchi-metricd]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-aodh-notifier-clone [openstack-aodh-notifier]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-heat-api-clone [openstack-heat-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-collector-clone [openstack-ceilometer-collector]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-glance-api-clone [openstack-glance-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-cinder-scheduler-clone [openstack-cinder-scheduler]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-api-clone [openstack-nova-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-consoleauth-clone [openstack-nova-consoleauth]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-sahara-api-clone [openstack-sahara-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-heat-api-cloudwatch-clone [openstack-heat-api-cloudwatch]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-sahara-engine-clone [openstack-sahara-engine]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-glance-registry-clone [openstack-glance-registry]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-gnocchi-statsd-clone [openstack-gnocchi-statsd]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-notification-clone [openstack-ceilometer-notification]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-cinder-api-clone [openstack-cinder-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-dhcp-agent-clone [neutron-dhcp-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-openvswitch-agent-clone [neutron-openvswitch-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-novncproxy-clone [openstack-nova-novncproxy]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: delay-clone [delay]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-server-clone [neutron-server]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-central-clone [openstack-ceilometer-central]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: httpd-clone [httpd]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-heat-api-cfn-clone [openstack-heat-api-cfn]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-conductor-clone [openstack-nova-conductor]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

Failed Actions:
 
 * openstack-aodh-evaluator_monitor_60000 on overcloud-controller-1 'not running' (7): call=78, status=complete, exitreason='none',
    last-rc-change='Sun Jun  5 07:06:55 2016', queued=0ms, exec=0ms
* openstack-ceilometer-central_start_0 on overcloud-controller-1 'not running' (7): call=316, status=complete, exitreason='none',
    last-rc-change='Sun Jun  5 07:13:25 2016', queued=0ms, exec=2113ms
* openstack-aodh-evaluator_monitor_60000 on overcloud-controller-0 'not running' (7): call=77, status=complete, exitreason='none',
    last-rc-change='Sun Jun  5 07:06:52 2016', queued=0ms, exec=0ms
* openstack-ceilometer-central_start_0 on overcloud-controller-0 'not running' (7): call=268, status=complete, exitreason='none',
    last-rc-change='Sun Jun  5 07:09:40 2016', queued=0ms, exec=2209ms
* openstack-aodh-evaluator_monitor_60000 on overcloud-controller-2 'not running' (7): call=76, status=complete, exitreason='none',
    last-rc-change='Sun Jun  5 07:06:51 2016', queued=0ms, exec=0ms
* openstack-ceilometer-central_start_0 on overcloud-controller-2 'not running' (7): call=312, status=complete, exitreason='none',
    last-rc-change='Sun Jun  5 07:13:30 2016', queued=0ms, exec=2119ms


PCSD Status:
  overcloud-controller-0: Online
  overcloud-controller-1: Online
  overcloud-controller-2: Online

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled



  Neutron reports

  


   System reports
  

  
 
  

   Running instances



*****************************************************
 Connections to VMs from undercloud
*****************************************************  
 [stack@undercloud ~]$ ssh -i oskey060516.pem fedora@192.0.2.151
Last login: Sun Jun  5 09:17:05 2016 from 192.0.2.1
[fedora@vf23dev01 ~]$ uname -a
Linux vf23dev01.localdomain 4.5.5-201.fc23.x86_64 #1 SMP Sat May 21 15:29:49 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
[fedora@vf23dev01 ~]$ Connection to 192.0.2.151 closed.
[stack@undercloud ~]$ ssh -i oskey060516.pem ubuntu@192.0.2.152
Welcome to Ubuntu 16.04 LTS (GNU/Linux 4.4.0-22-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

  Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud

14 packages can be updated.
5 updates are security updates.


Last login: Sun Jun  5 08:32:52 2016 from 192.0.2.1
ubuntu@ubuntudevs01:~$ uname -a
Linux ubuntudevs01 4.4.0-22-generic #40-Ubuntu SMP Thu May 12 22:03:46 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@ubuntudevs01:~$ logout
Connection to 192.0.2.152 closed.



   Hypervisors  on overcloud
  

  
   Server's CentOS 7.2 `top` report
  

Regarding details of ovecloud server's configuration

[root@overcloud-controller-0 network-scripts]# cat ifcfg-br-ex
# This file is autogenerated by os-net-config
DEVICE=br-ex
ONBOOT=yes
HOTPLUG=no
NM_CONTROLLED=no
DEVICETYPE=ovs
TYPE=OVSBridge
OVSBOOTPROTO=dhcp
OVSDHCPINTERFACES="eth0"
OVS_EXTRA="set bridge br-ex other-config:hwaddr=00:83:94:4b:f4:bf"

[root@overcloud-controller-0 network-scripts]# ifconfig
br-ex: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.0.2.10  netmask 255.255.255.0  broadcast 192.0.2.255
        inet6 fe80::283:94ff:fe4b:f4bf  prefixlen 64  scopeid 0x20<link>
        ether 00:83:94:4b:f4:bf  txqueuelen 0  (Ethernet)
        RX packets 1524142  bytes 482079467 (459.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1479958  bytes 289821172 (276.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::283:94ff:fe4b:f4bf  prefixlen 64  scopeid 0x20<link>
        ether 00:83:94:4b:f4:bf  txqueuelen 1000  (Ethernet)
        RX packets 1524492  bytes 482222219 (459.8 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1480362  bytes 289890148 (276.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

  

  Watch https://www.youtube.com/watch?v=8zFQG5mKwPk&feature=autoshare