Wednesday, July 27, 2016

TripleO QuickStart vs official Mitaka TripleO HA install via instack-virt-setup

   A final target of this post is to compare undercloud configuration been built by QuickStart and undercloud configuration been built per official documentation
for Mitaka stable , please see Attempt of official Mitaka TripleO HA install via instack-virt-setup
  Final version of instructions set may bee seen here :-
Stable Mitaka HA instack-virt-setup on CentOS 7.2 VIRTHOST

========================
TripleO QuickStart case
========================

First of all right before running `openstack overcloud deploy --templates .... `
Run on undercloud VM following commands :-

[stack@undercloud ~]$ sudo ovs-vsctl show
b8b5ecbc-dc8d-43b8-8f03-09896d1b08b3
    Bridge br-int
        fail_mode: secure
        Port int-br-ctlplane
            Interface int-br-ctlplane
                type: patch
                options: {peer=phy-br-ctlplane}
        Port br-int
            Interface br-int
                type: internal
        Port "tapd7a65b7a-48"
            tag: 1
            Interface "tapd7a65b7a-48"
                type: internal
    Bridge br-ctlplane
        Port "vlan10"
            tag: 10
            Interface "vlan10"
                type: internal
        Port br-ctlplane
            Interface br-ctlplane
                type: internal
        Port "eth1"
            Interface "eth1"
        Port phy-br-ctlplane
            Interface phy-br-ctlplane
                type: patch
                options: {peer=int-br-ctlplane}
    ovs_version: "2.5.0"

=============================

[root@undercloud ~]# ifconfig
br-ctlplane: flags=4163  mtu 1500
        inet 192.0.2.1  netmask 255.255.255.0  broadcast 192.0.2.255
        inet6 fe80::28e:5aff:fe16:9ba1  prefixlen 64  scopeid 0x20
        ether 00:8e:5a:16:9b:a1  txqueuelen 0  (Ethernet)
        RX packets 3383615  bytes 264121585 (251.8 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4873995  bytes 23750747704 (22.1 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163  mtu 1500
        inet 192.168.23.10  netmask 255.255.255.0  broadcast 192.168.23.255
        inet6 fe80::28e:5aff:fe16:9b9f  prefixlen 64  scopeid 0x20
        ether 00:8e:5a:16:9b:9f  txqueuelen 1000  (Ethernet)
        RX packets 48092  bytes 42203536 (40.2 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 35731  bytes 4188571 (3.9 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth1: flags=4163  mtu 1500
        inet6 fe80::28e:5aff:fe16:9ba1  prefixlen 64  scopeid 0x20
        ether 00:8e:5a:16:9b:a1  txqueuelen 1000  (Ethernet)
        RX packets 3385562  bytes 264368815 (252.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4876692  bytes 23773014677 (22.1 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10
        loop  txqueuelen 0  (Local Loopback)
        RX packets 3065638  bytes 25610179577 (23.8 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3065638  bytes 25610179577 (23.8 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virbr0: flags=4099  mtu 1500
        inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
        ether 52:54:00:eb:ef:39  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vlan10: flags=4163  mtu 1500
        inet 10.0.0.1  netmask 255.255.255.0  broadcast 10.0.0.255
        inet6 fe80::5ce2:8eff:fed9:2f89  prefixlen 64  scopeid 0x20
        ether 5e:e2:8e:d9:2f:89  txqueuelen 0  (Ethernet)
        RX packets 1154  bytes 176564 (172.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1759  bytes 22168381 (21.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

===============================================
Analyze code undercloud-post-install.sh following bellow
===============================================
[stack@undercloud ~]$ cat  undercloud-post-install.sh
#!/bin/bash

# Prepare the undercloud for deploy

set -eux

# Source in undercloud credentials.
source /home/stack/stackrc
######################
# Set of standard commands
######################

# Upload images to glance.
openstack overcloud image upload \

openstack baremetal import --json instackenv.json
openstack baremetal configure boot

# Perform introspection if requested.

. . . . . . . .

################################################
# Here follows critical VM network configuration portion
################################################

# enable NAT for "external" network
RULE="-s 10.0.0.1/24 ! -d 10.0.0.1/24 -j MASQUERADE"

if ! sudo iptables -t nat -C BOOTSTACK_MASQ $RULE; then
    sudo iptables -t nat -A BOOTSTACK_MASQ $RULE
    sudo sh -c 'iptables-save > /etc/sysconfig/iptables'
fi

sudo bash -c 'cat < /etc/sysconfig/network-scripts/ifcfg-vlan10
DEVICE=vlan10
ONBOOT=yes
DEVICETYPE=ovs
TYPE=OVSIntPort
BOOTPROTO=static
IPADDR=10.0.0.1
NETMASK=255.255.255.0
OVS_BRIDGE=br-ctlplane
OVS_OPTIONS="tag=10"
EOF'

sudo ifup ifcfg-vlan10

# clone the t-h-t templates if neede
d

=========================================
So finally up on overcloud-deployment completion :-
=========================================

[root@undercloud ~]# ip netns
qdhcp-74126965-fbac-483d-9d8d-1c2ff43a2bd2
[root@undercloud ~]# ip netns exec qdhcp-74126965-fbac-483d-9d8d-1c2ff43a2bd2 ifconfig
lo: flags=73  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10
        loop  txqueuelen 0  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tapd7a65b7a-48: flags=4163  mtu 1500
        inet 192.0.2.5  netmask 255.255.255.0  broadcast 192.0.2.255
        inet6 fe80::f816:3eff:fe9d:1a65  prefixlen 64  scopeid 0x20
        ether fa:16:3e:9d:1a:65  txqueuelen 0  (Ethernet)
        RX packets 1109  bytes 103765 (101.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 943  bytes 91866 (89.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@undercloud ~]# ip netns exec qdhcp-74126965-fbac-483d-9d8d-1c2ff43a2bd2 route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.0.2.1       0.0.0.0         UG    0      0        0 tapd7a65b7a-48
192.0.2.0       0.0.0.0         255.255.255.0   U     0      0        0 tapd7a65b7a-48


=======================================================
Get back to `ovs-vsctl show` on undercloud generated by QuickStart
=======================================================

Focus on device tapd7a65b7a-48 and vlan10

[stack@undercloud ~]$ sudo ovs-vsctl show
b8b5ecbc-dc8d-43b8-8f03-09896d1b08b3
    Bridge br-int
        fail_mode: secure
        Port int-br-ctlplane
            Interface int-br-ctlplane
                type: patch
                options: {peer=phy-br-ctlplane}  <=== veth pair connecting br-int
        Port br-int                                                               and br-ctlplane
            Interface br-int
                type: internal
        Port "tapd7a65b7a-48"
            tag: 1
            Interface "tapd7a65b7a-48"
                type: internal
    Bridge br-ctlplane
        Port "vlan10"
            tag: 10
            Interface "vlan10"
                type: internal
        Port br-ctlplane
            Interface br-ctlplane
                type: internal
        Port "eth1"
            Interface "eth1"
        Port phy-br-ctlplane
            Interface phy-br-ctlplane
                type: patch
                options: {peer=int-br-ctlplane}  <=== veth pair connecting
    ovs_version: "2.5.0"                                          connecting br-int and
                                                                             br-ctlplane

==================================================================
==================================================================

[stack@ServerCentOS72 ~]$ virsh list --all
 Id    Name                           State
----------------------------------------------------
 2     instack                        running
 -     baremetalbrbm_0                shut off
 -     baremetalbrbm_1                shut off
 -     baremetalbrbm_2                shut off
 -     baremetalbrbm_3                shut off

[stack@ServerCentOS72 ~]$ ssh root@192.168.122.193
Last login: Mon Jul 25 13:59:52 2016 from 192.168.122.1
[root@instack ~]# su - stack
Last login: Mon Jul 25 13:59:54 UTC 2016 on pts/5
[stack@instack ~]$ . stackrc
[stack@instack ~]$ sudo ovs-vsctl show
bc1c13cd-3651-4f79-87df-bdaf4f5fec01
    Bridge br-ctlplane
        Port br-ctlplane
            Interface br-ctlplane
                type: internal
        Port phy-br-ctlplane
            Interface phy-br-ctlplane
                type: patch
                options: {peer=int-br-ctlplane}
        Port "eth1"
            Interface "eth1"
        Port "vlan10"
            tag: 10
            Interface "vlan10"
                error: "could not open network device vlan10 (No such device)"

    Bridge br-int
        fail_mode: secure
        Port "tap41e6fddf-31"
            tag: 1
            Interface "tap41e6fddf-31"
                type: internal
        Port int-br-ctlplane
            Interface int-br-ctlplane
                type: patch
                options: {peer=phy-br-ctlplane}
        Port br-int
            Interface br-int
                type: internal
    ovs_version: "2.5.0"

[stack@instack ~]$ ifconfig
br-ctlplane: flags=4163  mtu 1500
        inet 192.0.2.1  netmask 255.255.255.0  broadcast 192.0.2.255
        inet6 fe80::297:fff:fe5c:c66c  prefixlen 64  scopeid 0x20
        ether 00:97:0f:5c:c6:6c  txqueuelen 0  (Ethernet)
        RX packets 13  bytes 1038 (1.0 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 12  bytes 816 (816.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163  mtu 1500
        inet 192.168.122.193  netmask 255.255.255.0  broadcast 192.168.122.255
        inet6 fe80::5054:ff:fe6f:906a  prefixlen 64  scopeid 0x20
        ether 52:54:00:6f:90:6a  txqueuelen 1000  (Ethernet)
        RX packets 1674  bytes 213273 (208.2 KiB)
        RX errors 0  dropped 9  overruns 0  frame 0
        TX packets 1078  bytes 163033 (159.2 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth1: flags=4163  mtu 1500
        inet6 fe80::297:fff:fe5c:c66c  prefixlen 64  scopeid 0x20
        ether 00:97:0f:5c:c6:6c  txqueuelen 1000  (Ethernet)
        RX packets 8  bytes 648 (648.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 14  bytes 1108 (1.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10
        loop  txqueuelen 0  (Local Loopback)
        RX packets 31888  bytes 10276736 (9.8 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 31888  bytes 10276736 (9.8 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

=======================================================
Thus any attempt to activate "Network Isolation"  having External Network
  running within
=======================================================

 openstack overcloud deploy --templates --libvirt-type qemu \
--control-scale 3 \
--compute-scale 1 \
-e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml \
--ntp-server pool.ntp.org

after committing all instructions from http://docs.openstack.org/developer/tripleo-docs/basic_deployment/basic_deployment_cli.html
is supposed to fail. Instack VM is missing device vlan10 supposed to become external interface attached as OVS port to br-ctlplane. What I believe was done in http://mariosandreou.com/tripleo/2016/06/17/deploy-tripleo-stable-mitaka.html  without explicitly advertising.

=================================================================
Get back to post mentioned in the header , we want Mitaka Tripleo deployment
to run on instack VM with "Network Isolation" setting up External network,
Network serving VXLAN tunnels . In regards of Ceph Nodes overcloud deployment "Network Isolation"  is obviously extremely important.
So, vlan10 device creation should be done with no doubts.
=================================================================

sudo bash -c 'cat <  /etc/sysconfig/network-scripts/ifcfg-vlan10
DEVICE=vlan10
ONBOOT=yes
DEVICETYPE=ovs
TYPE=OVSIntPort
BOOTPROTO=static
IPADDR=10.0.0.1
NETMASK=255.255.255.0
OVS_BRIDGE=br-ctlplane
OVS_OPTIONS="tag=10"
EOF'

sudo ifup ifcfg-vlan10

sudo iptables -A BOOTSTACK_MASQ -s 10.0.0.0/24 ! -d 10.0.0.0/24 -j MASQUERADE -t nat

=============================
Make sure updates are done
=============================

[boris@ServerCentOS72 ~]$ sudo su -
[sudo] password for boris:
Last login: Tue Jul 26 03:58:20 MSK 2016 on pts/0
[root@ServerCentOS72 ~]# su - stack
Last login: Tue Jul 26 03:58:45 MSK 2016 on pts/0
[stack@ServerCentOS72 ~]$ ssh root@192.168.122.193
Last login: Tue Jul 26 01:01:49 2016
[root@instack ~]# su - stack
Last login: Tue Jul 26 01:01:34 UTC 2016 on pts/0
[stack@instack ~]$ sudo ovs-vsctl show
bc1c13cd-3651-4f79-87df-bdaf4f5fec01
    Bridge br-ctlplane
        Port "eth1"
            Interface "eth1"
        Port br-ctlplane
            Interface br-ctlplane
                type: internal
        Port phy-br-ctlplane
            Interface phy-br-ctlplane
                type: patch
                options: {peer=int-br-ctlplane}
        Port "vlan10"
            tag: 10
            Interface "vlan10"
                type: internal
    Bridge br-int
        fail_mode: secure
        Port "tap41e6fddf-31"
            tag: 1
            Interface "tap41e6fddf-31"
                type: internal
        Port int-br-ctlplane
            Interface int-br-ctlplane
                type: patch
                options: {peer=phy-br-ctlplane}
        Port br-int
            Interface br-int
                type: internal
    ovs_version: "2.5.0"
[stack@instack ~]$ ifconfig
br-ctlplane: flags=4163  mtu 1500
        inet 192.0.2.1  netmask 255.255.255.0  broadcast 192.0.2.255
        inet6 fe80::297:fff:fe5c:c66c  prefixlen 64  scopeid 0x20
        ether 00:97:0f:5c:c6:6c  txqueuelen 0  (Ethernet)
        RX packets 2751712  bytes 205714576 (196.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2711617  bytes 12789727774 (11.9 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163  mtu 1500
        inet 192.168.122.193  netmask 255.255.255.0  broadcast 192.168.122.255
        inet6 fe80::5054:ff:fe6f:906a  prefixlen 64  scopeid 0x20
        ether 52:54:00:6f:90:6a  txqueuelen 1000  (Ethernet)
        RX packets 4767  bytes 590862 (577.0 KiB)
        RX errors 0  dropped 9  overruns 0  frame 0
        TX packets 3138  bytes 488880 (477.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth1: flags=4163  mtu 1500
        inet6 fe80::297:fff:fe5c:c66c  prefixlen 64  scopeid 0x20
        ether 00:97:0f:5c:c6:6c  txqueuelen 1000  (Ethernet)
        RX packets 2751684  bytes 205708317 (196.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2711674  bytes 12789742191 (11.9 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10
        loop  txqueuelen 0  (Local Loopback)
        RX packets 319388  bytes 1493930109 (1.3 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 319388  bytes 1493930109 (1.3 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vlan10: flags=4163  mtu 1500
        inet 10.0.0.1  netmask 255.255.255.0  broadcast 10.0.0.255
        inet6 fe80::1478:deff:fe20:7b86  prefixlen 64  scopeid 0x20
        ether 16:78:de:20:7b:86  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 12  bytes 816 (816.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

==================================================
Proceed as follows.   Create file network-environment template :-
==================================================
[stack@instack ~]$ cat network_env.yaml
{
    "parameter_defaults": {
        "ControlPlaneDefaultRoute": "192.0.2.1",
        "ControlPlaneSubnetCidr": "24",
        "DnsServers": [
            "192.168.23.1"
        ],
        "EC2MetadataIp": "192.0.2.1",
        "ExternalAllocationPools": [
            {
                "end": "10.0.0.250",
                "start": "10.0.0.4"
            }
        ],
        "ExternalNetCidr": "10.0.0.1/24",
        "NeutronExternalNetworkBridge": ""
    }
}

=====================
Then run :-
=====================
[stack@instack ~]$ source stackrc

[stack@instack ~]$ openstack overcloud deploy --templates --control-scale 3 \
  --compute-scale 1 \
  --libvirt-type qemu \
  --ntp-server pool.ntp.org  \
  -e  /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml \
  -e  /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
  -e  /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml \
  -e  $HOME/network_env.yaml
. . . .

Stack overcloud CREATE_COMPLETE
/home/stack/.ssh/known_hosts updated.
Original contents retained as /home/stack/.ssh/known_hosts.old
Skipping "horizon" postconfig because it wasn't found in the endpoint map output
PKI initialization in init-keystone is deprecated and will be removed.
Warning: Permanently added '192.0.2.16' (ECDSA) to the list of known hosts.
The following cert files already exist, use --rebuild to remove the existing files before regenerating:
/etc/keystone/ssl/certs/ca.pem already exists
/etc/keystone/ssl/private/signing_key.pem already exists
/etc/keystone/ssl/certs/signing_cert.pem already exists
Connection to 192.0.2.16 closed.
Overcloud Endpoint: http://10.0.0.4:5000/v2.0
Overcloud Deployed

[stack@instack ~]$ nova list
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                    | Status | Task State | Power State | Networks            |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| 068dcf61-1c07-49d3-97f9-66e0ff1896e4 | overcloud-controller-0  | ACTIVE | -          | Running     | ctlplane=192.0.2.19 |
| 1083bc50-4e30-4a8d-8a02-d60c35bab0b7 | overcloud-controller-1  | ACTIVE | -          | Running     | ctlplane=192.0.2.18 |
| 3d88de4e-2c25-4a7e-ac05-580d5e4532f5 | overcloud-controller-2  | ACTIVE | -          | Running     | ctlplane=192.0.2.20 |
| f4589428-ba17-44f5-b73c-db38af7963e9 | overcloud-novacompute-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.17 |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+

  
[stack@instack ~]$ neutron net-list
+--------------------------------------+--------------+--------------------------------------------+
| id                                   | name         | subnets                                    |
+--------------------------------------+--------------+--------------------------------------------+
| cc29c009-f2c8-457c-a92c-021acf650b78 | tenant       | 3afaf44d-19b5-46ac-8534-fe1520a14a1c       |
|                                      |              | 172.16.0.0/24                              |
| e8e8d778-1992-4ee2-9b4e-ac349e8d7985 | external     | 8ad5eeef-6860-4781-89e6-5132cf633013       |
|                                      |              | 10.0.0.0/24                                |
| 00bbd0fb-94b0-406a-b5b1-aa60b5526898 | internal_api | 1b84e01e-deb9-458c-990a-94d92f69f668       |
|                                      |              | 172.16.2.0/24                              |
| 65d426b0-0b02-4654-b598-1ba368a43d35 | storage      | 849719fb-877c-49f7-a606-959e4720011d       |
|                                      |              | 172.16.1.0/24                              |
| 687769a2-5438-434d-8085-2988e592755b | storage_mgmt | fc36759d-25ee-4cbe-908b-819dad6a222d       |
|                                      |              | 172.16.3.0/24                              |
| 584468f0-d26e-4a47-89a1-bca5847404fb | ctlplane     | 43d05014-098b-4eb5-8582-178404ff0e24       |
|                                      |              | 192.0.2.0/24                               |
+--------------------------------------+--------------+--------------------------------------------+

[stack@instack ~]$ ssh heat-admin@192.0.2.19
The authenticity of host '192.0.2.19 (192.0.2.19)' can't be established.
ECDSA key fingerprint is 61:31:94:19:12:51:a3:df:be:22:f6:0a:e7:dc:a1:d7.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.0.2.19' (ECDSA) to the list of known hosts.
Last login: Tue Jul 26 02:00:36 2016 from 192.0.2.1
[heat-admin@overcloud-controller-0 ~]$ sudo su -
[root@overcloud-controller-0 ~]# pcs status
Cluster name: tripleo_cluster
Last updated: Tue Jul 26 02:03:03 2016        Last change: Tue Jul 26 01:58:37 2016 by root via cibadmin on overcloud-controller-0
Stack: corosync
Current DC: overcloud-controller-2 (version 1.1.13-10.el7_2.2-44eb2dd) - partition with quorum
3 nodes and 127 resources configured

Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

Full list of resources:

 ip-192.0.2.16    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-0
 ip-172.16.2.5    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-1
 ip-172.16.3.4    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-2
 Clone Set: haproxy-clone [haproxy]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Master/Slave Set: galera-master [galera]
     Masters: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: memcached-clone [memcached]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 ip-10.0.0.4    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-0
 ip-172.16.2.4    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-1
 ip-172.16.1.4    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-2
 Clone Set: rabbitmq-clone [rabbitmq]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-core-clone [openstack-core]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Master/Slave Set: redis-master [redis]
     Masters: [ overcloud-controller-0 ]
     Slaves: [ overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: mongod-clone [mongod]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-aodh-evaluator-clone [openstack-aodh-evaluator]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-scheduler-clone [openstack-nova-scheduler]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-l3-agent-clone [neutron-l3-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-netns-cleanup-clone [neutron-netns-cleanup]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-ovs-cleanup-clone [neutron-ovs-cleanup]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 openstack-cinder-volume    (systemd:openstack-cinder-volume):    Started overcloud-controller-0
 Clone Set: openstack-heat-engine-clone [openstack-heat-engine]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-api-clone [openstack-ceilometer-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-aodh-listener-clone [openstack-aodh-listener]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-metadata-agent-clone [neutron-metadata-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-gnocchi-metricd-clone [openstack-gnocchi-metricd]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-aodh-notifier-clone [openstack-aodh-notifier]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-heat-api-clone [openstack-heat-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-collector-clone [openstack-ceilometer-collector]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-glance-api-clone [openstack-glance-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-cinder-scheduler-clone [openstack-cinder-scheduler]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-api-clone [openstack-nova-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-consoleauth-clone [openstack-nova-consoleauth]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-sahara-api-clone [openstack-sahara-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-heat-api-cloudwatch-clone [openstack-heat-api-cloudwatch]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-sahara-engine-clone [openstack-sahara-engine]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-glance-registry-clone [openstack-glance-registry]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-gnocchi-statsd-clone [openstack-gnocchi-statsd]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-notification-clone [openstack-ceilometer-notification]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-cinder-api-clone [openstack-cinder-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-dhcp-agent-clone [neutron-dhcp-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-openvswitch-agent-clone [neutron-openvswitch-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-novncproxy-clone [openstack-nova-novncproxy]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: delay-clone [delay]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-server-clone [neutron-server]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-central-clone [openstack-ceilometer-central]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: httpd-clone [httpd]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-heat-api-cfn-clone [openstack-heat-api-cfn]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-conductor-clone [openstack-nova-conductor]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

PCSD Status:
  overcloud-controller-0: Online
  overcloud-controller-1: Online
  overcloud-controller-2: Online

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

===============================================
Verification Galera in sync &&  rabbitmqctl cluster_status
===============================================