MySQL Fabric high availability testing

 

Preamble

MySQL replication is a proven well working MySQL high available solution that can also be used to scale-out your database backend layer. But you must redirect all writes to the master server, this is only suitable if your application is massively read only. If your application is not mainly read only (and trend is clearly going in this direction) then to scale-out you would have to use another method called data sharding.

Management of MySQL Replication can be cumbersome and changing a slave to be the new primary master can be difficult. We have already seen other products to help you doing it and MySQL Fabric is one of them.

To try to ease management of big MySQL farms (for HA or sharding) Oracle has created a product called MySQL Fabric. From official documentation:

MySQL Fabric is built around an extensible framework for managing farms of MySQL Servers. Currently two features have been implemented – High Availability and scaling out using data sharding. These features can be used in isolation or in combination.

For testing I have three virtual machines:

  • server1.domain.com (192.168.56.101) is my initial MySQL master server
  • server2.domain.com (192.168.56.102) is my initial MySQL slave server
  • server3.domain.com (192.168.56.103) is my MySQL Fabric server (that also contains a MySQL instance for the MySQL Fabric repository)

They are all running Oracle Linux Server release 7.0 64 bits and MySQL 5.6.23 community edition. I have compiled and installed MySQL Utilities 1.5.3.

On server1.domain.com and server2.domain.com I configure a MySQL Replication environment using GTID that we have already seen in a previous post.

Remark:
As we are planning to change primary and secondary role it is a good practice to issue reset master on slave server to avoid replication of unwanted commands while changing master to slave. Those unwanted commands may stop your MySQL Replication and you would then need to recover few errors…

mysql> reset master;
Query OK, 0 rows affected (0.06 sec)

Your application will then connect to the MySQL Fabric server so it is a good idea to make it highly available i.e. running in a virtual machine for example. Currently the only fabric aware supported connector to connect to MySQL Fabric are PHP, Python and Java.

This blog post is about MySQL Fabric high availability the MySQL Fabric data sharding part will be seen in a further post.

I also have to mention that MySQL Fabric documentation is, so far, really poor. The reference section for example is far below minimum required…

MySQL Fabric installation

Compiling and installing MySQL Utilities (MySQL Fabric is par of MySQL Utilities) is really straightforward, you need python and few libraries (python release 2.7.5 on OEL 7.0):

[mysql@server3 mysql-utilities-1.5.3]$ pwd
/mysql/mysql-utilities-1.5.3
[mysql@server3 mysql-utilities-1.5.3]$ ./setup.py build
[mysql@server3 mysql-utilities-1.5.3]$ ./setup.py install

As stated in the introduction the MySQL Fabric server also need a MySQL database to store MySQL Fabric repository. To be honest I have been a bit disappointed by this as it is another database to manage, I would have preferred the configuration stored in a text file much easier to save and restore…So in the MySQL Fabric instance repository I create a fabric account:

mysql> CREATE USER fabric@'%';
Query OK, 0 rows affected (0.00 sec)
 
mysql> GRANT ALL ON fabric.* TO fabric@'%';
Query OK, 0 rows affected (0.01 sec)

In default /etc/mysql/fabric.cfg configuration file I have just changed the [storage] section to reflect my server name and MySQL non-default port (3316):

address = server3.domain.com:3316

And in [protocol.xmlrpc] and [protocol.mysql] sections I have set the password to avoid any prompt (never do this on a production server), bind on real server IP address has also been changed:

password = admin

In [servers] secton I set a password for all the fabric account created on each node of the group:

password = secure_password

It gives:

[mysql@server3 ~]$ mysqlfabric manage setup --param=storage.user=fabric
[INFO] 1424193041.522855 - MainThread - Initializing persister: user (fabric), server (server3.domain.com:3316), database (fabric).
[INFO] 1424193043.024686 - MainThread - Initial password for admin/mysql set
Password set for admin/mysql from configuration file.
[INFO] 1424193043.030541 - MainThread - Password set for admin/mysql from configuration file.
[INFO] 1424193043.032204 - MainThread - Initial password for admin/xmlrpc set
Password set for admin/xmlrpc from configuration file.
[INFO] 1424193043.037428 - MainThread - Password set for admin/xmlrpc from configuration file.
No result returned

You can confirm it has been done with:

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| fabric             |
| mysql              |
| performance_schema |
+--------------------+
4 rows in set (0.00 sec)
 
mysql> use fabric; show tables;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
 
Database changed
+-------------------+
| Tables_in_fabric  |
+-------------------+
| checkpoints       |
| error_log         |
| group_replication |
| group_view        |
| groups            |
| log               |
| machines          |
| permissions       |
| proc_view         |
| providers         |
| role_permissions  |
| roles             |
| servers           |
| shard_maps        |
| shard_ranges      |
| shard_tables      |
| shards            |
| user_roles        |
| users             |
+-------------------+
19 rows in set (0.00 sec)

Then on all MySQL servers I create a fabric account with:

mysql> grant all on *.* to fabric@'%' identified by 'secure_password';
Query OK, 0 rows affected (0.10 sec)
 
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

As I plan to launch Fabric process with MySQL account I have to create the log file with correct rights in advance:

[root@server3 ~]# touch /var/log/fabric.log
[root@server3 ~]# chown mysql:dba /var/log/fabric.log

Start Fabric with (–daemonize to get back the hand on your session):

[mysql@server3 ~]$ mysqlfabric manage start --daemonize
[mysql@server3 ~]$ ps -ef | grep mysqlfabric
mysql     4420     1  0 10:35 ?        00:00:00 /bin/python /bin/mysqlfabric manage start --daemonize
mysql     4439  4392  0 10:35 pts/1    00:00:00 grep --color=auto mysqlfabric

MySQL Fabric configuration

I create a Fabric group called group01:

[mysql@server3 ~]$ mysqlfabric group create group01
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1
 
                                uuid finished success result
------------------------------------ -------- ------- ------
c6e94388-2348-47d6-9020-4a44ba8e15cc        1       1      1
 
state success          when                                                   description
----- ------- ------------- -------------------------------------------------------------
    3       2 1424257295.96 Triggered by <mysql.fabric.events.Event object at 0x129e710>.
    4       2 1424257296.05                             Executing action (_create_group).
    5       2 1424257296.06                              Executed action (_create_group).

To confirm it has been well done:

[mysql@server3 ~]$ mysqlfabric group lookup_groups
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1
 
group_id description failure_detector master_uuid
-------- ----------- ---------------- -----------
 group01        None                0        None

Then I add my master and slave server to the Fabric group:

[mysql@server3 ~]$ mysqlfabric group add group01 server1.domain.com:3316
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1
 
                                uuid finished success result
------------------------------------ -------- ------- ------
69d11425-45e1-467d-a20b-00440ab47f3a        1       1      1
 
state success          when                                                   description
----- ------- ------------- -------------------------------------------------------------
    3       2 1424258147.45 Triggered by <mysql.fabric.events.Event object at 0x117fb50>.
    4       2 1424258147.47                               Executing action (_add_server).
    5       2 1424258147.64                                Executed action (_add_server).
 
 
[mysql@server3 ~]$ mysqlfabric group add group01 server2.domain.com:3316
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1
 
                                uuid finished success result
------------------------------------ -------- ------- ------
256f0b0f-25ea-4225-a2c9-138170e15aa6        1       1      1
 
state success          when                                                   description
----- ------- ------------- -------------------------------------------------------------
    3       2 1424258157.05 Triggered by <mysql.fabric.events.Event object at 0x117fb50>.
    4       2 1424258157.05                               Executing action (_add_server).
    5       2  1424258157.2                                Executed action (_add_server).

To confirm they have been added (please note that they are both with status SECONDARY):

[mysql@server3 ~]$ mysqlfabric group lookup_servers group01
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1
 
                         server_uuid                 address    status      mode weight
------------------------------------ ----------------------- --------- --------- ------
40b23f27-b5fb-11e4-aadc-0800271a543a server1.domain.com:3316 SECONDARY READ_ONLY    1.0
58d087d6-b5fb-11e4-aadd-080027df5a8f server2.domain.com:3316 SECONDARY READ_ONLY    1.0

I promote my master server, in my opinion the parameter –salve_id is a bit difficult to understand at here you have to provide the server id of your master server:

[mysql@server3 ~]$ mysqlfabric group promote group01 --slave_id=40b23f27-b5fb-11e4-aadc-0800271a543a
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1
 
                                uuid finished success result
------------------------------------ -------- ------- ------
abd05779-674a-4b59-9cd1-f501540462b1        1       1      1
 
state success          when                                                   description
----- ------- ------------- -------------------------------------------------------------
    3       2 1424260633.47  Triggered by <mysql.fabric.events.Event object at 0xfeae50>.
    4       2 1424260633.47                      Executing action (_define_ha_operation).
    5       2 1424260633.48                       Executed action (_define_ha_operation).
    3       2 1424260633.48 Triggered by <mysql.fabric.events.Event object at 0x1165250>.
    4       2 1424260633.48                     Executing action (_check_candidate_fail).
    5       2 1424260633.51                      Executed action (_check_candidate_fail).
    3       2 1424260633.51 Triggered by <mysql.fabric.events.Event object at 0x11652d0>.
    4       2 1424260633.51                          Executing action (_wait_slave_fail).
    5       2 1424260633.62                           Executed action (_wait_slave_fail).
    3       2 1424260633.61 Triggered by <mysql.fabric.events.Event object at 0x1165410>.
    4       2 1424260633.62                      Executing action (_change_to_candidate).
    5       2 1424260633.85                       Executed action (_change_to_candidate).

Configuration is now correct:

[mysql@server3 ~]$ mysqlfabric group lookup_servers group01
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1
 
                         server_uuid                 address    status       mode weight
------------------------------------ ----------------------- --------- ---------- ------
40b23f27-b5fb-11e4-aadc-0800271a543a server1.domain.com:3316   PRIMARY READ_WRITE    1.0
58d087d6-b5fb-11e4-aadd-080027df5a8f server2.domain.com:3316 SECONDARY  READ_ONLY    1.0

Let’s activate automatic failure detection:

[mysql@server3 ~]$ mysqlfabric group activate group01
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1
 
                                uuid finished success result
------------------------------------ -------- ------- ------
e213740e-8d66-4342-a683-deb00855b48a        1       1      1
 
state success          when                                                   description
----- ------- ------------- -------------------------------------------------------------
    3       2 1424260783.66 Triggered by <mysql.fabric.events.Event object at 0x117fd10>.
    4       2 1424260783.68                           Executing action (_activate_group).
    5       2 1424260783.69                            Executed action (_activate_group).
 
[mysql@server3 ~]$ mysqlfabric group health group01
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1
 
                                uuid is_alive    status is_not_running is_not_configured io_not_running sql_not_running io_error sql_error
------------------------------------ -------- --------- -------------- ----------------- -------------- --------------- -------- ---------
40b23f27-b5fb-11e4-aadc-0800271a543a        1   PRIMARY              0                 0              0               0    False     False
58d087d6-b5fb-11e4-aadd-080027df5a8f        1 SECONDARY              0                 0              0               0    False     False
 
issue
-----

In order to avoid to use powerful admin account when connecting to Fabric I create a generic low profile account (fabric). To display what’s existing:

[mysql@server3 ~]$ mysqlfabric role list
  ID  Role Name   Description and Permissions
  --  ----------  ---------------------------
   1  superadmin  Role for Administrative users
                  + Full access to all core Fabric functionality
   2  useradmin   Role for users dealing with user administration
                  + User administration
                  + Role administration
   3  connector   Role for MySQL Connectors
                  + Access to dump commands
                  + Reporting to Fabric
No result returned
[mysql@server3 ~]$ mysqlfabric user list
Username  Protocol     Roles
--------- ------------ --------------------
admin     xmlrpc       superadmin
admin     mysql        superadmin
No result returned

To create it:

[mysql@server3 ~]$ mysqlfabric user add fabric
Add a new Fabric user
=====================
Username: fabric
Protocol (default xmlrpc):
Password:
Repeat Password:
 
Select role(s) for new user
  ID  Role Name   Description and Permissions
  --  ----------  ---------------------------
   1  superadmin  Role for Administrative users
                  + Full access to all core Fabric functionality
   2  useradmin   Role for users dealing with user administration
                  + User administration
                  + Role administration
   3  connector   Role for MySQL Connectors
                  + Access to dump commands
                  + Reporting to Fabric
 
Enter comma separated list of role IDs or names: 3
Fabric user added.
No result returned
[mysql@server3 ~]$ mysqlfabric user list
Username  Protocol     Roles
--------- ------------ --------------------
fabric    xmlrpc       connector
admin     xmlrpc       superadmin
admin     mysql        superadmin
No result returned

MySQL Fabric high availability testing

As stated in official Connector/J documentation: Fabric support is available in Connector/J 5.1.30 and later. For my testing I have used release 5.1.34. The Java code to connect to MySQL Fabric is almost the same as the one to connect to a traditional MySQL Replication environment. Even the limitation I have seen with Connector/J are exactly the same. Means for example that if you do not re-initiate the connection then a previously down server is not considered anymore and an error may raised if you try to connect to a non working server (even when catching error with catch try statement)…

As a reference the small Java code I have used:

package jdbcdemo5;
 
import java.sql.ResultSet;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
 
public class jdbcdemo5 {
  public static void main(String[] args) throws Exception {
    ResultSet rs;
    String variable_value;
    Connection conn = null;
    String JDBC_URL = "jdbc:mysql:fabric://server3.domain.com:32274/replicationdb?fabricServerGroup=group01&fabricUsername=fabric&fabricPassword=secure_password";
 
    System.out.println("\n------------ MySQL Connector/J and MySQL Fabric Testing ------------\n");
 
    try {
   		conn = DriverManager.getConnection(JDBC_URL,"yjaquier","secure_password");
   	}
   	catch (SQLException e) {
   		System.out.println("Connection Failed! Check output console");
   		System.out.println("Error cause: "+e.getCause());
   		System.out.println("Error message: "+e.getMessage());
   		return;
   	}
 
    for(int i=1; i <= 10000; i++) {
    	System.out.println("\nLoop "+i+": ");
    	System.out.println("Read only query...");
    	try {
    		conn.setReadOnly(true);
    		}
    	catch (SQLException e) {
    		System.out.println("Connection read only property set has failed...");
    	}
 
    	try {
    		rs = conn.createStatement().executeQuery("SELECT variable_value FROM information_schema.global_variables where variable_name='hostname'");
    		while (rs.next()) {
    			variable_value = rs.getString("variable_value");
    			System.out.println("variable_value : " + variable_value);
    		}
    	}
    	catch (SQLException e) {
    		System.out.println("Read/only query has failed...");
    	}
 
    	System.out.println("Read Write query...");
    	try {
    		conn.setReadOnly(false);
    	}
    	catch (SQLException e) {
    		System.out.println("Connection read write property set has has failed...");
    	}
 
    	try {
    		rs = conn.createStatement().executeQuery("SELECT variable_value FROM information_schema.global_variables where variable_name='hostname'");
    		while (rs.next()) {
    			variable_value = rs.getString("variable_value");
    			System.out.println("variable_value : " + variable_value);
    		}
    	}
    	catch (SQLException e) {
    		System.out.println("Read/write query has failed...");
    	}
    	Thread.sleep(2000);
    }
   	conn.close();
  }
}

Slave server unavailability

If your slave server fail MySQL Fabric will report (a running Java program will fail when doing a read only query):

[mysql@server3 ~]$ mysqlfabric group health group01
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1
 
                                uuid is_alive  status is_not_running is_not_configured io_not_running sql_not_running io_error sql_error
------------------------------------ -------- ------- -------------- ----------------- -------------- --------------- -------- ---------
40b23f27-b5fb-11e4-aadc-0800271a543a        1 PRIMARY              0                 0              0               0    False     False
58d087d6-b5fb-11e4-aadd-080027df5a8f        1  FAULTY              0                 0              0               0    False     False
 
issue
-----

Once you have recovered the situation at MySQL level the server will still no be automatically considered at MySQL Fabric level. Means that you java application will still ignore it. You will have to issue the here below MySQL Fabric commands:

[mysql@server3 ~]$ mysqlfabric server set_status 58d087d6-b5fb-11e4-aadd-080027df5a8f secondary
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1
 
ServerError: Cannot change server's (58d087d6-b5fb-11e4-aadd-080027df5a8f) status from (FAULTY) to (SECONDARY).
 
[mysql@server3 ~]$ mysqlfabric server set_status 58d087d6-b5fb-11e4-aadd-080027df5a8f spare
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1
 
                                uuid finished success result
------------------------------------ -------- ------- ------
79acb568-c97d-4a35-ad10-e457706d59a1        1       1      1
 
state success          when                                                   description
----- ------- ------------- -------------------------------------------------------------
    3       2 1424364356.98 Triggered by <mysql.fabric.events.Event object at 0x117ff10>.
    4       2 1424364356.98                        Executing action (_set_server_status).
    5       2 1424364357.13                         Executed action (_set_server_status).

Once the server has been moved from faulty to spare. Strangely at this stage your Java application will start to use it for read only query even if you have not yet moved it to secondary status:

[mysql@server3 ~]$ mysqlfabric group health group01
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1
 
                                uuid is_alive  status is_not_running is_not_configured io_not_running sql_not_running io_error sql_error
------------------------------------ -------- ------- -------------- ----------------- -------------- --------------- -------- ---------
40b23f27-b5fb-11e4-aadc-0800271a543a        1 PRIMARY              0                 0              0               0    False     False
58d087d6-b5fb-11e4-aadd-080027df5a8f        1   SPARE              0                 0              0               0    False     False
 
issue
-----

You can set it back to secondary:

[mysql@server3 ~]$ mysqlfabric server set_status 58d087d6-b5fb-11e4-aadd-080027df5a8f secondary
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1
 
                                uuid finished success result
------------------------------------ -------- ------- ------
68e3232b-1a22-470f-abd5-ad6763d0df70        1       1      1
 
state success          when                                                   description
----- ------- ------------- -------------------------------------------------------------
    3       2 1424364370.48 Triggered by <mysql.fabric.events.Event object at 0x117ff10>.
    4       2 1424364370.49                        Executing action (_set_server_status).
    5       2  1424364370.5                         Executed action (_set_server_status).
 
 
[mysql@server3 ~]$ mysqlfabric group health group01
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1
 
                                uuid is_alive    status is_not_running is_not_configured io_not_running sql_not_running io_error sql_error
------------------------------------ -------- --------- -------------- ----------------- -------------- --------------- -------- ---------
40b23f27-b5fb-11e4-aadc-0800271a543a        1   PRIMARY              0                 0              0               0    False     False
58d087d6-b5fb-11e4-aadd-080027df5a8f        1 SECONDARY              0                 0              0               0    False     False
 
issue
-----

Master server unavailability

Most interesting failover scenario I would say as here I’m clearly expecting MySQL Fabric to automatically manage the master/slave switch configuration:

If I kill my master server (my Java application got stuck !), MySQL Fabric is reporting:

[mysql@server3 ~]$ mysqlfabric group health group01
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1
 
                                uuid is_alive  status is_not_running is_not_configured io_not_running sql_not_running io_error sql_error
------------------------------------ -------- ------- -------------- ----------------- -------------- --------------- -------- ---------
40b23f27-b5fb-11e4-aadc-0800271a543a        0  FAULTY              0                 0              0               0    False     False
58d087d6-b5fb-11e4-aadd-080027df5a8f        1 PRIMARY              0                 0              0               0    False     False
 
issue
-----

So well done the old slave has been promoted as a master (with no slave) and I can continue inserting into it. Once the previous master server has been recovered I can set it as a spare at MySQL Fabric:

[mysql@server3 ~]$  mysqlfabric server set_status 40b23f27-b5fb-11e4-aadc-0800271a543a spare
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1
 
                                uuid finished success result
------------------------------------ -------- ------- ------
6e949ecc-db16-48e8-885c-ca06bffda314        1       1      1
 
state success          when                                                   description
----- ------- ------------- -------------------------------------------------------------
    3       2 1424447504.96 Triggered by <mysql.fabric.events.Event object at 0x12f0f50>.
    4       2 1424447504.96                        Executing action (_set_server_status).
    5       2 1424447505.21                         Executed action (_set_server_status).
 
 
[mysql@server3 ~]$ mysqlfabric group health group01
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1
 
                                uuid is_alive  status is_not_running is_not_configured io_not_running sql_not_running io_error sql_error
------------------------------------ -------- ------- -------------- ----------------- -------------- --------------- -------- ---------
40b23f27-b5fb-11e4-aadc-0800271a543a        1   SPARE              0                 0              0               0    False     False
58d087d6-b5fb-11e4-aadd-080027df5a8f        1 PRIMARY              0                 0              0               0    False     False
 
issue
-----

The MySQL Replication is re-instantiated immediately (and working) even before you have set its status to secondary. If you re-initiate the connection in your java application then server1.domain.com is considered active for read only queries:

[mysql@server3 ~]$ mysqlfabric server set_status 40b23f27-b5fb-11e4-aadc-0800271a543a secondary
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1
 
                                uuid finished success result
------------------------------------ -------- ------- ------
8af9d0b1-371d-4732-8c08-86a486e3b10e        1       1      1
 
state success          when                                                   description
----- ------- ------------- -------------------------------------------------------------
    3       2 1424447627.97 Triggered by <mysql.fabric.events.Event object at 0x12f0f50>.
    4       2 1424447627.97                        Executing action (_set_server_status).
    5       2 1424447627.99                         Executed action (_set_server_status).
 
 
[mysql@server3 ~]$ mysqlfabric group health group01
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1
 
                                uuid is_alive    status is_not_running is_not_configured io_not_running sql_not_running io_error sql_error
------------------------------------ -------- --------- -------------- ----------------- -------------- --------------- -------- ---------
40b23f27-b5fb-11e4-aadc-0800271a543a        1 SECONDARY              0                 0              0               0    False     False
58d087d6-b5fb-11e4-aadd-080027df5a8f        1   PRIMARY              0                 0              0               0    False     False
 
issue
-----

You can, if you wish, come back to the original situation with something like (again the –slave_id parameter is a bit misleading). Your java application will need to renew its connection to see it:

[mysql@server3 ~]$ mysqlfabric group promote group01 --slave_id=40b23f27-b5fb-11e4-aadc-0800271a543a
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1
 
                                uuid finished success result
------------------------------------ -------- ------- ------
feb9963b-d3ce-4adb-b94c-f18947c701e9        1       1      1
 
state success          when                                                   description
----- ------- ------------- -------------------------------------------------------------
    3       2 1424447736.25 Triggered by <mysql.fabric.events.Event object at 0x115ce90>.
    4       2 1424447736.25                      Executing action (_define_ha_operation).
    5       2 1424447736.28                       Executed action (_define_ha_operation).
    3       2 1424447736.26 Triggered by <mysql.fabric.events.Event object at 0x12d6390>.
    4       2 1424447736.28                   Executing action (_check_candidate_switch).
    5       2 1424447736.32                    Executed action (_check_candidate_switch).
    3       2 1424447736.31 Triggered by <mysql.fabric.events.Event object at 0x12d63d0>.
    4       2 1424447736.32                       Executing action (_block_write_switch).
    5       2 1424447736.35                        Executed action (_block_write_switch).
    3       2 1424447736.35 Triggered by <mysql.fabric.events.Event object at 0x12d6410>.
    4       2 1424447736.35                       Executing action (_wait_slaves_switch).
    5       2  1424447736.4                        Executed action (_wait_slaves_switch).
    3       2 1424447736.39 Triggered by <mysql.fabric.events.Event object at 0x12d6450>.
    4       2  1424447736.4                      Executing action (_change_to_candidate).
    5       2 1424447736.62                       Executed action (_change_to_candidate).
 
 
[mysql@server3 ~]$ mysqlfabric group health group01
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1
 
                                uuid is_alive    status is_not_running is_not_configured io_not_running sql_not_running io_error sql_error
------------------------------------ -------- --------- -------------- ----------------- -------------- --------------- -------- ---------
40b23f27-b5fb-11e4-aadc-0800271a543a        1   PRIMARY              0                 0              0               0    False     False
58d087d6-b5fb-11e4-aadd-080027df5a8f        1 SECONDARY              0                 0              0               0    False     False
 
issue
-----

References

One thought on “MySQL Fabric high availability testing

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes:

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>