Table of contents
Preamble
MySQL replication is a proven well working MySQL high available solution that can also be used to scale-out your database backend layer. But you must redirect all writes to the master server, this is only suitable if your application is massively read only. If your application is not mainly read only (and trend is clearly going in this direction) then to scale-out you would have to use another method called data sharding.
Management of MySQL Replication can be cumbersome and changing a slave to be the new primary master can be difficult. We have already seen other products to help you doing it and MySQL Fabric is one of them.
To try to ease management of big MySQL farms (for HA or sharding) Oracle has created a product called MySQL Fabric. From official documentation:
MySQL Fabric is built around an extensible framework for managing farms of MySQL Servers. Currently two features have been implemented – High Availability and scaling out using data sharding. These features can be used in isolation or in combination.
For testing I have three virtual machines:
- server1.domain.com (192.168.56.101) is my initial MySQL master server
- server2.domain.com (192.168.56.102) is my initial MySQL slave server
- server3.domain.com (192.168.56.103) is my MySQL Fabric server (that also contains a MySQL instance for the MySQL Fabric repository)
They are all running Oracle Linux Server release 7.0 64 bits and MySQL 5.6.23 community edition. I have compiled and installed MySQL Utilities 1.5.3.
On server1.domain.com and server2.domain.com I configure a MySQL Replication environment using GTID that we have already seen in a previous post.
Remark:
As we are planning to change primary and secondary role it is a good practice to issue reset master on slave server to avoid replication of unwanted commands while changing master to slave. Those unwanted commands may stop your MySQL Replication and you would then need to recover few errors…
mysql> reset master; Query OK, 0 rows affected (0.06 sec) |
Your application will then connect to the MySQL Fabric server so it is a good idea to make it highly available i.e. running in a virtual machine for example. Currently the only fabric aware supported connector to connect to MySQL Fabric are PHP, Python and Java.
This blog post is about MySQL Fabric high availability the MySQL Fabric data sharding part will be seen in a further post.
I also have to mention that MySQL Fabric documentation is, so far, really poor. The reference section for example is far below minimum required…
MySQL Fabric installation
Compiling and installing MySQL Utilities (MySQL Fabric is par of MySQL Utilities) is really straightforward, you need python and few libraries (python release 2.7.5 on OEL 7.0):
[mysql@server3 mysql-utilities-1.5.3]$ pwd /mysql/mysql-utilities-1.5.3 [mysql@server3 mysql-utilities-1.5.3]$ ./setup.py build [mysql@server3 mysql-utilities-1.5.3]$ ./setup.py install |
As stated in the introduction the MySQL Fabric server also need a MySQL database to store MySQL Fabric repository. To be honest I have been a bit disappointed by this as it is another database to manage, I would have preferred the configuration stored in a text file much easier to save and restore…So in the MySQL Fabric instance repository I create a fabric account:
mysql> CREATE USER fabric@'%'; Query OK, 0 rows affected (0.00 sec) mysql> GRANT ALL ON fabric.* TO fabric@'%'; Query OK, 0 rows affected (0.01 sec) |
In default /etc/mysql/fabric.cfg configuration file I have just changed the [storage] section to reflect my server name and MySQL non-default port (3316):
address = server3.domain.com:3316
And in [protocol.xmlrpc] and [protocol.mysql] sections I have set the password to avoid any prompt (never do this on a production server), bind on real server IP address has also been changed:
password = admin
In [servers] secton I set a password for all the fabric account created on each node of the group:
password = secure_password
It gives:
[mysql@server3 ~]$ mysqlfabric manage setup --param=storage.user=fabric [INFO] 1424193041.522855 - MainThread - Initializing persister: user (fabric), server (server3.domain.com:3316), database (fabric). [INFO] 1424193043.024686 - MainThread - Initial password for admin/mysql set Password set for admin/mysql from configuration file. [INFO] 1424193043.030541 - MainThread - Password set for admin/mysql from configuration file. [INFO] 1424193043.032204 - MainThread - Initial password for admin/xmlrpc set Password set for admin/xmlrpc from configuration file. [INFO] 1424193043.037428 - MainThread - Password set for admin/xmlrpc from configuration file. No result returned |
You can confirm it has been done with:
mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | fabric | | mysql | | performance_schema | +--------------------+ 4 rows in set (0.00 sec) mysql> use fabric; show tables; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed +-------------------+ | Tables_in_fabric | +-------------------+ | checkpoints | | error_log | | group_replication | | group_view | | groups | | log | | machines | | permissions | | proc_view | | providers | | role_permissions | | roles | | servers | | shard_maps | | shard_ranges | | shard_tables | | shards | | user_roles | | users | +-------------------+ 19 rows in set (0.00 sec) |
Then on all MySQL servers I create a fabric account with:
mysql> grant all on *.* to fabric@'%' identified by 'secure_password'; Query OK, 0 rows affected (0.10 sec) mysql> flush privileges; Query OK, 0 rows affected (0.00 sec) |
As I plan to launch Fabric process with MySQL account I have to create the log file with correct rights in advance:
[root@server3 ~]# touch /var/log/fabric.log [root@server3 ~]# chown mysql:dba /var/log/fabric.log |
Start Fabric with (–daemonize to get back the hand on your session):
[mysql@server3 ~]$ mysqlfabric manage start --daemonize [mysql@server3 ~]$ ps -ef | grep mysqlfabric mysql 4420 1 0 10:35 ? 00:00:00 /bin/python /bin/mysqlfabric manage start --daemonize mysql 4439 4392 0 10:35 pts/1 00:00:00 grep --color=auto mysqlfabric |
MySQL Fabric configuration
I create a Fabric group called group01:
[mysql@server3 ~]$ mysqlfabric group create group01 Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e Time-To-Live: 1 uuid finished success result ------------------------------------ -------- ------- ------ c6e94388-2348-47d6-9020-4a44ba8e15cc 1 1 1 state success when description ----- ------- ------------- ------------------------------------------------------------- 3 2 1424257295.96 Triggered by <mysql.fabric.events.Event object at 0x129e710>. 4 2 1424257296.05 Executing action (_create_group). 5 2 1424257296.06 Executed action (_create_group). |
To confirm it has been well done:
[mysql@server3 ~]$ mysqlfabric group lookup_groups Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e Time-To-Live: 1 group_id description failure_detector master_uuid -------- ----------- ---------------- ----------- group01 None 0 None |
Then I add my master and slave server to the Fabric group:
[mysql@server3 ~]$ mysqlfabric group add group01 server1.domain.com:3316 Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e Time-To-Live: 1 uuid finished success result ------------------------------------ -------- ------- ------ 69d11425-45e1-467d-a20b-00440ab47f3a 1 1 1 state success when description ----- ------- ------------- ------------------------------------------------------------- 3 2 1424258147.45 Triggered by <mysql.fabric.events.Event object at 0x117fb50>. 4 2 1424258147.47 Executing action (_add_server). 5 2 1424258147.64 Executed action (_add_server). [mysql@server3 ~]$ mysqlfabric group add group01 server2.domain.com:3316 Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e Time-To-Live: 1 uuid finished success result ------------------------------------ -------- ------- ------ 256f0b0f-25ea-4225-a2c9-138170e15aa6 1 1 1 state success when description ----- ------- ------------- ------------------------------------------------------------- 3 2 1424258157.05 Triggered by <mysql.fabric.events.Event object at 0x117fb50>. 4 2 1424258157.05 Executing action (_add_server). 5 2 1424258157.2 Executed action (_add_server). |
To confirm they have been added (please note that they are both with status SECONDARY):
[mysql@server3 ~]$ mysqlfabric group lookup_servers group01 Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e Time-To-Live: 1 server_uuid address status mode weight ------------------------------------ ----------------------- --------- --------- ------ 40b23f27-b5fb-11e4-aadc-0800271a543a server1.domain.com:3316 SECONDARY READ_ONLY 1.0 58d087d6-b5fb-11e4-aadd-080027df5a8f server2.domain.com:3316 SECONDARY READ_ONLY 1.0 |
I promote my master server, in my opinion the parameter –salve_id is a bit difficult to understand at here you have to provide the server id of your master server:
[mysql@server3 ~]$ mysqlfabric group promote group01 --slave_id=40b23f27-b5fb-11e4-aadc-0800271a543a Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e Time-To-Live: 1 uuid finished success result ------------------------------------ -------- ------- ------ abd05779-674a-4b59-9cd1-f501540462b1 1 1 1 state success when description ----- ------- ------------- ------------------------------------------------------------- 3 2 1424260633.47 Triggered by <mysql.fabric.events.Event object at 0xfeae50>. 4 2 1424260633.47 Executing action (_define_ha_operation). 5 2 1424260633.48 Executed action (_define_ha_operation). 3 2 1424260633.48 Triggered by <mysql.fabric.events.Event object at 0x1165250>. 4 2 1424260633.48 Executing action (_check_candidate_fail). 5 2 1424260633.51 Executed action (_check_candidate_fail). 3 2 1424260633.51 Triggered by <mysql.fabric.events.Event object at 0x11652d0>. 4 2 1424260633.51 Executing action (_wait_slave_fail). 5 2 1424260633.62 Executed action (_wait_slave_fail). 3 2 1424260633.61 Triggered by <mysql.fabric.events.Event object at 0x1165410>. 4 2 1424260633.62 Executing action (_change_to_candidate). 5 2 1424260633.85 Executed action (_change_to_candidate). |
Configuration is now correct:
[mysql@server3 ~]$ mysqlfabric group lookup_servers group01 Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e Time-To-Live: 1 server_uuid address status mode weight ------------------------------------ ----------------------- --------- ---------- ------ 40b23f27-b5fb-11e4-aadc-0800271a543a server1.domain.com:3316 PRIMARY READ_WRITE 1.0 58d087d6-b5fb-11e4-aadd-080027df5a8f server2.domain.com:3316 SECONDARY READ_ONLY 1.0 |
Let’s activate automatic failure detection:
[mysql@server3 ~]$ mysqlfabric group activate group01 Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e Time-To-Live: 1 uuid finished success result ------------------------------------ -------- ------- ------ e213740e-8d66-4342-a683-deb00855b48a 1 1 1 state success when description ----- ------- ------------- ------------------------------------------------------------- 3 2 1424260783.66 Triggered by <mysql.fabric.events.Event object at 0x117fd10>. 4 2 1424260783.68 Executing action (_activate_group). 5 2 1424260783.69 Executed action (_activate_group). [mysql@server3 ~]$ mysqlfabric group health group01 Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e Time-To-Live: 1 uuid is_alive status is_not_running is_not_configured io_not_running sql_not_running io_error sql_error ------------------------------------ -------- --------- -------------- ----------------- -------------- --------------- -------- --------- 40b23f27-b5fb-11e4-aadc-0800271a543a 1 PRIMARY 0 0 0 0 False False 58d087d6-b5fb-11e4-aadd-080027df5a8f 1 SECONDARY 0 0 0 0 False False issue ----- |
In order to avoid to use powerful admin account when connecting to Fabric I create a generic low profile account (fabric). To display what’s existing:
[mysql@server3 ~]$ mysqlfabric role list ID Role Name Description and Permissions -- ---------- --------------------------- 1 superadmin Role for Administrative users + Full access to all core Fabric functionality 2 useradmin Role for users dealing with user administration + User administration + Role administration 3 connector Role for MySQL Connectors + Access to dump commands + Reporting to Fabric No result returned [mysql@server3 ~]$ mysqlfabric user list Username Protocol Roles --------- ------------ -------------------- admin xmlrpc superadmin admin mysql superadmin No result returned |
To create it:
[mysql@server3 ~]$ mysqlfabric user add fabric Add a new Fabric user ===================== Username: fabric Protocol (default xmlrpc): Password: Repeat Password: Select role(s) for new user ID Role Name Description and Permissions -- ---------- --------------------------- 1 superadmin Role for Administrative users + Full access to all core Fabric functionality 2 useradmin Role for users dealing with user administration + User administration + Role administration 3 connector Role for MySQL Connectors + Access to dump commands + Reporting to Fabric Enter comma separated list of role IDs or names: 3 Fabric user added. No result returned [mysql@server3 ~]$ mysqlfabric user list Username Protocol Roles --------- ------------ -------------------- fabric xmlrpc connector admin xmlrpc superadmin admin mysql superadmin No result returned |
MySQL Fabric high availability testing
As stated in official Connector/J documentation: Fabric support is available in Connector/J 5.1.30 and later. For my testing I have used release 5.1.34. The Java code to connect to MySQL Fabric is almost the same as the one to connect to a traditional MySQL Replication environment. Even the limitation I have seen with Connector/J are exactly the same. Means for example that if you do not re-initiate the connection then a previously down server is not considered anymore and an error may raised if you try to connect to a non working server (even when catching error with catch try statement)…
As a reference the small Java code I have used:
package jdbcdemo5; import java.sql.ResultSet; import java.sql.Connection; import java.sql.DriverManager; import java.sql.SQLException; public class jdbcdemo5 { public static void main(String[] args) throws Exception { ResultSet rs; String variable_value; Connection conn = null; String JDBC_URL = "jdbc:mysql:fabric://server3.domain.com:32274/replicationdb?fabricServerGroup=group01&fabricUsername=fabric&fabricPassword=secure_password"; System.out.println("\n------------ MySQL Connector/J and MySQL Fabric Testing ------------\n"); try { conn = DriverManager.getConnection(JDBC_URL,"yjaquier","secure_password"); } catch (SQLException e) { System.out.println("Connection Failed! Check output console"); System.out.println("Error cause: "+e.getCause()); System.out.println("Error message: "+e.getMessage()); return; } for(int i=1; i <= 10000; i++) { System.out.println("\nLoop "+i+": "); System.out.println("Read only query..."); try { conn.setReadOnly(true); } catch (SQLException e) { System.out.println("Connection read only property set has failed..."); } try { rs = conn.createStatement().executeQuery("SELECT variable_value FROM information_schema.global_variables where variable_name='hostname'"); while (rs.next()) { variable_value = rs.getString("variable_value"); System.out.println("variable_value : " + variable_value); } } catch (SQLException e) { System.out.println("Read/only query has failed..."); } System.out.println("Read Write query..."); try { conn.setReadOnly(false); } catch (SQLException e) { System.out.println("Connection read write property set has has failed..."); } try { rs = conn.createStatement().executeQuery("SELECT variable_value FROM information_schema.global_variables where variable_name='hostname'"); while (rs.next()) { variable_value = rs.getString("variable_value"); System.out.println("variable_value : " + variable_value); } } catch (SQLException e) { System.out.println("Read/write query has failed..."); } Thread.sleep(2000); } conn.close(); } } |
If your slave server fail MySQL Fabric will report (a running Java program will fail when doing a read only query):
[mysql@server3 ~]$ mysqlfabric group health group01 Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e Time-To-Live: 1 uuid is_alive status is_not_running is_not_configured io_not_running sql_not_running io_error sql_error ------------------------------------ -------- ------- -------------- ----------------- -------------- --------------- -------- --------- 40b23f27-b5fb-11e4-aadc-0800271a543a 1 PRIMARY 0 0 0 0 False False 58d087d6-b5fb-11e4-aadd-080027df5a8f 1 FAULTY 0 0 0 0 False False issue ----- |
Once you have recovered the situation at MySQL level the server will still no be automatically considered at MySQL Fabric level. Means that you java application will still ignore it. You will have to issue the here below MySQL Fabric commands:
[mysql@server3 ~]$ mysqlfabric server set_status 58d087d6-b5fb-11e4-aadd-080027df5a8f secondary Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e Time-To-Live: 1 ServerError: Cannot change server's (58d087d6-b5fb-11e4-aadd-080027df5a8f) status from (FAULTY) to (SECONDARY). [mysql@server3 ~]$ mysqlfabric server set_status 58d087d6-b5fb-11e4-aadd-080027df5a8f spare Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e Time-To-Live: 1 uuid finished success result ------------------------------------ -------- ------- ------ 79acb568-c97d-4a35-ad10-e457706d59a1 1 1 1 state success when description ----- ------- ------------- ------------------------------------------------------------- 3 2 1424364356.98 Triggered by <mysql.fabric.events.Event object at 0x117ff10>. 4 2 1424364356.98 Executing action (_set_server_status). 5 2 1424364357.13 Executed action (_set_server_status). |
Once the server has been moved from faulty to spare. Strangely at this stage your Java application will start to use it for read only query even if you have not yet moved it to secondary status:
[mysql@server3 ~]$ mysqlfabric group health group01 Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e Time-To-Live: 1 uuid is_alive status is_not_running is_not_configured io_not_running sql_not_running io_error sql_error ------------------------------------ -------- ------- -------------- ----------------- -------------- --------------- -------- --------- 40b23f27-b5fb-11e4-aadc-0800271a543a 1 PRIMARY 0 0 0 0 False False 58d087d6-b5fb-11e4-aadd-080027df5a8f 1 SPARE 0 0 0 0 False False issue ----- |
You can set it back to secondary:
[mysql@server3 ~]$ mysqlfabric server set_status 58d087d6-b5fb-11e4-aadd-080027df5a8f secondary Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e Time-To-Live: 1 uuid finished success result ------------------------------------ -------- ------- ------ 68e3232b-1a22-470f-abd5-ad6763d0df70 1 1 1 state success when description ----- ------- ------------- ------------------------------------------------------------- 3 2 1424364370.48 Triggered by <mysql.fabric.events.Event object at 0x117ff10>. 4 2 1424364370.49 Executing action (_set_server_status). 5 2 1424364370.5 Executed action (_set_server_status). [mysql@server3 ~]$ mysqlfabric group health group01 Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e Time-To-Live: 1 uuid is_alive status is_not_running is_not_configured io_not_running sql_not_running io_error sql_error ------------------------------------ -------- --------- -------------- ----------------- -------------- --------------- -------- --------- 40b23f27-b5fb-11e4-aadc-0800271a543a 1 PRIMARY 0 0 0 0 False False 58d087d6-b5fb-11e4-aadd-080027df5a8f 1 SECONDARY 0 0 0 0 False False issue ----- |
Most interesting failover scenario I would say as here I’m clearly expecting MySQL Fabric to automatically manage the master/slave switch configuration:
If I kill my master server (my Java application got stuck !), MySQL Fabric is reporting:
[mysql@server3 ~]$ mysqlfabric group health group01 Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e Time-To-Live: 1 uuid is_alive status is_not_running is_not_configured io_not_running sql_not_running io_error sql_error ------------------------------------ -------- ------- -------------- ----------------- -------------- --------------- -------- --------- 40b23f27-b5fb-11e4-aadc-0800271a543a 0 FAULTY 0 0 0 0 False False 58d087d6-b5fb-11e4-aadd-080027df5a8f 1 PRIMARY 0 0 0 0 False False issue ----- |
So well done the old slave has been promoted as a master (with no slave) and I can continue inserting into it. Once the previous master server has been recovered I can set it as a spare at MySQL Fabric:
[mysql@server3 ~]$ mysqlfabric server set_status 40b23f27-b5fb-11e4-aadc-0800271a543a spare Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e Time-To-Live: 1 uuid finished success result ------------------------------------ -------- ------- ------ 6e949ecc-db16-48e8-885c-ca06bffda314 1 1 1 state success when description ----- ------- ------------- ------------------------------------------------------------- 3 2 1424447504.96 Triggered by <mysql.fabric.events.Event object at 0x12f0f50>. 4 2 1424447504.96 Executing action (_set_server_status). 5 2 1424447505.21 Executed action (_set_server_status). [mysql@server3 ~]$ mysqlfabric group health group01 Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e Time-To-Live: 1 uuid is_alive status is_not_running is_not_configured io_not_running sql_not_running io_error sql_error ------------------------------------ -------- ------- -------------- ----------------- -------------- --------------- -------- --------- 40b23f27-b5fb-11e4-aadc-0800271a543a 1 SPARE 0 0 0 0 False False 58d087d6-b5fb-11e4-aadd-080027df5a8f 1 PRIMARY 0 0 0 0 False False issue ----- |
The MySQL Replication is re-instantiated immediately (and working) even before you have set its status to secondary. If you re-initiate the connection in your java application then server1.domain.com is considered active for read only queries:
[mysql@server3 ~]$ mysqlfabric server set_status 40b23f27-b5fb-11e4-aadc-0800271a543a secondary Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e Time-To-Live: 1 uuid finished success result ------------------------------------ -------- ------- ------ 8af9d0b1-371d-4732-8c08-86a486e3b10e 1 1 1 state success when description ----- ------- ------------- ------------------------------------------------------------- 3 2 1424447627.97 Triggered by <mysql.fabric.events.Event object at 0x12f0f50>. 4 2 1424447627.97 Executing action (_set_server_status). 5 2 1424447627.99 Executed action (_set_server_status). [mysql@server3 ~]$ mysqlfabric group health group01 Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e Time-To-Live: 1 uuid is_alive status is_not_running is_not_configured io_not_running sql_not_running io_error sql_error ------------------------------------ -------- --------- -------------- ----------------- -------------- --------------- -------- --------- 40b23f27-b5fb-11e4-aadc-0800271a543a 1 SECONDARY 0 0 0 0 False False 58d087d6-b5fb-11e4-aadd-080027df5a8f 1 PRIMARY 0 0 0 0 False False issue ----- |
You can, if you wish, come back to the original situation with something like (again the –slave_id parameter is a bit misleading). Your java application will need to renew its connection to see it:
[mysql@server3 ~]$ mysqlfabric group promote group01 --slave_id=40b23f27-b5fb-11e4-aadc-0800271a543a Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e Time-To-Live: 1 uuid finished success result ------------------------------------ -------- ------- ------ feb9963b-d3ce-4adb-b94c-f18947c701e9 1 1 1 state success when description ----- ------- ------------- ------------------------------------------------------------- 3 2 1424447736.25 Triggered by <mysql.fabric.events.Event object at 0x115ce90>. 4 2 1424447736.25 Executing action (_define_ha_operation). 5 2 1424447736.28 Executed action (_define_ha_operation). 3 2 1424447736.26 Triggered by <mysql.fabric.events.Event object at 0x12d6390>. 4 2 1424447736.28 Executing action (_check_candidate_switch). 5 2 1424447736.32 Executed action (_check_candidate_switch). 3 2 1424447736.31 Triggered by <mysql.fabric.events.Event object at 0x12d63d0>. 4 2 1424447736.32 Executing action (_block_write_switch). 5 2 1424447736.35 Executed action (_block_write_switch). 3 2 1424447736.35 Triggered by <mysql.fabric.events.Event object at 0x12d6410>. 4 2 1424447736.35 Executing action (_wait_slaves_switch). 5 2 1424447736.4 Executed action (_wait_slaves_switch). 3 2 1424447736.39 Triggered by <mysql.fabric.events.Event object at 0x12d6450>. 4 2 1424447736.4 Executing action (_change_to_candidate). 5 2 1424447736.62 Executed action (_change_to_candidate). [mysql@server3 ~]$ mysqlfabric group health group01 Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e Time-To-Live: 1 uuid is_alive status is_not_running is_not_configured io_not_running sql_not_running io_error sql_error ------------------------------------ -------- --------- -------------- ----------------- -------------- --------------- -------- --------- 40b23f27-b5fb-11e4-aadc-0800271a543a 1 PRIMARY 0 0 0 0 False False 58d087d6-b5fb-11e4-aadd-080027df5a8f 1 SECONDARY 0 0 0 0 False False issue ----- |
References
- MySQL Fabric – adding High Availability to MySQL
- MySQL Fabric now Generally Available – Automating High Availability and Sharding for MySQL
