IT World http://blog.yannickjaquier.com RDBMS, Unix and many more... Mon, 16 Oct 2017 14:55:09 +0000 en-US hourly 1 https://wordpress.org/?v=4.8.2 Data Redaction (DBMS_REDACT) with 12cR1 (12.1.0.2) http://blog.yannickjaquier.com/oracle/data-redaction-dbms_redact-12cr1.html http://blog.yannickjaquier.com/oracle/data-redaction-dbms_redact-12cr1.html#respond Tue, 26 Sep 2017 15:02:37 +0000 http://blog.yannickjaquier.com/?p=3857

Table Of Contents

  1. Preamble
  2. Data redaction implementation
  3. Data redaction testing
  4. References
 

Preamble

Advanced Security enterprise edition paid option is made of two products:

Data Redaction conditionally hide on-the-fly sensitive data before it leaves the database. The picture available on Oracle product page (copyright Oracle) says it all:

data_redaction01
data_redaction01

The condition to hide figures is really open as you write it in PL/SQL. The one I will take as an example is an application with its own security model (LDAP or whatever) connecting to a database using the same applicative account. This is a real life example with any Java or web application.

As I have already tested Virtual private database (VPD), that is the term used for combination of fine grained access control (FGAC) with application contexts, I have asked myself the difference with Data Redaction. Fortunately it is well explained in official documentation in Oracle Data Redaction and Oracle Virtual Private Database chapter.

Whatever Oracle says I have feeling that Data Redaction that is new in 12cR1 and back ported in 11gR2 (11.2.0.4 only) is the new tool to use to hide sensitive information. Unfortunately VPD and FGAC are free while Data Redaction is not…

Testing of this blog post has been done using Oracle database enterprise edition 12.1.0.2.0 – 64bit running on Oracle Linux Server release 7.2 in a virtual machine.

Data redaction implementation

I create my application schema owner, identified externally. I also provide execute on Data Redaction package called DBMS_REDACT and capability to create a context (still in 12cR1 the create context does not exist):

SQL> create user app identified externally
     default tablespace users;

User created.

SQL> alter user app quota unlimited on users;

User altered.

SQL> grant connect,resource to app;

Grant succeeded.

SQL> grant execute on dbms_redact to app;

Grant succeeded.

SQL> grant create any context to app;

Grant succeeded.

I create and provide grants to the password authenticated user that will be used in my application:

SQL> create user app_read identified by secure_password;

User created.

SQL> grant connect to app_read;

Grant succeeded.

As app user I create my applicative table (really basic one). I also grant select and update to the account that will be used in my application:

SQL> create table employees (
     id number,
     firstname varchar2(50),
     lastname varchar2(50),
     salary number);

Table created.

SQL> grant select,update on employees to app_read;

Grant succeeded.

SQL> insert into employees values(1,'Yannick','Jaquier',10000);

1 row created.

SQL> commit;

Commit complete.

SQL> select * from employees;

        ID FIRSTNAME                                          LASTNAME                                               SALARY
---------- -------------------------------------------------- -------------------------------------------------- ----------
         1 Yannick                                            Jaquier                                                 10000

As you might guess the column we want to redact (hide) to a part of applicative users is salary !

To create the Data Redaction policy I will use an application context that we have already seen when testing FGAC so going a little bit faster on this part. As app account I create a context and a package to change its value based on client_identifier parameter value of userenv context, necessary grants are also provided. The rule is that any supervisor (supervisorxx value) can see salary while the other accounts cannot:

SQL> create or replace context my_context1 using my_context1_pkg;

Context created.

SQL> CREATE OR REPLACE PACKAGE my_context1_pkg IS
     PROCEDURE set_my_context1;
     END;
     /

Package created.

SQL> create or replace package body my_context1_pkg as
  procedure set_my_context1 is
  begin
    if lower(sys_context('userenv', 'client_identifier')) like 'supervisor%'
    then
      dbms_session.set_context('my_context1','salary_yes_no','DISPLAY_SALARY');
    else
      dbms_session.set_context('my_context1','salary_yes_no','DO_NOT_DISPLAY_SALARY');
		end if;
  end set_my_context1;
end;
/

Package body created.

SQL> grant execute on my_context1_pkg TO app_read;

Grant succeeded.

When adding a policy with DBMS_REDACT.ADD_POLICY one of the most important parameter is expression. Means that the real time masking will be performed only if the expression is TRUE. Here the example I would like to simulate is to hide salary when parameter salary_yes_no of my my_context1 context is set to DO_NOT_DISPLAY_SALARY or is not set (NULL value).

exec dbms_redact.add_policy(object_schema => 'app', object_name => 'employees', column_name => 'salary', -
policy_name => 'display_salary', function_type => dbms_redact.full, -
expression => 'sys_context(''my_context1'',''salary_yes_no'')=''DO_NOT_DISPLAY_SALARY'' or sys_context(''my_context1'',''salary_yes_no'') is null', -
policy_description => 'Hide salary column', -
column_description => 'Column with sensitive salary information');

The redaction policy is enabled by default (enable parameter is set to TRUE by default). The function_type parameter set the redaction masking function. DBMS_REDACT.FULL will simply set salary column to 0 but many other options are available like changing only first digit of a credit card number or a social security number. Please refer the official documentation for a complete description.

In case you want to perform multiple tests you can drop the policy with:

SQL> exec dbms_redact.drop_policy(object_schema => 'app', object_name => 'employees', policy_name => 'display_salary');

PL/SQL procedure successfully completed.

You have few dictionary tables to see what has been implemented:

SQL> set lines 200
SQL> col object_owner for a10
SQL> col object_name for a10
SQL> col policy_description for a20
SQL> col policy_name for a15
SQL> select object_owner,object_name,policy_name, enable,policy_description from redaction_policies;

OBJECT_OWN OBJECT_NAM POLICY_NAME     ENABLE  POLICY_DESCRIPTION
---------- ---------- --------------- ------- --------------------
APP        EMPLOYEES  display_salary  YES     Hide salary column

SQL> col column_name for a10
SQL> select object_owner,object_name,column_name,function_type from redaction_columns;

OBJECT_OWN OBJECT_NAM COLUMN_NAM FUNCTION_TYPE
---------- ---------- ---------- ---------------------------
APP        EMPLOYEES  SALARY     FULL REDACTION

Data redaction testing

For testing I will connect with app_read account and set CLIENT_IDENTIFIER parameter value of USERENV context with DBMS_SESSION.SET_IDENTIFIER procedure. CLIENT_IDENTIFIER parameter value simulate the applicative account that has been used to identify inside your Java/Web application (LDAP or whatever).

If you do not set CLIENT_IDENTIFIER value then salary is not displayed:

SQL> set lines 150
SQL> SELECT SYS_CONTEXT('my_context1','salary_yes_no') FROM dual;

SYS_CONTEXT('MY_CONTEXT1','SALARY_YES_NO')
------------------------------------------------------------------------------------------------------------------------------------------------------


SQL> select * from app.employees;

        ID FIRSTNAME                                          LASTNAME                                               SALARY
---------- -------------------------------------------------- -------------------------------------------------- ----------
         1 Yannick                                            Jaquier                                                     0

If you set CLIENT_IDENTIFIER value to an applicative account that is not allowed to see salaries:

SQL> EXEC DBMS_SESSION.SET_IDENTIFIER('operator01');

PL/SQL procedure successfully completed.

SQL> exec app.my_context1_pkg.set_my_context1;

PL/SQL procedure successfully completed.

SQL> SELECT SYS_CONTEXT('my_context1','salary_yes_no') FROM dual;

SYS_CONTEXT('MY_CONTEXT1','SALARY_YES_NO')
------------------------------------------------------------------------------------------------------------------------------------------------------
DO_NOT_DISPLAY_SALARY

SQL> select * from app.employees;

        ID FIRSTNAME                                          LASTNAME                                               SALARY
---------- -------------------------------------------------- -------------------------------------------------- ----------
         1 Yannick                                            Jaquier                                                     0

If you set CLIENT_IDENTIFIER value to an applicative account that has privilege to see salaries:

SQL> EXEC DBMS_SESSION.SET_IDENTIFIER('supervisor01');

PL/SQL procedure successfully completed.

SQL> exec app.my_context1_pkg.set_my_context1;

PL/SQL procedure successfully completed.

SQL> SELECT SYS_CONTEXT('my_context1','salary_yes_no') FROM dual;

SYS_CONTEXT('MY_CONTEXT1','SALARY_YES_NO')
------------------------------------------------------------------------------------------------------------------------------------------------------
DISPLAY_SALARY

SQL> select * from app.employees;

        ID FIRSTNAME                                          LASTNAME                                               SALARY
---------- -------------------------------------------------- -------------------------------------------------- ----------
         1 Yannick                                            Jaquier                                                 10000

One funny thing you might notice is even APP user, owner of the object, is not able to see the value of salary column. This can be solved granting below system privileges:

SQL> grant exempt redaction policy to app;

Grant succeeded.

You would have chosen DBMS_REDACT.RANDOM as a masking function the salary would be different each time you perform a select onto the employees table.

Even if you are not able to see salary column you can still update it:

SQL> SELECT SYS_CONTEXT('userenv','client_identifier') from dual;

SYS_CONTEXT('USERENV','CLIENT_IDENTIFIER')
------------------------------------------------------------------------------------------------------------------------------------------------------
operator01

SQL> SELECT SYS_CONTEXT('my_context1','salary_yes_no') FROM dual;

SYS_CONTEXT('MY_CONTEXT1','SALARY_YES_NO')
------------------------------------------------------------------------------------------------------------------------------------------------------
DO_NOT_DISPLAY_SALARY

SQL> update app.employees set salary=20000 where id=1;

1 row updated.

SQL> commit;

Commit complete.

SQL> select * from app.employees;

        ID FIRSTNAME                                          LASTNAME                                               SALARY
---------- -------------------------------------------------- -------------------------------------------------- ----------
         1 yannick                                            Jaquier                                                     0

If you control with schema owner app you will see that salary has been well set to 20,000…

References

]]>
http://blog.yannickjaquier.com/oracle/data-redaction-dbms_redact-12cr1.html/feed 0
Resolve ORA-01578 error with and without a backup http://blog.yannickjaquier.com/oracle/resolve-ora-01578-backup-no-backup.html http://blog.yannickjaquier.com/oracle/resolve-ora-01578-backup-no-backup.html#respond Mon, 28 Aug 2017 13:48:03 +0000 http://blog.yannickjaquier.com/?p=3835

Table Of Contents

  1. Preamble
  2. ORA-01578 with a backup
  3. ORA-01578 with no backup
  4. References
 

Preamble

Data corruption
Data corruption

ORA-01578: ORACLE data block corrupted is not an error message we receive often. Personally I have seen it only two times in my DBA life, thanks to the quality SAN we have with good RAID level. That’s why it is also complicated to practice on this storage error that has corrupted your most important production database…

In case this error happens you have two situations: either you have a backup and the only solution not to loose any data is to restore it or you have no backup (or the restore is not working for any reasons) and at this point data loss is clear but you might want to extract as much data as possible from impacted objects.

Of course if the impacted object is an index then you are lucky as a simple drop/recreate index will solve the issue.

No need to say that before rushing to recover your database you must resolve first, with no delay, the source of the hardware problem and replace the faulty part !

Testing has been done on Oracle Linux Server release 7.2 64bit with Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 – 64bit. The database is in ARCHIVELOG mode as any other production database.

ORA-01578 with a backup

I start by creating a dedicated tablespace for our test:

SQL> create tablespace bmr
     datafile '/u01/app/oracle/oradata/orcl/bmr01.dbf' size 5M autoextend on next 5M maxsize 500M
     logging online permanent
     blocksize 8192
     extent management local autoallocate
     segment space management auto;

Tablespace created.

Then I create a test table and fill it with 500 rows:

SQL> create table test1(val number, descr varchar2(200)) tablespace bmr;

Table created.

SQL> declare
i number;
begin
i:=1;
while (i <= 500)
loop
insert into test1 values (i,TO_CHAR(TO_DATE(i, 'j'), 'jsp'));
i:=i+1;
end loop;
end;
/

PL/SQL procedure successfully completed.

SQL> commit;

Commit complete.

I configure my Recovery Manager (RMAN) with below standard option:

configure controlfile autobackup on;
configure backup optimization on;
configure device type disk parallelism 2 backup type to compressed backupset;
configure controlfile autobackup format for device type disk to '/backup/%F';
configure channel device type disk format '/backup/%U';

I backup the database and archivelog files with:

RMAN> backup database plus archivelog delete input;

Then I add 500 more rows to the table, just to ensure I’m not loosing any figures after the end of recover process. I also gather statistics:

SQL> declare
i number;
begin
i:=501;
while (i <= 1000)
loop
insert into test1 values (i,TO_CHAR(TO_DATE(i, 'j'), 'jsp'));
i:=i+1;
end loop;
end;
/

PL/SQL procedure successfully completed.

SQL> commit;

Commit complete.

SQL> exec dbms_stats.gather_table_stats(USER,'test1');

PL/SQL procedure successfully completed.

SQL> select max(val) from test1;

  MAX(VAL)
----------
      1000

My table has below storage (5 data blocks starting at 131):

SQL> select extent_id,file_id,block_id,bytes,blocks from dba_extents where segment_name='TEST1';

 EXTENT_ID    FILE_ID   BLOCK_ID      BYTES     BLOCKS
---------- ---------- ---------- ---------- ----------
         0          5        128      65536          8
SQL> select num_rows, blocks, empty_blocks from user_tables where table_name='TEST1';

  NUM_ROWS     BLOCKS EMPTY_BLOCKS
---------- ---------- ------------
      1000          5            0

SQL> select distinct dbms_rowid.rowid_relative_fno(rowid) as file_no, dbms_rowid.rowid_block_number(rowid) as block_no from test1 order by 1,2;

   FILE_NO   BLOCK_NO
---------- ----------
         5        131
         5        132
         5        133
         5        134
         5        135

If you wonder why a difference between starting blocks (128 and 131), this is well explained in Overview of Extents and we clearly see the reserved blocks by Oracle in initial allocated extent.

Before any corrupted block test do not forget to purge the buffer cache or Oracle will not read again the block and you might see nothing as the blocks are already in buffer cache:

SQL> alter system flush buffer_cache;

System altered.

Let’s corrupt first data block with dd command. This trick is coming from my Recovery Manager training even if Oracle is doing things a bit differently. It might be cumbersome to be able to corrupt one block with all caching on modern computer (database, filesystem,..). I have finally be obliged to issue two dd commands and one sync:

[root@server1 ~]# dd if=/dev/zero of=/u01/app/oracle/oradata/orcl/bmr01.dbf bs=8192 conv=notrunc seek=131 count=1
1+0 records in
1+0 records out
8192 bytes (8.2 kB) copied, 0.000209918 s, 39.0 MB/s
[root@server1 ~]# sync
[root@server1 ~]# dd if=/dev/zero of=/u01/app/oracle/oradata/orcl/bmr01.dbf bs=8192 conv=notrunc seek=131 count=1
1+0 records in
1+0 records out
8192 bytes (8.2 kB) copied, 0.000209918 s, 39.0 MB/s

Now you should start to encounter error at Oracle level (if not flush the buffer cache):

SQL> alter system flush buffer_cache;

System altered.

SQL> select * from test1 order by val;
select * from test1 order by val
       *
ERROR at line 1:
ORA-01578: ORACLE data block corrupted (file # 5, block # 131)
ORA-01110: data file 5: '/u01/app/oracle/oradata/orcl/bmr01.dbf'

To understand which blocks are impacted you may use:

SQL> select * from v$database_block_corruption;

     FILE#     BLOCK#     BLOCKS CORRUPTION_CHANGE# CORRUPTIO     CON_ID
---------- ---------- ---------- ------------------ --------- ----------
         5        131          1                  0 ALL ZERO           0

Or DB verify utility:

[oracle@server1 ~]$ dbv file=/u01/app/oracle/oradata/orcl/bmr01.dbf

DBVERIFY: Release 12.1.0.2.0 - Production on Tue Jul 19 12:49:19 2016

Copyright (c) 1982, 2014, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE = /u01/app/oracle/oradata/orcl/bmr01.dbf
Page 131 is marked corrupt
Corrupt block relative dba: 0x01400083 (file 5, block 131)
Completely zero block found during dbv:



DBVERIFY - Verification complete

Total Pages Examined         : 640
Total Pages Processed (Data) : 4
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 0
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 130
Total Pages Processed (Seg)  : 0
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 505
Total Pages Marked Corrupt   : 1
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 2631270 (0.2631270)

You can also report corruption with RMAN:

RMAN> run {
 allocate channel channel01 type disk;
 allocate channel channel02 type disk;
 allocate channel channel03 type disk;
 allocate channel channel04 type disk;
 backup check logical validate database;
}
.
.
.
File Status Marked Corrupt Empty Blocks Blocks Examined High SCN
---- ------ -------------- ------------ --------------- ----------
5    FAILED 0              505          640             2631270
  File Name: /u01/app/oracle/oradata/orcl/bmr01.dbf
  Block Type Blocks Failing Blocks Processed
  ---------- -------------- ----------------
  Data       0              4
  Index      0              0
  Other      1              131

validate found one or more corrupt blocks
See trace file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_15528.trc for details
.
.
.

At this stage you can restore the entire datafile or use a cool feature called Block Media Recovery (BMR) and as its name stand for it will recover only the corrupted block. Pre 11gR1 the command is:

blockrecover datafile '...' block x;
blockrecover corruption list;

Starting with 11gR1 the new command is:

recover datafile '...' block x;
recover corruption list;

If you choose to restore the datafile then you have to put it offline so it has an impact for other objects in same datafile. BMR instead does not impact anything:

RMAN> alter database datafile '/u01/app/oracle/oradata/orcl/bmr01.dbf' offline;

Statement processed

RMAN> restore datafile 5;

Starting restore at 19-JUL-16
using channel ORA_DISK_1
using channel ORA_DISK_2

channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00005 to /u01/app/oracle/oradata/orcl/bmr01.dbf
channel ORA_DISK_1: reading from backup piece /backup/18rb3b6l_1_1
channel ORA_DISK_1: piece handle=/backup/18rb3b6l_1_1 tag=TAG20160719T124301
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
Finished restore at 19-JUL-16

RMAN> recover datafile 5;

Starting recover at 19-JUL-16
using channel ORA_DISK_1
using channel ORA_DISK_2

starting media recovery
media recovery complete, elapsed time: 00:00:00

Finished recover at 19-JUL-16

RMAN> alter database datafile '/u01/app/oracle/oradata/orcl/bmr01.dbf' online;

using target database control file instead of recovery catalog
Statement processed

BMR is the method that is chosen by Data Recovery Advisor:

RMAN> list failure;

using target database control file instead of recovery catalog
Database Role: PRIMARY

List of Database Failures
=========================

Failure ID Priority Status    Time Detected Summary
---------- -------- --------- ------------- -------
167        HIGH     OPEN      19-JUL-16     Datafile 5: '/u01/app/oracle/oradata/orcl/bmr01.dbf' contains one or more corrupt blocks

RMAN> advise failure;

Database Role: PRIMARY

List of Database Failures
=========================

Failure ID Priority Status    Time Detected Summary
---------- -------- --------- ------------- -------
167        HIGH     OPEN      19-JUL-16     Datafile 5: '/u01/app/oracle/oradata/orcl/bmr01.dbf' contains one or more corrupt blocks

analyzing automatic repair options; this may take some time
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=263 device type=DISK
allocated channel: ORA_DISK_2
channel ORA_DISK_2: SID=42 device type=DISK
analyzing automatic repair options complete

Mandatory Manual Actions
========================
no manual actions available

Optional Manual Actions
=======================
no manual actions available

Automated Repair Options
========================
Option Repair Description
------ ------------------
1      Perform block media recovery of block 131 in file 5
  Strategy: The repair includes complete media recovery with no data loss
  Repair script: /u01/app/oracle/diag/rdbms/orcl/orcl/hm/reco_2815027252.hm

RMAN> host 'cat /u01/app/oracle/diag/rdbms/orcl/orcl/hm/reco_2815027252.hm';

   # block media recovery
   recover datafile 5 block 131;
host command complete

RMAN> repair failure;

Strategy: The repair includes complete media recovery with no data loss
Repair script: /u01/app/oracle/diag/rdbms/orcl/orcl/hm/reco_2815027252.hm

contents of repair script:
   # block media recovery
   recover datafile 5 block 131;

Do you really want to execute the above repair (enter YES or NO)? y
executing repair script

Starting recover at 19-JUL-16
using channel ORA_DISK_1
using channel ORA_DISK_2

channel ORA_DISK_1: restoring block(s)
channel ORA_DISK_1: specifying block(s) to restore from backup set
restoring blocks of datafile 00005
channel ORA_DISK_1: reading from backup piece /backup/18rb3b6l_1_1
channel ORA_DISK_1: piece handle=/backup/18rb3b6l_1_1 tag=TAG20160719T124301
channel ORA_DISK_1: restored block(s) from backup piece 1
channel ORA_DISK_1: block restore complete, elapsed time: 00:00:01

starting media recovery
media recovery complete, elapsed time: 00:00:01

Finished recover at 19-JUL-16
repair failure complete

And the table is again fully accessible:

SQL> select count(*) from test1;

  COUNT(*)
----------
      1000

ORA-01578 with no backup

Of course this must not happen but you might encounter it after trying to restore a backup and realizing that your backup strategy was not so perfect… So the importance to validate your backup procedure as often as you can. In extreme situation the data center failure might be so huge that even your backups are not accessible because you simply do not offload your backup to a third party supplier. The situation we had was a SAN failure where was located the TSM database, in this situation no restore was even possible…

Worth to mention that you will encounter data loss, the below procedure should be followed only in last option to try to recover figures you can. Below as been executed with SYS as I have not succeeded to grant execute on DBMS_REPAIR to my DBA account (!!).

Start by creating the package required backend tables:

SQL> execute dbms_repair.admin_tables(table_name => 'ORPHAN_KEY_TABLE', table_type => dbms_repair.orphan_table, action => dbms_repair.create_action, tablespace => 'users');

PL/SQL procedure successfully completed.

SQL> execute dbms_repair.admin_tables(table_name => 'REPAIR_TABLE', table_type => dbms_repair.repair_table, action => dbms_repair.create_action, tablespace => 'users');

PL/SQL procedure successfully completed.

Check your object with:

SQL> set serveroutput on
SQL> declare
       corrupt_count binary_integer:=0;
     begin
       dbms_repair.check_object(schema_name => 'YJAQUIER', object_name => 'TEST1', repair_table_name => 'REPAIR_TABLE', corrupt_count => corrupt_count);
       dbms_output.put_line('Corrupted block(s): ' || to_char (corrupt_count));
     end;
     /

Corrupted block(s): 1

PL/SQL procedure successfully completed.

This fill REPAIR_TABLE:

SQL> col repair_description for a30
SQL> set lines 150
SQL> select relative_file_id,block_id,schema_name,object_name,fix_timestamp,repair_description from sys.repair_table;

RELATIVE_FILE_ID   BLOCK_ID SCHEMA_NAME                    OBJECT_NAME                    FIX_TIMES REPAIR_DESCRIPTION
---------------- ---------- ------------------------------ ------------------------------ --------- ------------------------------
               5        131 YJAQUIER                       TEST1                                    mark block software corrupt

Then I ask Oracle not to fetch this block anymore with:

SQL> set serveroutput on
SQL> SQL> exec DBMS_REPAIR.SKIP_CORRUPT_BLOCKS(schema_name => 'YJAQUIER', object_name => 'TEST1');

PL/SQL procedure successfully completed.

I can now read in my test table but I have lost 229 rows (!!):

SQL> select count(*) from test1;

  COUNT(*)
----------
       771

I can recover those rows in another table with something like:

SQL> create table test2 tablespace users as select * from test1;

Table created.

SQL> drop table test1;

Table dropped.

SQL> rename test2 to test1;

Table renamed.

If you check with DB Verify, RMAN or V$DATABASE_BLOCK_CORRUPTION you will still see a corrupted block. Either you wait Oracle to write a new object on it or you can accelerate the process following MOS note 336133.1:

First remove autoextend from your datafile:

SQL> alter database datafile '/u01/app/oracle/oradata/orcl/bmr01.dbf' autoextend off;

Database altered.

Create a test table in table with corrupted block avoiding deferred segment creation:

SQL> show parameter deferred_segment_creation

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
deferred_segment_creation            boolean     TRUE

SQL> create table clean1(n number, c varchar2(4000)) segment creation immediate nologging pctfree 99 tablespace bmr;

Table created.

Control extend size with:

SQL> set lines 150
SQL> select * from dba_free_space where file_id= 5 and 131 between block_id and block_id + blocks -1;

TABLESPACE_NAME                   FILE_ID   BLOCK_ID      BYTES     BLOCKS RELATIVE_FNO
------------------------------ ---------- ---------- ---------- ---------- ------------
BMR                                     5        128      65536          8            5

Allocate as many extent as you can with:

SQL> begin
       for i in 1..1000000 loop
         execute immediate 'alter table clean1 allocate extent (datafile '||'''/u01/app/oracle/oradata/orcl/bmr01.dbf''' ||'size 64k)';
       end loop;
     end ;
     /
begin
*
ERROR at line 1:
ORA-01653: unable to extend table YJAQUIER.CLEAN1 by 128 in tablespace BMR
ORA-06512: at line 3

Control the block has been allocated to your table with:

SQL> select segment_name, segment_type, owner from dba_extents where file_id = 5 and 131 between block_id and block_id + blocks -1;

no rows selected

If not restart the process with a clean2 table and so on…

In my case it succeeded with CLEAN2 table:

SQL> col segment_name for a15
SQL> col owner for a15
SQL> select segment_name, segment_type, owner from dba_extents where file_id = 5 and 131 between block_id and block_id + blocks -1;

SEGMENT_NAME    SEGMENT_TYPE       OWNER
--------------- ------------------ ---------------
CLEAN2          TABLE              YJAQUIER

Fill the table that has your corrupted block in its extents (CLEAN2 in my case):

SQL> begin 
       for i in 1..1000000000 loop 
         insert /*+ append */ into clean2 select i, lpad('reformat',3092, 'r') from dual; 
         commit ; 
       end loop; 
     end;
     /
begin
*
ERROR at line 1:
ORA-01653: unable to extend table YJAQUIER.CLEAN2 by 128 in tablespace BMR
ORA-06512: at line 3

Perform few checkpoint and logfile switch with:

SQL> alter system checkpoint;

System altered.

SQL> alter system switch logfile;

System altered.

DB Verify should not report error anymore:

[oracle@server1 ~]$ dbv file=/u01/app/oracle/oradata/orcl/bmr01.dbf

DBVERIFY: Release 12.1.0.2.0 - Production on Tue Jul 19 15:36:48 2016

Copyright (c) 1982, 2014, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE = /u01/app/oracle/oradata/orcl/bmr01.dbf


DBVERIFY - Verification complete

Total Pages Examined         : 640
Total Pages Processed (Data) : 118
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 0
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 471
Total Pages Processed (Seg)  : 0
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 51
Total Pages Marked Corrupt   : 0
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 2641854 (0.2641854)

To “purge” V$DATABASE_BLOCK_CORRUPTION issue:

RMAN> backup validate datafile 5;

Starting backup at 19-JUL-16
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=262 device type=DISK
allocated channel: ORA_DISK_2
channel ORA_DISK_2: SID=25 device type=DISK
channel ORA_DISK_1: starting compressed full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00005 name=/u01/app/oracle/oradata/orcl/bmr01.dbf
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
List of Datafiles
=================
File Status Marked Corrupt Empty Blocks Blocks Examined High SCN
---- ------ -------------- ------------ --------------- ----------
5    OK     0              51           647             2641854
  File Name: /u01/app/oracle/oradata/orcl/bmr01.dbf
  Block Type Blocks Failing Blocks Processed
  ---------- -------------- ----------------
  Data       0              118
  Index      0              0
  Other      0              471

Finished backup at 19-JUL-16

References

]]>
http://blog.yannickjaquier.com/oracle/resolve-ora-01578-backup-no-backup.html/feed 0
ProxySQL high availability tutorial with MariaDB replication http://blog.yannickjaquier.com/mysql/proxysql-high-availability-replication.html http://blog.yannickjaquier.com/mysql/proxysql-high-availability-replication.html#comments Mon, 24 Jul 2017 10:02:45 +0000 http://blog.yannickjaquier.com/?p=3812

Table Of Contents

  1. Preamble
  2. ProxySQL configuration
  3. ProxySQL testing
  4. ProxySQL high availability testing
    1. One slave dead
    2. Two slaves dead
    3. Master server dead
  5. ProxySQL additional features
    1. Query caching
    2. Query rewrite
  6. References
 

Preamble

In a recent webinar I attended I have seen mention of a Maxscale alternative called ProxySQL. Maxscale is a proven working high level proxy but ProxySQL author (René Cannaò) is quite dithyrambic on his product mainly on performance related benchmarks. So better I give a try to it but I will not make any comment on performance. Aim of the post is a simple ProxySQL implementation in a MariaDB replication architecture.

Testing has been done using three virtual machines running Oracle Enterprise Linux 7.2 64 bits:

  • server2.domain.com (192.168.56.102) is MariaDB 10.1.14 64 bits master server.
  • server3.domain.com (192.168.56.103) is MariaDB 10.1.14 64 bits slave server running two MariaDB slave instances.
  • server4.domain.com (192.168.56.104) is ProxySQL node with MysQL 5.7.13 64 bits for client binary.

The Java test application is running under Eclipse Java EE IDE for Web Developers version Neon Release (4.6.0) with MariaDB connector/J 1.4.6 and Oracle MySQL Connector/J 5.1.39.

The MariaDB replication architecture is made of:

  • server2.domain.com on port 3316 as master instance
  • server3.domain.com on port 3316 as first slave instance
  • server3.domain.com on port 3326 as second slave instance

I’m not re-entering in MariaDB replication implementation as we have already seen it.

ProxySQL release available at the time of this article is 1.2.0j. Last but not least ProxySQL can be used for query caching and query rewrite !!

ProxySQL configuration

To install ProxySQL either you compile it or you download a rpm for your release. I have used the one of Centos 7 (proxysql-1.2.0-1-centos7.x86_64.rpm) that is working fine on OEL 7 available at https://github.com/sysown/proxysql/releases.

[root@server4 tmp]# yum install proxysql-1.2.0-1-centos7.x86_64.rpm
Loaded plugins: ulninfo
Examining proxysql-1.2.0-1-centos7.x86_64.rpm: proxysql-1.2.0-1.x86_64
Marking proxysql-1.2.0-1-centos7.x86_64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package proxysql.x86_64 0:1.2.0-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

===========================================================================================================================================================================================================
 Package                                    Arch                                     Version                                      Repository                                                          Size
===========================================================================================================================================================================================================
Installing:
 proxysql                                   x86_64                                   1.2.0-1                                      /proxysql-1.2.0-1-centos7.x86_64                                    11 M

Transaction Summary
===========================================================================================================================================================================================================
Install  1 Package

Total size: 11 M
Installed size: 11 M
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : proxysql-1.2.0-1.x86_64                                                                                                                                                                 1/1
  Verifying  : proxysql-1.2.0-1.x86_64                                                                                                                                                                 1/1

Installed:
  proxysql.x86_64 0:1.2.0-1

Complete!

Start it:

[root@server4 ~]# service proxysql start
Starting ProxySQL: Main init phase0 completed in 3.9e-05 secs.
Main init global variables completed in 0.000236 secs.
Main daemonize phase1 completed in 3.2e-05 secs.
DONE!

I expected a graphical interface to customize options but ProxySQL is fully command line, not an issue ! ProxySQL information is stored is a MySQL back end database so any modification will be done through SQL language. To connect to ProxySQL back-end database you need a MySQL client. Either you install the one of your Linux distribution via standard repository or you install your own one. I have chosen to use MySQL 5.7.13 installed in /mysql/software/mysql01 but new security rules has made things a little bit harder:

[mysql@server4 ~]$ /mysql/software/mysql01/bin/mysql -u admin -padmin -h 127.0.0.1 -P6032
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2026 (HY000): SSL connection error: socket layer receive error

I initially tried the skip-ssl option that is working but deprecated:

[mysql@server4 ~]$ /mysql/software/mysql01/bin/mysql -u admin -padmin -h 127.0.0.1 -P6032 --skip-ssl
mysql: [Warning] Using a password on the command line interface can be insecure.
WARNING: --ssl is deprecated and will be removed in a future version. Use --ssl-mode instead.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 0
Server version: 5.1.30 (ProxySQL Admin Module)

Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

The working up-to-date option is ssl-mode:

[mysql@server4 ~]$ /mysql/software/mysql01/bin/mysql -u admin -padmin -h 127.0.0.1 -P6032 --ssl-mode=disabled
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.1.30 (ProxySQL Admin Module)

Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

Insert your MariaDB replication topology, load it live and save it to disk with:

mysql> insert into mysql_servers(hostgroup_id, hostname, port) values (0,'192.168.56.102',3316);
Query OK, 1 row affected (0.00 sec)

mysql> insert into mysql_servers(hostgroup_id, hostname, port) values (1,'192.168.56.103',3316);
Query OK, 1 row affected (0.01 sec)

mysql> insert into mysql_servers(hostgroup_id, hostname, port) values (1,'192.168.56.103',3326);
Query OK, 1 row affected (0.00 sec)

mysql> select * from mysql_servers;
+--------------+----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+
| hostgroup_id | hostname       | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms |
+--------------+----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+
| 0            | 192.168.56.102 | 3316 | ONLINE | 1      | 0           | 1000            | 0                   | 0       | 0              |
| 1            | 192.168.56.103 | 3316 | ONLINE | 1      | 0           | 1000            | 0                   | 0       | 0              |
| 1            | 192.168.56.103 | 3326 | ONLINE | 1      | 0           | 1000            | 0                   | 0       | 0              |
+--------------+----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+
3 rows in set (0.01 sec)

mysql> save mysql servers to disk;
Query OK, 0 rows affected (0.00 sec)

mysql> select * from disk.mysql_servers;
+--------------+----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+
| hostgroup_id | hostname       | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms |
+--------------+----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+
| 0            | 192.168.56.102 | 3316 | ONLINE | 1      | 0           | 1000            | 0                   | 0       | 0              |
| 1            | 192.168.56.103 | 3316 | ONLINE | 1      | 0           | 1000            | 0                   | 0       | 0              |
| 1            | 192.168.56.103 | 3326 | ONLINE | 1      | 0           | 1000            | 0                   | 0       | 0              |
+--------------+----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+
3 rows in set (0.00 sec)

mysql> load mysql servers to runtime;
Query OK, 0 rows affected (0.00 sec)

Second step is to create an applicative account on ProxySQL. This account must also be created on the MariaDB replication architecture. On master instance, account creation will be replicated on slaves:

grant all privileges on replicationdb.* to 'test'@'%' identified by 'secure_password';

And do it in ProxySQL repository database:

mysql> insert into mysql_users (username,password) values ('test','secure_password');
Query OK, 1 row affected (0.00 sec)

mysql> save mysql users to disk;
Query OK, 0 rows affected (0.00 sec)

mysql> select * from disk.mysql_users;
+----------+-----------------+--------+---------+-------------------+----------------+---------------+------------------------+--------------+---------+----------+-----------------+
| username | password        | active | use_ssl | default_hostgroup | default_schema | schema_locked | transaction_persistent | fast_forward | backend | frontend | max_connections |
+----------+-----------------+--------+---------+-------------------+----------------+---------------+------------------------+--------------+---------+----------+-----------------+
| test     | secure_password | 1      | 0       | 0                 | NULL           | 0             | 0                      | 0            | 1       | 1        | 10000           |
+----------+-----------------+--------+---------+-------------------+----------------+---------------+------------------------+--------------+---------+----------+-----------------+
1 row in set (0.00 sec)

mysql> load mysql users to runtime;
Query OK, 0 rows affected (0.00 sec)

Humm passwords are stored in clear, it’s a planned feature to encrypt them… Note the default hostgroup set to 0 (master server).

Let’s do a test of a read-only SQL command:

[root@server4 ~]# /mysql/software/mysql01/bin/mysql --user=test --password=secure_password --host=127.0.0.1 --port=6033 -e 'SELECT @@hostname,@@port' --ssl-mode=disabled
mysql: [Warning] Using a password on the command line interface can be insecure.
+--------------------+--------+
| @@hostname         | @@port |
+--------------------+--------+
| server2.domain.com |   3316 |
+--------------------+--------+

If you execute above command multiple times only master server is chosen. Read/write split is not implemented by default and if you want a Maxscale behavior you have to create query routing rules…

I have also activated ProxySQL backend servers monitoring:

mysql> SELECT * FROM global_variables where variable_name like 'mysql-monitor%';
+----------------------------------------+---------------------------------------------------+
| variable_name                          | variable_value                                    |
+----------------------------------------+---------------------------------------------------+
| mysql-monitor_history                  | 600000                                            |
| mysql-monitor_connect_interval         | 120000                                            |
| mysql-monitor_connect_timeout          | 200                                               |
| mysql-monitor_ping_interval            | 60000                                             |
| mysql-monitor_ping_max_failures        | 3                                                 |
| mysql-monitor_ping_timeout             | 100                                               |
| mysql-monitor_read_only_interval       | 1000                                              |
| mysql-monitor_read_only_timeout        | 100                                               |
| mysql-monitor_replication_lag_interval | 10000                                             |
| mysql-monitor_replication_lag_timeout  | 1000                                              |
| mysql-monitor_username                 | monitor                                           |
| mysql-monitor_password                 | monitor                                           |
| mysql-monitor_query_variables          | SELECT * FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES |
| mysql-monitor_query_status             | SELECT * FROM INFORMATION_SCHEMA.GLOBAL_STATUS    |
| mysql-monitor_query_interval           | 60000                                             |
| mysql-monitor_query_timeout            | 100                                               |
| mysql-monitor_timer_cached             | true                                              |
| mysql-monitor_writer_is_also_reader    | true                                              |
+----------------------------------------+---------------------------------------------------+
18 rows in set (0.01 sec)

mysql> update global_variables set variable_value='secure_password' where variable_name='mysql-monitor_password';
Query OK, 1 row affected (0.00 sec)

mysql> load mysql variables to runtime;
Query OK, 0 rows affected (0.00 sec)

mysql> save mysql variables to disk;
Query OK, 64 rows affected (0.02 sec)

I create the ProxySQL monitor account on master server, it will be automatically replicated to slaves:

MariaDB [(none)]> grant replication client on *.* to 'monitor'@'%' identified by 'secure_password';
Query OK, 0 rows affected (0.00 sec)

After a small period you can control it runs successfully with:

mysql> select * from monitor.mysql_server_replication_lag_log order by time_start desc limit 10;
+----------------+------+------------------+--------------+----------+-------+
| hostname       | port | time_start       | success_time | repl_lag | error |
+----------------+------+------------------+--------------+----------+-------+
| 192.168.56.102 | 3316 | 1467715539097299 | 810          | NULL     | NULL  |
| 192.168.56.103 | 3316 | 1467715539097299 | 621          | 0        | NULL  |
| 192.168.56.103 | 3326 | 1467715539097299 | 1022         | 0        | NULL  |
| 192.168.56.102 | 3316 | 1467715529097105 | 780          | NULL     | NULL  |
| 192.168.56.103 | 3316 | 1467715529097105 | 816          | 0        | NULL  |
| 192.168.56.103 | 3326 | 1467715529097105 | 1024         | 0        | NULL  |
| 192.168.56.102 | 3316 | 1467715519096898 | 786          | NULL     | NULL  |
| 192.168.56.103 | 3316 | 1467715519096898 | 655          | 0        | NULL  |
| 192.168.56.103 | 3326 | 1467715519096898 | 1111         | 0        | NULL  |
| 192.168.56.102 | 3316 | 1467715509096195 | 885          | NULL     | NULL  |
+----------------+------+------------------+--------------+----------+-------+
10 rows in set (0.00 sec)

mysql> select * from monitor.mysql_server_ping_log order by time_start desc limit 10;
+----------------+------+------------------+-------------------+------------+
| hostname       | port | time_start       | ping_success_time | ping_error |
+----------------+------+------------------+-------------------+------------+
| 192.168.56.102 | 3316 | 1467715518924380 | 721               | NULL       |
| 192.168.56.103 | 3316 | 1467715518924380 | 525               | NULL       |
| 192.168.56.103 | 3326 | 1467715518924380 | 701               | NULL       |
| 192.168.56.102 | 3316 | 1467715458924137 | 682               | NULL       |
| 192.168.56.103 | 3316 | 1467715458924137 | 517               | NULL       |
| 192.168.56.103 | 3326 | 1467715458924137 | 703               | NULL       |
| 192.168.56.102 | 3316 | 1467715398923945 | 685               | NULL       |
| 192.168.56.103 | 3316 | 1467715398923945 | 531               | NULL       |
| 192.168.56.103 | 3326 | 1467715398923945 | 843               | NULL       |
| 192.168.56.102 | 3316 | 1467715338923725 | 678               | NULL       |
+----------------+------+------------------+-------------------+------------+
10 rows in set (0.00 sec)

mysql> select * from monitor.mysql_server_connect_log order by time_start desc limit 10;
+----------------+------+------------------+----------------------+---------------+
| hostname       | port | time_start       | connect_success_time | connect_error |
+----------------+------+------------------+----------------------+---------------+
| 192.168.56.102 | 3316 | 1467715518941492 | 1524                 | NULL          |
| 192.168.56.103 | 3316 | 1467715518941492 | 2105                 | NULL          |
| 192.168.56.103 | 3326 | 1467715518941492 | 2232                 | NULL          |
| 192.168.56.102 | 3316 | 1467715398941430 | 1734                 | NULL          |
| 192.168.56.103 | 3316 | 1467715398941430 | 2450                 | NULL          |
| 192.168.56.103 | 3326 | 1467715398941430 | 1357                 | NULL          |
| 192.168.56.102 | 3316 | 1467715278941119 | 1437                 | NULL          |
| 192.168.56.103 | 3316 | 1467715278941119 | 1937                 | NULL          |
| 192.168.56.103 | 3326 | 1467715278941119 | 4414                 | NULL          |
| 192.168.56.102 | 3316 | 1467715158941062 | 1806                 | NULL          |
+----------------+------+------------------+----------------------+---------------+
10 rows in set (0.00 sec)

I have not well understood if mysql_replication_hostgroups table must be configured, even without it, it should work:

mysql> insert into mysql_replication_hostgroups values(0,1);
Query OK, 1 row affected (0.01 sec)

mysql> save mysql servers to disk;
Query OK, 0 rows affected (0.01 sec)

mysql> load mysql servers to runtime;
Query OK, 0 rows affected (0.01 sec)

mysql> select * from mysql_replication_hostgroups;
+------------------+------------------+
| writer_hostgroup | reader_hostgroup |
+------------------+------------------+
| 0                | 1                |
+------------------+------------------+
1 row in set (0.00 sec)

mysql> select * from disk.mysql_replication_hostgroups;
+------------------+------------------+
| writer_hostgroup | reader_hostgroup |
+------------------+------------------+
| 0                | 1                |
+------------------+------------------+
1 row in set (0.01 sec)

If you hit strange behavior, like I had, showing duplicated nodes across hostgroup:

mysql> SELECT * FROM mysql_servers;
+--------------+----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+
| hostgroup_id | hostname       | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms |
+--------------+----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+
| 0            | 192.168.56.102 | 3316 | ONLINE | 1      | 0           | 1000            | 10                  | 0       | 10             |
| 0            | 192.168.56.103 | 3326 | ONLINE | 1      | 0           | 1000            | 10                  | 0       | 10             |
| 0            | 192.168.56.103 | 3316 | ONLINE | 1      | 0           | 1000            | 10                  | 0       | 10             |
| 1            | 192.168.56.103 | 3326 | ONLINE | 1      | 0           | 1000            | 10                  | 0       | 10             |
| 1            | 192.168.56.103 | 3316 | ONLINE | 1      | 0           | 1000            | 10                  | 0       | 10             |
+--------------+----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+
5 rows in set (0.00 sec)

mysql> select * from disk.mysql_server;
+--------------+----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+
| hostgroup_id | hostname       | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms |
+--------------+----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+
| 0            | 192.168.56.102 | 3316 | ONLINE | 1      | 0           | 1000            | 10                  | 0       | 10             |
| 1            | 192.168.56.103 | 3316 | ONLINE | 1      | 0           | 1000            | 10                  | 0       | 10             |
| 1            | 192.168.56.103 | 3326 | ONLINE | 1      | 0           | 1000            | 10                  | 0       | 10             |
+--------------+----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+
3 rows in set (0.01 sec)

You have to set your slaves read only with:

MariaDB [(none)]> set global read_only=ON;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> show variables like 'read_only';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| read_only     | ON    |
+---------------+-------+
1 row in set (0.00 sec)

For below error message when using MySQL Connector/J 5.1.39:

Error message: Unknown system variable 'language'

You have to set ProxySQL MySQL version above 5.1.31 with something like (mysql-server_version is just a display for ProxySQL):

mysql> set mysql-server_version='5.7.13';
Query OK, 1 row affected (0.00 sec)

mysql> load mysql variables to runtime;
Query OK, 0 rows affected (0.00 sec)

mysql> save mysql variables to disk;
Query OK, 64 rows affected (0.01 sec)

ProxySQL testing

As stated in ProxySQL website to simulate read/write split you have to create query routing rules, just using what is given in ProxySQL web site:

mysql> insert into mysql_query_rules(active,match_pattern,destination_hostgroup,apply) values (1,'^SELECT.*FOR UPDATE$',0,1);
Query OK, 1 row affected (0.00 sec)

mysql> insert into mysql_query_rules(active,match_pattern,destination_hostgroup,apply) values (1,'^SELECT',1,1);
Query OK, 1 row affected (0.00 sec)

mysql> select rule_id, match_pattern,destination_hostgroup, apply from mysql_query_rules;
+---------+----------------------+-----------------------+-------+
| rule_id | match_pattern        | destination_hostgroup | apply |
+---------+----------------------+-----------------------+-------+
| 1       | ^SELECT.*FOR UPDATE$ | 0                     | 1     |
| 2       | ^SELECT              | 1                     | 1     |
+---------+----------------------+-----------------------+-------+
2 rows in set (0.00 sec)

mysql> save mysql query rules to disk;
Query OK, 0 rows affected (0.00 sec)

mysql> load mysql query rules to runtime;
Query OK, 0 rows affected (0.00 sec)

This example query routing rules listed in official documentation are routing select queries (read-only) to hostgroup 1 (slave servers) and select for update queries (read-write) to master server. Queries not passing any of the rules goes to default hostgroup that is master server. With a classical MySQL client connection it gives:

[root@server4 ~]# /mysql/software/mysql01/bin/mysql --user=test --password=secure_password --host=127.0.0.1 --port=6033 -e 'SELECT @@hostname,@@port' --ssl-mode=disabled
mysql: [Warning] Using a password on the command line interface can be insecure.
+--------------------+--------+
| @@hostname         | @@port |
+--------------------+--------+
| server3.domain.com |   3326 |
+--------------------+--------+
[root@server4 ~]# /mysql/software/mysql01/bin/mysql --user=test --password=secure_password --host=127.0.0.1 --port=6033 -e 'SELECT @@hostname,@@port' --ssl-mode=disabled
mysql: [Warning] Using a password on the command line interface can be insecure.
+--------------------+--------+
| @@hostname         | @@port |
+--------------------+--------+
| server3.domain.com |   3316 |
+--------------------+--------+

To further test it I have used the typical Java application I have used in many other blog posts. Here I initiate a connection and loop to perform a read-write and a read-only query. Obviously I am not disconnecting and reconnecting after each loop mainly because a standard Java application would not do it.

To be honest I have faced lots of issue making this Java application work and round robin onto my slave servers, fortunately I have been helped by René directly at this forum thread https://groups.google.com/forum/#!topic/proxysql/Rzwd55xIQGk. First René explained that due to @ symbol ProxySQL multiplexing is disabled to ensure consistency on users’ variables. So he suggested to change my test query by something like:

SELECT (SELECT VARIABLE_VALUE FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES WHERE VARIABLE_NAME='hostname') `hostname`,(SELECT VARIABLE_VALUE FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES WHERE VARIABLE_NAME='port') `port`;

Then, directed by René, I realized that I stupidly changed the value of mysql-multiplexing to false (default value is true), so changed it back to original value:

mysql> set mysql-multiplexing='true';
Query OK, 1 row affected (0.01 sec)

mysql> load mysql variables to runtime;
Query OK, 0 rows affected (0.01 sec)

mysql> save mysql variables to disk;
Query OK, 64 rows affected (0.01 sec)

Finally below Java code (you can de-comment the Connector/J you wish to use, Java code is compliant with both):

package jdbcdemo7;

import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.util.Properties;
import org.mariadb.jdbc.Driver;
//import com.mysql.jdbc.Driver;

public class jdbcdemo7 {
	public static void main(String[] args) throws Exception {
		Driver driver = new Driver();
		Properties props = new Properties();
		ResultSet rs;
		String variable_value;
		Connection conn = null;
		String JDBC_URL = "jdbc:mysql://address=(protocol=tcp)(host=192.168.56.104)(port=6033)";

		props.put("useSSL", "false");
		props.put("user", "test");
		props.put("password", "secure_password");

		System.out.println("\n------------ MariaDB Connector/J and ProxySQL Testing ------------\n");

		System.out.println("Trying connection...");
		try {
			conn = driver.connect(JDBC_URL, props);
		}
			catch (SQLException e) {
			System.out.println("Connection Failed!");
			System.out.println("Error cause: "+e.getCause());
			System.out.println("Error message: "+e.getMessage());
			return;
		}

		System.out.println("Connection established...");

		for(int i=1; i <= 50; i++) {
			System.out.println("\nQuery "+i+": ");
			// Read write query that can be performed ONLY on master server
			System.out.println("Read Write query...");
			try {
				rs = conn.createStatement().executeQuery("select (select variable_value from information_schema.global_variables where variable_name=\'hostname\') || \' on port \' || (select variable_value from information_schema.global_variables where variable_name=\'port\') variable_value for update");
				while (rs.next()) {
					variable_value = rs.getString("variable_value");
					System.out.println("variable_value : " + variable_value);
				}
			}
			catch (SQLException e) {
				System.out.println("Read/write query has failed...");
			}
	    

				// Read Only statement (that can also be done on master server if all slaves are down)
			System.out.println("Read Only query...");

			try {
				rs = conn.createStatement().executeQuery("select (select variable_value from information_schema.global_variables where variable_name=\'hostname\') || \' on port \' || (select variable_value from information_schema.global_variables where variable_name=\'port\') variable_value");
				while (rs.next()) {
					variable_value = rs.getString("variable_value");
					System.out.println("variable_value : " + variable_value);
				}
			}
			catch (SQLException e) {
				System.out.println("Read only query has failed...");
			}

			Thread.sleep(1000);
		}
		conn.close();
	}
}

To provide under Eclipse below nice result:

proxysql01
proxysql01

Few statistics tables are also available:

mysql> select * from stats_mysql_connection_pool;
+-----------+----------------+----------+--------+----------+----------+--------+---------+---------+-----------------+-----------------+------------+
| hostgroup | srv_host       | srv_port | status | ConnUsed | ConnFree | ConnOK | ConnERR | Queries | Bytes_data_sent | Bytes_data_recv | Latency_ms |
+-----------+----------------+----------+--------+----------+----------+--------+---------+---------+-----------------+-----------------+------------+
| 0         | 192.168.56.102 | 3316     | ONLINE | 0        | 0        | 23     | 0       | 1012    | 73908           | 61508           | 0          |
| 1         | 192.168.56.103 | 3316     | ONLINE | 0        | 0        | 32     | 0       | 307     | 18160           | 10868           | 0          |
| 1         | 192.168.56.103 | 3326     | ONLINE | 0        | 0        | 40     | 0       | 480     | 28544           | 17024           | 0          |
+-----------+----------------+----------+--------+----------+----------+--------+---------+---------+-----------------+-----------------+------------+
3 rows in set (0.00 sec)

mysql> select hostgroup, sum(queries), sum(bytes_data_sent), sum(bytes_data_recv) from stats_mysql_connection_pool group by hostgroup;
+-----------+--------------+----------------------+----------------------+
| hostgroup | SUM(Queries) | SUM(Bytes_data_sent) | SUM(Bytes_data_recv) |
+-----------+--------------+----------------------+----------------------+
| 0         | 667          | 48373                | 26196                |
| 1         | 664          | 40256                | 23712                |
+-----------+--------------+----------------------+----------------------+
2 rows in set (0.00 sec)

For the complete list of statistics tables use:

mysql> show tables from stats;
+--------------------------------+
| tables                         |
+--------------------------------+
| stats_mysql_query_rules        |
| stats_mysql_commands_counters  |
| stats_mysql_processlist        |
| stats_mysql_connection_pool    |
| stats_mysql_query_digest       |
| stats_mysql_query_digest_reset |
| stats_mysql_global             |
+--------------------------------+
7 rows in set (0.00 sec)

ProxySQL high availability testing

One slave dead

If I shutdown one of the slave above Java application is not throwing any error and continue to run transparently on only remaining slave.

mysql> SELECT * FROM stats_mysql_connection_pool;
+-----------+----------------+----------+---------+----------+----------+--------+---------+---------+-----------------+-----------------+------------+
| hostgroup | srv_host       | srv_port | status  | ConnUsed | ConnFree | ConnOK | ConnERR | Queries | Bytes_data_sent | Bytes_data_recv | Latency_ms |
+-----------+----------------+----------+---------+----------+----------+--------+---------+---------+-----------------+-----------------+------------+
| 0         | 192.168.56.102 | 3316     | ONLINE  | 0        | 1        | 1      | 0       | 28      | 6317            | 1192            | 587        |
| 1         | 192.168.56.103 | 3316     | ONLINE  | 0        | 1        | 1      | 0       | 18      | 4086            | 648             | 530        |
| 1         | 192.168.56.103 | 3326     | SHUNNED | 0        | 0        | 1      | 12      | 8       | 1816            | 288             | 901        |
+-----------+----------------+----------+---------+----------+----------+--------+---------+---------+-----------------+-----------------+------------+
3 rows in set (0.00 sec)

Once the dead slave is back to live ProxySQL use it automatically.

Two slaves dead

If I shutdown my two slaves then read only queries are no more possible:

mysql> SELECT * FROM stats_mysql_connection_pool;
+-----------+----------------+----------+---------+----------+----------+--------+---------+---------+-----------------+-----------------+------------+
| hostgroup | srv_host       | srv_port | status  | ConnUsed | ConnFree | ConnOK | ConnERR | Queries | Bytes_data_sent | Bytes_data_recv | Latency_ms |
+-----------+----------------+----------+---------+----------+----------+--------+---------+---------+-----------------+-----------------+------------+
| 0         | 192.168.56.102 | 3316     | ONLINE  | 0        | 1        | 1      | 0       | 92      | 21549           | 3496            | 919        |
| 1         | 192.168.56.103 | 3316     | SHUNNED | 0        | 0        | 1      | 18      | 76      | 17252           | 2700            | 896        |
| 1         | 192.168.56.103 | 3326     | SHUNNED | 0        | 0        | 2      | 84      | 13      | 2951            | 432             | 901        |
+-----------+----------------+----------+---------+----------+----------+--------+---------+---------+-----------------+-----------------+------------+
3 rows in set (0.00 sec)

With few ProxySQL commands it would be possible to correct this and allocate master server in hostgroup 1 to allow it to resolve read only queries.

When at least one of the slaves is back to life application return in normal running state...

Master server dead

Of course read write queries are no more possible when master server is down ! statistics are a bit strange:

mysql> SELECT * FROM stats_mysql_connection_pool;
+-----------+----------------+----------+--------------+----------+----------+--------+---------+---------+-----------------+-----------------+------------+
| hostgroup | srv_host       | srv_port | status       | ConnUsed | ConnFree | ConnOK | ConnERR | Queries | Bytes_data_sent | Bytes_data_recv | Latency_ms |
+-----------+----------------+----------+--------------+----------+----------+--------+---------+---------+-----------------+-----------------+------------+
| 0         | 192.168.56.102 | 3316     | OFFLINE_HARD | 0        | 0        | 1      | 0       | 127     | 29532           | 4940            | 622        |
| 1         | 192.168.56.103 | 3316     | ONLINE       | 0        | 1        | 2      | 131     | 84      | 19068           | 2988            | 759        |
| 1         | 192.168.56.103 | 3326     | ONLINE       | 0        | 1        | 3      | 198     | 22      | 4994            | 756             | 660        |
| 1         | 192.168.56.102 | 3316     | ONLINE       | 0        | 0        | 0      | 0       | 0       | 0               | 0               | 0          |
+-----------+----------------+----------+--------------+----------+----------+--------+---------+---------+-----------------+-----------------+------------+
4 rows in set (0.00 sec)

mysql> SELECT * FROM mysql_servers;
+--------------+----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+
| hostgroup_id | hostname       | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms |
+--------------+----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+
| 1            | 192.168.56.103 | 3316 | ONLINE | 1      | 0           | 1000            | 10                  | 0       | 10             |
| 1            | 192.168.56.103 | 3326 | ONLINE | 1      | 0           | 1000            | 10                  | 0       | 10             |
| 0            | 192.168.56.102 | 3316 | ONLINE | 1      | 0           | 1000            | 10                  | 0       | 10             |
| 1            | 192.168.56.102 | 3316 | ONLINE | 1      | 0           | 1000            | 10                  | 0       | 10             |
+--------------+----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+
4 rows in set (0.00 sec)

In this situation either you promote a slave to new master role and ProxySQL must be tuned, nothing complex here. Or you are able to restart your master server, most probable situation.

When my master server was again alive I have seen that ProxySQL has added it to read only group (hostgroup 1) and now read only queries are also served by it. I have resolved this weird situation with:

mysql> delete from mysql_servers where hostgroup_id=1 and hostname='192.168.56.102';
Query OK, 1 row affected (0.00 sec)

mysql> load mysql servers to runtime;
Query OK, 0 rows affected (0.01 sec)

mysql> select * from mysql_servers;
+--------------+----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+
| hostgroup_id | hostname       | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms |
+--------------+----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+
| 1            | 192.168.56.103 | 3316 | ONLINE | 1      | 0           | 1000            | 10                  | 0       | 10             |
| 1            | 192.168.56.103 | 3326 | ONLINE | 1      | 0           | 1000            | 10                  | 0       | 10             |
| 0            | 192.168.56.102 | 3316 | ONLINE | 1      | 0           | 1000            | 10                  | 0       | 10             |
+--------------+----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+
3 rows in set (0.00 sec)

mysql> select * from stats_mysql_connection_pool;
+-----------+----------------+----------+--------------+----------+----------+--------+---------+---------+-----------------+-----------------+------------+
| hostgroup | srv_host       | srv_port | status       | ConnUsed | ConnFree | ConnOK | ConnERR | Queries | Bytes_data_sent | Bytes_data_recv | Latency_ms |
+-----------+----------------+----------+--------------+----------+----------+--------+---------+---------+-----------------+-----------------+------------+
| 0         | 192.168.56.102 | 3316     | ONLINE       | 0        | 1        | 1      | 0       | 53      | 12614           | 1908            | 682        |
| 1         | 192.168.56.103 | 3316     | ONLINE       | 0        | 1        | 2      | 131     | 110     | 24970           | 3924            | 1477       |
| 1         | 192.168.56.103 | 3326     | ONLINE       | 0        | 1        | 3      | 198     | 47      | 10669           | 1656            | 1101       |
| 1         | 192.168.56.102 | 3316     | OFFLINE_HARD | 0        | 0        | 1      | 4       | 13      | 2951            | 468             | 682        |
+-----------+----------------+----------+--------------+----------+----------+--------+---------+---------+-----------------+-----------------+------------+
4 rows in set (0.01 sec)

If you restart ProxySQL everything is cleared...

ProxySQL additional features

Query caching

Use this trick to reset statistics:

select 1 from stats_mysql_query_digest_reset;

One nice feature is query caching that reminds me the benefit I have seen with Memcached. After a run of my Java test application with 1000 loop I get below obvious statistics:

mysql> select hostgroup,count_star,min_time,max_time,sum_time,digest_text from stats_mysql_query_digest;
+-----------+------------+----------+----------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| hostgroup | count_star | min_time | max_time | sum_time | digest_text                                                                                                                                                                                                           |
+-----------+------------+----------+----------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 1         | 1000       | 1052     | 4287     | 1778360  | select (select variable_value from information_schema.global_variables where variable_name=?) || ?|| (select variable_value from information_schema.global_variables where variable_name=?) variable_value            |
| 0         | 1000       | 1018     | 2209     | 1554333  | select (select variable_value from information_schema.global_variables where variable_name=?) || ?|| (select variable_value from information_schema.global_variables where variable_name=?) variable_value for update |
| 0         | 1          | 536      | 536      | 536      | set session autocommit=?                                                                                                                                                                                              |
| 0         | 1          | 1722     | 1722     | 1722     | SHOW VARIABLES WHERE Variable_name in (?, ?, ?, ?)                                                                                                                                                                    |
+-----------+------------+----------+----------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
4 rows in set (0.00 sec)

Let's cache for 10 seconds the read only query that is executed on hostgroup 1:

mysql> select * from mysql_query_rules;
+---------+--------+----------+------------+--------+-------------+------------+------------+--------+--------------+----------------------+----------------------+---------+-----------------+-----------------------+-----------+-----------+---------+---------+-------+----------------+------------------+-----------+-----+-------+
| rule_id | active | username | schemaname | flagIN | client_addr | proxy_addr | proxy_port | digest | match_digest | match_pattern        | negate_match_pattern | flagOUT | replace_pattern | destination_hostgroup | cache_ttl | reconnect | timeout | retries | delay | mirror_flagOUT | mirror_hostgroup | error_msg | log | apply |
+---------+--------+----------+------------+--------+-------------+------------+------------+--------+--------------+----------------------+----------------------+---------+-----------------+-----------------------+-----------+-----------+---------+---------+-------+----------------+------------------+-----------+-----+-------+
| 1       | 1      | NULL     | NULL       | 0      | NULL        | NULL       | NULL       | NULL   | NULL         | ^SELECT.*FOR UPDATE$ | 0                    | NULL    | NULL            | 0                     | NULL      | NULL      | NULL    | NULL    | NULL  | NULL           | NULL             | NULL      | NULL | 1     |
| 2       | 1      | NULL     | NULL       | 0      | NULL        | NULL       | NULL       | NULL   | NULL         | ^SELECT              | 0                    | NULL    | NULL            | 1                     | NULL      | NULL      | NULL    | NULL    | NULL  | NULL           | NULL             | NULL      | NULL | 1     |
+---------+--------+----------+------------+--------+-------------+------------+------------+--------+--------------+----------------------+----------------------+---------+-----------------+-----------------------+-----------+-----------+---------+---------+-------+----------------+------------------+-----------+-----+-------+
2 rows in set (0.00 sec)

mysql> update mysql_query_rules set cache_ttl=10000 where rule_id=2;
Query OK, 1 row affected (0.00 sec)

mysql> load mysql query rules to runtime;
Query OK, 0 rows affected (0.00 sec)

If I re-execute my Java test application we can see that on the 1000 loop only the first one is executed on my MariaDB backend layer, all the remaining ones are served directly by ProxySQL under the special hostgroup -1:

mysql> select hostgroup,count_star,min_time,max_time,sum_time,digest_text from stats_mysql_query_digest;
+-----------+------------+----------+----------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| hostgroup | count_star | min_time | max_time | sum_time | digest_text                                                                                                                                                                                                           |
+-----------+------------+----------+----------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| -1        | 999        | 0        | 0        | 0        | select (select variable_value from information_schema.global_variables where variable_name=?) || ?|| (select variable_value from information_schema.global_variables where variable_name=?) variable_value            |
| 1         | 1001       | 1052     | 4287     | 1780439  | select (select variable_value from information_schema.global_variables where variable_name=?) || ?|| (select variable_value from information_schema.global_variables where variable_name=?) variable_value            |
| 0         | 2000       | 1018     | 2858     | 3236456  | select (select variable_value from information_schema.global_variables where variable_name=?) || ?|| (select variable_value from information_schema.global_variables where variable_name=?) variable_value for update |
| 0         | 2          | 536      | 546      | 1082     | set session autocommit=?                                                                                                                                                                                              |
| 0         | 2          | 1722     | 1775     | 3497     | SHOW VARIABLES WHERE Variable_name in (?, ?, ?, ?)                                                                                                                                                                    |
+-----------+------------+----------+----------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
5 rows in set (0.01 sec)

Query rewrite

Let's imagine I have my application running a query to get maximum value of a column of below table:

MariaDB [(none)]> show create table replicationdb.test1;
+-------+-------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table                                                                                                                  |
+-------+-------------------------------------------------------------------------------------------------------------------------------+
| test1 | CREATE TABLE `test1` (
  `val` int(11) DEFAULT NULL,
  `descr` varchar(50) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf32 |
+-------+-------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

MariaDB [(none)]> explain extended select max(val) from replicationdb.test1;
+------+-------------+-------+------+---------------+------+---------+------+------+----------+-------+
| id   | select_type | table | type | possible_keys | key  | key_len | ref  | rows | filtered | Extra |
+------+-------------+-------+------+---------------+------+---------+------+------+----------+-------+
|    1 | SIMPLE      | test1 | ALL  | NULL          | NULL | NULL    | NULL |    2 |   100.00 |       |
+------+-------------+-------+------+---------------+------+---------+------+------+----------+-------+
1 row in set, 1 warning (0.00 sec)

As you know this is going to do a full table scan on replicationdb.test1 table to get this value. Now let's imagine this table is static and I want to change the query each time it is fired to ProxySQL by something like (I assume I have three rows in my table):

select 3

Let's add needed rule, as rule 2 will be fired before newly created rule 10 I deactivate it as well. Here the rule you enter is following regexp way of working so almost no limitation in what it is possible to do:

mysql> insert into mysql_query_rules (rule_id,active,match_pattern,replace_pattern,apply) values (10,1,'^select max\(val\) from replicationdb.test1$','select 3',1);
Query OK, 1 row affected (0.00 sec)

mysql> update mysql_query_rules set active=0 where rule_id=2;
Query OK, 1 row affected (0.01 sec)

mysql> select * from mysql_query_rules;
+---------+--------+----------+------------+--------+-------------+------------+------------+--------+--------------+----------------------------------------------+----------------------+---------+-----------------+-----------------------+-----------+-----------+---------+---------+-------+----------------+------------------+-----------+-----+-------+
| rule_id | active | username | schemaname | flagIN | client_addr | proxy_addr | proxy_port | digest | match_digest | match_pattern                                | negate_match_pattern | flagOUT | replace_pattern | destination_hostgroup | cache_ttl | reconnect | timeout | retries | delay | mirror_flagOUT | mirror_hostgroup | error_msg | log | apply |
+---------+--------+----------+------------+--------+-------------+------------+------------+--------+--------------+----------------------------------------------+----------------------+---------+-----------------+-----------------------+-----------+-----------+---------+---------+-------+----------------+------------------+-----------+-----+-------+
| 1       | 1      | NULL     | NULL       | 0      | NULL        | NULL       | NULL       | NULL   | NULL         | ^SELECT.*FOR UPDATE$                         | 0                    | NULL    | NULL            | 0                     | NULL      | NULL      | NULL    | NULL    | NULL  | NULL           | NULL             | NULL      | NULL | 1     |
| 2       | 0      | NULL     | NULL       | 0      | NULL        | NULL       | NULL       | NULL   | NULL         | ^SELECT                                      | 0                    | NULL    | NULL            | 1                     | NULL      | NULL      | NULL    | NULL    | NULL  | NULL           | NULL             | NULL      | NULL | 1     |
| 10      | 1      | NULL     | NULL       | 0      | NULL        | NULL       | NULL       | NULL   | NULL         | ^select max\(val\) from replicationdb.test1$ | 0                    | NULL    | select 3        | NULL                  | NULL      | NULL      | NULL    | NULL    | NULL  | NULL           | NULL             | NULL      | NULL | 1     |
+---------+--------+----------+------------+--------+-------------+------------+------------+--------+--------------+----------------------------------------------+----------------------+---------+-----------------+-----------------------+-----------+-----------+---------+---------+-------+----------------+------------------+-----------+-----+-------+
3 rows in set (0.00 sec)

mysql> load mysql query rules to runtime;
Query OK, 0 rows affected (0.00 sec)

Now I execute the query with:

[mysql@server4 ~]$ /mysql/software/mysql01/bin/mysql --user=test --password=secure_password --host=127.0.0.1 --port=6033 -e "SELECT MAX(val) from replicationdb.test1" --ssl-mode=disabled
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------+
| MAX(val) |
+----------+
|        3 |
+----------+

I get my result, but has it been transformed by ProxySQL ?:

mysql> select hostgroup,count_star,min_time,max_time,sum_time,digest_text from stats_mysql_query_digest;
+-----------+------------+----------+----------+----------+------------------------------------------+
| hostgroup | count_star | min_time | max_time | sum_time | digest_text                              |
+-----------+------------+----------+----------+----------+------------------------------------------+
| 0         | 1          | 3080     | 3080     | 3080     | SELECT MAX(val) from replicationdb.test1 |
| 0         | 1          | 0        | 0        | 0        | select @@version_comment limit ?         |
+-----------+------------+----------+----------+----------+------------------------------------------+
2 rows in set (0.00 sec)

mysql> select * from stats_mysql_query_rules;
+---------+------+
| rule_id | hits |
+---------+------+
| 1       | 0    |
| 10      | 1    |
+---------+------+
2 rows in set (0.01 sec)

The hits column increases so we can say query rewrite is in action !

References

]]>
http://blog.yannickjaquier.com/mysql/proxysql-high-availability-replication.html/feed 2
Orchestrator MySQL replication topology tool tutorial http://blog.yannickjaquier.com/mysql/orchestrator-tutorial.html http://blog.yannickjaquier.com/mysql/orchestrator-tutorial.html#respond Mon, 26 Jun 2017 10:38:28 +0000 http://blog.yannickjaquier.com/?p=3793

Table Of Contents

  1. Preamble
  2. Orchestrator installation
  3. Orchestrator testing
  4. References
 

Preamble

MySQL replication is simple to put in place but it could be a little bit harder when it comes to monitoring and promoting slave servers to master role. Orchestrator is super easy to put in place and if you do not like command line tools or have not subscribed to MONyog this is the tool to give a try. Orchestrator also forbid you operations that would result in an error.

I have already tested multiple solution like Master High Availability Manager and tools for MySQL (MHA), MySQL Utilities, Multi-Master Replication Manager for MySQL (MMM), Maxscale and MySQL Fabric. Orchestrator simply aim at graphically or command line representing your configuration and perform few re-configuration. It is anyway not a pure monitoring tool as SNMP traps cannot be send to your favorite corporate monitoring tool.

In below post I have used three virtual machines running Oracle Linux Server release 7.2 64 bits. I expected to do my tests with a simple master-slave configuration but realized this was not really interesting so created two slaves of my master instance. So I have:

  • server2.domain.com (192.168.56.102) is MariaDB 10.1.14 64 bits master server.
  • server3.domain.com (192.168.56.103) is MariaDB 10.1.14 64 bits slave server running two MariaDB instances.
  • server4.domain.com (192.168.56.104) is Orchestrator node with MySQL 5.7.13 64 bits for repository database.

For MariaDB instances I have:

  • server2.domain.com on port 3316 as master instance
  • server3.domain.com on port 3316 as first slave instance
  • server3.domain.com on port 3326 as second slave instance

Orchestrator installation

I’m not re-entering in MySQL replication (more specifically here MariaDB) as we have seen it recently in a previous post.

Orchestrator requires a MySQL backend database to store its information. I have to say that I’m a bit disappointed by that as I would have expected information to be saved in a JSON file or something similar. But according to Orchestrator author (Shlomi Noach) this is planned !

In your MySQL 5.7 repository instance create a database and an account to use it:

mysql> CREATE DATABASE IF NOT EXISTS orchestrator;
Query OK, 1 row affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON `orchestrator`.* TO 'orchestrator'@'%' IDENTIFIED BY 'secure_password';
Query OK, 0 rows affected (0.00 sec)

Copy and edit the Orchestrator configuration file:

[root@server4 ~]# cd /usr/local/orchestrator/
[root@server4 orchestrator]# cp orchestrator-sample.conf.json /etc/orchestrator.conf.json
[root@server4 orchestrator]# vi /etc/orchestrator.conf.json

What I have changed inside to match my configuration:

  "MySQLOrchestratorHost": "127.0.0.1",
  "MySQLOrchestratorPort": 3316,
  "MySQLOrchestratorDatabase": "orchestrator",
  "MySQLOrchestratorUser": ""orchestrator",
  "MySQLOrchestratorPassword": "secure_password",

For better display I have also customized:

  "RemoveTextFromHostnameDisplay": ".domain.com",

And also remove the default verbose output, re-activate it if you face any issue:

  "Debug": false,

Log file in case of issue is located at /var/log/orchestrator.log.

On all MariaDB instance create the Orchestrator account with:

GRANT SUPER, PROCESS, REPLICATION SLAVE, RELOAD ON *.* TO 'orchestrator'@'%' IDENTIFIED BY 'secure_password';

To start Orchestrator you need to move first to its directory because if you issue:

[root@server4 ~]# /usr/local/orchestrator/orchestrator --debug http &

You will get below error message:

html/template: "templates/layout" is undefined

Because resources directory is not accessible by daemon at startup so better use:

[root@server4 ~]# cd  /usr/local/orchestrator/
[root@server4 orchestrator]# ./orchestrator --debug http &
2016-06-24 15:05:20 INFO starting orchestrator
2016-06-24 15:05:20 INFO Read config: /etc/orchestrator.conf.json
2016-06-24 15:05:20 DEBUG Initializing orchestrator
2016-06-24 15:05:20 DEBUG Migrating database schema
2016-06-24 15:05:20 FATAL Cannot initiate orchestrator: Error 1067: Invalid default value for 'end_timestamp'

If you encounter this error this is because of the value of SQL mode for your backend repository instance and more specifically the NO_ZERO_DATE option:

mysql> show variables like 'sql_mode';
+---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Variable_name | Value                                                                                                                                                                |
+---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| sql_mode      | PIPES_AS_CONCAT,STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION |
+---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.01 sec)

So change sql_mode to something like:

sql_mode=oracle

Which gives:

mysql> show variables like 'sql_mode';
+---------------+----------------------------------------------------------------------------------------------------------------------+
| Variable_name | Value                                                                                                                |
+---------------+----------------------------------------------------------------------------------------------------------------------+
| sql_mode      | PIPES_AS_CONCAT,ANSI_QUOTES,IGNORE_SPACE,ORACLE,NO_KEY_OPTIONS,NO_TABLE_OPTIONS,NO_FIELD_OPTIONS,NO_AUTO_CREATE_USER |
+---------------+----------------------------------------------------------------------------------------------------------------------+
1 row in set (0.04 sec)

When started access your Orchestrator node on default port 3000 and in Discover menu start to add the node of your topology. At then end you should get something like:

orchestrator01
orchestrator01

If you click on gear icon along server name you have access to few customization, for master server:

orchestrator02
orchestrator02

For slave server (read only option is directly accessible):

orchestrator03
orchestrator03

You can also use commend line to display it:

[root@server4 ~]# /usr/local/orchestrator/orchestrator -c topology -i server2:3316 cli
server2.domain.com:3316   [0s,ok,10.1.14-MariaDB,rw,ROW]
+ server3.domain.com:3316 [0s,ok,10.1.14-MariaDB,rw,ROW,GTID]
+ server3.domain.com:3326 [0s,ok,10.1.14-MariaDB,rw,ROW,GTID]

Orchestrator testing

I have faced issues when trying to drag and drop my slaves, for example if I try to make server3.domain.com:3326 a slave of server3.domain.com:3316 then it is not possible:

orchestrator04
orchestrator04

If my master server fails Orchestrator report it but I have no option to restart it. To do this it would require an agent or a system connection to remote, Orchestrator agent is available but it would be a much complex implementation. Here, same, if I try to make make server3.domain.com:3326 a slave of server3.domain.com:3316 then it is not possible:

orchestrator05
orchestrator05

As no error message with graphical interface I have tried command line and now error message is clear:

[root@server4 ~]# /usr/local/orchestrator/orchestrator -c relocate -i server3.domain.com:3326 -d server3.domain.com:3316
2016-06-29 15:45:51 DEBUG Initializing orchestrator
2016-06-29 15:45:51 ERROR server3.domain.com:3326 cannot replicate from server3.domain.com:3316. Reason: instance does not have log_slave_updates enabled: server3.domain.com:3316
2016-06-29 15:45:51 FATAL 2016-06-29 15:45:51 ERROR server3.domain.com:3326 cannot replicate from server3.domain.com:3316. Reason: instance does not have log_slave_updates enabled: server3.domain.com:3316

So I activated log_slave_updates parameter in my.cnf file of all my instances (in case my master becomes a slave) as the parameter is not dynamic:

MariaDB [(none)]> show variables like '%log_slave_updates%';
+-------------------+-------+
| Variable_name     | Value |
+-------------------+-------+
| log_slave_updates | OFF   |
+-------------------+-------+
1 row in set (0.00 sec)

MariaDB [(none)]> set global log_slave_updates=on;
ERROR 1238 (HY000): Variable 'log_slave_updates' is a read only variable

Once activated we see a difference in printing with the stripes along the servers either graphical or command line:

[root@server4 ~]# /usr/local/orchestrator/orchestrator -c topology -i server2:3316 cli
server2.domain.com:3316   [0s,ok,10.1.14-MariaDB,rw,ROW,>>]
+ server3.domain.com:3316 [0s,ok,10.1.14-MariaDB,rw,ROW,>>,GTID]
+ server3.domain.com:3326 [0s,ok,10.1.14-MariaDB,rw,ROW,>>,GTID]

If my master fail (or in case you want to cascade the slaves) I make server3.domain.com:3316 my new master and stop replication on it (you would do the same in case you have only one slave). I set server3.domain.com:3326 a slave of server3.domain.com:3116:

orchestrator06
orchestrator06
orchestrator07
orchestrator07

But the graphical interface does not allow me to put back server2.domain.com:3316 a slave and even command line is asking me to do it manually:

[root@server4 ~]# /usr/local/orchestrator/orchestrator -c relocate -i server2.domain.com:3316 -d server3.domain.com:3316
2016-06-30 15:12:36 ERROR Relocating server2.domain.com:3316 below server3.domain.com:3316 turns to be too complex; please do it manually
2016-06-30 15:12:36 FATAL 2016-06-30 15:12:36 ERROR Relocating server2.domain.com:3316 below server3.domain.com:3316 turns to be too complex; please do it manually

So you end up with this configuration made of two clusters:

orchestrator08
orchestrator08

And the main one is made of two servers:

orchestrator09
orchestrator09

Obviously nothing cannot be done command line to recover original situation but for this Orchestrator is of no help…

References

]]>
http://blog.yannickjaquier.com/mysql/orchestrator-tutorial.html/feed 0
Directory naming configuration and usage (ldap.ora) – part 3 http://blog.yannickjaquier.com/oracle/directory-naming-configuration-usage-3.html http://blog.yannickjaquier.com/oracle/directory-naming-configuration-usage-3.html#comments Fri, 02 Jun 2017 09:33:18 +0000 http://blog.yannickjaquier.com/?p=3771

Table Of Contents

  1. Preamble
  2. Directory naming configuration
  3. References
 

Preamble

In first part we have seen WebLogic installation and configuration, in second part we have seen Oracle Internet directory (OID) installation and configuration. This final third part is about configuring directory naming for your client to make use of Oracle connect descriptors you have inserted in LDAP directory.

Directory naming configuration

We are finally ready to add Oracle connect descriptors in our newly created OID. What I recommend to do is to use the SQL*Net layer of your Oracle database repository.

In $ORACLE_HOME/network/admin directory of your repository database activate directory naming by creating a ldap.ora file that should looks like:

DIRECTORY_SERVERS = (server1.domain.com:3060:3131)
DIRECTORY_SERVER_TYPE = OID
DEFAULT_ADMIN_CONTEXT = "dc=sgp, dc=st, dc=com"

And modify below parameter in sqlnet.ora file in same directory to handle LDAP directory. I prefer to keep TNSNAMES first to let a chance to user to overwrite LDAP entries, could be dangerous but more flexible for them. This sqlnet.ora file should be modified for all clients that you wish to use your directory server.:

NAMES.DIRECTORY_PATH= (TNSNAMES, LDAP)

Execute Network Manager (netmgr):

directory_naming57
directory_naming57

Expand directory part and connect with cn=orcladmin account and password you specified when installing OID:

directory_naming58
directory_naming58

Press green arrow in left toolbar when selection is Service Naming to create a new LDAP Oracle connect descriptor. You can even create one for your repository database as a test:

directory_naming59
directory_naming59

Protocol for your Oracle connect descriptor:

directory_naming60
directory_naming60

Server and port where is the database:

directory_naming61
directory_naming61

Service of the listener:

directory_naming62
directory_naming62

Test or finish to validate:

directory_naming63
directory_naming63

It now appears as a new entry under directory structure:

directory_naming64
directory_naming64

If you rush to test it it will most probably fail:

[oracle@server1 admin]$ tnsping orcl

TNS Ping Utility for Linux: Version 12.1.0.2.0 - Production on 13-JUN-2016 16:14:33

Copyright (c) 1997, 2014, Oracle.  All rights reserved.

Used parameter files:
/u01/app/oracle/product/12.1.0/dbhome_1/network/admin/sqlnet.ora

TNS-03505: Failed to resolve name

We have to change one property of OID to allow anonymous bind. And Enterprise Manager we have just configured will be of great help. Connect to it, still on http://server1.domain.com:7001/em, expand Identity and Access and select oid1. In management menu choose Administration and Server Properties as shown below:

directory_naming65
directory_naming65

Allow Anonymous bind as below and push Apply button to save:

directory_naming66
directory_naming66

If you do not have the graphical interface (in the case on no WebLogic) follow MOS note 947285.1. Create a text file like:

[oracle@server1 ~]$ cat ~oracle/anonymousbind.ldif
dn: cn=oid1,cn=osdldapd,cn=subconfigsubentry
changetype: modify
replace: orclAnonymousBindsFlag
orclAnonymousBindsFlag: 1

Then execute it with ldapmodify binary to change your LDAP directory (OID) property:

[oracle@server1 ~]$ export ORACLE_HOME=/u01/Middleware/Oracle_IDM1
[oracle@server1 ~]$ export PATH=$PATH:$ORACLE_HOME/bin
[oracle@server1 ~]$ ldapmodify -D cn=orcladmin -q -p 3060 -h server1.domain.com -f ~oracle/anonymousbind.ldif
Please enter bind password:
modifying entry cn=oid1,cn=osdldapd,cn=subconfigsubentry

Then when retesting the newly created OID service naming it should positively answer:

[oracle@server1 admin]$ tnsping orcl

TNS Ping Utility for Linux: Version 12.1.0.2.0 - Production on 13-JUN-2016 16:15:57

Copyright (c) 1997, 2014, Oracle.  All rights reserved.

Used parameter files:
/u01/app/oracle/product/12.1.0/dbhome_1/network/admin/sqlnet.ora

Used LDAP adapter to resolve the alias
Attempting to contact (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=server1.domain.com)(PORT=1531)))(CONNECT_DATA=(SERVICE_NAME=orcl)))
OK (10 msec)

Notice the Used LDAP adapter to resolve the alias sentence to confirm Oracle connect descriptor has been resolved by your directory and not by your local tnsnames.ora file…

If you do not like Network Manager to handle your Oracle connect descriptors you can also use a generic LDAP browser as OID behaves like a normal LDAP directory. Here is an example with the first free one I have found searching on Google (JXplorer):

directory_naming67
directory_naming67

This graphical tool can also be used to modify LDAP property of your directory, for example the famous anonymous bind one from 2 to 1:

directory_naming71
directory_naming71

References

]]>
http://blog.yannickjaquier.com/oracle/directory-naming-configuration-usage-3.html/feed 1
Directory naming configuration and usage (ldap.ora) – part 2 http://blog.yannickjaquier.com/oracle/directory-naming-configuration-usage-2.html http://blog.yannickjaquier.com/oracle/directory-naming-configuration-usage-2.html#comments Fri, 02 Jun 2017 09:32:59 +0000 http://blog.yannickjaquier.com/?p=3770

Table Of Contents

  1. Preamble
  2. OID repository creation
  3. OID installation
  4. OID configuration
  5. References
 

Preamble

We have seen in first part how to install and configure WebLogic application server. This second part is about configuring the database repository with Repository Creation utility (RCU) and installing Oracle Internet Directory (OID).

OID repository creation

Before you can install Oracle Internet Directory (OID) you must first fill your repository database with required schemas. On the Oracle Identity Management 11g Downloads page download below utility:

  • Oracle Fusion Middleware Repository Creation Utility 11g (11.1.1.9.0)

The database I have used is an Enterprise Edition (we use only this edition) in 12.1.0.2.0, set processes and open_cursors parameters to 500 to avoid any warning. I’m not detailing this part as it should be more than obvious for you with, for example, Database Configuration Assistant (DBCA).

The downloaded file (ofm_rcu_linux_11.1.1.9.0_64_disk1_1of1.zip) can be unzip anywhere. In ./rcuHome/bin directory invoke Repository Creation Utility with rcu binary:

directory_naming21
directory_naming21

Choose to create the repository:

directory_naming22
directory_naming22

Supply your database information with, preferably, a SYS connection. After having supplied the connection credentials few checks are performed, if all is good press Ok or correct any raised problems:

directory_naming23
directory_naming23

Choose OID repository creation:

directory_naming24
directory_naming24

Choose a password for the accounts (ODS and ODSSM):

directory_naming25
directory_naming25

I have changed only the temporary tablespace:

directory_naming26
directory_naming26
directory_naming27
directory_naming27

Do not change the default tablespace or you will end up with:

ORA-01917: user or role 'ODS' does not exist

The tablespaces are hard coded in creation scripts so if you change to something else they are not created and RCU will fail… Wondering why they have left the capability to change then…

Summary of what will be done:

directory_naming28
directory_naming28

Ending window if all goes well:

directory_naming29
directory_naming29

OID installation

On the Oracle Identity Management 11g Downloads page download below files:

  • Identity Management (11.1.1.9.0)

Unzip the two files (ofm_idm_linux_11.1.1.9.0_64_disk1_1of2.zip and ofm_idm_linux_11.1.1.9.0_64_disk1_2of2.zip) and execute runInstaller in Disk1 directory:

directory_naming31
directory_naming31

For proxy reason and internet access restriction on my server I usually chose to avoid software update:

directory_naming32
directory_naming32

Choose install and configure:

directory_naming33
directory_naming33

Pre-requisites check, even if OEL 7 is certified I had few warning about too recent packages, I have obviously decided to ignore:

directory_naming34
directory_naming34

Specify credentials for your WebLogic domain:

directory_naming35
directory_naming35

I had this warning that I have also decided to ignore:

directory_naming36
directory_naming36

If you have decided not to install WebLogic you must choose below option instead:

directory_naming68
directory_naming68

Specify installation directories, WebLogic home and application name your have chosen (I kept at maximum default values):

directory_naming37
directory_naming37

The window is slightly different if you have chosen not to install WebLogic:

directory_naming69
directory_naming69

Again I already received plenty of security alerts:

directory_naming38
directory_naming38

Choose OID and I have also chosen Oracle Directory Service Manager (ODSM) but you will see later on that I have not been able to use it so not mandatory at all:

directory_naming39
directory_naming39

The window is much simpler in the case of no WebLogic:

directory_naming70
directory_naming70

Automatic port configuration but if you have special port requirements it can be customized

directory_naming40
directory_naming40

OID database schema that has been created with RCU:

directory_naming41
directory_naming41

OID domain. I have chosen the one I setup with OID 10g for ascending compatibility, choose anything you like that match your company. You also choose the password of LDAP administrator (cn=orcladmin):

directory_naming42
directory_naming42

Summary window:

directory_naming43
directory_naming43

You classically end up installation with execution of /u01/Middleware/Oracle_IDM1/oracleRoot.sh script as root…

directory_naming44
directory_naming44

OID configuration

While configuring all components I got an error saying my WebLogic server was not reachable. When moving to widow where I executed it I discovered it has failed…

Configuration of component follow installation process:

directory_naming45
directory_naming45

Ending successful window:

directory_naming46
directory_naming46

You can now control with Oracle Process Manager and Notification Server (OPMN) tool that everything is started and running well:

[oracle@server1 ~]$ /u01/Middleware/asinst_1/bin/opmnctl status

Processes in Instance: asinst_1
---------------------------------+--------------------+---------+---------
ias-component                    | process-type       |     pid | status
---------------------------------+--------------------+---------+---------
oid1                             | oidldapd           |   26401 | Alive
oid1                             | oidldapd           |   26397 | Alive
oid1                             | oidmon             |   26389 | Alive
EMAGENT                          | EMAGENT            |   25953 | Alive

OID is listening on port 3060 (LDAP, insecure) and 3131 (LDAPS, secure):

[oracle@server1 ~]$ netstat -an | grep LISTEN | grep -e 3060 -e 3131
tcp6       0      0 :::3060                 :::*                    LISTEN
tcp6       0      0 :::3131                 :::*                    LISTEN

You have also Oracle Directory Services Manager (ODSM) at http://server1.domain.com:7005/odsm but it was so buggy that I have not been able to really use it:

directory_naming47
directory_naming47

One tool that is nice to configure afterwards is Enterprise Manager. To do so execute the configuration utility of your WebLogic domain with /u01/Middleware/oracle_common/common/bin/config.sh and choose to configure an existing domain:

directory_naming48
directory_naming48

Chose your domain in directory list:

directory_naming49
directory_naming49

Add Oracle Enterprise Manager component:

directory_naming50
directory_naming50

Application location for your domain:

directory_naming51
directory_naming51

Let default:

directory_naming52
directory_naming52

Summary window:

directory_naming53
directory_naming53

I had a warning for an used port that is the one of WebLogic, maybe I should have stopped WebLogic before re-configuring it

directory_naming54
directory_naming54

Ending window:

directory_naming55
directory_naming55

Then stop and restart WebLogic by exiting and restarting /u01/Middleware/user_projects/domains/base_domain/startWebLogic.sh. You should then see at http://server1.domain.com:7001/em Enterprise Manager console that will help us to graphically modify an important setup of OID:

directory_naming56
directory_naming56

We are almost done move on to next part of the series (see references section below) to finally configure directory naming with your clients…

References

]]>
http://blog.yannickjaquier.com/oracle/directory-naming-configuration-usage-2.html/feed 1