IT World http://blog.yannickjaquier.com RDBMS, Unix and many more... Fri, 01 Dec 2017 16:52:28 +0000 en-US hourly 1 https://wordpress.org/?v=4.8.4 Oracle Label Security (OLS) 12c installation and configuration http://blog.yannickjaquier.com/oracle/oracle-label-security-ols-12c-setup.html http://blog.yannickjaquier.com/oracle/oracle-label-security-ols-12c-setup.html#respond Mon, 20 Nov 2017 15:16:16 +0000 http://blog.yannickjaquier.com/?p=3918

Table of contents

Preamble

Oracle Label Security (OLS) is not a new feature as it has been release in Oracle 9iR1. This feature is base on Virtual Private Database (VPD) technology that has even been released in Oracle 8i. OLS is a paid option of the database enterprise edition:

Oracle Virtual Private Database (VPD) is provided at no additional cost with the Enterprise Edition of Oracle Database. Oracle Label Security is an add-on security option for the Oracle Database Enterprise Edition.

When you want to protect rows of the same table you might do-it-yourself with a security column and you will check that the connected user has rights to see it with something like:

0 < (select count(*) from security_table sec where sec.name = 'BO_username' and fact_table.security__code like sec.code)

The wildcard (all regions, all groups, ...) can be simulated with underscore (_) that will perfectly work with LIKE SQL operator.

The drawback is an extended complexity in all SQL as the check will be added in WHERE clause. It is also quite easy to bypass the security if you are able to update the running SQL (forbid SQL editing in BO is a must in this case).

We have seen VPD in a previous post, that is generic term for Fine-Grained Access Control (FGAC), application context and global application context. VPD policies are made with PL/SQL while, as Oracle claims, Oracle Label Security is an out-of-the-box solution for row level security.

The high level picture of OLS model is this Oracle picture (copyright Oracle):

ols01
ols01

This can also be seen as a 3D security model (copyright Oracle):

ols02
ols02

Testing of this post has been done using Oracle Enterprise edition 12cR1 (12.1.0.2) running on Oracle Linux 7.2 64 bits running in a VirtualBox quest. The Cloud Control 13cR1 images have been done using the Oracle provided VirtualBox guest.

Oracle Label Security installation

Check if OLS is active on your database:

SQL> select value from v$option where parameter = 'Oracle Label Security';

VALUE
----------------------------------------------------------------
FALSE

If not you can activate it using DataBase Configuration Assistant (DBCA):

ols03
ols03

Or command line with:

SQL> EXEC LBACSYS.OLS_ENFORCEMENT.ENABLE_OLS;
BEGIN LBACSYS.OLS_ENFORCEMENT.ENABLE_OLS; END;

*
ERROR at line 1:
ORA-12459: Oracle Label Security not configured
ORA-06512: at "LBACSYS.OLS_ENFORCEMENT", line 3
ORA-06512: at "LBACSYS.OLS_ENFORCEMENT", line 25
ORA-06512: at line 1


SQL> !oerr ORA 12459
12459, 00000, "Oracle Label Security not configured"
// *Cause:  An administrative operation was attempted without configuring
//          Oracle Label Security.
// *Action: Consult the Oracle Label Security documentation for information
//          on how to configure Oracle Label Security.

We need to configure OLS before enabling it (database must be restarted):

SQL> select status from dba_ols_status where name = 'OLS_CONFIGURE_STATUS';

STATU
-----
FALSE

SQL> EXEC LBACSYS.CONFIGURE_OLS;

PL/SQL procedure successfully completed.

SQL> EXEC LBACSYS.OLS_ENFORCEMENT.ENABLE_OLS;

PL/SQL procedure successfully completed.

SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.

Total System Global Area  838860800 bytes
Fixed Size                  2929936 bytes
Variable Size             603982576 bytes
Database Buffers          226492416 bytes
Redo Buffers                5455872 bytes
Database mounted.
Database opened.

Check the status:

SQL> select status from dba_ols_status where name = 'OLS_CONFIGURE_STATUS';

STATU
-----
TRUE

SQL> select value from v$option where parameter = 'Oracle Label Security';

VALUE
----------------------------------------------------------------
TRUE

Unlock OLS database user (LBACSYS) and change its password:

SQL> alter user lbacsys identified by "secure_password" account unlock;

User altered.

Oracle Label Security test data model

The schema owning the data model is an OS authenticated account, I grant it the LBAC_DBA role as this account will administer OLS directly:

SQL> create user app identified externally;

User created.

SQL> grant connect,resource to app;

Grant succeeded.

SQL> grant unlimited tablespace to app;

Grant succeeded.

SQL> grant lbac_dba to app;

Grant succeeded.

The data model is made of three table. One fact table and two dimension table: SALES, PRODUT_GROUP and REGION. The creation script is self explaining I hope (no indexes except the one for primary keys as the post is not performance related):

set pages 20
create table product_group(
  code varchar2(4) primary key,
  descr varchar2(20))
tablespace users;

insert into product_group values('0001','IoT');
insert into product_group values('0002','Mems');
insert into product_group values('0003','Smart Driving');

create table region(
  code varchar2(4) primary key,
  descr varchar2(30))
tablespace users;

set define '#'
insert into region values('0001','America');
insert into region values('0002','Asia Pacific');
insert into region values('0003','Japan & Korea');
insert into region values('0004','Greater China');
insert into region values('0005','Europe Middle East & Africa');

create table sales(
  product_group__code varchar2(4),
  region__code varchar2(4),
  val number,
	constraint product_group__code_fk foreign key (product_group__code) references product_group(code),
	constraint region__code_fk foreign key (region__code) references region(code))
tablespace users;

insert into sales values('0001','0005',1500);
insert into sales values('0002','0005',10000);
insert into sales values('0003','0005',500);
insert into sales values('0001','0001',5000);
insert into sales values('0002','0001',7500);
insert into sales values('0003','0001',400);
insert into sales values('0001','0003',4000);
insert into sales values('0002','0003',10400);
insert into sales values('0003','0003',400);
insert into sales values('0001','0004',3000);
insert into sales values('0002','0004',5000);
insert into sales values('0003','0004',200);
commit;

You can get sales by product group and region using this below classical query:

SQL> select app.product_group.descr, app.region.descr, sum(app.sales.val)
from app.product_group, app.region, app.sales
where app.sales.product_group__code=app.product_group.code
and app.sales.region__code=app.region.code
group by app.product_group.descr, app.region.descr
order by 1,2,3;

DESCR                DESCR                          SUM(SALES.VAL)
-------------------- ------------------------------ --------------
IoT                  America                                  5000
IoT                  Europe Middle East & Africa              1500
IoT                  Greater China                            3000
IoT                  Japan & Korea                            4000
Mems                 America                                  7500
Mems                 Europe Middle East & Africa             10000
Mems                 Greater China                            5000
Mems                 Japan & Korea                           10400
Smart Driving        America                                   400
Smart Driving        Europe Middle East & Africa               500
Smart Driving        Greater China                             200
Smart Driving        Japan & Korea                             400

12 rows selected.

I also create a classical password authenticated account that would be used in your applciation:

SQL> create user app_read identified by "secure_password";

User created.

SQL> grant connect to app_read;

Grant succeeded.

SQL> grant select on app.sales to app_read;

Grant succeeded.

SQL> grant select on app.region to app_read;

Grant succeeded.

SQL> grant select on app.product_group to app_read;

Grant succeeded.

I also grant few execute privileges on OLS packages to APP account:

SQL> grant execute on sa_policy_admin to app;

Grant succeeded.

SQL> grant execute on to_lbac_data_label to app;

Grant succeeded.

Oracle Label Security setup

Policy creation

As LBACSYS user I create policy sales_ols_pol using all enforcement options. I choose to call the OLS column ols_col and to keep it visible:

SQL> exec sa_sysdba.create_policy(policy_name => 'sales_ols_pol', column_name => 'ols_col', default_options => 'all_control');

PL/SQL procedure successfully completed.

SQL> col policy_options for a50 word_wrapped
SQL> col column_name for a10
SQL> select * from dba_sa_policies;

POLICY_NAME                    COLUMN_NAM STATUS   POLICY_OPTIONS                                     POLIC
------------------------------ ---------- -------- -------------------------------------------------- -----
SALES_OLS_POL                  OLS_COL    ENABLED  READ_CONTROL, INSERT_CONTROL, UPDATE_CONTROL,      FALSE
                                                   DELETE_CONTROL, LABEL_DEFAULT, LABEL_UPDATE,
                                                   CHECK_CONTROL

Remark
HIDE option hide the OLS column in protected tables (use 'all_control,hide').

To allow APP account to manage its own Label Security I grant to it the policy_name_dba role:

SQL> grant sales_ols_pol_dba to app;

Grant succeeded.

Levels creation

I define two levels, a public and a confidential one. Remember that level_num value define the sensitivity ranking:

SQL> exec sa_components.create_level(policy_name => 'sales_ols_pol', level_num => 10, short_name => 'P', long_name => 'PUBLIC');

PL/SQL procedure successfully completed.

SQL> exec sa_components.create_level(policy_name => 'sales_ols_pol', level_num => 20, short_name => 'C', long_name => 'CONFIDENTIAL');

PL/SQL procedure successfully completed.

SQL> col long_name for a20
SQL> select * from dba_sa_levels order by level_num;

POLICY_NAME                     LEVEL_NUM SHORT_NAME                     LONG_NAME
------------------------------ ---------- ------------------------------ --------------------
SALES_OLS_POL                          10 P                              PUBLIC
SALES_OLS_POL                          20 C                              CONFIDENTIAL

Compartments creation

I define three compartments that are my product groups. Comp_num parameter determines the order in which compartments are listed in labels. Names are not case sensitive, will be inserted in uppercase:

SQL> exec sa_components.create_compartment(policy_name => 'sales_ols_pol',comp_num => '10', short_name => 'IOT', long_name => 'IOT');

PL/SQL procedure successfully completed.

SQL> exec sa_components.create_compartment(policy_name => 'sales_ols_pol',comp_num => '20', short_name => 'MEMS', long_name => 'MEMS');

PL/SQL procedure successfully completed.

SQL> exec sa_components.create_compartment(policy_name => 'sales_ols_pol',comp_num => '30', short_name => 'SD', long_name => 'SMART DRIVING');

PL/SQL procedure successfully completed.

SQL> select * from dba_sa_compartments order by comp_num;

POLICY_NAME                      COMP_NUM SHORT_NAME                     LONG_NAME
------------------------------ ---------- ------------------------------ --------------------
SALES_OLS_POL                          10 IOT                            IOT
SALES_OLS_POL                          20 MEMS                           MEMS
SALES_OLS_POL                          30 SD                             SMART DRIVING

Groups creation

I define three main groups and two sub-group of a main groups so five in total. Names are not case sensitive, will be inserted in uppercase. Special character like & not allowed so replacing with words:

SQL> exec sa_components.create_group(policy_name => 'sales_ols_pol', group_num => 1000, short_name => 'USA', long_name => 'AMERICA');

PL/SQL procedure successfully completed.

SQL> exec sa_components.create_group(policy_name => 'sales_ols_pol', group_num => 2000, short_name => 'AP', long_name => 'ASIA PACIFIC');

PL/SQL procedure successfully completed.

SQL> exec sa_components.create_group(policy_name => 'sales_ols_pol', group_num => 2010, short_name => 'JK', long_name => 'JAPAN AND KOREA', parent_name=> 'AP');

PL/SQL procedure successfully completed.

SQL> exec sa_components.create_group(policy_name => 'sales_ols_pol', group_num => 2020, short_name => 'GC', long_name => 'GREATER CHINA', parent_name=> 'AP');

PL/SQL procedure successfully completed.

SQL> exec sa_components.create_group(policy_name => 'sales_ols_pol', group_num => 3000, short_name => 'EMEA', long_name => 'EUROPE MIDDLE EAST AND AFRICA');

PL/SQL procedure successfully completed.
SQL> col long_name for a30
SQL> select group_num,short_name,long_name,parent_num,parent_name from dba_sa_groups order by group_num;

 GROUP_NUM SHORT_NAME                     LONG_NAME                      PARENT_NUM PARENT_NAME
---------- ------------------------------ ------------------------------ ---------- ------------------------------
      1000 USA                            AMERICA
      2000 AP                             ASIA PACIFIC
      2010 JK                             JAPAN AND KOREA                      2000 AP
      2020 GC                             GREATER CHINA                        2000 AP
      3000 EMEA                           EUROPE MIDDLE EAST AND AFRICA

You can also view created hierarchy using:

SQL> col group_name for a40
SQL> select * from DBA_SA_GROUP_HIERARCHY;

POLICY_NAME                    HIERARCHY_LEVEL GROUP_NAME
------------------------------ --------------- ----------------------------------------
SALES_OLS_POL                                1   USA - AMERICA
SALES_OLS_POL                                1   AP - ASIA PACIFIC
SALES_OLS_POL                                2     JK - JAPAN AND KOREA
SALES_OLS_POL                                2     GC - GREATER CHINA
SALES_OLS_POL                                1   EMEA - EUROPE MIDDLE EAST AND AFRICA

Label function

This is the most tricky part. You need to write what Oracle call a label function. This function will compute a label for the row based on the values of inserted columns. The returned label must return short name of levels, compartments and groups. Here below Iot product group, as next future business, is defined as confidential figures. The non-sexy case could have been replaced by select into. Refer to Oracle documentation for another example:

create or replace function gen_sales_label(product_group__code varchar2, region__code varchar2)
return lbacsys.lbac_label
as
  i_label varchar2(80);
begin
  /************* determine level *************/
  if product_group__code='0001' then --IOT
    i_label := 'C:';
  else
    i_label := 'P:';
  end if;

  /************* determine compartment *************/
  case product_group__code
	  when '0001' then i_label := i_label || 'IOT:';
	  when '0002' then i_label := i_label || 'MEMS:';
	  when '0003' then i_label := i_label || 'SD:';
  end case;

  /************* determine groups *************/
	case region__code
	  when '0001' then i_label := i_label || 'USA';
	  when '0002' then i_label := i_label || 'AP';
	  when '0003' then i_label := i_label || 'JK';
	  when '0004' then i_label := i_label || 'GC';
	  when '0005' then i_label := i_label || 'EMEA';
  end case;

  return to_lbac_data_label('sales_ols_pol',i_label);
end;
/

As APP user I apply the OLS policy to my sales table using my label function:

SQL> exec sa_policy_admin.apply_table_policy(policy_name => 'sales_ols_pol', schema_name => 'app', table_name => 'sales', -
table_options => 'all_control', label_function => 'app.gen_sales_label(:new.product_group__code,:new.region__code)');

PL/SQL procedure successfully completed.

You can notice it adds a new column to your table (if not using HIDE option):

SQL> desc sales
 Name                                                                                Null?    Type
 ----------------------------------------------------------------------------------- -------- --------------------------------------------------------
 PRODUCT_GROUP__CODE                                                                          VARCHAR2(4)
 REGION__CODE                                                                                 VARCHAR2(4)
 VAL                                                                                          NUMBER
 OLS_COL                                                                                      NUMBER(10)

To remove the policy you can use:

SQL> exec sa_policy_admin.remove_table_policy(policy_name => 'sales_ols_pol', schema_name => 'app', table_name => 'sales', drop_column=> true);

PL/SQL procedure successfully completed.

If you get ORA-12446 error message:

exec sa_policy_admin.apply_table_policy(policy_name => 'sales_ols_pol', schema_name => 'app', table_name => 'sales', table_options  => 'all_control', -
> label_function => 'app.gen_sales_label(:new.product_group__code,:new.region__code)');
BEGIN sa_policy_admin.apply_table_policy(policy_name => 'sales_ols_pol', schema_name => 'app', table_name => 'sales', table_options  => 'all_control',
label_function => 'app.gen_sales_label(:new.product_group__code,:new.region__code)'); END;

*
ERROR at line 1:
ORA-12446: Insufficient authorization for administration of policy
sales_ols_pol
ORA-06512: at "LBACSYS.LBAC_POLICY_ADMIN", line 385
ORA-06512: at line 1

Then grant sales_ols_pol_dba role to app or execute the apply table policy with LBACSYS account.

If you get ORA-12433:

exec sa_policy_admin.apply_table_policy(policy_name => 'sales_ols_pol', schema_name => 'app', table_name => 'sales', table_options  => 'all_control', -
> label_function => 'app.gen_sales_label(:new.product_group__code,:new.region__code)');
BEGIN sa_policy_admin.apply_table_policy(policy_name => 'sales_ols_pol', schema_name => 'app', table_name => 'sales', table_options  => 'all_control',
label_function => 'app.gen_sales_label(:new.product_group__code,:new.region__code)'); END;

*
ERROR at line 1:
ORA-12433: create trigger failed, policy not applied
ORA-06512: at "LBACSYS.LBAC_POLICY_ADMIN", line 385
ORA-06512: at line 1

Then grant execute on app.gen_sales_label to LBACSYS. Clearly the error message is not at all self-explaining !

You could have done all this using Cloud Control 13cR1:

ols04
ols04

Oracle Label Security testing

At that stage APP and APP_READ user are not able to see anymore figures in SALES table because the OLS_COL is empty and also because we have not defined users' security:

SQL> select * from sales;

no rows selected

Update the OLS_COL column simulating a full update of your table with something like:

SQL> update sales set product_group__code=product_group__code;

12 rows updated.

SQL> commit;

Commit complete.

SQL> select * from sales;

PROD REGI        VAL    OLS_COL
---- ---- ---------- ----------
0001 0005       1500 1000000061
0002 0005      10000 1000000062
0003 0005        500 1000000063
0001 0001       5000 1000000064
0002 0001       7500 1000000065
0003 0001        400 1000000066
0001 0003       4000 1000000067
0002 0003      10400 1000000068
0003 0003        400 1000000069
0001 0004       3000 1000000070
0002 0004       5000 1000000071
0003 0004        200 1000000072

12 rows selected.

As LBACSYS we allow APP to bypass OLS and APP_READ to change its label and privileges to another user (by default the account still see nothing):

SQL> exec sa_user_admin.set_user_privs('sales_ols_pol','app','full');

PL/SQL procedure successfully completed.

SQL> exec sa_user_admin.set_user_privs('sales_ols_pol','app_read','profile_access');

PL/SQL procedure successfully completed.

Same as in real life each applicative users will not connect to the database with their own Oracle account. Either they are authenticated with an LDAP account or with an applicative security. So what we define is a set of accounts not linked to any database user. Those accounts name will have obvious name to ease understanding of what I plan to test:

Account Privileges
sales_p_ww Worldwide access to public information
sales_c_ww Worldwide access to public and confidential information
sales_p_jk Japan & Korea access to public information
sales_p_ap Asia Pacific access to public information
sales_c_ap Asia Pacific access to public and confidential information

With Cloud Control 13cR1 the user list is:

ols05
ols05

To simulate those application accounts we will use sa_session.set_access_profile procedure to set APP_READ behaving like our applicative users but I could have used a context as we have already seen with something like SYS_CONTEXT('userenv','CLIENT_IDENTIFIER').

First test is an account with public worldwide access, IoT information must not be display:

SQL> exec sa_user_admin.set_levels(policy_name => 'sales_ols_pol', user_name => 'sales_p_ww', max_level => 'P');

PL/SQL procedure successfully completed.

SQL> exec sa_user_admin.set_compartments(policy_name => 'sales_ols_pol', user_name => 'sales_p_ww', read_comps => 'MEMS,SD');

PL/SQL procedure successfully completed.

SQL> exec sa_user_admin.set_groups(policy_name => 'sales_ols_pol', user_name => 'sales_p_ww', read_groups => 'EMEA,AP,USA');

PL/SQL procedure successfully completed.

SQL> exec sa_session.set_access_profile(policy_name => 'sales_ols_pol', user_name => 'sales_p_ww');

PL/SQL procedure successfully completed.

SQL> select sa_session.sa_user_name(policy_name => 'sales_ols_pol') from dual;

SA_SESSION.SA_USER_NAME(POLICY_NAME=>'SALES_OLS_POL')
--------------------------------------------------------------------------------
SALES_P_WW

SQL> select sa_session.label(policy_name => 'sales_ols_pol') from dual;

SA_SESSION.LABEL(POLICY_NAME=>'SALES_OLS_POL')
--------------------------------------------------------------------------------
P:MEMS,SD:USA,AP,JK,GC,EMEA

SQL> select sa_session.comp_read(policy_name => 'sales_ols_pol') from dual;

SA_SESSION.COMP_READ(POLICY_NAME=>'SALES_OLS_POL')
--------------------------------------------------------------------------------
MEMS,SD

SQL> select app.product_group.descr, app.region.descr, sum(app.sales.val)
from app.product_group, app.region, app.sales
where app.sales.product_group__code=app.product_group.code
and app.sales.region__code=app.region.code
group by app.product_group.descr, app.region.descr
order by 1,2,3;

DESCR                DESCR                          SUM(APP.SALES.VAL)
-------------------- ------------------------------ ------------------
Mems                 America                                      7500
Mems                 Europe Middle East & Africa                 10000
Mems                 Greater China                                5000
Mems                 Japan & Korea                               10400
Smart Driving        America                                       400
Smart Driving        Europe Middle East & Africa                   500
Smart Driving        Greater China                                 200
Smart Driving        Japan & Korea                                 400

8 rows selected.

Second test is an account with worldwide all level access (full sales table in other words):

SQL> exec sa_user_admin.set_levels(policy_name => 'sales_ols_pol', user_name => 'sales_c_ww', max_level => 'C');

PL/SQL procedure successfully completed.

SQL> exec sa_user_admin.set_compartments(policy_name => 'sales_ols_pol', user_name => 'sales_c_ww', read_comps => 'IOT,MEMS,SD');

PL/SQL procedure successfully completed.

SQL> exec sa_user_admin.set_groups(policy_name => 'sales_ols_pol', user_name => 'sales_c_ww', read_groups => 'EMEA,AP,USA');

PL/SQL procedure successfully completed.

SQL> exec sa_session.set_access_profile(policy_name => 'sales_ols_pol', user_name => 'sales_c_ww');

PL/SQL procedure successfully completed.

SQL> select app.product_group.descr, app.region.descr, sum(app.sales.val)
from app.product_group, app.region, app.sales
where app.sales.product_group__code=app.product_group.code
and app.sales.region__code=app.region.code
group by app.product_group.descr, app.region.descr
order by 1,2,3;

DESCR                DESCR                          SUM(APP.SALES.VAL)
-------------------- ------------------------------ ------------------
IoT                  America                                      5000
IoT                  Europe Middle East & Africa                  1500
IoT                  Greater China                                3000
IoT                  Japan & Korea                                4000
Mems                 America                                      7500
Mems                 Europe Middle East & Africa                 10000
Mems                 Greater China                                5000
Mems                 Japan & Korea                               10400
Smart Driving        America                                       400
Smart Driving        Europe Middle East & Africa                   500
Smart Driving        Greater China                                 200
Smart Driving        Japan & Korea                                 400

12 rows selected.

Third test is an account with Japan and Korea access and only public information. You can also define directly the label of the user using sa_user_admin.set_user_labels:

SQL> exec sa_user_admin.set_user_labels(policy_name => 'sales_ols_pol', user_name => 'sales_p_jk', max_read_label => 'P:MEMS,SD:JK');

PL/SQL procedure successfully completed.

SQL> select app.product_group.descr, app.region.descr, sum(app.sales.val)
from app.product_group, app.region, app.sales
where app.sales.product_group__code=app.product_group.code
and app.sales.region__code=app.region.code
group by app.product_group.descr, app.region.descr
order by 1,2,3;

DESCR                DESCR                          SUM(APP.SALES.VAL)
-------------------- ------------------------------ ------------------
Mems                 Japan & Korea                               10400
Smart Driving        Japan & Korea                                 400

Fourth test is an account with Asia Pacific access and only public information. Aim here is to see if two sub-groups are well displayed:

SQL> exec sa_user_admin.set_user_labels(policy_name => 'sales_ols_pol', user_name => 'sales_p_ap', max_read_label => 'P:MEMS,SD:AP');

PL/SQL procedure successfully completed.

SQL> select app.product_group.descr, app.region.descr, sum(app.sales.val)
from app.product_group, app.region, app.sales
where app.sales.product_group__code=app.product_group.code
and app.sales.region__code=app.region.code
group by app.product_group.descr, app.region.descr
order by 1,2,3;

DESCR                DESCR                          SUM(APP.SALES.VAL)
-------------------- ------------------------------ ------------------
Mems                 Greater China                                5000
Mems                 Japan & Korea                               10400
Smart Driving        Greater China                                 200
Smart Driving        Japan & Korea                                 400

Fifth test is an account with Asia Pacific access and all levels. Aim is again to see if two sub-groups are well displayed:

SQL> exec sa_user_admin.set_user_labels(policy_name => 'sales_ols_pol', user_name => 'sales_c_ap', max_read_label => 'C:IOT,MEMS,SD:AP');

PL/SQL procedure successfully completed.

SQL> select /* Yannick */ app.product_group.descr, app.region.descr, sum(app.sales.val)
from app.product_group, app.region, app.sales
where app.sales.product_group__code=app.product_group.code
and app.sales.region__code=app.region.code
group by app.product_group.descr, app.region.descr
order by 1,2,3;

DESCR                DESCR                          SUM(APP.SALES.VAL)
-------------------- ------------------------------ ------------------
IoT                  Greater China                                3000
IoT                  Japan & Korea                                4000
Mems                 Greater China                                5000
Mems                 Japan & Korea                               10400
Smart Driving        Greater China                                 200
Smart Driving        Japan & Korea                                 400

6 rows selected.

I also wanted to see how Oracle is handling those query and if any transformations are applied to the query before executing them. We can see than SQL queries remain unchanged and only a filter is applied. See Predicate Information number 9 in below explain plan extract:

SQL_ID  8qt810d4tkg1x, child number 0
-------------------------------------
select /* Yannick */ app.product_group.descr, app.region.descr, 
sum(app.sales.val) from app.product_group, app.region, app.sales where 
app.sales.product_group__code=app.product_group.code and 
app.sales.region__code=app.region.code group by 
app.product_group.descr, app.region.descr order by 1,2,3
 
Plan hash value: 4206122969
 
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Id  | Operation                        | Name          | Starts | E-Rows |E-Bytes| Cost (%CPU)| E-Time   | A-Rows |   A-Time   | Buffers | Reads  |  OMem |  1Mem |  O/1/M   |
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                 |               |      1 |        |       |    11 (100)|          |     12 |00:00:00.01 |      16 |      6 |       |       |          |
|   1 |  SORT ORDER BY                   |               |      1 |     11 |   583 |    11  (28)| 00:00:01 |     12 |00:00:00.01 |      16 |      6 |  2048 |  2048 |     1/0/0|
|   2 |   HASH GROUP BY                  |               |      1 |     11 |   583 |    11  (28)| 00:00:01 |     12 |00:00:00.01 |      16 |      6 |   930K|   930K|     1/0/0|
|*  3 |    FILTER                        |               |      1 |        |       |            |          |     12 |00:00:00.01 |      16 |      6 |       |       |          |
|*  4 |     HASH JOIN                    |               |      1 |     12 |   636 |     9  (12)| 00:00:01 |     12 |00:00:00.01 |      16 |      6 |  1393K|  1393K|     1/0/0|
|   5 |      MERGE JOIN                  |               |      1 |     12 |   396 |     6  (17)| 00:00:01 |     12 |00:00:00.01 |       9 |      4 |       |       |          |
|   6 |       TABLE ACCESS BY INDEX ROWID| PRODUCT_GROUP |      1 |      3 |    39 |     2   (0)| 00:00:01 |      3 |00:00:00.01 |       2 |      1 |       |       |          |
|   7 |        INDEX FULL SCAN           | SYS_C0010354  |      1 |      3 |       |     1   (0)| 00:00:01 |      3 |00:00:00.01 |       1 |      1 |       |       |          |
|*  8 |       SORT JOIN                  |               |      3 |     12 |   240 |     4  (25)| 00:00:01 |     12 |00:00:00.01 |       7 |      3 |  2048 |  2048 |     1/0/0|
|*  9 |        TABLE ACCESS FULL         | SALES         |      1 |     12 |   240 |     3   (0)| 00:00:01 |     12 |00:00:00.01 |       7 |      3 |       |       |          |
|  10 |      TABLE ACCESS FULL           | REGION        |      1 |      5 |   100 |     3   (0)| 00:00:01 |      5 |00:00:00.01 |       7 |      2 |       |       |          |
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
Query Block Name / Object Alias (identified by operation id):
-------------------------------------------------------------
 
   1 - SEL$F5BB74E1
   6 - SEL$F5BB74E1 / PRODUCT_GROUP@SEL$1
   7 - SEL$F5BB74E1 / PRODUCT_GROUP@SEL$1
   9 - SEL$F5BB74E1 / SALES@SEL$2
  10 - SEL$F5BB74E1 / REGION@SEL$1
 
Predicate Information (identified by operation id):
---------------------------------------------------
 
   3 - filter(TO_NUMBER(SYS_CONTEXT('LBAC$0_LAB','LBAC$MAXLABEL'))>=TO_NUMBER(SYS_CONTEXT('LBAC$0_LAB','LBAC$MINLABEL')))
   4 - access("REGION__CODE"="REGION"."CODE")
   8 - access("PRODUCT_GROUP__CODE"="PRODUCT_GROUP"."CODE")
       filter("PRODUCT_GROUP__CODE"="PRODUCT_GROUP"."CODE")
   9 - filter(("OLS_COL">=TO_NUMBER(SYS_CONTEXT('LBAC$0_LAB','LBAC$MINLABEL')) AND "OLS_COL"<=TO_NUMBER(SYS_CONTEXT('LBAC$0_LAB','LBAC$MAXLABEL')) AND 
              TO_NUMBER(SYS_CONTEXT('LBAC$LABELS',TO_CHAR("OLS_COL")))>=0))
 
Column Projection Information (identified by operation id):
-----------------------------------------------------------
 
   1 - (#keys=3) "PRODUCT_GROUP"."DESCR"[VARCHAR2,20], "REGION"."DESCR"[VARCHAR2,30], SUM("VAL")[22]
   2 - "PRODUCT_GROUP"."DESCR"[VARCHAR2,20], "REGION"."DESCR"[VARCHAR2,30], SUM("VAL")[22]
   3 - "PRODUCT_GROUP"."DESCR"[VARCHAR2,20], "REGION"."DESCR"[VARCHAR2,30], "VAL"[NUMBER,22], "REGION"."DESCR"[VARCHAR2,30]
   4 - (#keys=1) "PRODUCT_GROUP"."DESCR"[VARCHAR2,20], "REGION"."DESCR"[VARCHAR2,30], "VAL"[NUMBER,22], "REGION"."DESCR"[VARCHAR2,30]
   5 - "PRODUCT_GROUP"."DESCR"[VARCHAR2,20], "REGION__CODE"[VARCHAR2,4], "VAL"[NUMBER,22]
   6 - "PRODUCT_GROUP"."CODE"[VARCHAR2,4], "PRODUCT_GROUP"."DESCR"[VARCHAR2,20]
   7 - "PRODUCT_GROUP".ROWID[ROWID,10], "PRODUCT_GROUP"."CODE"[VARCHAR2,4]
   8 - (#keys=1) "PRODUCT_GROUP__CODE"[VARCHAR2,4], "REGION__CODE"[VARCHAR2,4], "VAL"[NUMBER,22]
   9 - "PRODUCT_GROUP__CODE"[VARCHAR2,4], "REGION__CODE"[VARCHAR2,4], "VAL"[NUMBER,22]
  10 - "REGION"."CODE"[VARCHAR2,4], "REGION"."DESCR"[VARCHAR2,30]
 
Note
-----
   - this is an adaptive plan

Oracle Label Security auditing

Oracle Label Security policies can be audited using SA_AUDIT_ADMIN package. First ensure audit_trail parameter is not set to none value:

SQL> show parameter audit_trail

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
audit_trail                          string      DB

I activate all possible audit for my policy by access:

SQL> exec sa_audit_admin.audit(policy_name => 'sales_ols_pol', audit_type => 'BY ACCESS');

PL/SQL procedure successfully completed.

SQL> exec sa_audit_admin.audit(policy_name => 'sales_ols_pol', audit_option => 'PRIVILEGES', audit_type => 'BY ACCESS');

PL/SQL procedure successfully completed.

SQL> col user_name for a30
SQL> select * from dba_sa_audit_options;

POLICY_NAME                    USER_NAME                      APY REM SET PRV
------------------------------ ------------------------------ --- --- --- ---
SALES_OLS_POL                  ALL_USERS                      A/A A/A A/A A/A

I also activate policy label recording with:

SQL> exec sa_audit_admin.audit_label(policy_name => 'sales_ols_pol');

PL/SQL procedure successfully completed.

SQL> set serveroutput on
begin
  if sa_audit_admin.audit_label_enabled('sales_ols_pol')
    then dbms_output.put_line('OLS sales_ols_pol labels are being audited.');
  else
    dbms_output.put_line('OLS sales_ols_pol labels not being audited.');
  end if;
end;
/
OLS sales_ols_pol labels are being audited.

PL/SQL procedure successfully completed.

I also create the dedicated view to display audit records:

SQL> exec sa_audit_admin.create_view(policy_name => 'sales_ols_pol');

PL/SQL procedure successfully completed.

SQL> desc dba_sales_ols_pol_audit_trail
 Name                                                                                Null?    Type
 ----------------------------------------------------------------------------------- -------- --------------------------------------------------------
 USERNAME                                                                                     VARCHAR2(128)
 USERHOST                                                                                     VARCHAR2(128)
 TERMINAL                                                                                     VARCHAR2(255)
 TIMESTAMP                                                                                    DATE
 OWNER                                                                                        VARCHAR2(128)
 OBJ_NAME                                                                                     VARCHAR2(128)
 ACTION                                                                              NOT NULL NUMBER
 ACTION_NAME                                                                                  VARCHAR2(47)
 COMMENT_TEXT                                                                                 VARCHAR2(4000)
 SESSIONID                                                                           NOT NULL NUMBER
 ENTRYID                                                                             NOT NULL NUMBER
 STATEMENTID                                                                         NOT NULL NUMBER
 RETURNCODE                                                                          NOT NULL NUMBER
 EXTENDED_TIMESTAMP                                                                           TIMESTAMP(6) WITH TIME ZONE
 OLS_COL                                                                                      VARCHAR2(4000)

If I select in SALES table with APP account I get:

SQL> col comment_text for a40
SQL> col username for a10
SQL> select username,timestamp,action_name,comment_text from lbacsys.dba_sales_ols_pol_audit_trail;

USERNAME   TIMESTAMP ACTION_NAME                                     COMMENT_TEXT
---------- --------- ----------------------------------------------- ----------------------------------------
APP        04-NOV-16 PRIVILEGED ACTION                               SALES_OLS_POL: BYPASSALL PRIVILEGE SET

But if I select with APP_READ using SA_SESSION.SET_ACCESS_PROFILE package then it does not generate any audit record, so a bit disappointed...

Auditing can also be managed with Cloud Control:

ols06
ols06

Oracle Label Security cleaning

Cleaning everything simply means dropping the policy with LBACSYS account:

SQL> exec sa_sysdba.drop_policy(policy_name => 'sales_ols_pol', drop_column => true);

PL/SQL procedure successfully completed.

References

]]>
http://blog.yannickjaquier.com/oracle/oracle-label-security-ols-12c-setup.html/feed 0
Database Vault 12cR1 installation and configuration http://blog.yannickjaquier.com/oracle/database-vault-12cr1-installation.html http://blog.yannickjaquier.com/oracle/database-vault-12cr1-installation.html#respond Mon, 23 Oct 2017 06:45:10 +0000 http://blog.yannickjaquier.com/?p=3884

Table of contents

Preamble

I have already tested Oracle Database Vault in 11gR2 but as we are starting the study for a new project containing HR figures I wanted to re-test Database Vault 12cR1 (12.1.0.2) to see how things improved. Worth to re-mention that Database Vault is a paid option of Oracle Database Enterprise Edition.

I have two virtual machine to do the testing, one is running my 12cR1 (12.1.0.2) Enterprise edition database under Oracle Linux Server release 7.2. The second virtual machine is a VirtualBox image that I have directly downloaded at:
http://www.oracle.com/technetwork/oem/enterprise-manager/downloads/oem-templates-2767917.html

What this VirtualBox image provides is a complete running Cloud Control 13cR1 (13.1.0.0.0) environment. Even if Cloud Control 13cR2 is available at the time of writing this blog post it is not yet available for download as a running image.

I will also briefly test Privilege Analysis, a new feature introduced with Database Vault 12cR1. Privilege Analysis does not require you enable Database Vault to use it but its licensing is part of Database Vault.

Database Vault 12cR1 installation

Versus 11gR2 Database Vault is already installed but not activated as Oracle says:

Starting with Oracle Database 12c, Oracle Database Vault is installed by default but not enabled. Customers can enable it using DBCA or from the command line using SQL*Plus in a matter of minutes.

Can be confirmed with:

SQL> set lines 150
SQL> select parameter, value from v$option where parameter = 'Oracle Database Vault';

PARAMETER                                                        VALUE
---------------------------------------------------------------- ----------------------------------------------------------------
Oracle Database Vault                                            FALSE

Following the official documentation to activate it I executed (the 12cR1 default accounts have moved from dbvxxx to dbv_xxx):

SQL> exec DVSYS.CONFIGURE_DV (dvowner_uname => 'dbv_owner', dvacctmgr_uname => 'dbv_acctmgr');
BEGIN DVSYS.CONFIGURE_DV (dvowner_uname => 'dbv_owner', dvacctmgr_uname => 'dbv_acctmgr'); END;

*
ERROR at line 1:
ORA-01918: user 'dbv_owner' does not exist
ORA-06512: at "DVSYS.DBMS_MACUTL", line 34
ORA-06512: at "DVSYS.DBMS_MACUTL", line 389
ORA-06512: at "DVSYS.CONFIGURE_DV", line 126
ORA-06512: at line 1

Fortunately MOS note How To Enable Database Vault in a 12c database ? (Doc ID 2112167.1) comes to the rescue and accounts must be created first:

SQL> create user dbv_owner identified by "secure_password";

User created.

SQL> create user dbv_acctmgr identified by "secure_password";

User created.

Vault activation is then straightforward:

SQL> exec DVSYS.CONFIGURE_DV (dvowner_uname => 'dbv_owner', dvacctmgr_uname => 'dbv_acctmgr');

PL/SQL procedure successfully completed.

SQL> connect dbv_owner/"secure_password"
Connected.
SQL> exec dbms_macadm.enable_dv;

PL/SQL procedure successfully completed.

Restart the database and you can see it’s there:

SQL> set lines 150
SQL> select parameter, value from v$option where parameter = 'Oracle Database Vault';

PARAMETER                                                        VALUE
---------------------------------------------------------------- ----------------------------------------------------------------
Oracle Database Vault                                            TRUE

To be able to use Cloud Control 13cR1 I have granted dv_admin role to my personal DBA account:

SQL> connect dbv_owner/"secure_password"
Connected.
SQL> grant dv_admin to yjaquier;

Grant succeeded.

Database Vault 12cR1 testing

The aim of my testing is to protect objects owned by and OS authenticated account (APP) from high privileges users but to keep the access for accounts to which grants have been given (APP_READ). The data model in itself is simple. I start with accounts creation with dbv_acctmgr account (except RESOURCE role that must be granted with SYS account):

SQL> connect dbv_acctmgr/"secure_password"
Connected.
SQL> create user app identified externally default tablespace users;
 
USER created.
 
SQL> alter user app quota unlimited on users;
 
USER altered.
 
SQL> grant connect to app;
 
GRANT succeeded.
 
SQL> create user app_read identified by secure_password;
 
USER created.
 
SQL> grant connect to app_read;
 
GRANT succeeded.

SQL> connect / as sysdba
Connected.
SQL> grant resource to app;
 
GRANT succeeded.

Remark
If you have activated Database Vault before creating the data model you start to see the added complexity of the product. The new account to manage accounts is dbv_acctmgr except for few privileges related to the default realms created when implementing Database Vault:

  • RESOURCE role is protected in Oracle System Privilege and Role Management Realm realm and accessible only by SYS
  • CONNECT role is protected in Database Vault Account Management realm and accessible only by DV_ACCTMGR

To see the Oracle defined realms either you use Cloud Control under security/Database Vault menu and check the “Show Oracle defined Realms” checkbox in Administration tab and Realms sub-menu:

database_vault_12cr1_01
database_vault_12cr1_01

Or you use below query (there is no real column to extract only Oracle defined realms):

SQL> set pages 50
SQL> col name for a50
SQL> col description for a80 word_wrapped
SQL> select name, description from DVSYS.DV$REALM where realm_type is null;

NAME                                               DESCRIPTION
-------------------------------------------------- --------------------------------------------------------------------------------
Oracle Database Vault                              Defines the realm for the Oracle Database Vault schemas - DVSYS, DVF and LBACSYS
                                                   where Database Vault access control configuration and roles are contained.

Database Vault Account Management                  Defines the realm for administrators who create and manage database accounts and
                                                   profiles.

Oracle Enterprise Manager                          Defines the Enterprise Manager monitoring and management realm.
Oracle Default Schema Protection Realm             Defines the realm for the Oracle Default schemas.
Oracle System Privilege and Role Management Realm  Defines the realm to control granting of system privileges and database
                                                   administrator roles.

Oracle Default Component Protection Realm          Defines the realm to protect default components of the Oracle database.

6 rows selected.

I connect with APP account and create the employees table and grant select and update on it to APP_READ:

SQL> set lines 150
SQL> CREATE TABLE employees (
     id NUMBER,
     firstname VARCHAR2(50),
     lastname VARCHAR2(50),
     salary NUMBER);
 
TABLE created.
 
SQL> GRANT select, update ON employees TO app_read;
 
GRANT succeeded.
 
SQL> INSERT INTO employees VALUES(1,'Yannick','Jaquier',10000);
 
1 ROW created.
 
SQL> COMMIT;
 
COMMIT complete.
 
SQL> SELECT * FROM employees;
 
        ID FIRSTNAME                                          LASTNAME                                               SALARY
---------- -------------------------------------------------- -------------------------------------------------- ----------
         1 Yannick                                            Jaquier                                                 10000

To create the realm you can use Cloud Control, click on create Administration tab and Realms sub-menu in Security/Database Vault:

database_vault_12cr1_02
database_vault_12cr1_02

Enter a name and a description, its original status and what you wan to audit:

database_vault_12cr1_03
database_vault_12cr1_03

Choose the objects you’d like to protect, let rest by default:

database_vault_12cr1_04
database_vault_12cr1_04

In PL/SQL it gives the two below command, parameter realm_type is set to 0 to avoid creation of a mandatory realms which means that objects owner (APP) will still have full access to its objects. I also activate the realms right after its creation (enabled parameter):

SQL> exec dbms_macadm.create_realm(realm_name => 'APP schema', description => 'Protect APP Schema ', enabled => 'Y', audit_options => 1, realm_type =>'0' );

PL/SQL procedure successfully completed.

SQL> exec dbms_macadm.add_object_to_realm(realm_name => 'APP schema', object_owner => 'APP', object_name => '%', object_type => '%' );

PL/SQL procedure successfully completed.

Even if I do not add any users to the realms the accounts that have select privileges on APP schema can still perform select while DBA like accounts (including SYS and SYSTEM) cannot see figures anymore:

SQL> set lines 150
SQL> show user
USER is "YJAQUIER"
SQL> select * from app.employees;
select * from app.employees
                  *
ERROR at line 1:
ORA-01031: insufficient privileges

SQL> connect app_read/"secure_password"
SQL> show user
USER is "APP_READ"
SQL> select * from app.employees;

        ID FIRSTNAME                                          LASTNAME                                               SALARY
---------- -------------------------------------------------- -------------------------------------------------- ----------
         1 Yannick                                            Jaquier                                                 10000

If you make the realms mandatory:

SQL> exec dbms_macadm.update_realm(realm_name => 'APP schema', description => 'Protect APP Schema ', enabled => 'Y', realm_type =>'1');

PL/SQL procedure successfully completed.

Then APP_READ schema cannot anymore select on employees table:

SQL> show user
USER is "APP_READ"
SQL> select * from app.employees;
select * from app.employees
                  *
ERROR at line 1:
ORA-01031: insufficient privileges

So the simplest first installation is a non mandatory realm but even if you try to make it simple you will see that APP user cannot create any more objects in its own schema:

SQL> show user
USER is "APP"
SQL> create table test01 (val number);
create table test01 (val number)
*
ERROR at line 1:
ORA-47401: Realm violation for CREATE TABLE on APP.TEST01

If you grant the APP account as a participant to its own realm (can also be done with Cloud Control):

SQL> exec dbms_macadm.add_auth_to_realm(realm_name => 'APP schema', grantee => 'APP', rule_set_name => '', auth_options => DBMS_MACUTL.G_REALM_AUTH_PARTICIPANT);

PL/SQL procedure successfully completed.

Then you can create object but you cannot grant them to other accounts:

SQL> create table test01(val number);

Table created.

SQL> insert into test01 values(1);

1 row created.

SQL> commit;

Commit complete.

SQL> grant select on test01 to app_read;
grant select on test01 to app_read
                *
ERROR at line 1:
ORA-47401: Realm violation for GRANT on APP.TEST01

If you grant APP to be a owner of its own realm (you need to remove it before re-adding it with another role), can be done with Cloud Control as well:

SQL> exec dbms_macadm.add_auth_to_realm(realm_name => 'APP schema', grantee => 'APP', rule_set_name => '', auth_options => DBMS_MACUTL.G_REALM_AUTH_OWNER);
BEGIN dbms_macadm.add_auth_to_realm(realm_name => 'APP schema', grantee => 'APP', rule_set_name => '', auth_options => DBMS_MACUTL.G_REALM_AUTH_OWNER); END;

*
ERROR at line 1:
ORA-47260: Realm Authorization to APP schema for Realm APP already defined
ORA-06512: at "DVSYS.DBMS_MACADM", line 1903
ORA-06512: at "DVSYS.DBMS_MACADM", line 1968
ORA-06512: at line 1


SQL> exec dbms_macadm.delete_auth_from_realm(realm_name => 'APP schema', grantee => 'APP');

PL/SQL procedure successfully completed.

SQL> exec dbms_macadm.add_auth_to_realm(realm_name => 'APP schema', grantee => 'APP', rule_set_name => '', auth_options => DBMS_MACUTL.G_REALM_AUTH_OWNER);

PL/SQL procedure successfully completed.

Then APP can grant its own objects to others (and grantee can see figures):

SQL> grant select on test01 to app_read;

Grant succeeded.

SQL> connect app_read/"secure_password"
SQL> select * from app.test01;

       VAL
----------
         1

To remove everything simply execute:

SQL> exec dbms_macadm.delete_realm(realm_name => 'APP schema');

PL/SQL procedure successfully completed.

Database Vault 12cR1 reporting

The two views, well three views, to control what has been changed in Vault or who has tried to violate one of the Vault enforcement are DVSYS.DV$CONFIGURATION_AUDIT and DVSYS.DV$ENFORCEMENT_AUDIT based on DVSYS.AUDIT_TRAIL$:

SQL> col username for a10
SQL> col action_name for a30
SQL> col action_object_name for a20
SQL> set pages 100
SQL> alter session set nls_date_format="dd-mon-yyyy hh24:mi:ss";

Session altered.
SQL> select username,timestamp,action_name,action_object_name from dvsys.dv$configuration_audit order by id# desc;

USERNAME   TIMESTAMP            ACTION_NAME                    ACTION_OBJECT_NAME
---------- -------------------- ------------------------------ --------------------
YJAQUIER   17-oct-2016 17:36:53 Add Realm Auth Audit           APP schema
YJAQUIER   17-oct-2016 17:36:50 Delete Realm Auth Audit        APP schema
YJAQUIER   17-oct-2016 17:33:47 Add Realm Auth Audit           APP schema
YJAQUIER   17-oct-2016 17:30:31 Add Realm Object Audit         APP schema
YJAQUIER   17-oct-2016 17:30:24 Realm Creation Audit           APP schema
YJAQUIER   17-oct-2016 17:17:31 Realm Deletion Audit           APP schema
YJAQUIER   14-oct-2016 17:55:59 Add Realm Auth Audit           APP schema
YJAQUIER   14-oct-2016 17:55:59 Delete Realm Auth Audit        APP schema
YJAQUIER   14-oct-2016 17:48:56 Add Realm Auth Audit           APP schema
YJAQUIER   14-oct-2016 17:48:56 Delete Realm Auth Audit        APP schema
YJAQUIER   14-oct-2016 16:51:04 Add Realm Auth Audit           APP schema
YJAQUIER   14-oct-2016 15:52:19 Add Realm Object Audit         APP schema
YJAQUIER   14-oct-2016 15:52:14 Realm Creation Audit           APP schema
YJAQUIER   14-oct-2016 15:48:54 Realm Deletion Audit           APP schema
YJAQUIER   14-oct-2016 13:07:04 Add Realm Object Audit         APP schema
YJAQUIER   14-oct-2016 13:06:59 Realm Creation Audit           APP schema
YJAQUIER   14-oct-2016 13:06:34 Realm Deletion Audit           APP schema
YJAQUIER   14-oct-2016 12:52:40 Realm Update Audit             APP schema
YJAQUIER   14-oct-2016 12:44:01 Delete Realm Auth Audit        APP schema
YJAQUIER   14-oct-2016 12:30:20 Realm Update Audit             APP schema
YJAQUIER   14-oct-2016 12:30:10 Realm Update Audit             APP schema
YJAQUIER   14-oct-2016 12:23:58 Add Realm Auth Audit           APP schema
YJAQUIER   14-oct-2016 12:14:33 Realm Update Audit             APP schema
YJAQUIER   14-oct-2016 10:50:13 Realm Update Audit             APP schema
YJAQUIER   14-oct-2016 10:15:03 Add Realm Object Audit         APP schema
YJAQUIER   14-oct-2016 10:14:55 Realm Creation Audit           APP schema
DBV_OWNER  13-oct-2016 16:40:49 Enable DV enforcement Audit

27 rows selected.

SQL> col action_command for a60
SQL> select username,timestamp,action_name,action_object_name,action_command from dvsys.dv$enforcement_audit order by id# desc;

USERNAME   TIMESTAMP            ACTION_NAME                    ACTION_OBJECT_NAME   ACTION_COMMAND
---------- -------------------- ------------------------------ -------------------- ------------------------------------------------------------
APP        17-oct-2016 17:34:31 Realm Violation Audit          APP schema           GRANT SELECT ON TEST01 TO APP_READ
APP        17-oct-2016 17:33:10 Realm Violation Audit          APP schema           CREATE TABLE TEST01 (VAL NUMBER)
APP        17-oct-2016 17:32:19 Realm Violation Audit          APP schema           DROP TABLE TEST1
YJAQUIER   17-oct-2016 17:31:45 Realm Violation Audit          APP schema           SELECT * FROM APP.EMPLOYEES
DBV_OWNER  17-oct-2016 15:44:36 Realm Violation Audit          Database Vault Accou GRANT CONNECT TO APP_READ
                                                               nt Management

DBV_OWNER  17-oct-2016 15:42:06 Realm Violation Audit          Oracle System Privil GRANT RESOURCE TO TEST
                                                               ege and Role Managem
                                                               ent Realm

DBV_ACCTMG 17-oct-2016 15:41:52 Realm Violation Audit          Oracle System Privil GRANT RESOURCE TO TEST
R                                                              ege and Role Managem
                                                               ent Realm

DBV_OWNER  17-oct-2016 15:41:45 Realm Violation Audit          Database Vault Accou GRANT CONNECT TO TEST
                                                               nt Management

DBV_OWNER  17-oct-2016 15:41:39 Realm Violation Audit          Oracle System Privil GRANT RESOURCE TO TEST
                                                               ege and Role Managem
                                                               ent Realm

DBV_OWNER  17-oct-2016 15:40:33 Realm Violation Audit          Database Vault Accou GRANT CONNECT,RESOURCE TO TEST
                                                               nt Management

SYS        17-oct-2016 15:40:19 Realm Violation Audit          Database Vault Accou GRANT CONNECT,RESOURCE TO TEST
                                                               nt Management

DBV_ACCTMG 17-oct-2016 15:40:02 Realm Violation Audit          Oracle System Privil GRANT CONNECT,RESOURCE TO TEST
R                                                              ege and Role Managem
                                                               ent Realm
.
.
.

But it cannot compete with the really nice Cloud Control Database Vault reporting. You can also click on each slice of the pie to get further details:

database_vault_12cr1_05
database_vault_12cr1_05

Database Vault 12cR1 errors

When playing with my own account granted and removed to my test realm and when de-activating, re-creating my test realm I reached a strange situation. I was able to select from APP.EMPLOYEES table with my own account while technically I should not have been able to do so. A simple flush shared pool solved the bad situation (and end my half day investigation):

SQL> show user
USER is "YJAQUIER"
SQL> select * from app.employees;

        ID FIRSTNAME                                          LASTNAME                                               SALARY
---------- -------------------------------------------------- -------------------------------------------------- ----------
         1 Yannick                                            Jaquier                                                 10000

SQL> alter system flush shared_pool;

System altered.

SQL> select * from app.employees;
select * from app.employees
                  *
ERROR at line 1:
ORA-01031: insufficient privileges

Oracle Database Vault 12cR1 Privilege Analysis

Database Vault 12cR1 introduce a new feature called Privilege Analysis. As the name stand for it helps you to analyze the used and unused privileges inside your database. A typical example is an applicative account which you can study to see which privileges (system or objects) it is using and the one it is NOT using. May really helps you to revoke too high privileges in a safer manner.

In Cloud Control in security menu choose Privilege Analysis:

database_vault_12cr1_06
database_vault_12cr1_06

To create a Privilege Analysis policy either you use Cloud Control or PL/SQL. I will create a policy to check privileges APP_READ account uses and dig in the ones it is not using. Remember we have granted update on APP.EMPLOYEES that I will not use in my testing:

SQL> exec DBMS_PRIVILEGE_CAPTURE.CREATE_CAPTURE(name => 'APP_READ policy', description => 'Check APP_READ privileges', type => DBMS_PRIVILEGE_CAPTURE.G_CONTEXT, -
> condition => 'SYS_CONTEXT (''USERENV'',''CURRENT_SCHEMA'') = ''APP_READ''' );

PL/SQL procedure successfully completed.

SQL> col context for a60
SQL> select type,enabled,context from DBA_PRIV_CAPTURES where name='APP_READ policy';

TYPE             E CONTEXT
---------------- - ------------------------------------------------------------
CONTEXT          N SYS_CONTEXT ('USERENV','CURRENT_SCHEMA') = 'APP_READ'

Enable the policy with:

SQL> exec dbms_privilege_capture.enable_capture(name => 'APP_READ policy');

PL/SQL procedure successfully completed.

Perform connect and few select with APP_READ account but do not update APP.EMPLOYESS table to show APP_READ does not need this privileges ! Once done disable the policy:

SQL> exec dbms_privilege_capture.disable_capture(name => 'APP_READ policy');

PL/SQL procedure successfully completed.

With Cloud Control it gives something like:

database_vault_12cr1_07
database_vault_12cr1_07

Generate result report with:

SQL> exec dbms_privilege_capture.generate_result(name => 'APP_READ policy');

PL/SQL procedure successfully completed.

And fetch the figures with (refer to Database Vault official documentation for long list of available views), this is extracted and modified from official documentation. List of used privileges:

SQL> col username format a10
SQL> col sys_priv format a16
SQL> col object_owner format a13
SQL> col object_name format a23
SQL> select username, nvl(sys_priv,obj_priv) as priv, object_owner, object_name from dba_used_privs where username='APP_READ';

USERNAME   PRIV                                     OBJECT_OWNER  OBJECT_NAME
---------- ---------------------------------------- ------------- -----------------------
APP_READ   SELECT                                   APP           EMPLOYEES
APP_READ   SELECT                                   APP           TEST01
APP_READ   SELECT                                   SYS           DUAL
APP_READ   CREATE SESSION
APP_READ   EXECUTE                                  SYS           DBMS_APPLICATION_INFO
APP_READ   SELECT                                   SYS           DUAL
APP_READ   SELECT                                   SYSTEM        PRODUCT_PRIVS

7 rows selected.

List of unused privileges, we clearly see that update on app.employees has not been used so might be revoked:

SQL> select username, nvl(sys_priv,obj_priv) as priv, object_owner, object_name from dba_unused_privs where username='APP_READ';

USERNAME   PRIV                                     OBJECT_OWNER  OBJECT_NAME
---------- ---------------------------------------- ------------- -----------------------
APP_READ   UPDATE                                   APP           EMPLOYEES
APP_READ   SET CONTAINER

More easily the reports can be generated from Cloud Control:

database_vault_12cr1_08
database_vault_12cr1_08

Clean Privileges Analysis with:

SQL> exec dbms_privilege_capture.drop_capture(name => 'APP_READ policy');

PL/SQL procedure successfully completed.

References

  • Master Note For Oracle Database Vault (Doc ID 1195205.1)
  • How To Enable Database Vault in a 12c database ? (Doc ID 2112167.1)
]]>
http://blog.yannickjaquier.com/oracle/database-vault-12cr1-installation.html/feed 0
Data Redaction (DBMS_REDACT) with 12cR1 (12.1.0.2) http://blog.yannickjaquier.com/oracle/data-redaction-dbms_redact-12cr1.html http://blog.yannickjaquier.com/oracle/data-redaction-dbms_redact-12cr1.html#respond Tue, 26 Sep 2017 15:02:37 +0000 http://blog.yannickjaquier.com/?p=3857

Table of contents

Preamble

Advanced Security enterprise edition paid option is made of two products:

Data Redaction conditionally hide on-the-fly sensitive data before it leaves the database. The picture available on Oracle product page (copyright Oracle) says it all:

data_redaction01
data_redaction01

The condition to hide figures is really open as you write it in PL/SQL. The one I will take as an example is an application with its own security model (LDAP or whatever) connecting to a database using the same applicative account. This is a real life example with any Java or web application.

As I have already tested Virtual private database (VPD), that is the term used for combination of fine grained access control (FGAC) with application contexts, I have asked myself the difference with Data Redaction. Fortunately it is well explained in official documentation in Oracle Data Redaction and Oracle Virtual Private Database chapter.

Whatever Oracle says I have feeling that Data Redaction that is new in 12cR1 and back ported in 11gR2 (11.2.0.4 only) is the new tool to use to hide sensitive information. Unfortunately VPD and FGAC are free while Data Redaction is not…

Testing of this blog post has been done using Oracle database enterprise edition 12.1.0.2.0 – 64bit running on Oracle Linux Server release 7.2 in a virtual machine.

Data redaction implementation

I create my application schema owner, identified externally. I also provide execute on Data Redaction package called DBMS_REDACT and capability to create a context (still in 12cR1 the create context does not exist):

SQL> create user app identified externally
     default tablespace users;

User created.

SQL> alter user app quota unlimited on users;

User altered.

SQL> grant connect,resource to app;

Grant succeeded.

SQL> grant execute on dbms_redact to app;

Grant succeeded.

SQL> grant create any context to app;

Grant succeeded.

I create and provide grants to the password authenticated user that will be used in my application:

SQL> create user app_read identified by secure_password;

User created.

SQL> grant connect to app_read;

Grant succeeded.

As app user I create my applicative table (really basic one). I also grant select and update to the account that will be used in my application:

SQL> create table employees (
     id number,
     firstname varchar2(50),
     lastname varchar2(50),
     salary number);

Table created.

SQL> grant select,update on employees to app_read;

Grant succeeded.

SQL> insert into employees values(1,'Yannick','Jaquier',10000);

1 row created.

SQL> commit;

Commit complete.

SQL> select * from employees;

        ID FIRSTNAME                                          LASTNAME                                               SALARY
---------- -------------------------------------------------- -------------------------------------------------- ----------
         1 Yannick                                            Jaquier                                                 10000

As you might guess the column we want to redact (hide) to a part of applicative users is salary !

To create the Data Redaction policy I will use an application context that we have already seen when testing FGAC so going a little bit faster on this part. As app account I create a context and a package to change its value based on client_identifier parameter value of userenv context, necessary grants are also provided. The rule is that any supervisor (supervisorxx value) can see salary while the other accounts cannot:

SQL> create or replace context my_context1 using my_context1_pkg;

Context created.

SQL> CREATE OR REPLACE PACKAGE my_context1_pkg IS
     PROCEDURE set_my_context1;
     END;
     /

Package created.

SQL> create or replace package body my_context1_pkg as
  procedure set_my_context1 is
  begin
    if lower(sys_context('userenv', 'client_identifier')) like 'supervisor%'
    then
      dbms_session.set_context('my_context1','salary_yes_no','DISPLAY_SALARY');
    else
      dbms_session.set_context('my_context1','salary_yes_no','DO_NOT_DISPLAY_SALARY');
		end if;
  end set_my_context1;
end;
/

Package body created.

SQL> grant execute on my_context1_pkg TO app_read;

Grant succeeded.

When adding a policy with DBMS_REDACT.ADD_POLICY one of the most important parameter is expression. Means that the real time masking will be performed only if the expression is TRUE. Here the example I would like to simulate is to hide salary when parameter salary_yes_no of my my_context1 context is set to DO_NOT_DISPLAY_SALARY or is not set (NULL value).

exec dbms_redact.add_policy(object_schema => 'app', object_name => 'employees', column_name => 'salary', -
policy_name => 'display_salary', function_type => dbms_redact.full, -
expression => 'sys_context(''my_context1'',''salary_yes_no'')=''DO_NOT_DISPLAY_SALARY'' or sys_context(''my_context1'',''salary_yes_no'') is null', -
policy_description => 'Hide salary column', -
column_description => 'Column with sensitive salary information');

The redaction policy is enabled by default (enable parameter is set to TRUE by default). The function_type parameter set the redaction masking function. DBMS_REDACT.FULL will simply set salary column to 0 but many other options are available like changing only first digit of a credit card number or a social security number. Please refer the official documentation for a complete description.

In case you want to perform multiple tests you can drop the policy with:

SQL> exec dbms_redact.drop_policy(object_schema => 'app', object_name => 'employees', policy_name => 'display_salary');

PL/SQL procedure successfully completed.

You have few dictionary tables to see what has been implemented:

SQL> set lines 200
SQL> col object_owner for a10
SQL> col object_name for a10
SQL> col policy_description for a20
SQL> col policy_name for a15
SQL> select object_owner,object_name,policy_name, enable,policy_description from redaction_policies;

OBJECT_OWN OBJECT_NAM POLICY_NAME     ENABLE  POLICY_DESCRIPTION
---------- ---------- --------------- ------- --------------------
APP        EMPLOYEES  display_salary  YES     Hide salary column

SQL> col column_name for a10
SQL> select object_owner,object_name,column_name,function_type from redaction_columns;

OBJECT_OWN OBJECT_NAM COLUMN_NAM FUNCTION_TYPE
---------- ---------- ---------- ---------------------------
APP        EMPLOYEES  SALARY     FULL REDACTION

Data redaction testing

For testing I will connect with app_read account and set CLIENT_IDENTIFIER parameter value of USERENV context with DBMS_SESSION.SET_IDENTIFIER procedure. CLIENT_IDENTIFIER parameter value simulate the applicative account that has been used to identify inside your Java/Web application (LDAP or whatever).

If you do not set CLIENT_IDENTIFIER value then salary is not displayed:

SQL> set lines 150
SQL> SELECT SYS_CONTEXT('my_context1','salary_yes_no') FROM dual;

SYS_CONTEXT('MY_CONTEXT1','SALARY_YES_NO')
------------------------------------------------------------------------------------------------------------------------------------------------------


SQL> select * from app.employees;

        ID FIRSTNAME                                          LASTNAME                                               SALARY
---------- -------------------------------------------------- -------------------------------------------------- ----------
         1 Yannick                                            Jaquier                                                     0

If you set CLIENT_IDENTIFIER value to an applicative account that is not allowed to see salaries:

SQL> EXEC DBMS_SESSION.SET_IDENTIFIER('operator01');

PL/SQL procedure successfully completed.

SQL> exec app.my_context1_pkg.set_my_context1;

PL/SQL procedure successfully completed.

SQL> SELECT SYS_CONTEXT('my_context1','salary_yes_no') FROM dual;

SYS_CONTEXT('MY_CONTEXT1','SALARY_YES_NO')
------------------------------------------------------------------------------------------------------------------------------------------------------
DO_NOT_DISPLAY_SALARY

SQL> select * from app.employees;

        ID FIRSTNAME                                          LASTNAME                                               SALARY
---------- -------------------------------------------------- -------------------------------------------------- ----------
         1 Yannick                                            Jaquier                                                     0

If you set CLIENT_IDENTIFIER value to an applicative account that has privilege to see salaries:

SQL> EXEC DBMS_SESSION.SET_IDENTIFIER('supervisor01');

PL/SQL procedure successfully completed.

SQL> exec app.my_context1_pkg.set_my_context1;

PL/SQL procedure successfully completed.

SQL> SELECT SYS_CONTEXT('my_context1','salary_yes_no') FROM dual;

SYS_CONTEXT('MY_CONTEXT1','SALARY_YES_NO')
------------------------------------------------------------------------------------------------------------------------------------------------------
DISPLAY_SALARY

SQL> select * from app.employees;

        ID FIRSTNAME                                          LASTNAME                                               SALARY
---------- -------------------------------------------------- -------------------------------------------------- ----------
         1 Yannick                                            Jaquier                                                 10000

One funny thing you might notice is even APP user, owner of the object, is not able to see the value of salary column. This can be solved granting below system privileges:

SQL> grant exempt redaction policy to app;

Grant succeeded.

You would have chosen DBMS_REDACT.RANDOM as a masking function the salary would be different each time you perform a select onto the employees table.

Even if you are not able to see salary column you can still update it:

SQL> SELECT SYS_CONTEXT('userenv','client_identifier') from dual;

SYS_CONTEXT('USERENV','CLIENT_IDENTIFIER')
------------------------------------------------------------------------------------------------------------------------------------------------------
operator01

SQL> SELECT SYS_CONTEXT('my_context1','salary_yes_no') FROM dual;

SYS_CONTEXT('MY_CONTEXT1','SALARY_YES_NO')
------------------------------------------------------------------------------------------------------------------------------------------------------
DO_NOT_DISPLAY_SALARY

SQL> update app.employees set salary=20000 where id=1;

1 row updated.

SQL> commit;

Commit complete.

SQL> select * from app.employees;

        ID FIRSTNAME                                          LASTNAME                                               SALARY
---------- -------------------------------------------------- -------------------------------------------------- ----------
         1 yannick                                            Jaquier                                                     0

If you control with schema owner app you will see that salary has been well set to 20,000…

References

]]>
http://blog.yannickjaquier.com/oracle/data-redaction-dbms_redact-12cr1.html/feed 0
Resolve ORA-01578 error with and without a backup http://blog.yannickjaquier.com/oracle/resolve-ora-01578-backup-no-backup.html http://blog.yannickjaquier.com/oracle/resolve-ora-01578-backup-no-backup.html#respond Mon, 28 Aug 2017 13:48:03 +0000 http://blog.yannickjaquier.com/?p=3835

Table of contents

Preamble

Data corruption
Data corruption

ORA-01578: ORACLE data block corrupted is not an error message we receive often. Personally I have seen it only two times in my DBA life, thanks to the quality SAN we have with good RAID level. That’s why it is also complicated to practice on this storage error that has corrupted your most important production database…

In case this error happens you have two situations: either you have a backup and the only solution not to loose any data is to restore it or you have no backup (or the restore is not working for any reasons) and at this point data loss is clear but you might want to extract as much data as possible from impacted objects.

Of course if the impacted object is an index then you are lucky as a simple drop/recreate index will solve the issue.

No need to say that before rushing to recover your database you must resolve first, with no delay, the source of the hardware problem and replace the faulty part !

Testing has been done on Oracle Linux Server release 7.2 64bit with Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 – 64bit. The database is in ARCHIVELOG mode as any other production database.

ORA-01578 with a backup

I start by creating a dedicated tablespace for our test:

SQL> create tablespace bmr
     datafile '/u01/app/oracle/oradata/orcl/bmr01.dbf' size 5M autoextend on next 5M maxsize 500M
     logging online permanent
     blocksize 8192
     extent management local autoallocate
     segment space management auto;

Tablespace created.

Then I create a test table and fill it with 500 rows:

SQL> create table test1(val number, descr varchar2(200)) tablespace bmr;

Table created.

SQL> declare
i number;
begin
i:=1;
while (i <= 500)
loop
insert into test1 values (i,TO_CHAR(TO_DATE(i, 'j'), 'jsp'));
i:=i+1;
end loop;
end;
/

PL/SQL procedure successfully completed.

SQL> commit;

Commit complete.

I configure my Recovery Manager (RMAN) with below standard option:

configure controlfile autobackup on;
configure backup optimization on;
configure device type disk parallelism 2 backup type to compressed backupset;
configure controlfile autobackup format for device type disk to '/backup/%F';
configure channel device type disk format '/backup/%U';

I backup the database and archivelog files with:

RMAN> backup database plus archivelog delete input;

Then I add 500 more rows to the table, just to ensure I’m not loosing any figures after the end of recover process. I also gather statistics:

SQL> declare
i number;
begin
i:=501;
while (i <= 1000)
loop
insert into test1 values (i,TO_CHAR(TO_DATE(i, 'j'), 'jsp'));
i:=i+1;
end loop;
end;
/

PL/SQL procedure successfully completed.

SQL> commit;

Commit complete.

SQL> exec dbms_stats.gather_table_stats(USER,'test1');

PL/SQL procedure successfully completed.

SQL> select max(val) from test1;

  MAX(VAL)
----------
      1000

My table has below storage (5 data blocks starting at 131):

SQL> select extent_id,file_id,block_id,bytes,blocks from dba_extents where segment_name='TEST1';

 EXTENT_ID    FILE_ID   BLOCK_ID      BYTES     BLOCKS
---------- ---------- ---------- ---------- ----------
         0          5        128      65536          8
SQL> select num_rows, blocks, empty_blocks from user_tables where table_name='TEST1';

  NUM_ROWS     BLOCKS EMPTY_BLOCKS
---------- ---------- ------------
      1000          5            0

SQL> select distinct dbms_rowid.rowid_relative_fno(rowid) as file_no, dbms_rowid.rowid_block_number(rowid) as block_no from test1 order by 1,2;

   FILE_NO   BLOCK_NO
---------- ----------
         5        131
         5        132
         5        133
         5        134
         5        135

If you wonder why a difference between starting blocks (128 and 131), this is well explained in Overview of Extents and we clearly see the reserved blocks by Oracle in initial allocated extent.

Before any corrupted block test do not forget to purge the buffer cache or Oracle will not read again the block and you might see nothing as the blocks are already in buffer cache:

SQL> alter system flush buffer_cache;

System altered.

Let’s corrupt first data block with dd command. This trick is coming from my Recovery Manager training even if Oracle is doing things a bit differently. It might be cumbersome to be able to corrupt one block with all caching on modern computer (database, filesystem,..). I have finally be obliged to issue two dd commands and one sync:

[root@server1 ~]# dd if=/dev/zero of=/u01/app/oracle/oradata/orcl/bmr01.dbf bs=8192 conv=notrunc seek=131 count=1
1+0 records in
1+0 records out
8192 bytes (8.2 kB) copied, 0.000209918 s, 39.0 MB/s
[root@server1 ~]# sync
[root@server1 ~]# dd if=/dev/zero of=/u01/app/oracle/oradata/orcl/bmr01.dbf bs=8192 conv=notrunc seek=131 count=1
1+0 records in
1+0 records out
8192 bytes (8.2 kB) copied, 0.000209918 s, 39.0 MB/s

Now you should start to encounter error at Oracle level (if not flush the buffer cache):

SQL> alter system flush buffer_cache;

System altered.

SQL> select * from test1 order by val;
select * from test1 order by val
       *
ERROR at line 1:
ORA-01578: ORACLE data block corrupted (file # 5, block # 131)
ORA-01110: data file 5: '/u01/app/oracle/oradata/orcl/bmr01.dbf'

To understand which blocks are impacted you may use:

SQL> select * from v$database_block_corruption;

     FILE#     BLOCK#     BLOCKS CORRUPTION_CHANGE# CORRUPTIO     CON_ID
---------- ---------- ---------- ------------------ --------- ----------
         5        131          1                  0 ALL ZERO           0

Or DB verify utility:

[oracle@server1 ~]$ dbv file=/u01/app/oracle/oradata/orcl/bmr01.dbf

DBVERIFY: Release 12.1.0.2.0 - Production on Tue Jul 19 12:49:19 2016

Copyright (c) 1982, 2014, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE = /u01/app/oracle/oradata/orcl/bmr01.dbf
Page 131 is marked corrupt
Corrupt block relative dba: 0x01400083 (file 5, block 131)
Completely zero block found during dbv:



DBVERIFY - Verification complete

Total Pages Examined         : 640
Total Pages Processed (Data) : 4
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 0
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 130
Total Pages Processed (Seg)  : 0
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 505
Total Pages Marked Corrupt   : 1
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 2631270 (0.2631270)

You can also report corruption with RMAN:

RMAN> run {
 allocate channel channel01 type disk;
 allocate channel channel02 type disk;
 allocate channel channel03 type disk;
 allocate channel channel04 type disk;
 backup check logical validate database;
}
.
.
.
File Status Marked Corrupt Empty Blocks Blocks Examined High SCN
---- ------ -------------- ------------ --------------- ----------
5    FAILED 0              505          640             2631270
  File Name: /u01/app/oracle/oradata/orcl/bmr01.dbf
  Block Type Blocks Failing Blocks Processed
  ---------- -------------- ----------------
  Data       0              4
  Index      0              0
  Other      1              131

validate found one or more corrupt blocks
See trace file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_15528.trc for details
.
.
.

At this stage you can restore the entire datafile or use a cool feature called Block Media Recovery (BMR) and as its name stand for it will recover only the corrupted block. Pre 11gR1 the command is:

blockrecover datafile '...' block x;
blockrecover corruption list;

Starting with 11gR1 the new command is:

recover datafile '...' block x;
recover corruption list;

If you choose to restore the datafile then you have to put it offline so it has an impact for other objects in same datafile. BMR instead does not impact anything:

RMAN> alter database datafile '/u01/app/oracle/oradata/orcl/bmr01.dbf' offline;

Statement processed

RMAN> restore datafile 5;

Starting restore at 19-JUL-16
using channel ORA_DISK_1
using channel ORA_DISK_2

channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00005 to /u01/app/oracle/oradata/orcl/bmr01.dbf
channel ORA_DISK_1: reading from backup piece /backup/18rb3b6l_1_1
channel ORA_DISK_1: piece handle=/backup/18rb3b6l_1_1 tag=TAG20160719T124301
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
Finished restore at 19-JUL-16

RMAN> recover datafile 5;

Starting recover at 19-JUL-16
using channel ORA_DISK_1
using channel ORA_DISK_2

starting media recovery
media recovery complete, elapsed time: 00:00:00

Finished recover at 19-JUL-16

RMAN> alter database datafile '/u01/app/oracle/oradata/orcl/bmr01.dbf' online;

using target database control file instead of recovery catalog
Statement processed

BMR is the method that is chosen by Data Recovery Advisor:

RMAN> list failure;

using target database control file instead of recovery catalog
Database Role: PRIMARY

List of Database Failures
=========================

Failure ID Priority Status    Time Detected Summary
---------- -------- --------- ------------- -------
167        HIGH     OPEN      19-JUL-16     Datafile 5: '/u01/app/oracle/oradata/orcl/bmr01.dbf' contains one or more corrupt blocks

RMAN> advise failure;

Database Role: PRIMARY

List of Database Failures
=========================

Failure ID Priority Status    Time Detected Summary
---------- -------- --------- ------------- -------
167        HIGH     OPEN      19-JUL-16     Datafile 5: '/u01/app/oracle/oradata/orcl/bmr01.dbf' contains one or more corrupt blocks

analyzing automatic repair options; this may take some time
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=263 device type=DISK
allocated channel: ORA_DISK_2
channel ORA_DISK_2: SID=42 device type=DISK
analyzing automatic repair options complete

Mandatory Manual Actions
========================
no manual actions available

Optional Manual Actions
=======================
no manual actions available

Automated Repair Options
========================
Option Repair Description
------ ------------------
1      Perform block media recovery of block 131 in file 5
  Strategy: The repair includes complete media recovery with no data loss
  Repair script: /u01/app/oracle/diag/rdbms/orcl/orcl/hm/reco_2815027252.hm

RMAN> host 'cat /u01/app/oracle/diag/rdbms/orcl/orcl/hm/reco_2815027252.hm';

   # block media recovery
   recover datafile 5 block 131;
host command complete

RMAN> repair failure;

Strategy: The repair includes complete media recovery with no data loss
Repair script: /u01/app/oracle/diag/rdbms/orcl/orcl/hm/reco_2815027252.hm

contents of repair script:
   # block media recovery
   recover datafile 5 block 131;

Do you really want to execute the above repair (enter YES or NO)? y
executing repair script

Starting recover at 19-JUL-16
using channel ORA_DISK_1
using channel ORA_DISK_2

channel ORA_DISK_1: restoring block(s)
channel ORA_DISK_1: specifying block(s) to restore from backup set
restoring blocks of datafile 00005
channel ORA_DISK_1: reading from backup piece /backup/18rb3b6l_1_1
channel ORA_DISK_1: piece handle=/backup/18rb3b6l_1_1 tag=TAG20160719T124301
channel ORA_DISK_1: restored block(s) from backup piece 1
channel ORA_DISK_1: block restore complete, elapsed time: 00:00:01

starting media recovery
media recovery complete, elapsed time: 00:00:01

Finished recover at 19-JUL-16
repair failure complete

And the table is again fully accessible:

SQL> select count(*) from test1;

  COUNT(*)
----------
      1000

ORA-01578 with no backup

Of course this must not happen but you might encounter it after trying to restore a backup and realizing that your backup strategy was not so perfect… So the importance to validate your backup procedure as often as you can. In extreme situation the data center failure might be so huge that even your backups are not accessible because you simply do not offload your backup to a third party supplier. The situation we had was a SAN failure where was located the TSM database, in this situation no restore was even possible…

Worth to mention that you will encounter data loss, the below procedure should be followed only in last option to try to recover figures you can. Below as been executed with SYS as I have not succeeded to grant execute on DBMS_REPAIR to my DBA account (!!).

Start by creating the package required backend tables:

SQL> execute dbms_repair.admin_tables(table_name => 'ORPHAN_KEY_TABLE', table_type => dbms_repair.orphan_table, action => dbms_repair.create_action, tablespace => 'users');

PL/SQL procedure successfully completed.

SQL> execute dbms_repair.admin_tables(table_name => 'REPAIR_TABLE', table_type => dbms_repair.repair_table, action => dbms_repair.create_action, tablespace => 'users');

PL/SQL procedure successfully completed.

Check your object with:

SQL> set serveroutput on
SQL> declare
       corrupt_count binary_integer:=0;
     begin
       dbms_repair.check_object(schema_name => 'YJAQUIER', object_name => 'TEST1', repair_table_name => 'REPAIR_TABLE', corrupt_count => corrupt_count);
       dbms_output.put_line('Corrupted block(s): ' || to_char (corrupt_count));
     end;
     /

Corrupted block(s): 1

PL/SQL procedure successfully completed.

This fill REPAIR_TABLE:

SQL> col repair_description for a30
SQL> set lines 150
SQL> select relative_file_id,block_id,schema_name,object_name,fix_timestamp,repair_description from sys.repair_table;

RELATIVE_FILE_ID   BLOCK_ID SCHEMA_NAME                    OBJECT_NAME                    FIX_TIMES REPAIR_DESCRIPTION
---------------- ---------- ------------------------------ ------------------------------ --------- ------------------------------
               5        131 YJAQUIER                       TEST1                                    mark block software corrupt

Then I ask Oracle not to fetch this block anymore with:

SQL> set serveroutput on
SQL> SQL> exec DBMS_REPAIR.SKIP_CORRUPT_BLOCKS(schema_name => 'YJAQUIER', object_name => 'TEST1');

PL/SQL procedure successfully completed.

I can now read in my test table but I have lost 229 rows (!!):

SQL> select count(*) from test1;

  COUNT(*)
----------
       771

I can recover those rows in another table with something like:

SQL> create table test2 tablespace users as select * from test1;

Table created.

SQL> drop table test1;

Table dropped.

SQL> rename test2 to test1;

Table renamed.

If you check with DB Verify, RMAN or V$DATABASE_BLOCK_CORRUPTION you will still see a corrupted block. Either you wait Oracle to write a new object on it or you can accelerate the process following MOS note 336133.1:

First remove autoextend from your datafile:

SQL> alter database datafile '/u01/app/oracle/oradata/orcl/bmr01.dbf' autoextend off;

Database altered.

Create a test table in table with corrupted block avoiding deferred segment creation:

SQL> show parameter deferred_segment_creation

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
deferred_segment_creation            boolean     TRUE

SQL> create table clean1(n number, c varchar2(4000)) segment creation immediate nologging pctfree 99 tablespace bmr;

Table created.

Control extend size with:

SQL> set lines 150
SQL> select * from dba_free_space where file_id= 5 and 131 between block_id and block_id + blocks -1;

TABLESPACE_NAME                   FILE_ID   BLOCK_ID      BYTES     BLOCKS RELATIVE_FNO
------------------------------ ---------- ---------- ---------- ---------- ------------
BMR                                     5        128      65536          8            5

Allocate as many extent as you can with:

SQL> begin
       for i in 1..1000000 loop
         execute immediate 'alter table clean1 allocate extent (datafile '||'''/u01/app/oracle/oradata/orcl/bmr01.dbf''' ||'size 64k)';
       end loop;
     end ;
     /
begin
*
ERROR at line 1:
ORA-01653: unable to extend table YJAQUIER.CLEAN1 by 128 in tablespace BMR
ORA-06512: at line 3

Control the block has been allocated to your table with:

SQL> select segment_name, segment_type, owner from dba_extents where file_id = 5 and 131 between block_id and block_id + blocks -1;

no rows selected

If not restart the process with a clean2 table and so on…

In my case it succeeded with CLEAN2 table:

SQL> col segment_name for a15
SQL> col owner for a15
SQL> select segment_name, segment_type, owner from dba_extents where file_id = 5 and 131 between block_id and block_id + blocks -1;

SEGMENT_NAME    SEGMENT_TYPE       OWNER
--------------- ------------------ ---------------
CLEAN2          TABLE              YJAQUIER

Fill the table that has your corrupted block in its extents (CLEAN2 in my case):

SQL> begin 
       for i in 1..1000000000 loop 
         insert /*+ append */ into clean2 select i, lpad('reformat',3092, 'r') from dual; 
         commit ; 
       end loop; 
     end;
     /
begin
*
ERROR at line 1:
ORA-01653: unable to extend table YJAQUIER.CLEAN2 by 128 in tablespace BMR
ORA-06512: at line 3

Perform few checkpoint and logfile switch with:

SQL> alter system checkpoint;

System altered.

SQL> alter system switch logfile;

System altered.

DB Verify should not report error anymore:

[oracle@server1 ~]$ dbv file=/u01/app/oracle/oradata/orcl/bmr01.dbf

DBVERIFY: Release 12.1.0.2.0 - Production on Tue Jul 19 15:36:48 2016

Copyright (c) 1982, 2014, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE = /u01/app/oracle/oradata/orcl/bmr01.dbf


DBVERIFY - Verification complete

Total Pages Examined         : 640
Total Pages Processed (Data) : 118
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 0
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 471
Total Pages Processed (Seg)  : 0
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 51
Total Pages Marked Corrupt   : 0
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 2641854 (0.2641854)

To “purge” V$DATABASE_BLOCK_CORRUPTION issue:

RMAN> backup validate datafile 5;

Starting backup at 19-JUL-16
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=262 device type=DISK
allocated channel: ORA_DISK_2
channel ORA_DISK_2: SID=25 device type=DISK
channel ORA_DISK_1: starting compressed full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00005 name=/u01/app/oracle/oradata/orcl/bmr01.dbf
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
List of Datafiles
=================
File Status Marked Corrupt Empty Blocks Blocks Examined High SCN
---- ------ -------------- ------------ --------------- ----------
5    OK     0              51           647             2641854
  File Name: /u01/app/oracle/oradata/orcl/bmr01.dbf
  Block Type Blocks Failing Blocks Processed
  ---------- -------------- ----------------
  Data       0              118
  Index      0              0
  Other      0              471

Finished backup at 19-JUL-16

References

]]>
http://blog.yannickjaquier.com/oracle/resolve-ora-01578-backup-no-backup.html/feed 0
ProxySQL high availability tutorial with MariaDB replication http://blog.yannickjaquier.com/mysql/proxysql-high-availability-replication.html http://blog.yannickjaquier.com/mysql/proxysql-high-availability-replication.html#comments Mon, 24 Jul 2017 10:02:45 +0000 http://blog.yannickjaquier.com/?p=3812

Table of contents

Preamble

In a recent webinar I attended I have seen mention of a Maxscale alternative called ProxySQL. Maxscale is a proven working high level proxy but ProxySQL author (René Cannaò) is quite dithyrambic on his product mainly on performance related benchmarks. So better I give a try to it but I will not make any comment on performance. Aim of the post is a simple ProxySQL implementation in a MariaDB replication architecture.

Testing has been done using three virtual machines running Oracle Enterprise Linux 7.2 64 bits:

  • server2.domain.com (192.168.56.102) is MariaDB 10.1.14 64 bits master server.
  • server3.domain.com (192.168.56.103) is MariaDB 10.1.14 64 bits slave server running two MariaDB slave instances.
  • server4.domain.com (192.168.56.104) is ProxySQL node with MysQL 5.7.13 64 bits for client binary.

The Java test application is running under Eclipse Java EE IDE for Web Developers version Neon Release (4.6.0) with MariaDB connector/J 1.4.6 and Oracle MySQL Connector/J 5.1.39.

The MariaDB replication architecture is made of:

  • server2.domain.com on port 3316 as master instance
  • server3.domain.com on port 3316 as first slave instance
  • server3.domain.com on port 3326 as second slave instance

I’m not re-entering in MariaDB replication implementation as we have already seen it.

ProxySQL release available at the time of this article is 1.2.0j. Last but not least ProxySQL can be used for query caching and query rewrite !!

ProxySQL configuration

To install ProxySQL either you compile it or you download a rpm for your release. I have used the one of Centos 7 (proxysql-1.2.0-1-centos7.x86_64.rpm) that is working fine on OEL 7 available at https://github.com/sysown/proxysql/releases.

[root@server4 tmp]# yum install proxysql-1.2.0-1-centos7.x86_64.rpm
Loaded plugins: ulninfo
Examining proxysql-1.2.0-1-centos7.x86_64.rpm: proxysql-1.2.0-1.x86_64
Marking proxysql-1.2.0-1-centos7.x86_64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package proxysql.x86_64 0:1.2.0-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

===========================================================================================================================================================================================================
 Package                                    Arch                                     Version                                      Repository                                                          Size
===========================================================================================================================================================================================================
Installing:
 proxysql                                   x86_64                                   1.2.0-1                                      /proxysql-1.2.0-1-centos7.x86_64                                    11 M

Transaction Summary
===========================================================================================================================================================================================================
Install  1 Package

Total size: 11 M
Installed size: 11 M
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : proxysql-1.2.0-1.x86_64                                                                                                                                                                 1/1
  Verifying  : proxysql-1.2.0-1.x86_64                                                                                                                                                                 1/1

Installed:
  proxysql.x86_64 0:1.2.0-1

Complete!

Start it:

[root@server4 ~]# service proxysql start
Starting ProxySQL: Main init phase0 completed in 3.9e-05 secs.
Main init global variables completed in 0.000236 secs.
Main daemonize phase1 completed in 3.2e-05 secs.
DONE!

I expected a graphical interface to customize options but ProxySQL is fully command line, not an issue ! ProxySQL information is stored is a MySQL back end database so any modification will be done through SQL language. To connect to ProxySQL back-end database you need a MySQL client. Either you install the one of your Linux distribution via standard repository or you install your own one. I have chosen to use MySQL 5.7.13 installed in /mysql/software/mysql01 but new security rules has made things a little bit harder:

[mysql@server4 ~]$ /mysql/software/mysql01/bin/mysql -u admin -padmin -h 127.0.0.1 -P6032
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2026 (HY000): SSL connection error: socket layer receive error

I initially tried the skip-ssl option that is working but deprecated:

[mysql@server4 ~]$ /mysql/software/mysql01/bin/mysql -u admin -padmin -h 127.0.0.1 -P6032 --skip-ssl
mysql: [Warning] Using a password on the command line interface can be insecure.
WARNING: --ssl is deprecated and will be removed in a future version. Use --ssl-mode instead.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 0
Server version: 5.1.30 (ProxySQL Admin Module)

Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

The working up-to-date option is ssl-mode:

[mysql@server4 ~]$ /mysql/software/mysql01/bin/mysql -u admin -padmin -h 127.0.0.1 -P6032 --ssl-mode=disabled
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.1.30 (ProxySQL Admin Module)

Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

Insert your MariaDB replication topology, load it live and save it to disk with:

mysql> insert into mysql_servers(hostgroup_id, hostname, port) values (0,'192.168.56.102',3316);
Query OK, 1 row affected (0.00 sec)

mysql> insert into mysql_servers(hostgroup_id, hostname, port) values (1,'192.168.56.103',3316);
Query OK, 1 row affected (0.01 sec)

mysql> insert into mysql_servers(hostgroup_id, hostname, port) values (1,'192.168.56.103',3326);
Query OK, 1 row affected (0.00 sec)

mysql> select * from mysql_servers;
+--------------+----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+
| hostgroup_id | hostname       | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms |
+--------------+----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+
| 0            | 192.168.56.102 | 3316 | ONLINE | 1      | 0           | 1000            | 0                   | 0       | 0              |
| 1            | 192.168.56.103 | 3316 | ONLINE | 1      | 0           | 1000            | 0                   | 0       | 0              |
| 1            | 192.168.56.103 | 3326 | ONLINE | 1      | 0           | 1000            | 0                   | 0       | 0              |
+--------------+----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+
3 rows in set (0.01 sec)

mysql> save mysql servers to disk;
Query OK, 0 rows affected (0.00 sec)

mysql> select * from disk.mysql_servers;
+--------------+----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+
| hostgroup_id | hostname       | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms |
+--------------+----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+
| 0            | 192.168.56.102 | 3316 | ONLINE | 1      | 0           | 1000            | 0                   | 0       | 0              |
| 1            | 192.168.56.103 | 3316 | ONLINE | 1      | 0           | 1000            | 0                   | 0       | 0              |
| 1            | 192.168.56.103 | 3326 | ONLINE | 1      | 0           | 1000            | 0                   | 0       | 0              |
+--------------+----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+
3 rows in set (0.00 sec)

mysql> load mysql servers to runtime;
Query OK, 0 rows affected (0.00 sec)

Second step is to create an applicative account on ProxySQL. This account must also be created on the MariaDB replication architecture. On master instance, account creation will be replicated on slaves:

grant all privileges on replicationdb.* to 'test'@'%' identified by 'secure_password';

And do it in ProxySQL repository database:

mysql> insert into mysql_users (username,password) values ('test','secure_password');
Query OK, 1 row affected (0.00 sec)

mysql> save mysql users to disk;
Query OK, 0 rows affected (0.00 sec)

mysql> select * from disk.mysql_users;
+----------+-----------------+--------+---------+-------------------+----------------+---------------+------------------------+--------------+---------+----------+-----------------+
| username | password        | active | use_ssl | default_hostgroup | default_schema | schema_locked | transaction_persistent | fast_forward | backend | frontend | max_connections |
+----------+-----------------+--------+---------+-------------------+----------------+---------------+------------------------+--------------+---------+----------+-----------------+
| test     | secure_password | 1      | 0       | 0                 | NULL           | 0             | 0                      | 0            | 1       | 1        | 10000           |
+----------+-----------------+--------+---------+-------------------+----------------+---------------+------------------------+--------------+---------+----------+-----------------+
1 row in set (0.00 sec)

mysql> load mysql users to runtime;
Query OK, 0 rows affected (0.00 sec)

Humm passwords are stored in clear, it’s a planned feature to encrypt them… Note the default hostgroup set to 0 (master server).

Let’s do a test of a read-only SQL command:

[root@server4 ~]# /mysql/software/mysql01/bin/mysql --user=test --password=secure_password --host=127.0.0.1 --port=6033 -e 'SELECT @@hostname,@@port' --ssl-mode=disabled
mysql: [Warning] Using a password on the command line interface can be insecure.
+--------------------+--------+
| @@hostname         | @@port |
+--------------------+--------+
| server2.domain.com |   3316 |
+--------------------+--------+

If you execute above command multiple times only master server is chosen. Read/write split is not implemented by default and if you want a Maxscale behavior you have to create query routing rules…

I have also activated ProxySQL backend servers monitoring:

mysql> SELECT * FROM global_variables where variable_name like 'mysql-monitor%';
+----------------------------------------+---------------------------------------------------+
| variable_name                          | variable_value                                    |
+----------------------------------------+---------------------------------------------------+
| mysql-monitor_history                  | 600000                                            |
| mysql-monitor_connect_interval         | 120000                                            |
| mysql-monitor_connect_timeout          | 200                                               |
| mysql-monitor_ping_interval            | 60000                                             |
| mysql-monitor_ping_max_failures        | 3                                                 |
| mysql-monitor_ping_timeout             | 100                                               |
| mysql-monitor_read_only_interval       | 1000                                              |
| mysql-monitor_read_only_timeout        | 100                                               |
| mysql-monitor_replication_lag_interval | 10000                                             |
| mysql-monitor_replication_lag_timeout  | 1000                                              |
| mysql-monitor_username                 | monitor                                           |
| mysql-monitor_password                 | monitor                                           |
| mysql-monitor_query_variables          | SELECT * FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES |
| mysql-monitor_query_status             | SELECT * FROM INFORMATION_SCHEMA.GLOBAL_STATUS    |
| mysql-monitor_query_interval           | 60000                                             |
| mysql-monitor_query_timeout            | 100                                               |
| mysql-monitor_timer_cached             | true                                              |
| mysql-monitor_writer_is_also_reader    | true                                              |
+----------------------------------------+---------------------------------------------------+
18 rows in set (0.01 sec)

mysql> update global_variables set variable_value='secure_password' where variable_name='mysql-monitor_password';
Query OK, 1 row affected (0.00 sec)

mysql> load mysql variables to runtime;
Query OK, 0 rows affected (0.00 sec)

mysql> save mysql variables to disk;
Query OK, 64 rows affected (0.02 sec)

I create the ProxySQL monitor account on master server, it will be automatically replicated to slaves:

MariaDB [(none)]> grant replication client on *.* to 'monitor'@'%' identified by 'secure_password';
Query OK, 0 rows affected (0.00 sec)

After a small period you can control it runs successfully with:

mysql> select * from monitor.mysql_server_replication_lag_log order by time_start desc limit 10;
+----------------+------+------------------+--------------+----------+-------+
| hostname       | port | time_start       | success_time | repl_lag | error |
+----------------+------+------------------+--------------+----------+-------+
| 192.168.56.102 | 3316 | 1467715539097299 | 810          | NULL     | NULL  |
| 192.168.56.103 | 3316 | 1467715539097299 | 621          | 0        | NULL  |
| 192.168.56.103 | 3326 | 1467715539097299 | 1022         | 0        | NULL  |
| 192.168.56.102 | 3316 | 1467715529097105 | 780          | NULL     | NULL  |
| 192.168.56.103 | 3316 | 1467715529097105 | 816          | 0        | NULL  |
| 192.168.56.103 | 3326 | 1467715529097105 | 1024         | 0        | NULL  |
| 192.168.56.102 | 3316 | 1467715519096898 | 786          | NULL     | NULL  |
| 192.168.56.103 | 3316 | 1467715519096898 | 655          | 0        | NULL  |
| 192.168.56.103 | 3326 | 1467715519096898 | 1111         | 0        | NULL  |
| 192.168.56.102 | 3316 | 1467715509096195 | 885          | NULL     | NULL  |
+----------------+------+------------------+--------------+----------+-------+
10 rows in set (0.00 sec)

mysql> select * from monitor.mysql_server_ping_log order by time_start desc limit 10;
+----------------+------+------------------+-------------------+------------+
| hostname       | port | time_start       | ping_success_time | ping_error |
+----------------+------+------------------+-------------------+------------+
| 192.168.56.102 | 3316 | 1467715518924380 | 721               | NULL       |
| 192.168.56.103 | 3316 | 1467715518924380 | 525               | NULL       |
| 192.168.56.103 | 3326 | 1467715518924380 | 701               | NULL       |
| 192.168.56.102 | 3316 | 1467715458924137 | 682               | NULL       |
| 192.168.56.103 | 3316 | 1467715458924137 | 517               | NULL       |
| 192.168.56.103 | 3326 | 1467715458924137 | 703               | NULL       |
| 192.168.56.102 | 3316 | 1467715398923945 | 685               | NULL       |
| 192.168.56.103 | 3316 | 1467715398923945 | 531               | NULL       |
| 192.168.56.103 | 3326 | 1467715398923945 | 843               | NULL       |
| 192.168.56.102 | 3316 | 1467715338923725 | 678               | NULL       |
+----------------+------+------------------+-------------------+------------+
10 rows in set (0.00 sec)

mysql> select * from monitor.mysql_server_connect_log order by time_start desc limit 10;
+----------------+------+------------------+----------------------+---------------+
| hostname       | port | time_start       | connect_success_time | connect_error |
+----------------+------+------------------+----------------------+---------------+
| 192.168.56.102 | 3316 | 1467715518941492 | 1524                 | NULL          |
| 192.168.56.103 | 3316 | 1467715518941492 | 2105                 | NULL          |
| 192.168.56.103 | 3326 | 1467715518941492 | 2232                 | NULL          |
| 192.168.56.102 | 3316 | 1467715398941430 | 1734                 | NULL          |
| 192.168.56.103 | 3316 | 1467715398941430 | 2450                 | NULL          |
| 192.168.56.103 | 3326 | 1467715398941430 | 1357                 | NULL          |
| 192.168.56.102 | 3316 | 1467715278941119 | 1437                 | NULL          |
| 192.168.56.103 | 3316 | 1467715278941119 | 1937                 | NULL          |
| 192.168.56.103 | 3326 | 1467715278941119 | 4414                 | NULL          |
| 192.168.56.102 | 3316 | 1467715158941062 | 1806                 | NULL          |
+----------------+------+------------------+----------------------+---------------+
10 rows in set (0.00 sec)

I have not well understood if mysql_replication_hostgroups table must be configured, even without it, it should work:

mysql> insert into mysql_replication_hostgroups values(0,1);
Query OK, 1 row affected (0.01 sec)

mysql> save mysql servers to disk;
Query OK, 0 rows affected (0.01 sec)

mysql> load mysql servers to runtime;
Query OK, 0 rows affected (0.01 sec)

mysql> select * from mysql_replication_hostgroups;
+------------------+------------------+
| writer_hostgroup | reader_hostgroup |
+------------------+------------------+
| 0                | 1                |
+------------------+------------------+
1 row in set (0.00 sec)

mysql> select * from disk.mysql_replication_hostgroups;
+------------------+------------------+
| writer_hostgroup | reader_hostgroup |
+------------------+------------------+
| 0                | 1                |
+------------------+------------------+
1 row in set (0.01 sec)

If you hit strange behavior, like I had, showing duplicated nodes across hostgroup:

mysql> SELECT * FROM mysql_servers;
+--------------+----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+
| hostgroup_id | hostname       | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms |
+--------------+----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+
| 0            | 192.168.56.102 | 3316 | ONLINE | 1      | 0           | 1000            | 10                  | 0       | 10             |
| 0            | 192.168.56.103 | 3326 | ONLINE | 1      | 0           | 1000            | 10                  | 0       | 10             |
| 0            | 192.168.56.103 | 3316 | ONLINE | 1      | 0           | 1000            | 10                  | 0       | 10             |
| 1            | 192.168.56.103 | 3326 | ONLINE | 1      | 0           | 1000            | 10                  | 0       | 10             |
| 1            | 192.168.56.103 | 3316 | ONLINE | 1      | 0           | 1000            | 10                  | 0       | 10             |
+--------------+----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+
5 rows in set (0.00 sec)

mysql> select * from disk.mysql_server;
+--------------+----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+
| hostgroup_id | hostname       | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms |
+--------------+----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+
| 0            | 192.168.56.102 | 3316 | ONLINE | 1      | 0           | 1000            | 10                  | 0       | 10             |
| 1            | 192.168.56.103 | 3316 | ONLINE | 1      | 0           | 1000            | 10                  | 0       | 10             |
| 1            | 192.168.56.103 | 3326 | ONLINE | 1      | 0           | 1000            | 10                  | 0       | 10             |
+--------------+----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+
3 rows in set (0.01 sec)

You have to set your slaves read only with:

MariaDB [(none)]> set global read_only=ON;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> show variables like 'read_only';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| read_only     | ON    |
+---------------+-------+
1 row in set (0.00 sec)

For below error message when using MySQL Connector/J 5.1.39:

Error message: Unknown system variable 'language'

You have to set ProxySQL MySQL version above 5.1.31 with something like (mysql-server_version is just a display for ProxySQL):

mysql> set mysql-server_version='5.7.13';
Query OK, 1 row affected (0.00 sec)

mysql> load mysql variables to runtime;
Query OK, 0 rows affected (0.00 sec)

mysql> save mysql variables to disk;
Query OK, 64 rows affected (0.01 sec)

ProxySQL testing

As stated in ProxySQL website to simulate read/write split you have to create query routing rules, just using what is given in ProxySQL web site:

mysql> insert into mysql_query_rules(active,match_pattern,destination_hostgroup,apply) values (1,'^SELECT.*FOR UPDATE$',0,1);
Query OK, 1 row affected (0.00 sec)

mysql> insert into mysql_query_rules(active,match_pattern,destination_hostgroup,apply) values (1,'^SELECT',1,1);
Query OK, 1 row affected (0.00 sec)

mysql> select rule_id, match_pattern,destination_hostgroup, apply from mysql_query_rules;
+---------+----------------------+-----------------------+-------+
| rule_id | match_pattern        | destination_hostgroup | apply |
+---------+----------------------+-----------------------+-------+
| 1       | ^SELECT.*FOR UPDATE$ | 0                     | 1     |
| 2       | ^SELECT              | 1                     | 1     |
+---------+----------------------+-----------------------+-------+
2 rows in set (0.00 sec)

mysql> save mysql query rules to disk;
Query OK, 0 rows affected (0.00 sec)

mysql> load mysql query rules to runtime;
Query OK, 0 rows affected (0.00 sec)

This example query routing rules listed in official documentation are routing select queries (read-only) to hostgroup 1 (slave servers) and select for update queries (read-write) to master server. Queries not passing any of the rules goes to default hostgroup that is master server. With a classical MySQL client connection it gives:

[root@server4 ~]# /mysql/software/mysql01/bin/mysql --user=test --password=secure_password --host=127.0.0.1 --port=6033 -e 'SELECT @@hostname,@@port' --ssl-mode=disabled
mysql: [Warning] Using a password on the command line interface can be insecure.
+--------------------+--------+
| @@hostname         | @@port |
+--------------------+--------+
| server3.domain.com |   3326 |
+--------------------+--------+
[root@server4 ~]# /mysql/software/mysql01/bin/mysql --user=test --password=secure_password --host=127.0.0.1 --port=6033 -e 'SELECT @@hostname,@@port' --ssl-mode=disabled
mysql: [Warning] Using a password on the command line interface can be insecure.
+--------------------+--------+
| @@hostname         | @@port |
+--------------------+--------+
| server3.domain.com |   3316 |
+--------------------+--------+

To further test it I have used the typical Java application I have used in many other blog posts. Here I initiate a connection and loop to perform a read-write and a read-only query. Obviously I am not disconnecting and reconnecting after each loop mainly because a standard Java application would not do it.

To be honest I have faced lots of issue making this Java application work and round robin onto my slave servers, fortunately I have been helped by René directly at this forum thread https://groups.google.com/forum/#!topic/proxysql/Rzwd55xIQGk. First René explained that due to @ symbol ProxySQL multiplexing is disabled to ensure consistency on users’ variables. So he suggested to change my test query by something like:

SELECT (SELECT VARIABLE_VALUE FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES WHERE VARIABLE_NAME='hostname') `hostname`,(SELECT VARIABLE_VALUE FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES WHERE VARIABLE_NAME='port') `port`;

Then, directed by René, I realized that I stupidly changed the value of mysql-multiplexing to false (default value is true), so changed it back to original value:

mysql> set mysql-multiplexing='true';
Query OK, 1 row affected (0.01 sec)

mysql> load mysql variables to runtime;
Query OK, 0 rows affected (0.01 sec)

mysql> save mysql variables to disk;
Query OK, 64 rows affected (0.01 sec)

Finally below Java code (you can de-comment the Connector/J you wish to use, Java code is compliant with both):

package jdbcdemo7;

import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.util.Properties;
import org.mariadb.jdbc.Driver;
//import com.mysql.jdbc.Driver;

public class jdbcdemo7 {
	public static void main(String[] args) throws Exception {
		Driver driver = new Driver();
		Properties props = new Properties();
		ResultSet rs;
		String variable_value;
		Connection conn = null;
		String JDBC_URL = "jdbc:mysql://address=(protocol=tcp)(host=192.168.56.104)(port=6033)";

		props.put("useSSL", "false");
		props.put("user", "test");
		props.put("password", "secure_password");

		System.out.println("\n------------ MariaDB Connector/J and ProxySQL Testing ------------\n");

		System.out.println("Trying connection...");
		try {
			conn = driver.connect(JDBC_URL, props);
		}
			catch (SQLException e) {
			System.out.println("Connection Failed!");
			System.out.println("Error cause: "+e.getCause());
			System.out.println("Error message: "+e.getMessage());
			return;
		}

		System.out.println("Connection established...");

		for(int i=1; i <= 50; i++) {
			System.out.println("\nQuery "+i+": ");
			// Read write query that can be performed ONLY on master server
			System.out.println("Read Write query...");
			try {
				rs = conn.createStatement().executeQuery("select (select variable_value from information_schema.global_variables where variable_name=\'hostname\') || \' on port \' || (select variable_value from information_schema.global_variables where variable_name=\'port\') variable_value for update");
				while (rs.next()) {
					variable_value = rs.getString("variable_value");
					System.out.println("variable_value : " + variable_value);
				}
			}
			catch (SQLException e) {
				System.out.println("Read/write query has failed...");
			}
	    

				// Read Only statement (that can also be done on master server if all slaves are down)
			System.out.println("Read Only query...");

			try {
				rs = conn.createStatement().executeQuery("select (select variable_value from information_schema.global_variables where variable_name=\'hostname\') || \' on port \' || (select variable_value from information_schema.global_variables where variable_name=\'port\') variable_value");
				while (rs.next()) {
					variable_value = rs.getString("variable_value");
					System.out.println("variable_value : " + variable_value);
				}
			}
			catch (SQLException e) {
				System.out.println("Read only query has failed...");
			}

			Thread.sleep(1000);
		}
		conn.close();
	}
}

To provide under Eclipse below nice result:

proxysql01
proxysql01

Few statistics tables are also available:

mysql> select * from stats_mysql_connection_pool;
+-----------+----------------+----------+--------+----------+----------+--------+---------+---------+-----------------+-----------------+------------+
| hostgroup | srv_host       | srv_port | status | ConnUsed | ConnFree | ConnOK | ConnERR | Queries | Bytes_data_sent | Bytes_data_recv | Latency_ms |
+-----------+----------------+----------+--------+----------+----------+--------+---------+---------+-----------------+-----------------+------------+
| 0         | 192.168.56.102 | 3316     | ONLINE | 0        | 0        | 23     | 0       | 1012    | 73908           | 61508           | 0          |
| 1         | 192.168.56.103 | 3316     | ONLINE | 0        | 0        | 32     | 0       | 307     | 18160           | 10868           | 0          |
| 1         | 192.168.56.103 | 3326     | ONLINE | 0        | 0        | 40     | 0       | 480     | 28544           | 17024           | 0          |
+-----------+----------------+----------+--------+----------+----------+--------+---------+---------+-----------------+-----------------+------------+
3 rows in set (0.00 sec)

mysql> select hostgroup, sum(queries), sum(bytes_data_sent), sum(bytes_data_recv) from stats_mysql_connection_pool group by hostgroup;
+-----------+--------------+----------------------+----------------------+
| hostgroup | SUM(Queries) | SUM(Bytes_data_sent) | SUM(Bytes_data_recv) |
+-----------+--------------+----------------------+----------------------+
| 0         | 667          | 48373                | 26196                |
| 1         | 664          | 40256                | 23712                |
+-----------+--------------+----------------------+----------------------+
2 rows in set (0.00 sec)

For the complete list of statistics tables use:

mysql> show tables from stats;
+--------------------------------+
| tables                         |
+--------------------------------+
| stats_mysql_query_rules        |
| stats_mysql_commands_counters  |
| stats_mysql_processlist        |
| stats_mysql_connection_pool    |
| stats_mysql_query_digest       |
| stats_mysql_query_digest_reset |
| stats_mysql_global             |
+--------------------------------+
7 rows in set (0.00 sec)

ProxySQL high availability testing

One slave dead

If I shutdown one of the slave above Java application is not throwing any error and continue to run transparently on only remaining slave.

mysql> SELECT * FROM stats_mysql_connection_pool;
+-----------+----------------+----------+---------+----------+----------+--------+---------+---------+-----------------+-----------------+------------+
| hostgroup | srv_host       | srv_port | status  | ConnUsed | ConnFree | ConnOK | ConnERR | Queries | Bytes_data_sent | Bytes_data_recv | Latency_ms |
+-----------+----------------+----------+---------+----------+----------+--------+---------+---------+-----------------+-----------------+------------+
| 0         | 192.168.56.102 | 3316     | ONLINE  | 0        | 1        | 1      | 0       | 28      | 6317            | 1192            | 587        |
| 1         | 192.168.56.103 | 3316     | ONLINE  | 0        | 1        | 1      | 0       | 18      | 4086            | 648             | 530        |
| 1         | 192.168.56.103 | 3326     | SHUNNED | 0        | 0        | 1      | 12      | 8       | 1816            | 288             | 901        |
+-----------+----------------+----------+---------+----------+----------+--------+---------+---------+-----------------+-----------------+------------+
3 rows in set (0.00 sec)

Once the dead slave is back to live ProxySQL use it automatically.

Two slaves dead

If I shutdown my two slaves then read only queries are no more possible:

mysql> SELECT * FROM stats_mysql_connection_pool;
+-----------+----------------+----------+---------+----------+----------+--------+---------+---------+-----------------+-----------------+------------+
| hostgroup | srv_host       | srv_port | status  | ConnUsed | ConnFree | ConnOK | ConnERR | Queries | Bytes_data_sent | Bytes_data_recv | Latency_ms |
+-----------+----------------+----------+---------+----------+----------+--------+---------+---------+-----------------+-----------------+------------+
| 0         | 192.168.56.102 | 3316     | ONLINE  | 0        | 1        | 1      | 0       | 92      | 21549           | 3496            | 919        |
| 1         | 192.168.56.103 | 3316     | SHUNNED | 0        | 0        | 1      | 18      | 76      | 17252           | 2700            | 896        |
| 1         | 192.168.56.103 | 3326     | SHUNNED | 0        | 0        | 2      | 84      | 13      | 2951            | 432             | 901        |
+-----------+----------------+----------+---------+----------+----------+--------+---------+---------+-----------------+-----------------+------------+
3 rows in set (0.00 sec)

With few ProxySQL commands it would be possible to correct this and allocate master server in hostgroup 1 to allow it to resolve read only queries.

When at least one of the slaves is back to life application return in normal running state...

Master server dead

Of course read write queries are no more possible when master server is down ! statistics are a bit strange:

mysql> SELECT * FROM stats_mysql_connection_pool;
+-----------+----------------+----------+--------------+----------+----------+--------+---------+---------+-----------------+-----------------+------------+
| hostgroup | srv_host       | srv_port | status       | ConnUsed | ConnFree | ConnOK | ConnERR | Queries | Bytes_data_sent | Bytes_data_recv | Latency_ms |
+-----------+----------------+----------+--------------+----------+----------+--------+---------+---------+-----------------+-----------------+------------+
| 0         | 192.168.56.102 | 3316     | OFFLINE_HARD | 0        | 0        | 1      | 0       | 127     | 29532           | 4940            | 622        |
| 1         | 192.168.56.103 | 3316     | ONLINE       | 0        | 1        | 2      | 131     | 84      | 19068           | 2988            | 759        |
| 1         | 192.168.56.103 | 3326     | ONLINE       | 0        | 1        | 3      | 198     | 22      | 4994            | 756             | 660        |
| 1         | 192.168.56.102 | 3316     | ONLINE       | 0        | 0        | 0      | 0       | 0       | 0               | 0               | 0          |
+-----------+----------------+----------+--------------+----------+----------+--------+---------+---------+-----------------+-----------------+------------+
4 rows in set (0.00 sec)

mysql> SELECT * FROM mysql_servers;
+--------------+----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+
| hostgroup_id | hostname       | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms |
+--------------+----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+
| 1            | 192.168.56.103 | 3316 | ONLINE | 1      | 0           | 1000            | 10                  | 0       | 10             |
| 1            | 192.168.56.103 | 3326 | ONLINE | 1      | 0           | 1000            | 10                  | 0       | 10             |
| 0            | 192.168.56.102 | 3316 | ONLINE | 1      | 0           | 1000            | 10                  | 0       | 10             |
| 1            | 192.168.56.102 | 3316 | ONLINE | 1      | 0           | 1000            | 10                  | 0       | 10             |
+--------------+----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+
4 rows in set (0.00 sec)

In this situation either you promote a slave to new master role and ProxySQL must be tuned, nothing complex here. Or you are able to restart your master server, most probable situation.

When my master server was again alive I have seen that ProxySQL has added it to read only group (hostgroup 1) and now read only queries are also served by it. I have resolved this weird situation with:

mysql> delete from mysql_servers where hostgroup_id=1 and hostname='192.168.56.102';
Query OK, 1 row affected (0.00 sec)

mysql> load mysql servers to runtime;
Query OK, 0 rows affected (0.01 sec)

mysql> select * from mysql_servers;
+--------------+----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+
| hostgroup_id | hostname       | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms |
+--------------+----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+
| 1            | 192.168.56.103 | 3316 | ONLINE | 1      | 0           | 1000            | 10                  | 0       | 10             |
| 1            | 192.168.56.103 | 3326 | ONLINE | 1      | 0           | 1000            | 10                  | 0       | 10             |
| 0            | 192.168.56.102 | 3316 | ONLINE | 1      | 0           | 1000            | 10                  | 0       | 10             |
+--------------+----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+
3 rows in set (0.00 sec)

mysql> select * from stats_mysql_connection_pool;
+-----------+----------------+----------+--------------+----------+----------+--------+---------+---------+-----------------+-----------------+------------+
| hostgroup | srv_host       | srv_port | status       | ConnUsed | ConnFree | ConnOK | ConnERR | Queries | Bytes_data_sent | Bytes_data_recv | Latency_ms |
+-----------+----------------+----------+--------------+----------+----------+--------+---------+---------+-----------------+-----------------+------------+
| 0         | 192.168.56.102 | 3316     | ONLINE       | 0        | 1        | 1      | 0       | 53      | 12614           | 1908            | 682        |
| 1         | 192.168.56.103 | 3316     | ONLINE       | 0        | 1        | 2      | 131     | 110     | 24970           | 3924            | 1477       |
| 1         | 192.168.56.103 | 3326     | ONLINE       | 0        | 1        | 3      | 198     | 47      | 10669           | 1656            | 1101       |
| 1         | 192.168.56.102 | 3316     | OFFLINE_HARD | 0        | 0        | 1      | 4       | 13      | 2951            | 468             | 682        |
+-----------+----------------+----------+--------------+----------+----------+--------+---------+---------+-----------------+-----------------+------------+
4 rows in set (0.01 sec)

If you restart ProxySQL everything is cleared...

ProxySQL additional features

Query caching

Use this trick to reset statistics:

select 1 from stats_mysql_query_digest_reset;

One nice feature is query caching that reminds me the benefit I have seen with Memcached. After a run of my Java test application with 1000 loop I get below obvious statistics:

mysql> select hostgroup,count_star,min_time,max_time,sum_time,digest_text from stats_mysql_query_digest;
+-----------+------------+----------+----------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| hostgroup | count_star | min_time | max_time | sum_time | digest_text                                                                                                                                                                                                           |
+-----------+------------+----------+----------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 1         | 1000       | 1052     | 4287     | 1778360  | select (select variable_value from information_schema.global_variables where variable_name=?) || ?|| (select variable_value from information_schema.global_variables where variable_name=?) variable_value            |
| 0         | 1000       | 1018     | 2209     | 1554333  | select (select variable_value from information_schema.global_variables where variable_name=?) || ?|| (select variable_value from information_schema.global_variables where variable_name=?) variable_value for update |
| 0         | 1          | 536      | 536      | 536      | set session autocommit=?                                                                                                                                                                                              |
| 0         | 1          | 1722     | 1722     | 1722     | SHOW VARIABLES WHERE Variable_name in (?, ?, ?, ?)                                                                                                                                                                    |
+-----------+------------+----------+----------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
4 rows in set (0.00 sec)

Let's cache for 10 seconds the read only query that is executed on hostgroup 1:

mysql> select * from mysql_query_rules;
+---------+--------+----------+------------+--------+-------------+------------+------------+--------+--------------+----------------------+----------------------+---------+-----------------+-----------------------+-----------+-----------+---------+---------+-------+----------------+------------------+-----------+-----+-------+
| rule_id | active | username | schemaname | flagIN | client_addr | proxy_addr | proxy_port | digest | match_digest | match_pattern        | negate_match_pattern | flagOUT | replace_pattern | destination_hostgroup | cache_ttl | reconnect | timeout | retries | delay | mirror_flagOUT | mirror_hostgroup | error_msg | log | apply |
+---------+--------+----------+------------+--------+-------------+------------+------------+--------+--------------+----------------------+----------------------+---------+-----------------+-----------------------+-----------+-----------+---------+---------+-------+----------------+------------------+-----------+-----+-------+
| 1       | 1      | NULL     | NULL       | 0      | NULL        | NULL       | NULL       | NULL   | NULL         | ^SELECT.*FOR UPDATE$ | 0                    | NULL    | NULL            | 0                     | NULL      | NULL      | NULL    | NULL    | NULL  | NULL           | NULL             | NULL      | NULL | 1     |
| 2       | 1      | NULL     | NULL       | 0      | NULL        | NULL       | NULL       | NULL   | NULL         | ^SELECT              | 0                    | NULL    | NULL            | 1                     | NULL      | NULL      | NULL    | NULL    | NULL  | NULL           | NULL             | NULL      | NULL | 1     |
+---------+--------+----------+------------+--------+-------------+------------+------------+--------+--------------+----------------------+----------------------+---------+-----------------+-----------------------+-----------+-----------+---------+---------+-------+----------------+------------------+-----------+-----+-------+
2 rows in set (0.00 sec)

mysql> update mysql_query_rules set cache_ttl=10000 where rule_id=2;
Query OK, 1 row affected (0.00 sec)

mysql> load mysql query rules to runtime;
Query OK, 0 rows affected (0.00 sec)

If I re-execute my Java test application we can see that on the 1000 loop only the first one is executed on my MariaDB backend layer, all the remaining ones are served directly by ProxySQL under the special hostgroup -1:

mysql> select hostgroup,count_star,min_time,max_time,sum_time,digest_text from stats_mysql_query_digest;
+-----------+------------+----------+----------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| hostgroup | count_star | min_time | max_time | sum_time | digest_text                                                                                                                                                                                                           |
+-----------+------------+----------+----------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| -1        | 999        | 0        | 0        | 0        | select (select variable_value from information_schema.global_variables where variable_name=?) || ?|| (select variable_value from information_schema.global_variables where variable_name=?) variable_value            |
| 1         | 1001       | 1052     | 4287     | 1780439  | select (select variable_value from information_schema.global_variables where variable_name=?) || ?|| (select variable_value from information_schema.global_variables where variable_name=?) variable_value            |
| 0         | 2000       | 1018     | 2858     | 3236456  | select (select variable_value from information_schema.global_variables where variable_name=?) || ?|| (select variable_value from information_schema.global_variables where variable_name=?) variable_value for update |
| 0         | 2          | 536      | 546      | 1082     | set session autocommit=?                                                                                                                                                                                              |
| 0         | 2          | 1722     | 1775     | 3497     | SHOW VARIABLES WHERE Variable_name in (?, ?, ?, ?)                                                                                                                                                                    |
+-----------+------------+----------+----------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
5 rows in set (0.01 sec)

Query rewrite

Let's imagine I have my application running a query to get maximum value of a column of below table:

MariaDB [(none)]> show create table replicationdb.test1;
+-------+-------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table                                                                                                                  |
+-------+-------------------------------------------------------------------------------------------------------------------------------+
| test1 | CREATE TABLE `test1` (
  `val` int(11) DEFAULT NULL,
  `descr` varchar(50) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf32 |
+-------+-------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

MariaDB [(none)]> explain extended select max(val) from replicationdb.test1;
+------+-------------+-------+------+---------------+------+---------+------+------+----------+-------+
| id   | select_type | table | type | possible_keys | key  | key_len | ref  | rows | filtered | Extra |
+------+-------------+-------+------+---------------+------+---------+------+------+----------+-------+
|    1 | SIMPLE      | test1 | ALL  | NULL          | NULL | NULL    | NULL |    2 |   100.00 |       |
+------+-------------+-------+------+---------------+------+---------+------+------+----------+-------+
1 row in set, 1 warning (0.00 sec)

As you know this is going to do a full table scan on replicationdb.test1 table to get this value. Now let's imagine this table is static and I want to change the query each time it is fired to ProxySQL by something like (I assume I have three rows in my table):

select 3

Let's add needed rule, as rule 2 will be fired before newly created rule 10 I deactivate it as well. Here the rule you enter is following regexp way of working so almost no limitation in what it is possible to do:

mysql> insert into mysql_query_rules (rule_id,active,match_pattern,replace_pattern,apply) values (10,1,'^select max\(val\) from replicationdb.test1$','select 3',1);
Query OK, 1 row affected (0.00 sec)

mysql> update mysql_query_rules set active=0 where rule_id=2;
Query OK, 1 row affected (0.01 sec)

mysql> select * from mysql_query_rules;
+---------+--------+----------+------------+--------+-------------+------------+------------+--------+--------------+----------------------------------------------+----------------------+---------+-----------------+-----------------------+-----------+-----------+---------+---------+-------+----------------+------------------+-----------+-----+-------+
| rule_id | active | username | schemaname | flagIN | client_addr | proxy_addr | proxy_port | digest | match_digest | match_pattern                                | negate_match_pattern | flagOUT | replace_pattern | destination_hostgroup | cache_ttl | reconnect | timeout | retries | delay | mirror_flagOUT | mirror_hostgroup | error_msg | log | apply |
+---------+--------+----------+------------+--------+-------------+------------+------------+--------+--------------+----------------------------------------------+----------------------+---------+-----------------+-----------------------+-----------+-----------+---------+---------+-------+----------------+------------------+-----------+-----+-------+
| 1       | 1      | NULL     | NULL       | 0      | NULL        | NULL       | NULL       | NULL   | NULL         | ^SELECT.*FOR UPDATE$                         | 0                    | NULL    | NULL            | 0                     | NULL      | NULL      | NULL    | NULL    | NULL  | NULL           | NULL             | NULL      | NULL | 1     |
| 2       | 0      | NULL     | NULL       | 0      | NULL        | NULL       | NULL       | NULL   | NULL         | ^SELECT                                      | 0                    | NULL    | NULL            | 1                     | NULL      | NULL      | NULL    | NULL    | NULL  | NULL           | NULL             | NULL      | NULL | 1     |
| 10      | 1      | NULL     | NULL       | 0      | NULL        | NULL       | NULL       | NULL   | NULL         | ^select max\(val\) from replicationdb.test1$ | 0                    | NULL    | select 3        | NULL                  | NULL      | NULL      | NULL    | NULL    | NULL  | NULL           | NULL             | NULL      | NULL | 1     |
+---------+--------+----------+------------+--------+-------------+------------+------------+--------+--------------+----------------------------------------------+----------------------+---------+-----------------+-----------------------+-----------+-----------+---------+---------+-------+----------------+------------------+-----------+-----+-------+
3 rows in set (0.00 sec)

mysql> load mysql query rules to runtime;
Query OK, 0 rows affected (0.00 sec)

Now I execute the query with:

[mysql@server4 ~]$ /mysql/software/mysql01/bin/mysql --user=test --password=secure_password --host=127.0.0.1 --port=6033 -e "SELECT MAX(val) from replicationdb.test1" --ssl-mode=disabled
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------+
| MAX(val) |
+----------+
|        3 |
+----------+

I get my result, but has it been transformed by ProxySQL ?:

mysql> select hostgroup,count_star,min_time,max_time,sum_time,digest_text from stats_mysql_query_digest;
+-----------+------------+----------+----------+----------+------------------------------------------+
| hostgroup | count_star | min_time | max_time | sum_time | digest_text                              |
+-----------+------------+----------+----------+----------+------------------------------------------+
| 0         | 1          | 3080     | 3080     | 3080     | SELECT MAX(val) from replicationdb.test1 |
| 0         | 1          | 0        | 0        | 0        | select @@version_comment limit ?         |
+-----------+------------+----------+----------+----------+------------------------------------------+
2 rows in set (0.00 sec)

mysql> select * from stats_mysql_query_rules;
+---------+------+
| rule_id | hits |
+---------+------+
| 1       | 0    |
| 10      | 1    |
+---------+------+
2 rows in set (0.01 sec)

The hits column increases so we can say query rewrite is in action !

References

]]>
http://blog.yannickjaquier.com/mysql/proxysql-high-availability-replication.html/feed 4
Orchestrator MySQL replication topology tool tutorial http://blog.yannickjaquier.com/mysql/orchestrator-tutorial.html http://blog.yannickjaquier.com/mysql/orchestrator-tutorial.html#respond Mon, 26 Jun 2017 10:38:28 +0000 http://blog.yannickjaquier.com/?p=3793

Table of contents

Preamble

MySQL replication is simple to put in place but it could be a little bit harder when it comes to monitoring and promoting slave servers to master role. Orchestrator is super easy to put in place and if you do not like command line tools or have not subscribed to MONyog this is the tool to give a try. Orchestrator also forbid you operations that would result in an error.

I have already tested multiple solution like Master High Availability Manager and tools for MySQL (MHA), MySQL Utilities, Multi-Master Replication Manager for MySQL (MMM), Maxscale and MySQL Fabric. Orchestrator simply aim at graphically or command line representing your configuration and perform few re-configuration. It is anyway not a pure monitoring tool as SNMP traps cannot be send to your favorite corporate monitoring tool.

In below post I have used three virtual machines running Oracle Linux Server release 7.2 64 bits. I expected to do my tests with a simple master-slave configuration but realized this was not really interesting so created two slaves of my master instance. So I have:

  • server2.domain.com (192.168.56.102) is MariaDB 10.1.14 64 bits master server.
  • server3.domain.com (192.168.56.103) is MariaDB 10.1.14 64 bits slave server running two MariaDB instances.
  • server4.domain.com (192.168.56.104) is Orchestrator node with MySQL 5.7.13 64 bits for repository database.

For MariaDB instances I have:

  • server2.domain.com on port 3316 as master instance
  • server3.domain.com on port 3316 as first slave instance
  • server3.domain.com on port 3326 as second slave instance

Orchestrator installation

I’m not re-entering in MySQL replication (more specifically here MariaDB) as we have seen it recently in a previous post.

Orchestrator requires a MySQL backend database to store its information. I have to say that I’m a bit disappointed by that as I would have expected information to be saved in a JSON file or something similar. But according to Orchestrator author (Shlomi Noach) this is planned !

In your MySQL 5.7 repository instance create a database and an account to use it:

mysql> CREATE DATABASE IF NOT EXISTS orchestrator;
Query OK, 1 row affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON `orchestrator`.* TO 'orchestrator'@'%' IDENTIFIED BY 'secure_password';
Query OK, 0 rows affected (0.00 sec)

Copy and edit the Orchestrator configuration file:

[root@server4 ~]# cd /usr/local/orchestrator/
[root@server4 orchestrator]# cp orchestrator-sample.conf.json /etc/orchestrator.conf.json
[root@server4 orchestrator]# vi /etc/orchestrator.conf.json

What I have changed inside to match my configuration:

  "MySQLOrchestratorHost": "127.0.0.1",
  "MySQLOrchestratorPort": 3316,
  "MySQLOrchestratorDatabase": "orchestrator",
  "MySQLOrchestratorUser": ""orchestrator",
  "MySQLOrchestratorPassword": "secure_password",

For better display I have also customized:

  "RemoveTextFromHostnameDisplay": ".domain.com",

And also remove the default verbose output, re-activate it if you face any issue:

  "Debug": false,

Log file in case of issue is located at /var/log/orchestrator.log.

On all MariaDB instance create the Orchestrator account with:

GRANT SUPER, PROCESS, REPLICATION SLAVE, RELOAD ON *.* TO 'orchestrator'@'%' IDENTIFIED BY 'secure_password';

To start Orchestrator you need to move first to its directory because if you issue:

[root@server4 ~]# /usr/local/orchestrator/orchestrator --debug http &

You will get below error message:

html/template: "templates/layout" is undefined

Because resources directory is not accessible by daemon at startup so better use:

[root@server4 ~]# cd  /usr/local/orchestrator/
[root@server4 orchestrator]# ./orchestrator --debug http &
2016-06-24 15:05:20 INFO starting orchestrator
2016-06-24 15:05:20 INFO Read config: /etc/orchestrator.conf.json
2016-06-24 15:05:20 DEBUG Initializing orchestrator
2016-06-24 15:05:20 DEBUG Migrating database schema
2016-06-24 15:05:20 FATAL Cannot initiate orchestrator: Error 1067: Invalid default value for 'end_timestamp'

If you encounter this error this is because of the value of SQL mode for your backend repository instance and more specifically the NO_ZERO_DATE option:

mysql> show variables like 'sql_mode';
+---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Variable_name | Value                                                                                                                                                                |
+---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| sql_mode      | PIPES_AS_CONCAT,STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION |
+---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.01 sec)

So change sql_mode to something like:

sql_mode=oracle

Which gives:

mysql> show variables like 'sql_mode';
+---------------+----------------------------------------------------------------------------------------------------------------------+
| Variable_name | Value                                                                                                                |
+---------------+----------------------------------------------------------------------------------------------------------------------+
| sql_mode      | PIPES_AS_CONCAT,ANSI_QUOTES,IGNORE_SPACE,ORACLE,NO_KEY_OPTIONS,NO_TABLE_OPTIONS,NO_FIELD_OPTIONS,NO_AUTO_CREATE_USER |
+---------------+----------------------------------------------------------------------------------------------------------------------+
1 row in set (0.04 sec)

When started access your Orchestrator node on default port 3000 and in Discover menu start to add the node of your topology. At then end you should get something like:

orchestrator01
orchestrator01

If you click on gear icon along server name you have access to few customization, for master server:

orchestrator02
orchestrator02

For slave server (read only option is directly accessible):

orchestrator03
orchestrator03

You can also use commend line to display it:

[root@server4 ~]# /usr/local/orchestrator/orchestrator -c topology -i server2:3316 cli
server2.domain.com:3316   [0s,ok,10.1.14-MariaDB,rw,ROW]
+ server3.domain.com:3316 [0s,ok,10.1.14-MariaDB,rw,ROW,GTID]
+ server3.domain.com:3326 [0s,ok,10.1.14-MariaDB,rw,ROW,GTID]

Orchestrator testing

I have faced issues when trying to drag and drop my slaves, for example if I try to make server3.domain.com:3326 a slave of server3.domain.com:3316 then it is not possible:

orchestrator04
orchestrator04

If my master server fails Orchestrator report it but I have no option to restart it. To do this it would require an agent or a system connection to remote, Orchestrator agent is available but it would be a much complex implementation. Here, same, if I try to make make server3.domain.com:3326 a slave of server3.domain.com:3316 then it is not possible:

orchestrator05
orchestrator05

As no error message with graphical interface I have tried command line and now error message is clear:

[root@server4 ~]# /usr/local/orchestrator/orchestrator -c relocate -i server3.domain.com:3326 -d server3.domain.com:3316
2016-06-29 15:45:51 DEBUG Initializing orchestrator
2016-06-29 15:45:51 ERROR server3.domain.com:3326 cannot replicate from server3.domain.com:3316. Reason: instance does not have log_slave_updates enabled: server3.domain.com:3316
2016-06-29 15:45:51 FATAL 2016-06-29 15:45:51 ERROR server3.domain.com:3326 cannot replicate from server3.domain.com:3316. Reason: instance does not have log_slave_updates enabled: server3.domain.com:3316

So I activated log_slave_updates parameter in my.cnf file of all my instances (in case my master becomes a slave) as the parameter is not dynamic:

MariaDB [(none)]> show variables like '%log_slave_updates%';
+-------------------+-------+
| Variable_name     | Value |
+-------------------+-------+
| log_slave_updates | OFF   |
+-------------------+-------+
1 row in set (0.00 sec)

MariaDB [(none)]> set global log_slave_updates=on;
ERROR 1238 (HY000): Variable 'log_slave_updates' is a read only variable

Once activated we see a difference in printing with the stripes along the servers either graphical or command line:

[root@server4 ~]# /usr/local/orchestrator/orchestrator -c topology -i server2:3316 cli
server2.domain.com:3316   [0s,ok,10.1.14-MariaDB,rw,ROW,>>]
+ server3.domain.com:3316 [0s,ok,10.1.14-MariaDB,rw,ROW,>>,GTID]
+ server3.domain.com:3326 [0s,ok,10.1.14-MariaDB,rw,ROW,>>,GTID]

If my master fail (or in case you want to cascade the slaves) I make server3.domain.com:3316 my new master and stop replication on it (you would do the same in case you have only one slave). I set server3.domain.com:3326 a slave of server3.domain.com:3116:

orchestrator06
orchestrator06
orchestrator07
orchestrator07

But the graphical interface does not allow me to put back server2.domain.com:3316 a slave and even command line is asking me to do it manually:

[root@server4 ~]# /usr/local/orchestrator/orchestrator -c relocate -i server2.domain.com:3316 -d server3.domain.com:3316
2016-06-30 15:12:36 ERROR Relocating server2.domain.com:3316 below server3.domain.com:3316 turns to be too complex; please do it manually
2016-06-30 15:12:36 FATAL 2016-06-30 15:12:36 ERROR Relocating server2.domain.com:3316 below server3.domain.com:3316 turns to be too complex; please do it manually

So you end up with this configuration made of two clusters:

orchestrator08
orchestrator08

And the main one is made of two servers:

orchestrator09
orchestrator09

Obviously nothing cannot be done command line to recover original situation but for this Orchestrator is of no help…

References

]]>
http://blog.yannickjaquier.com/mysql/orchestrator-tutorial.html/feed 0