IT World https://blog.yannickjaquier.com RDBMS, Unix and many more... Tue, 25 Jan 2022 11:10:53 +0000 en-US hourly 1 https://wordpress.org/?v=5.8.3 Database Resident Connection Pooling (DRCP) hands-on https://blog.yannickjaquier.com/oracle/database-resident-connection-pooling-drcp-hands-on.html https://blog.yannickjaquier.com/oracle/database-resident-connection-pooling-drcp-hands-on.html#respond Fri, 28 Jan 2022 08:10:07 +0000 https://blog.yannickjaquier.com/?p=5319 Preamble Database Resident Connection Pooling (DRCP) is not more than a connection pool inside the database. Connection pooling is usually a feature provided by middle tier processes like I have already done in a pure Java program or in one of the well know servlet containers / application servers like Tomcat/JBoss or else. So why […]

The post Database Resident Connection Pooling (DRCP) hands-on appeared first on IT World.

]]>

Table of contents

Preamble

Database Resident Connection Pooling (DRCP) is not more than a connection pool inside the database. Connection pooling is usually a feature provided by middle tier processes like I have already done in a pure Java program or in one of the well know servlet containers / application servers like Tomcat/JBoss or else.

So why implementing this connection pooling directly inside the database ? First to be able to use for application where it is not available, typical example provided by Oracle is web application where you often need small and quick database access to retrieve information. Ultimately we can also imagine clubbing multiple middle tier pools inside the same DRCP pool.

The clear added value is a decrease in database memory footprint as well as a decrease in CPU consumption. Each client process is using CPU ! One also very interesting added value is to keep the session open and your web application will not have the burden of creating a session each time it needs to connect to the database.

Each connection to the pool is handle by a Connection Broker that then delegate the client process to the chosen server process. Once the client disconnect or after a timeout the server process is released to the pool:

drcp01
drcp01

In 19.10 they have added two new initialization parameters to control number of processes to handle session authentication (different from the number of minimum and maximum sessions inside the pool):

SQL> show parameter min_auth_servers

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
min_auth_servers                     integer     1
SQL> show parameter max_auth_servers

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
max_auth_servers                     integer     25

In 19.11 they have added DRCP_DEDICATED_OPT parameter to have the pool behave like a dedicated server (so as a one-to-one client/server) in case you are below the maximum number of pool connection. This is again a trade between performance and resources consumption (the V$AUTHPOOL_STATS view is NOT existing, probably a bug):

SQL> show parameter drcp_dedicated_opt

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
drcp_dedicated_opt                   string      YES

My testing has been done on a 19.12 Oracle database running on a bare metal server with 12 cores and 64GB of memory.

Database Resident Connection Pooling creation

I have started to create my pool directly in the pluggable database I’m using but the message is clear, you must create the pool in the root container:

SQL> exec dbms_connection_pool.start_pool(pool_name => 'pool01');
BEGIN dbms_connection_pool.start_pool(pool_name => 'pool01'); END;

*
ERROR at line 1:
ORA-56515: DRCP: Operation not allowed from a Pluggable Database
ORA-06512: at "SYS.DBMS_CONNECTION_POOL", line 4
ORA-06512: at line 1

You also cannot (yet) change the pool name and only the default name (SYS_DEFAULT_CONNECTION_POOL) is accepted. As there is a parameter pool_name I have no doubt that one day we will be able to create multiple pools. So currently all you pooled database connection will go in same pool, cannot split them and create a bigger pool for a more important application:

SQL> exec dbms_connection_pool.start_pool(pool_name => 'pool01');
BEGIN dbms_connection_pool.start_pool(pool_name => 'pool01'); END;

*
ERROR at line 1:
ORA-56501: DRCP: Pool startup failed
ORA-56500: DRCP: Pool not found
ORA-06512: at "SYS.DBMS_CONNECTION_POOL", line 4
ORA-06512: at line 1

Before (or after) starting the pool you can modify its parameters with either DBMS_CONNECTION_POOL.ALTER_PARAM or DBMS_CONNECTION_POOL.CONFIGURE_POOL:

SQL> exec dbms_connection_pool.configure_pool(pool_name => 'SYS_DEFAULT_CONNECTION_POOL', minsize => 1, maxsize => 10);

PL/SQL procedure successfully completed.

SQL> exec dbms_connection_pool.alter_param(pool_name => 'SYS_DEFAULT_CONNECTION_POOL', param_name => 'incrsize', param_value => 1);

PL/SQL procedure successfully completed.

Thne start the pool with:

SQL> exec dbms_connection_pool.start_pool(pool_name => 'SYS_DEFAULT_CONNECTION_POOL');

PL/SQL procedure successfully completed.
SQL> set lines 200
SQL> col connection_pool for a30
SQL> select connection_pool, status, minsize, maxsize, incrsize from dba_cpool_info;

CONNECTION_POOL                STATUS              MINSIZE    MAXSIZE   INCRSIZE
------------------------------ ---------------- ---------- ---------- ----------
SYS_DEFAULT_CONNECTION_POOL    ACTIVE                    1         10          1

At Linux level it will create background processes, called connection broker to handle client request and free server processes:

[oracle@server01 ~]$ ps -ef | grep ora_l00 | grep -v grep
oracle   10516     1  0 15:24 ?        00:00:00 ora_l000_orcl
oracle   10522     1  0 15:24 ?        00:00:00 ora_l001_orcl

Database Resident Connection Pooling testing

Even if the pool has been created in the root container you can obviously use it to connect to your pluggable database. You can either use easy connect string or define an TNS string in your tnsnames.ora:

pdb1_pool =
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)
                      (HOST=server01.domain.com)
                      (PORT=1531))
                      (CONNECT_DATA=(SERVICE_NAME=pdb1)
                      (SERVER=POOLED)))

To test it I have used the Oracle client I have on my Windows desktop as well as my own account (YJAQUIER) and a test account (TEST01):

PS C:\Users\yjaquier> sqlplus yjaquier/secure_password@server01.domain.com:1531/pdb1:pooled

SQL*Plus: Release 19.0.0.0.0 - Production on Wed Oct 6 15:58:54 2021
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.

Last Successful login time: Wed Oct 06 2021 15:37:45 +02:00

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.12.0.0.0

SQL>

Or

PS C:\Users\yjaquier> tnsping pdb1_pool

TNS Ping Utility for 64-bit Windows: Version 19.0.0.0.0 - Production on 12-OCT-2021 15:06:55

Copyright (c) 1997, 2019, Oracle.  All rights reserved.

Used parameter files:
C:\app\client\product\19.0.0\client_1\network\admin\sqlnet.ora


Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp) (HOST=server01.domain.com) (PORT=1531)) (CONNECT_DATA=(SERVICE_NAME=pdb1) (SERVER=POOLED)))
OK (110 msec)
PS C:\Users\yjaquier> sqlplus test01/test01@pdb1_pool

SQL*Plus: Release 19.0.0.0.0 - Production on Tue Oct 12 15:06:59 2021
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.

Last Successful login time: Tue Oct 12 2021 14:57:58 +02:00

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.12.0.0.0

SQL>

Finally you have few views to follow connections and their respective pool usage (feel free to describe them to see all available columns):

SQL> set lines 200
SQL> col username for a15
SQL> col cclass_name for a15
SQL> select username, cclass_name, process_id, program, machine from v$cpool_conn_info;

USERNAME        CCLASS_NAME     PROCESS_ID               PROGRAM                                          MACHINE
--------------- --------------- ------------------------ ------------------------------------------------ ----------------------------------------------------------------
YJAQUIER        YJAQUIER.SHARED 21904:9308               sqlplus.exe                                      COMPANY\DESKTOP007
TEST01          TEST01.SHARED   9860:2396                sqlplus.exe                                      COMPANY\DESKTOP007

SQL> col pool_name for a30
SQL> select pool_name, num_open_servers, num_busy_servers, num_requests from v$cpool_stats;

POOL_NAME                      NUM_OPEN_SERVERS NUM_BUSY_SERVERS NUM_REQUESTS
------------------------------ ---------------- ---------------- ------------
SYS_DEFAULT_CONNECTION_POOL                   3                0            7

SQL> select * from v$cpool_cc_info;

POOL_NAME                      CCLASS_NAME         CON_ID
------------------------------ --------------- ----------
SYS_DEFAULT_CONNECTION_POOL    YJAQUIER.SHARED          0
SYS_DEFAULT_CONNECTION_POOL    TEST01.SHARED            0

SQL> select cclass_name, num_requests, num_hits, num_misses from v$cpool_cc_stats;

CCLASS_NAME     NUM_REQUESTS   NUM_HITS NUM_MISSES
--------------- ------------ ---------- ----------
YJAQUIER.SHARED            5          0          5
TEST01.SHARED              4          0          4

Simply stop the pool with:

SQL> exec dbms_connection_pool.stop_pool(pool_name => 'SYS_DEFAULT_CONNECTION_POOL');

PL/SQL procedure successfully completed.

References

The post Database Resident Connection Pooling (DRCP) hands-on appeared first on IT World.

]]>
https://blog.yannickjaquier.com/oracle/database-resident-connection-pooling-drcp-hands-on.html/feed 0
Gradual Database Password Rollover hands-on https://blog.yannickjaquier.com/oracle/gradual-database-password-rollover-hands-on.html https://blog.yannickjaquier.com/oracle/gradual-database-password-rollover-hands-on.html#respond Tue, 28 Dec 2021 10:14:33 +0000 https://blog.yannickjaquier.com/?p=5273 Preamble One of my Oracle contact (thanks Marco !) sent me an interesting feature for us: Gradual Database Password Rollover. This feature simply allow you to temporarily allow two different passwords for an account. The period where the two passwords are usable is self define. This feature is really interesting as you do not need […]

The post Gradual Database Password Rollover hands-on appeared first on IT World.

]]>

Table of contents

Preamble

One of my Oracle contact (thanks Marco !) sent me an interesting feature for us: Gradual Database Password Rollover. This feature simply allow you to temporarily allow two different passwords for an account. The period where the two passwords are usable is self define.

This feature is really interesting as you do not need to have your DBA and your application manager to work in concert to change in database and in application at the same time. The DBA can make the change in advance and then the application manager change when (s)he is ready to do it. We can even imagine the DBA to change the password over the week and take the opportunity of an applicative patch over the week-end to restart the application with the new password.

While digging a bit on this feature I have (re-)discovered in the new features guide that Oracle keep a list of the Release Update (RU) features that are (mainly) backported from 21c to 19c:

gradual_database_password_rollover01
gradual_database_password_rollover01

This document is worth consulting after every new RU as now Oracle does not wait a major release to give access to new features. By the way looking at the list I have seen plenty of cool other stuff to investigate…

My testing has been done a 19.12 Oracle database (minimum RU to test this feature) running on a Red Hat Enterprise Linux Server release 7.9 (Maipo) bare metal server (12 cores and 64GB of memory).

Gradual Database Password Rollover testing

The feature comes with a new parameter in database profiles called PASSWORD_ROLLOVER_TIME. The minimum value you can specify for PASSWORD_ROLLOVER_TIME is one hour and it obviously sets the period where the two passwords will be usable. The granularity is 1 second and you supply the value in fraction of days i.e. 1/24 for one hour, 1/1440 for one minute and 1/86400 for one second.

To test it I create a dummy profile:

SQL> create profile profile01
     limit password_rollover_time 1/24;

Profile created.
SQL> set lines 200
SQL> col profile for a15
SQL> col limit for a20
SQL> select *
     from dba_profiles
     where profile='PROFILE01'
     and resource_name='PASSWORD_ROLLOVER_TIME';

PROFILE         RESOURCE_NAME                    RESOURCE LIMIT                COM INH IMP
--------------- -------------------------------- -------- -------------------- --- --- ---
PROFILE01       PASSWORD_ROLLOVER_TIME           PASSWORD 3600                 NO  NO  NO

I have done the testing on my own personal account (yjaquier). I change the profile of my account to use the newly created one:

SQL> alter user yjaquier profile profile01;

User altered.

And then I change the password of my account:

SQL> alter user yjaquier identified by "new_secure_password";

User altered.

For a period of one hour I can use the old and the new password:

SQL> connect yjaquier/secure_password@pdb1
Connected.
SQL> connect yjaquier/new_secure_password@pdb1
Connected.

The account status has taken a newly created value:

SQL> select account_status from dba_users where username='YJAQUIER';

ACCOUNT_STATUS
--------------------------------
OPEN & IN ROLLOVER

And obviously after 1 hour the old password can no longer be used:

SQL> connect yjaquier/new_secure_password@pdb1
Connected.
SQL> connect yjaquier/secure_password@pdb1
ERROR:
ORA-01017: invalid username/password; logon denied


Warning: You are no longer connected to ORACLE.

And the account status has come back to its original state:

SQL> select account_status from dba_users where username='YJAQUIER';

ACCOUNT_STATUS
--------------------------------
OPEN

You can also reduce the password rollover period with a new ALTER USER option. If you are sure that everything now use the new password you can end up immediately the rollover period with:

SQL> select account_status from dba_users where username='YJAQUIER';

ACCOUNT_STATUS
--------------------------------
OPEN & IN ROLLOVER

SQL> alter user yjaquier expire password rollover period;

User altered.

SQL> select account_status from dba_users where username='YJAQUIER';

ACCOUNT_STATUS
--------------------------------
OPEN

To drop the profile use:

SQL> drop profile profile01;

Profile dropped.

The post Gradual Database Password Rollover hands-on appeared first on IT World.

]]>
https://blog.yannickjaquier.com/oracle/gradual-database-password-rollover-hands-on.html/feed 0
Secure Oracle application design and the ANY privileges https://blog.yannickjaquier.com/oracle/secure-oracle-application-design-and-the-any-privileges.html https://blog.yannickjaquier.com/oracle/secure-oracle-application-design-and-the-any-privileges.html#respond Sun, 28 Nov 2021 09:29:45 +0000 https://blog.yannickjaquier.com/?p=5246 Preamble The non existing privilege to grant someone else the right to create objects in your schema is a recurring debate where I work and most probably in many other places. The recurrent question is “can you grant me the right to create a table in another schema ?” and as you know the only […]

The post Secure Oracle application design and the ANY privileges appeared first on IT World.

]]>

Table of contents

Preamble

database-security
database-security

The non existing privilege to grant someone else the right to create objects in your schema is a recurring debate where I work and most probably in many other places. The recurrent question is “can you grant me the right to create a table in another schema ?” and as you know the only possible solution is to grant the high privilege CREATE ANY TABLE.

With this CREATE ANY TABLE high privilege comes a bunch of other “required” privileges like DROP ANY TABLE, ALTER ANY TABLE and so on. By laziness, and to be fair, also because sometimes it is not that easy to achieve comes the SELECT ANY TABLE, UPDATE ANY TABLE, INSERT ANY TABLE and DELETE ANY TABLE privileges. For this last bunch of privileges you can still argue that the owner of the object(s) can grant what’s required and close the thread. But for the CREATE ANY TABLE there is no trivial solution…

One excellent reference when designing your application with an Oracle database as a backend is the website of Pete Finnigan and in particular this presentation:

Design For Your Database Applications – Least privilege, data and ownership

I tend to mostly agree on all (hum) but for some parts I would simplify and act differently, let me clarify…

The not so bad solution to avoid CREATE ANY TABLE

The least privilege approach is obviously a must and the recent privilege analysis feature that has been made free can be of great help.

The application must also NOT connect with the owner of the database objects and you must have an applicative dedicated account with only minimum required privileges (read only or whatever). The password of this account must not be shared with end users and ideally you change it regularly with an automatic solution (CyberArk Application Access Manager (AAM)). The password of this account must not be found in configuration file of your applicative servers and you should use a password vault to sore it (Azure Key Vault for example).

Inside the database you must also split the different parts of the application or the many applications is their own dedicated database account owner. If all your applications reside is same database account owner then it is impossible to segregate data access between batches and scripts that are using the only database account owner to connect to database (not so many hopefully).

I also like to set this database account owner an Operating System authenticated account (in live, not necessarily on QA/Dev/T&I) to avoid remote connection through SQL*Net to it. In other word the account authentication is backported to the operating system and not to the database. Said differently you need to have an OS account with exact same name (os_authent_prefix) as the database and authenticate to it before being able to connect to the database. Not being able to use, for example, SQL Developer or Toad to access those accounts might be seen as a limitation but this is more for me an increased security and it avoid mistakes patching live while thinking being on a development platform (true story !):

Example of an OS authenticated account:

SQL> create user app_owner identified externally;

User created.

Remark:
For increased security there is also an option to add a certificate to those OS authenticated account. You also have the option to create account identified by your enterprise LDAP directory (IDENTIFIED GLOBALLY) but this is a bit more complex to manage and requires more infrastructure.

Then remains the usual case where you need to patch your production data model and so where you need to connect with the schema owner. This access might be required by internal people or by a third party to whom you bought the software.

I rate not a big deal to share this password for short period under control and to immediately change it once the job has been done ! If the schema owner is an OS authenticated account you might already have a strong access control (CyberArk, for example, with/without sudo in parallel) to the operating system of your servers so again not a big deal.

Another solution is the usage of Oracle proxy users but I rate this an extra complexity and in any case at the end you have an additional account able to connect “as if” your database schema owner so for me a new security hole to handle…

Oracle proxy user

In case you decide to go for Oracle proxy users a small test case to demonstrate it. I create my database schema owner with. The SELECT_CATALOG_ROLE role is just an example for the proxy user:

SQL> create user app_owner identified by "password";

User created.

SQL> grant connect, resource, select_catalog_role to app_owner;

Grant succeeded.

SQL> alter user app_owner quota unlimited on users;

User altered.

Then the third party/developer account. Note this account has no privilege, even not the one to connect (CREATE SESSION). This account will be able to proxy my database schema owner but only with CONNECT and RESOURCE role, not with SELECT_CATALOG_ROLE:

SQL> create user supplier_owner identified by "password";

User created.

SQL> alter user app_owner grant connect through supplier_owner with role connect, resource;

User altered.

SQL> set lines 200
SQL> col proxy for a20
SQL> col client for a20
SQL> select * from proxy_users;

PROXY                CLIENT               AUT FLAGS
-------------------- -------------------- --- -----------------------------------
SUPPLIER_OWNER       APP_OWNER            NO  PROXY MAY ACTIVATE ROLE

I create a test table and confirm I can select dictionary views:

SQL> connect app_owner/password@pdb1

SQL> create table test01(val number, descr varchar2(20)) tablespace users;

Table created.

SQL> insert into test01 values(1, 'One');

1 row created.

SQL> commit;

Commit complete.

SQL> select count(*) from v$session;

  COUNT(*)
----------
        95

SQL> select table_name from user_tables;

TABLE_NAME
--------------------------------------------------------------------------------
TEST01

Now I connect with supplier_owner and proxy app_owner account. I confirm I have no access to dictionary view and I create a new table:

SQL> connect supplier_owner[app_owner]/password@pdb1
Connected.
SQL> show user
USER is "APP_OWNER"
SQL> select * from test01;

       VAL DESCR
---------- --------------------
         1 One

SQL> select count(*) from v$session;
select count(*) from v$session
                     *
ERROR at line 1:
ORA-00942: table or view does not exist

SQL> create table test02(val number, descr varchar2(20));

Table created.

As expected the second test table is owned by app_owner:

SQL> connect app_owner/password@pdb1

SQL> select table_name from user_tables;

TABLE_NAME
--------------------------------------------------------------------------------
TEST01
TEST02

Conclusion

Overall for a production database, and moreover for one containing sensitive figures the ANY privileges should not be used for any account. The only exception we have done on this is for an internal application that people use to patch figures on live database where they have no direct access. Yes time to time business people need to correct mistakes on production figures, but this patching application has an Enterprise Directory authentication and we log all what is done.

But for this patching application we implemented Secure external password store (SEPS) and the password of those powerful accounts is ultra secure (more than 15 characters, uppercase, lowercase, number, ponctuations symbols) and not known by anyone (the DBA that set it has immediately forgotten it).

References

The post Secure Oracle application design and the ANY privileges appeared first on IT World.

]]>
https://blog.yannickjaquier.com/oracle/secure-oracle-application-design-and-the-any-privileges.html/feed 0
PHP OCI8 and PDO_OCI extensions installation to access Oracle https://blog.yannickjaquier.com/linux/php-oci8-and-pdo_oci-extensions-installation-to-access-oracle.html https://blog.yannickjaquier.com/linux/php-oci8-and-pdo_oci-extensions-installation-to-access-oracle.html#respond Thu, 28 Oct 2021 08:45:17 +0000 https://blog.yannickjaquier.com/?p=5224 Preamble We have recently upgraded our internal DBA web site where we display publicly internal information and where we have few PHP pages connecting to an internal MySQL repository as well as few performance pages where we access Oracle databases. I know with recent web evolution (Vue, React, Node.js, …) it might sound a little […]

The post PHP OCI8 and PDO_OCI extensions installation to access Oracle appeared first on IT World.

]]>

Table of contents

Preamble

We have recently upgraded our internal DBA web site where we display publicly internal information and where we have few PHP pages connecting to an internal MySQL repository as well as few performance pages where we access Oracle databases. I know with recent web evolution (Vue, React, Node.js, …) it might sound a little bit weird to have PHP pages but to be honest to create what we know call Rest API I have not found anything better than PHP pages. The language is super rich and in any case to access a database backend your script must run on the server side and not on the client. Of course I could have a Node.js running on my web server but again this is so much easier to do this in PHP that I think I will continue to do it for a while…

In the past, to access MySQL, I was using the original MySQL API with procedures like mysql_connect, mysql_select_db and mysql_fetch_array. This extension has been made deprecated as of PHP 5.5.0, and has been removed as of PHP 7.0.0. To prepare it I have asked myself what to use between mysqli or PDO_MySQL. At that time I have decided to use PDO for the common way of working between different flavor of databases (MySQL and Oracle in my case).

When we have setup the new server I expected to configure OCI8 and PDO (PHP Data Objects) by just downloading a package on the official RedHat depot but as usual it did not go as expected and I have spent quite a few hours to sort it out…

Even if I personally like a lot the PDO approach for having a common langage regardless of the backend database Oracle is recommending to use OCI8 instead for its superior feature:

For the moment Oracle still recommends using PHP OCI8 in preference to PDO_OCI because OCI8 has superior features and can also take advantage of various Oracle client library connection and caching functionality.

I tend to say that if your are not chasing performance or top of the edge feature I would recommend PDO, particularly if like me you access multiple database flavors…

My server is a virtual machine running Red Hat Enterprise Linux Server release 7.9 (Maipo) and as the time of writing this blog post using Apache 2.4.37 and PHP Version 7.2.24. As a strong pre-requisite for all extensions (OCI or PDO_OCI) you obviously must have an Oracle client on your web server. This can be the normal, thick, client or the thin instant client. This installation has been done by a teammate and he has chosen the normal client 12.2.0.1.0 (obsolete already but whatever).

OCI8 from pecl

One option to install OCI8 modules is to go with pecl tool. To get pecl do:

[root@server ~]# dnf install php-pear

Download oci8 for PHP 7:

[root@server tmp]# php -v
PHP 7.2.24 (cli) (built: Oct 22 2019 08:28:36) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies
[root@server tmp]# wget https://pecl.php.net/get/oci8-2.2.0.tgz
[root@server tmp]# pecl install oci8-2.2.0.tgz
11 source files, building
running: phpize
Can't find PHP headers in /usr/include/php
The php-devel package is required for use of this command.
ERROR: `phpize' failed

Install php-devel:

[root@server tmp]# dnf install php-devel

This time installation went well. Supply your Oracle home at the prompt and specify if you are using an instant client or not (display has been cut for clarity):

[root@server tmp]# pecl install oci8-2.2.0.tgz
11 source files, building
running: phpize
Configuring for:
PHP Api Version:         20170718
Zend Module Api No:      20170718
Zend Extension Api No:   320170718
Please provide the path to the ORACLE_HOME directory. Use 'instantclient,/path/to/instant/client/lib' if you're compiling with Oracle Instant Client [autodetect] : /oracle/software
building in /var/tmp/pear-build-rootIZwygS/oci8-2.2.0
running: /var/tmp/oci8/configure --with-php-config=/usr/bin/php-config --with-oci8=/oracle/software
checking for grep that handles long lines and -e... /usr/bin/grep
.
.
.
checking whether to build static libraries... no
configure: creating ./config.status
config.status: creating config.h
config.status: executing libtool commands
running: make
.
.
.
----------------------------------------------------------------------
Libraries have been installed in:
   /var/tmp/pear-build-rootIZwygS/oci8-2.2.0/modules

If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the '-LLIBDIR'
flag during linking and do at least one of the following:
   - add LIBDIR to the 'LD_LIBRARY_PATH' environment variable
     during execution
   - add LIBDIR to the 'LD_RUN_PATH' environment variable
     during linking
   - use the '-Wl,-rpath -Wl,LIBDIR' linker flag
   - have your system administrator add LIBDIR to '/etc/ld.so.conf'

See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------

Build complete.
Don't forget to run 'make test'.

running: make INSTALL_ROOT="/var/tmp/pear-build-rootIZwygS/install-oci8-2.2.0" install
Installing shared extensions:     /var/tmp/pear-build-rootIZwygS/install-oci8-2.2.0/usr/lib64/php/modules/
running: find "/var/tmp/pear-build-rootIZwygS/install-oci8-2.2.0" | xargs ls -dils
 104162   0 drwxr-xr-x 3 root root     17 Aug  4 12:30 /var/tmp/pear-build-rootIZwygS/install-oci8-2.2.0
2235876   0 drwxr-xr-x 3 root root     19 Aug  4 12:30 /var/tmp/pear-build-rootIZwygS/install-oci8-2.2.0/usr
3280107   0 drwxr-xr-x 3 root root     17 Aug  4 12:30 /var/tmp/pear-build-rootIZwygS/install-oci8-2.2.0/usr/lib64
 110674   0 drwxr-xr-x 3 root root     21 Aug  4 12:30 /var/tmp/pear-build-rootIZwygS/install-oci8-2.2.0/usr/lib64/php
1144553   0 drwxr-xr-x 2 root root     21 Aug  4 12:30 /var/tmp/pear-build-rootIZwygS/install-oci8-2.2.0/usr/lib64/php/modules
1144554 876 -rwxr-xr-x 1 root root 894144 Aug  4 12:30 /var/tmp/pear-build-rootIZwygS/install-oci8-2.2.0/usr/lib64/php/modules/oci8.so

Build process completed successfully
Installing '/usr/lib64/php/modules/oci8.so'
install ok: channel://pecl.php.net/oci8-2.2.0
configuration option "php_ini" is not set to php.ini location
You should add "extension=oci8.so" to php.ini

The oci8.so module is directly in good directory and you just need to change your PHP configuration, by adding a new file in /etc/php.d directory as below, to have it loaded:

[root@server php.d]# ll /usr/lib64/php/modules/oci8.so
-rw-r--r-- 1 root root 894144 Aug  4 12:30 /usr/lib64/php/modules/oci8.so
[root@server php.d]# chmod a+x /usr/lib64/php/modules/oci8.so
[root@server php.d]# cat /etc/php.d/30-oci8.ini
; Enable oci8 extension module
extension=oci8.so
[root@server php.d]# systemctl restart httpd
[root@server php.d]# systemctl restart php-fpm
[root@server php.d]# php -m
[PHP Modules]
bz2
calendar
Core
ctype
curl
date
dom
exif
fileinfo
filter
ftp
gettext
hash
iconv
json
libxml
mbstring
mysqli
mysqlnd
oci8
openssl
pcntl
pcre
PDO
pdo_mysql
pdo_sqlite
Phar
posix
readline
Reflection
session
shmop
SimpleXML
sockets
SPL
sqlite3
standard
sysvmsg
sysvsem
sysvshm
tokenizer
wddx
xml
xmlreader
xmlwriter
xsl
zlib

[Zend Modules]

[root@server php.d]# php --ri oci8

oci8

OCI8 Support => enabled
OCI8 DTrace Support => disabled
OCI8 Version => 2.2.0
Oracle Run-time Client Library Version => 12.2.0.1.0
Oracle Compile-time Version => 12.2
Compile-time ORACLE_HOME => /oracle/software
Libraries Used => -Wl,-rpath,/oracle/software/lib -L/oracle/software/lib  -lclntsh

Directive => Local Value => Master Value
oci8.max_persistent => -1 => -1
oci8.persistent_timeout => -1 => -1
oci8.ping_interval => 60 => 60
oci8.privileged_connect => Off => Off
oci8.statement_cache_size => 20 => 20
oci8.default_prefetch => 100 => 100
oci8.old_oci_close_semantics => Off => Off
oci8.connection_class => no value => no value
oci8.events => Off => Off

Statistics =>
Active Persistent Connections => 0
Active Connections => 0

I highly recommend to create a phpinfo() file on your web server to be able to quickly see how PHP is configured:

php_oci8
php_oci8

I have taken the example script in PHP official documentation and as it was not working I have tried directly on the server with:

[root@server html]# php oci8.php
PHP Warning:  oci_connect(): OCIEnvNlsCreate() failed. There is something wrong with your system - please check that ORACLE_HOME and LD_LIBRARY_PATH are set and point to the right directories in /www/html/oci8.php on line 3
PHP Warning:  oci_connect(): Error while trying to retrieve text for error ORA-01804
 in /www/html/oci8.php on line 3
PHP Fatal error:   in /www/html/oci8.php on line 6

This above message is really strange and I solved it with:

[ora12c@server ~]$ cd $ORACLE_HOME
[ora12c@server software]$ find . -name libociicus.so
./instantclient/light/libociicus.so
[ora12c@server software]$ cp ./instantclient/light/libociicus.so lib
[ora12c@server software]$ ll ./lib/libociicus.so
-rw-r----- 1 ora12c dba 6352804 Aug  4 15:06 ./lib/libociicus.so

Then issuing again the script interactively worked but not via Apache with, so, an http request !!

As suggested I have tried to avoid cleaning of environment variables when forking php-fpm processes with (/etc/php-fpm.d/www.conf file):

[root@server php-fpm.d]# vi www.conf
[root@server php-fpm.d]# grep clear www.conf
; by clearing the environment in workers before env vars specified in this
clear_env = no
[root@server php-fpm.d]# systemctl restart php-fpm
[root@server php-fpm.d]# systemctl restart httpd

But it did not solve the problem as explained in few blog posts…. So I have finally changed www.conf again and added:

env[ORACLE_HOME] = /oracle/software
env[LD_LIBRAY_PATH] = /oracle/software/lib

This time all worked well and we now have a running OCI8 module !!

PDO_OCI

When you go to official PDO_OCI page on pecl the warning message is quite clear:

PDO_OCI
This package is not maintained anymore and has been superseded. Package has moved to channel http://www.php.net/pdo_oci package ext/pdo_oci.

So in other words the package is inside the source code of PHP. Okay cool, but then if like me (and most probably 99.9% of people) your PHP is coming from the RPM of the official depot of your Linux distribution ? System admin also do not like you recompile on server as it requires tons of headers that are coming with devel packages. So ideally you should do it on a separate server but as we will see I had nothing more to install so let’s say even your preferred system administrator teammate would be happy !

I have downloaded the PHP source code of the exact same version of my running PHP 7.2.24. It also raise the question when you upgrade PHP, I will continue to try to find a more elegant solution and/or if someone can suggest in comment section I would be happy to know.

Unzip the source code and go in ext/pdo_oci directory, export the ORACLE_HOME environment variable and use phpize (https://www.php.net/manual/en/install.pecl.phpize.php). I have cut the display as it would be too long.

Run phpize and ./configure command:

[root@server php-7.2.24]# cd ext/pdo_oci/
[root@server pdo_oci]# export ORACLE_HOME=/oracle/software
[root@server pdo_oci]# ll
total 88
-rwxrwxr-x 1 root root  7649 Oct 22  2019 config.m4
-rwxrwxr-x 1 root root  1494 Oct 22  2019 config.w32
-rwxrwxr-x 1 root root    40 Oct 22  2019 CREDITS
-rw-rw-r-- 1 root root 21968 Oct 22  2019 oci_driver.c
-rw-rw-r-- 1 root root 22353 Oct 22  2019 oci_statement.c
-rw-rw-r-- 1 root root  2326 Oct 22  2019 package2.xml
-rw-rw-r-- 1 root root  4172 Oct 22  2019 pdo_oci.c
-rw-rw-r-- 1 root root  1708 Oct 22  2019 php_pdo_oci.h
-rw-rw-r-- 1 root root  3135 Oct 22  2019 php_pdo_oci_int.h
drwxrwxr-x 2 root root  4096 Oct 22  2019 tests
[root@server pdo_oci]# phpize
Configuring for:
PHP Api Version:         20170718
Zend Module Api No:      20170718
Zend Extension Api No:   320170718
[root@server pdo_oci]# ll
total 1540
-rw-r--r-- 1 root root  86562 Aug  4 16:10 acinclude.m4
-rw-r--r-- 1 root root 418161 Aug  4 16:10 aclocal.m4
drwxr-xr-x 2 root root     54 Aug  4 16:10 autom4te.cache
drwxr-xr-x 2 root root    115 Aug  4 16:10 build
-rwxr-xr-x 1 root root  42938 Aug  4 16:10 config.guess
-rw-r--r-- 1 root root   1988 Aug  4 16:10 config.h.in
-rwxrwxr-x 1 root root   7649 Oct 22  2019 config.m4
-rwxr-xr-x 1 root root  35967 Aug  4 16:10 config.sub
-rwxr-xr-x 1 root root 455182 Aug  4 16:10 configure
-rw-r--r-- 1 root root   4684 Aug  4 16:10 configure.ac
-rwxrwxr-x 1 root root   1494 Oct 22  2019 config.w32
-rwxrwxr-x 1 root root     40 Oct 22  2019 CREDITS
-rw-r--r-- 1 root root      0 Aug  4 16:10 install-sh
-rw-r--r-- 1 root root 324152 Aug  4 16:10 ltmain.sh
-rw-r--r-- 1 root root   7156 Aug  4 16:10 Makefile.global
-rw-r--r-- 1 root root      0 Aug  4 16:10 missing
-rw-r--r-- 1 root root      0 Aug  4 16:10 mkinstalldirs
-rw-rw-r-- 1 root root  21968 Oct 22  2019 oci_driver.c
-rw-rw-r-- 1 root root  22353 Oct 22  2019 oci_statement.c
-rw-rw-r-- 1 root root   2326 Oct 22  2019 package2.xml
-rw-rw-r-- 1 root root   4172 Oct 22  2019 pdo_oci.c
-rw-rw-r-- 1 root root   1708 Oct 22  2019 php_pdo_oci.h
-rw-rw-r-- 1 root root   3135 Oct 22  2019 php_pdo_oci_int.h
-rw-r--r-- 1 root root  85864 Aug  4 16:10 run-tests.php
drwxrwxr-x 2 root root   4096 Oct 22  2019 tests
[root@server pdo_oci]# ./configure
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for a sed that does not truncate output... /usr/bin/sed
checking for cc... cc
checking whether the C compiler works... yes
.
.
.
checking whether to build shared libraries... yes
checking whether to build static libraries... no
configure: creating ./config.status
config.status: creating config.h
config.status: executing libtool commands

Finally issue make and make test commands:

[root@server pdo_oci]# make
.
.
.
Libraries have been installed in:
   /tmp/php-7.2.24/ext/pdo_oci/modules

If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the '-LLIBDIR'
flag during linking and do at least one of the following:
   - add LIBDIR to the 'LD_LIBRARY_PATH' environment variable
     during execution
   - add LIBDIR to the 'LD_RUN_PATH' environment variable
     during linking
   - use the '-Wl,-rpath -Wl,LIBDIR' linker flag
   - have your system administrator add LIBDIR to '/etc/ld.so.conf'

See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------

Build complete.
Don't forget to run 'make test'.

[root@server pdo_oci]# make test
.
.
.
Number of tests :   33                 0
Tests skipped   :   33 (100.0%) --------
Tests warned    :    0 (  0.0%) (  0.0%)
Tests failed    :    0 (  0.0%) (  0.0%)
Expected fail   :    0 (  0.0%) (  0.0%)
Tests passed    :    0 (  0.0%) (  0.0%)
---------------------------------------------------------------------
Time taken      :    2 seconds
=====================================================================

This report can be automatically sent to the PHP QA team at
http://qa.php.net/reports and http://news.php.net/php.qa.reports
This gives us a better understanding of PHP's behavior.
If you don't want to send the report immediately you can choose
option "s" to save it.  You can then email it to qa-reports@lists.php.net later.
Do you want to send this report now? [Yns]: n

Then the compiled module is in modules directory. Copy it to target directory (/usr/lib64/php/modules/) and configure php-fpm to use it:

[root@server pdo_oci]# ll modules
total 204
-rw-r--r-- 1 root root    951 Aug  4 16:11 pdo_oci.la
-rwxr-xr-x 1 root root 202832 Aug  4 16:11 pdo_oci.so
[root@server pdo_oci]# cp modules/pdo_oci.so /usr/lib64/php/modules/
[root@server pdo_oci]# cat /etc/php.d/30-pdo_oci.ini
; Enable pdo_oci extension module
extension=pdo_oci
[root@server pdo_oci]# systemctl restart php-fpm
[root@server pdo_oci]# php --ri pdo

PDO

PDO support => enabled
PDO drivers => mysql, oci, sqlite

This can also be seen in phpinfo():

php_pdo_oci
php_pdo_oci

OCI8 from PHP source

When installing PDO_OCI I have realized that the extension directory also contains an oci8 directory that we could also install with the same strategy and not go through pecl. What is the difference with the version in the source code versus the one on pecl repository ? I feel like the one in PHP source is closer to what’s really running on your server and the one on pecl might have some bug correction…

Let’s see if we can compile and use it. We export ORACLE_HOME, run phpize and issue ./configure command:

[root@server php-7.2.24]# cd ext/oci8
[root@server oci8]# export ORACLE_HOME=/oracle/software
[root@server oci8]# ll
total 444
-rw-rw-r-- 1 root root  16048 Oct 22  2019 config.m4
-rw-rw-r-- 1 root root   3845 Oct 22  2019 config.w32
-rw-rw-r-- 1 root root    173 Oct 22  2019 CREDITS
-rw-rw-r-- 1 root root   3204 Oct 22  2019 LICENSE
-rw-rw-r-- 1 root root 119803 Oct 22  2019 oci8.c
-rw-rw-r-- 1 root root  24183 Oct 22  2019 oci8_collection.c
-rw-rw-r-- 1 root root   2334 Oct 22  2019 oci8_dtrace.d
-rw-rw-r-- 1 root root   5199 Oct 22  2019 oci8_failover.c
-rw-rw-r-- 1 root root  70256 Oct 22  2019 oci8_interface.c
-rw-rw-r-- 1 root root  27847 Oct 22  2019 oci8_lob.c
-rw-rw-r-- 1 root root  60056 Oct 22  2019 oci8_statement.c
-rw-rw-r-- 1 root root  48000 Oct 22  2019 package.xml
-rw-rw-r-- 1 root root   2703 Oct 22  2019 php_oci8.h
-rw-rw-r-- 1 root root  25333 Oct 22  2019 php_oci8_int.h
-rw-rw-r-- 1 root root    976 Oct 22  2019 README
drwxrwxr-x 2 root root  12288 Oct 22  2019 tests
[root@server oci8]# phpize
Configuring for:
PHP Api Version:         20170718
Zend Module Api No:      20170718
Zend Extension Api No:   320170718
[root@server oci8]# ll
total 1892
-rw-r--r-- 1 root root  86562 Aug  4 16:33 acinclude.m4
-rw-r--r-- 1 root root 418161 Aug  4 16:33 aclocal.m4
drwxr-xr-x 2 root root     54 Aug  4 16:33 autom4te.cache
drwxr-xr-x 2 root root    115 Aug  4 16:33 build
-rwxr-xr-x 1 root root  42938 Aug  4 16:33 config.guess
-rw-r--r-- 1 root root   2244 Aug  4 16:33 config.h.in
-rw-rw-r-- 1 root root  16048 Oct 22  2019 config.m4
-rwxr-xr-x 1 root root  35967 Aug  4 16:33 config.sub
-rwxr-xr-x 1 root root 454262 Aug  4 16:33 configure
-rw-r--r-- 1 root root   4684 Aug  4 16:33 configure.ac
-rw-rw-r-- 1 root root   3845 Oct 22  2019 config.w32
-rw-rw-r-- 1 root root    173 Oct 22  2019 CREDITS
-rw-r--r-- 1 root root      0 Aug  4 16:33 install-sh
-rw-rw-r-- 1 root root   3204 Oct 22  2019 LICENSE
-rw-r--r-- 1 root root 324152 Aug  4 16:33 ltmain.sh
-rw-r--r-- 1 root root   7156 Aug  4 16:33 Makefile.global
-rw-r--r-- 1 root root      0 Aug  4 16:33 missing
-rw-r--r-- 1 root root      0 Aug  4 16:33 mkinstalldirs
-rw-rw-r-- 1 root root 119803 Oct 22  2019 oci8.c
-rw-rw-r-- 1 root root  24183 Oct 22  2019 oci8_collection.c
-rw-rw-r-- 1 root root   2334 Oct 22  2019 oci8_dtrace.d
-rw-rw-r-- 1 root root   5199 Oct 22  2019 oci8_failover.c
-rw-rw-r-- 1 root root  70256 Oct 22  2019 oci8_interface.c
-rw-rw-r-- 1 root root  27847 Oct 22  2019 oci8_lob.c
-rw-rw-r-- 1 root root  60056 Oct 22  2019 oci8_statement.c
-rw-rw-r-- 1 root root  48000 Oct 22  2019 package.xml
-rw-rw-r-- 1 root root   2703 Oct 22  2019 php_oci8.h
-rw-rw-r-- 1 root root  25333 Oct 22  2019 php_oci8_int.h
-rw-rw-r-- 1 root root    976 Oct 22  2019 README
-rw-r--r-- 1 root root  85864 Aug  4 16:33 run-tests.php
drwxrwxr-x 2 root root  12288 Oct 22  2019 tests
[root@server oci8]# ./configure
checking for grep that handles long lines and -e... /usr/bin/grep
.
.
.
checking whether to build static libraries... no
configure: creating ./config.status
config.status: creating config.h
config.status: executing libtool commands

We run make and make test commands. My make test command is full of error but at the end I have been able to use the module:

[root@server oci8]# make
.
.
.
----------------------------------------------------------------------
Libraries have been installed in:
   /tmp/php-7.2.24/ext/oci8/modules

If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the '-LLIBDIR'
flag during linking and do at least one of the following:
   - add LIBDIR to the 'LD_LIBRARY_PATH' environment variable
     during execution
   - add LIBDIR to the 'LD_RUN_PATH' environment variable
     during linking
   - use the '-Wl,-rpath -Wl,LIBDIR' linker flag
   - have your system administrator add LIBDIR to '/etc/ld.so.conf'

See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------

Build complete.
Don't forget to run 'make test'.

[root@server oci8]# make test

Build complete.
Don't forget to run 'make test'.


=====================================================================
PHP         : /usr/bin/php
PHP_SAPI    : cli
PHP_VERSION : 7.2.24
ZEND_VERSION: 3.2.0
PHP_OS      : Linux - Linux server 4.18.0-305.3.1.el8_4.x86_64 #1 SMP Mon May 17 10:08:25 EDT 2021 x86_64
INI actual  : /tmp/php-7.2.24/ext/oci8/tmp-php.ini
More .INIs  :
CWD         : /tmp/php-7.2.24/ext/oci8
Extra dirs  :
VALGRIND    : Not used
=====================================================================
TIME START 2021-08-04 14:34:19
=====================================================================
FAIL oci_bind_array_by_name() and invalid values 1 [tests/array_bind_001.phpt]
FAIL oci_bind_array_by_name() and invalid values 2 [tests/array_bind_002.phpt]
FAIL oci_bind_array_by_name() and invalid values 3 [tests/array_bind_003.phpt]
FAIL oci_bind_array_by_name() and invalid values 4 [tests/array_bind_004.phpt]
FAIL oci_bind_array_by_name() and invalid values 5 [tests/array_bind_005.phpt]
FAIL oci_bind_array_by_name(), SQLT_CHR and default max_length [tests/array_bind_006.phpt]
FAIL oci_bind_array_by_name() and invalid values 7 [tests/array_bind_007.phpt]
.
.
.
---------------------------------------------------------------------

Number of tests :  363               358
Tests skipped   :    5 (  1.4%) --------
Tests warned    :    0 (  0.0%) (  0.0%)
Tests failed    :  346 ( 95.3%) ( 96.6%)
Expected fail   :    0 (  0.0%) (  0.0%)
Tests passed    :   12 (  3.3%) (  3.4%)
---------------------------------------------------------------------
Time taken      :   55 seconds
=====================================================================

=====================================================================
FAILED TEST SUMMARY
---------------------------------------------------------------------
oci_bind_array_by_name() and invalid values 1 [tests/array_bind_001.phpt]
oci_bind_array_by_name() and invalid values 2 [tests/array_bind_002.phpt]
oci_bind_array_by_name() and invalid values 3 [tests/array_bind_003.phpt]
.
.
.
Basic XMLType test #2 [tests/xmltype_02.phpt]
=====================================================================

You may have found a problem in PHP.
This report can be automatically sent to the PHP QA team at
http://qa.php.net/reports and http://news.php.net/php.qa.reports
This gives us a better understanding of PHP's behavior.
If you don't want to send the report immediately you can choose
option "s" to save it.  You can then email it to qa-reports@lists.php.net later.
Do you want to send this report now? [Yns]: n
make: *** [Makefile:136: test] Error 1

Same as for PDO_OCI the compiled module is in modules directory:

[root@server oci8]# ll modules
total 880
-rw-r--r-- 1 root root    930 Aug  4 16:34 oci8.la
-rwxr-xr-x 1 root root 894256 Aug  4 16:34 oci8.so
[root@server oci8]# ll /usr/lib64/php/modules/oci8.so
-rwxr-xr-x 1 root root 894144 Aug  4 12:30 /usr/lib64/php/modules/oci8.so

I have installed it in /usr/lib64/php/modules and it also works well. As said you are sure it is the same version as your running PHP version, 7.2.24 in my case.

References

The post PHP OCI8 and PDO_OCI extensions installation to access Oracle appeared first on IT World.

]]>
https://blog.yannickjaquier.com/linux/php-oci8-and-pdo_oci-extensions-installation-to-access-oracle.html/feed 0
DBMS_DATA_MINING package for Machine Learning inside the database https://blog.yannickjaquier.com/oracle/dbms_data_mining-package-for-machine-learning-inside-the-database.html https://blog.yannickjaquier.com/oracle/dbms_data_mining-package-for-machine-learning-inside-the-database.html#respond Tue, 28 Sep 2021 08:03:30 +0000 https://blog.yannickjaquier.com/?p=5243 Preamble Since the feature what was formerly known as Advanced Analytics and now called Machine Learning, Spatial and Graph has been made freely available: explained here, here or here it was time for me to make a try of this highly hype Machine Learning feature. The Machine Learning part is accessible through the DBMS_DATA_MINING PL/SQL […]

The post DBMS_DATA_MINING package for Machine Learning inside the database appeared first on IT World.

]]>

Table of contents

Preamble

Since the feature what was formerly known as Advanced Analytics and now called Machine Learning, Spatial and Graph has been made freely available: explained here, here or here it was time for me to make a try of this highly hype Machine Learning feature. The Machine Learning part is accessible through the DBMS_DATA_MINING PL/SQL package. No doubt that with the number of free Machine Learning tools out there it was not sustainable for Oracle to make this option non free…

Needless to say that it was really hard for me to get an idea of what to do and even harder to reach even a small objective. We have all read lots and lots of articles on how machine learning is helping to analyze medical images for tumor research, fraud detection or even speech recognition. Here as we are in an Oracle database the source of information would be ideally structured (versus non-structure as for images) but again it was clearly not a piece of cake to organize my mind and find where to start.

Then I remembered a nice web site called Kaggle and I have tried to find on it a popular dataset with few tasks associated and ideally people who have kindly submitted and shared their work to be able to compare if I am able to reach a similar result. On Kaggle people mainly work with Python in what we call notebooks. The dataset I have chosen is the Water Quality (https://www.kaggle.com/adityakadiwal/water-potability) one. The associated task is:

Predict if water is safe for Human consumption:
Create a model to determine if the sample tested from the water body is fit for human consumption or not.
This dataset may require you to treat missing value if any and check for data imbalance.

My testing has been luckily done on a powerful bare metal test server made of 12 cores and 64GB of RAM running Red Hat Enterprise Linux Server release 7.8 (Maipo). My Oracle test database is a pluggable database (pdb1) running Oracle 19.12 (July 2021 Release Update).

Loading dataset

Download and transfer the csv file to your database server. When you transfer this dataset to your database server you might have to convert it with dos2unix tool to have strange hidden characters at the end of each line (you can spend a day on a stupid thing like this). Then put the file in any folder you like (the Unix account used to run the database must be able to read it) and load it as an external table with a code like below.

I start by creating an account for me and a directory and giving full privileges on this directory to my account:

SQL> create user yjaquier identified by "secure_password";

User created.

SQL> grant dba to yjaquier;

Grant succeeded.

SQL> create or replace directory directory01 as '/home/oracle/';

Directory created.

SQL> grant read,write on directory directory01 to yjaquier;

Grant succeeded.

Finally create the external table with:

SQL> connect yjaquier/"secure_password"@pdb1
Connected.

SQL>
create table water_potability_csv (
  ph number,
  hardness number,
  solids number,
  chloramines number,
  sulfate number,
  conductivity number,
  organic_carbon number,
  trihalomethanes number,
  turbidity number,
  potability number(1)
)
organization external
(
  type oracle_loader
  default directory directory01
  access parameters
  (
    records delimited by newline skip 1 logfile 'water_potability.log' badfile 'water_potability.bad' discardfile 'water_potability.dsc'
    fields terminated by ','
    missing field values are null
  )
  location ('water_potability.csv')
)
reject limit unlimited;

Table created.

You can control it has been well loaded by directly selecting the external table:

SQL> set lines 200
SQL> select count(*) from water_potability_csv;

  COUNT(*)
----------
      3276

SQL> select * from water_potability_csv fetch first 5 rows only;

        PH   HARDNESS     SOLIDS CHLORAMINES    SULFATE CONDUCTIVITY ORGANIC_CARBON TRIHALOMETHANES  TURBIDITY POTABILITY
---------- ---------- ---------- ----------- ---------- ------------ -------------- --------------- ---------- ----------
           204.890455  20791.319  7.30021187 368.516441   564.308654     10.3797831      86.9909705 2.96313538          0
3.71608008 129.422921 18630.0579  6.63524588              592.885359     15.1800131      56.3290763 4.50065627          0
8.09912419 224.236259 19909.5417   9.2758836              418.606213     16.8686369      66.4200925 3.05593375          0
8.31676588 214.373394 22018.4174  8.05933238 356.886136   363.266516     18.4365245      100.341674 4.62877054          0
9.09222346 181.101509 17978.9863  6.54659997 310.135738   398.410813     11.5582794      31.9979927 4.07507543          0

You can also control in the filesystem of the directory we created above that there is no water_potability.bad file and control in water_potability.log that everything goes well (confirmed by the number of loaded rows in my case).

For the case_id_column_name parameter of the dbms_data_mining.create_model procedure I have realized that I needed to add an sequence id kind of column on my dataset table:

SQL> create table water_potability (
  id NUMBER GENERATED ALWAYS AS IDENTITY,
  ph number,
  hardness number,
  solids number,
  chloramines number,
  sulfate number,
  conductivity number,
  organic_carbon number,
  trihalomethanes number,
  turbidity number,
  potability number(1)
);

SQL> insert into water_potability(ph, hardness, solids, chloramines, sulfate, conductivity, organic_carbon, trihalomethanes, turbidity, potability)
     select * from water_potability_csv;

3276 rows created.

SQL> commit;

Commit complete.

Remark:
If you need more advanced transformation Oracle has implemented this in DBMS_DATA_MINING_TRANSFORM package.

Dataset queries and charts

The multiple answers of the Kaggle thread helps to see that we can almost get same repartition results in SQL (chartless in SQL obviously):

SQL> select decode(potability,0,'Not potable','Potable') as potability,
     round(ratio_to_report(count(*)) over ()*100) as percentage
     from water_potability group by potability;

POTABILITY  PERCENTAGE
----------- ----------
Potable             39
Not potable         61

To display few charts you could do in Python (Matplotlib) with cx_Oracle connector or use free Power BI Desktop that I recently installed to test MariaDB ColumnStore (https://blog.yannickjaquier.com/mysql/mariadb-columnstore-installation-and-testing-part-1.html). I connected to my database using ODBC and EZConnect and imported in Power BI Desktop my water_potability table…

Potability samples repartition:

dbms_data_mining01
dbms_data_mining01

Hardness repartition:

dbms_data_mining02
dbms_data_mining02

The dataset has deliberately wrong figures and part of the task is to clean figures to replace the null values. One traditional approach is to replace those null values by median value which is not a complex task in Python with Pandas. I could also do this in SQL with queries like:

SQL> select median(ph) as median_ph from water_potability where potability=0 and ph is not null;

 MEDIAN_PH
----------
7.03545552

But Oracle in their DBMS_DATA_MINING package has their own automatic data cleaning algorithms that I’m planning to activate so skipping this task for now…

Model creation and testing

You need an option table as described in official documentation:

SQL> create table model_settings(setting_name varchar2(30), setting_value varchar2(30));

Table created.

One typical activity is to split the dataset in training sample and testing sample, not to train your data model on the testing sample. One rule of thumb is to split in 80%/20% and do the training on the 80% and test the accuracy of your model on the remaining 20%:

SQL> create table water_potability_training
     as select * from water_potability SAMPLE (80);

Table created.

SQL> create table water_potability_testing
     as select * from water_potability
     minus select * from water_potability_training;

Table created.

SQL> select count(*) from water_potability_training;

  COUNT(*)
----------
      2619

SQL> select count(*) from water_potability_testing;

  COUNT(*)
----------
       657

SQL> select count(*) from water_potability;

  COUNT(*)
----------
    3276

When playing with different models and their associated parameters if you are too optimistic then you can end up with a very long running time for the CREATE_MODEL procedure. This running time was also serial (using one thread) so if you wish to use more power of your database server (see conclusion for the pros and cons) I have also changed the attributes of my table to allow parallel operation and put in in memory with:

SQL> alter table water_potability_training parallel inmemory;

Table altered.

Remark:
I recall that InMemory paid Enterprise option is free if you use less than 16GB. To be sure you are compliant you can even set the limit to 16GB with inmemory_size parameter.

Choose your model and set its parameters. Be very careful all the insertion must be in a PL/SQL block or you will get error message like (ORA-06553: PLS-221: ‘ALGO_NAME’ is not a procedure or is undefined). I have chosen the Random Forest algorithm as this is the one that provided better result from people who submitted a task answer on Kaggle (it would have been difficult alone to decide which one to choose). When using Random Forest algorithm you can also tweak the Decision Tree algorithm parameters:

SQL>
begin
  -- Clean the table before starting (TRUNCATE cannot be used in PL/SQL)
  delete from model_settings;
  -- Choose your model
  insert into model_settings values(dbms_data_mining.algo_name, dbms_data_mining.algo_random_forest);
  -- Automatic data preparation activation
  insert into model_settings values(dbms_data_mining.prep_auto, dbms_data_mining.prep_auto_on);
  -- Missing value will be replaced by mean value
  insert into model_settings values (dbms_data_mining.odms_missing_value_treatment, dbms_data_mining.odms_missing_value_mean_mode);
  -- Algorithm Settings: Random Forest
  insert into model_settings values (dbms_data_mining.rfor_mtry, 0);
  insert into model_settings values (dbms_data_mining.rfor_num_trees, 100);
  insert into model_settings values (dbms_data_mining.rfor_sampling_ratio, 1);
  insert into model_settings values (dbms_data_mining.tree_term_max_depth, 50);
  commit;
end;
/

PL/SQL procedure successfully completed.

SQL> select * from model_settings;

SETTING_NAME                   SETTING_VALUE
------------------------------ ------------------------------
ALGO_NAME                      ALGO_RANDOM_FOREST
PREP_AUTO                      ON
ODMS_MISSING_VALUE_TREATMENT   ODMS_MISSING_VALUE_MEAN_MODE
RFOR_MTRY                      0
RFOR_NUM_TREES                 100
RFOR_SAMPLING_RATIO            1

6 rows selected.

Finally create the model, it will also training it so the execution time is linked to chosen algorithm and its parameters:

-- create the model using the specified settings 
begin
  dbms_data_mining.create_model(
    model_name          => 'water_potability_model',
    mining_function     => dbms_data_mining.classification,
    data_table_name     => 'water_potability_training',
    case_id_column_name => 'id',
    target_column_name  => 'potability',
    settings_table_name => 'model_settings');
end;
/

PL/SQL procedure successfully completed.

If you plan to make multiple test by playing with model parameters you must drop the model first before creating a new one:

SQL> exec dbms_data_mining.drop_model('water_potability_model');

PL/SQL procedure successfully completed.

Remark:
The procedure is also creating plenty of DM$xxWATER_POTABILITY_MODEL tables.

I have finally used below query to apply my model on the 20% sample of testing data. The best accuracy I have been able to get is 65% with the Random Forest algorithm:

SQL>
select
  predicted,
  round(ratio_to_report(count(*)) over ()*100) as percentage
from (
  select
    case when potability=predicted_potability then 'Good' else 'Bad' end as predicted
  from (
    select 
      t.*,
      prediction (water_potability_model using *) predicted_potability
    from water_potability_testing t)
  )
group by predicted;

PRED PERCENTAGE
---- ----------
Good         65
Bad          35

This is here one huge difficulty I have found (at least for me) is which algorithm to choose ? You might not have time and/or energy/resource to test them all. Then when you have chosen your algorithm when you paly with its parameters (Random Forest algorithm for me) I have also experimented that better is enemy of good enough as each time I have tried to add more trees or fraction of the training data to be randomly sampled it has given a worse result…More or less each time it has ended up with a worst accuracy but with a drastic increase in CPU consumption.

Last but not least, I was honestly expecting an higher accuracy as people on Kaggle are going up to 80%. Maybe I doing something wrong or my whole exercise is wrong. Do not hesitate to comment if you see something stupid in my logic…

Conclusion

I don’t really know what to think with this Oracle database Machine Learning feature. In one hand Oracle has made something easy to use and you obviously use it with a language you already know very well: SQL. Of course the Machine Learning language is Python so if you are in this domain Python is most probably your best friend.

On the other hand you do machine learning at the cost of the Oracle database while doing Python and cx_Oracle is almost *free* (separate server, even a virtual machine, with all free components).

I have taken a screenshot of my server while creating a model and it can be like this for more than 10 minutes from my simple trial dataset (if you keep the default algorithm parameters you will not have this situation):

dbms_data_mining03
dbms_data_mining03

Then of course the feature has become free and, for me, it is clearly a must if Oracle expect people to use it as the free offer is really generous (Scikit-learn, tensorflow, Spark ML, …). Before deciding to use it you have to balance the additional CPU consumption you will make on your database server versus having a dedicated server with Python and offloading your figures to it… I would say that if your dataset is huge and computing a model is fast then this option is interesting. You can also live see the result of your model applied to your figures with a simple SQL statement…

DBMS_PREDICTIVE_ANALYTICS all in one bonus package

I have also tried with DBMS_PREDICTIVE_ANALYTICS package that is more easy to implement for noob on Machine Learning like me. if you have already use the DBMS_DATA_MINING package then this one is much simpler to implement but you have obviously less control over it.

I start with the EXPLAIN procedure that I initially did not even consider but at the end it provides interesting information on which columns Oracle will use to make a prediction of your target column. To not have the required added ID column you could also issue it on the WATER_POTABILITY_CSV external table if you like:

SQL> exec dbms_predictive_analytics.explain(data_table_name => 'water_potability_training', explain_column_name => 'potability', result_table_name => 'water_potability_explain');

PL/SQL procedure successfully completed.

SQL> set lines 200
SQL> col attribute_name for a15
SQL> col attribute_subname for a15
SQL> select * from water_potability_explain;

ATTRIBUTE_NAME  ATTRIBUTE_SUBNA EXPLANATORY_VALUE       RANK
--------------- --------------- ----------------- ----------
ID                                     .552324937          1
SULFATE                                .007351846          2
PH                                              0          3
TRIHALOMETHANES                                 0          3
TURBIDITY                                       0          3
CONDUCTIVITY                                    0          3
CHLORAMINES                                     0          3
ORGANIC_CARBON                                  0          3
HARDNESS                                        0          3
SOLIDS                                          0          3

10 rows selected.

It’s a bit disturbing, and if I understand it well, but apparently only SULFATE column is taken into account to make a prediction. This might explain the poor result I’ll get…

Then I start with a similar execution as with DBMS_DATA_MINING package. Here no model to choose and I would even not need to split my table in training and testing data sets:

SQL> set serveroutput on size 999999
SQL>
DECLARE 
    v_accuracy NUMBER(10,9); 
BEGIN 
    DBMS_PREDICTIVE_ANALYTICS.PREDICT( 
        accuracy             => v_accuracy, 
        data_table_name      => 'water_potability_training', 
        case_id_column_name  => 'id', 
        target_column_name   => 'potability', 
        result_table_name    => 'water_potability_predict_result'); 
    DBMS_OUTPUT.PUT_LINE('Accuracy = ' || v_accuracy); 
END; 
/

Accuracy = .055282742

PL/SQL procedure successfully completed.

Remark:
The returned accuracy is clearly not good…

To check the predicted value versus the real one:

SQL> col probability for 9.999999999
SQL> select a.potability, b.prediction,b.probability
     from water_potability a, water_potability_predict_result b
     where a.id=b.id
     fetch first 10 rows only;

POTABILITY PREDICTION  PROBABILITY
---------- ---------- ------------
        0          0   .610151794
        0          0   .610151734
        0          1   .389848346
        0          1   .389848362
        0          1   .389848299
        0          1   .389848292
        0          1   .389848368
        0          1   .389848334
        0          1   .389848339
        0          0   .610151736

10 rows selected.  

Finally same query as DBMS_DATA_MINING package to see that result is not really good:

SQL>
select
  predicted,
  round(ratio_to_report(count(*)) over ()*100) as percentage
from (
  select
    case when potability=prediction then 'Good' else 'Bad' end as predicted
  from (
    select
      a.potability,
      b.prediction
    from water_potability_training a, water_potability_predict_result b
    where a.id = b.id)
  )
group by predicted;

PRED PERCENTAGE
---- ----------
Good         52
Bad          48

If I query eliminating where the probability is not good the accuracy improve and we almost reach previous result but we eliminated few predictions:

SQL>
select
  predicted,
  round(ratio_to_report(count(*)) over ()*100) as percentage
from (
  select
    case when potability=prediction then 'Good' else 'Bad' end as predicted
  from (
    select
      a.potability,
      b.prediction
    from water_potability_training a, water_potability_predict_result b
    where a.id = b.id and probability > 0.5)
  )
group by predicted;

PRED PERCENTAGE
---- ----------
Bad          36
Good         64

References

The post DBMS_DATA_MINING package for Machine Learning inside the database appeared first on IT World.

]]>
https://blog.yannickjaquier.com/oracle/dbms_data_mining-package-for-machine-learning-inside-the-database.html/feed 0
PostgreSQL graphical monitoring tools comparison https://blog.yannickjaquier.com/postgresql/postgresql-graphical-monitoring-tools-comparison.html https://blog.yannickjaquier.com/postgresql/postgresql-graphical-monitoring-tools-comparison.html#respond Sat, 28 Aug 2021 13:08:49 +0000 https://blog.yannickjaquier.com/?p=5187 Preamble Monitoring tools is part of our journey to slowly but constantly increasing our internal expertise on PostgreSQL. So far this is not part of our IT standards but fact is that that few applications already started to use it. In parallel of PostgreSQL community edition you might have heard about Enterprise DB that propose […]

The post PostgreSQL graphical monitoring tools comparison appeared first on IT World.

]]>

Table of contents

Preamble

Monitoring tools is part of our journey to slowly but constantly increasing our internal expertise on PostgreSQL. So far this is not part of our IT standards but fact is that that few applications already started to use it.

In parallel of PostgreSQL community edition you might have heard about Enterprise DB that propose a commercial offer on PostgreSQL said to be compatible with Oracle. With the clear goal to reduce your Oracle fees in migrating to EnterpriseDB. So far this is not yet a goal for us and we just aim to stick to community PostgreSQL and increase our internal knowledge before going further…

So I have decided to start to have a look and increase my knowledge on enterprise features that are key for us:

This blog post will be on PostgreSQL monitoring tools as this is the most appealing part and is the part that helps you not to be blind in front of problems…

I plan to start small this blog post and enrich it if I test new monitoring tools in future. The plan is also to compare them versus the commercial offer of EnterpriseDB: PostgreSQL Enterprise Manager. If trial keys are available I also plan to add other commercial products. I will also mainly focus on on-premise product as our databases are mostly on-premise.

Looking at the free and open source products available out there I tend to say that it sounds difficult to me for paid competition…

Preference so far:

  1. PGWatch
  2. Percona Monitoring and Management
  3. Postgres Enterprise Manager
  4. OmniDB
  5. pgAdmin

pgAdmin

PgAdmin that is one of the most famous free and open source PostgreSQL monitoring tool.

Step zero, if like me you are behind a corporate proxy, is to configure your Python pip to go on internet by creating the /etc/pip.conf file similar to below (I have also decided to use a dedicated Linux account, pgadmin, to run it):

[pgadmin@server ~]$ cat /etc/pip.conf
[global]
extra-index-url=https://www.piwheels.org/simple
proxy = http://account:password@proxy_server.domain.com:proxy_port/
trusted-host = pypi.python.org pypi.org www.piwheels.org  files.pythonhosted.org

Create and activate a Python vitual environment with (Install Python 3 on your server using your Linux distribution repository):

[pgadmin@server ~]$ cd /www/pgadmin/
[pgadmin@server pgadmin]$ python3 -m venv pgadmin4
[pgadmin@server pgadmin]$ source pgadmin4/bin/activate

Most probably you will have to upgrade pip if you get below error message:

WARNING: You are using pip version 21.1.2; however, version 21.1.3 is available.
You should consider upgrading via the '/www/pgadmin/pgadmin4/bin/python3 -m pip install --upgrade pip' command.

Use:

(pgadmin4) [pgadmin@server ~]$ pip install --upgrade pip

Create and give ownership to your pgAdmin Linux account to directory /var/log/pgadmin.

Finally install pgAdmin 4 with (current version as a time of writing this article is 5.4):

pip install pgadmin4

Execute it with:

(pgadmin4) [pgadmin@server pgadmin]$ nohup pgadmin4 > pgadmin4.out &
[1] 22676

To generate a bit of activity and understand how it works I have initialized pgBench. I have decided to create a dedicated database for pgBench, this must be done upfront using:

create database pgbenchdb;

Then I created the pgbench datamodel using (you must plan for 5GB data storage):

pgbench --host=server2.domain.com --port=5433 --user=postgres --initialize --scale=100 pgbenchdb

If you mess up or want to delete the pgbench table use:

pgbench --host=server2.domain.com --port=5433 --user=postgres --initialize --init-steps=d pgbenchdb

Most probably you will fill the WAL directory so use something like (never do this is production ! But here I assume you are using a test server where you don’t care about recovery). First display mode to see what PosgreSQL would do:

[postgres@server ~]$ pg_archivecleanup -d -n /postgres/13/data/pg_wal/ 0000000300000000000000D0
pg_archivecleanup: keeping WAL file "/postgres/13/data/pg_wal//0000000300000000000000D0" and later
/postgres/13/data/pg_wal//0000000300000000000000B9
pg_archivecleanup: file "/postgres/13/data/pg_wal//0000000300000000000000B9" would be removed
/postgres/13/data/pg_wal//0000000300000000000000BA
pg_archivecleanup: file "/postgres/13/data/pg_wal//0000000300000000000000BA" would be removed
/postgres/13/data/pg_wal//0000000300000000000000BB
pg_archivecleanup: file "/postgres/13/data/pg_wal//0000000300000000000000BB" would be removed
/postgres/13/data/pg_wal//0000000300000000000000BC
pg_archivecleanup: file "/postgres/13/data/pg_wal//0000000300000000000000BC" would be removed
/postgres/13/data/pg_wal//0000000300000000000000BD
pg_archivecleanup: file "/postgres/13/data/pg_wal//0000000300000000000000BD" would be removed
/postgres/13/data/pg_wal//0000000300000000000000BE
pg_archivecleanup: file "/postgres/13/data/pg_wal//0000000300000000000000BE" would be removed
/postgres/13/data/pg_wal//0000000300000000000000BF
pg_archivecleanup: file "/postgres/13/data/pg_wal//0000000300000000000000BF" would be removed
/postgres/13/data/pg_wal//0000000300000000000000C0
pg_archivecleanup: file "/postgres/13/data/pg_wal//0000000300000000000000C0" would be removed
/postgres/13/data/pg_wal//0000000300000000000000C1
pg_archivecleanup: file "/postgres/13/data/pg_wal//0000000300000000000000C1" would be removed
/postgres/13/data/pg_wal//0000000300000000000000C2
pg_archivecleanup: file "/postgres/13/data/pg_wal//0000000300000000000000C2" would be removed
/postgres/13/data/pg_wal//0000000300000000000000C3
pg_archivecleanup: file "/postgres/13/data/pg_wal//0000000300000000000000C3" would be removed
/postgres/13/data/pg_wal//0000000300000000000000C4
pg_archivecleanup: file "/postgres/13/data/pg_wal//0000000300000000000000C4" would be removed
/postgres/13/data/pg_wal//0000000300000000000000C5
pg_archivecleanup: file "/postgres/13/data/pg_wal//0000000300000000000000C5" would be removed
/postgres/13/data/pg_wal//0000000300000000000000C6
pg_archivecleanup: file "/postgres/13/data/pg_wal//0000000300000000000000C6" would be removed
/postgres/13/data/pg_wal//0000000300000000000000C7
pg_archivecleanup: file "/postgres/13/data/pg_wal//0000000300000000000000C7" would be removed
/postgres/13/data/pg_wal//0000000300000000000000C8
pg_archivecleanup: file "/postgres/13/data/pg_wal//0000000300000000000000C8" would be removed
/postgres/13/data/pg_wal//0000000300000000000000C9
pg_archivecleanup: file "/postgres/13/data/pg_wal//0000000300000000000000C9" would be removed
/postgres/13/data/pg_wal//0000000300000000000000CA
pg_archivecleanup: file "/postgres/13/data/pg_wal//0000000300000000000000CA" would be removed
/postgres/13/data/pg_wal//0000000300000000000000CB
pg_archivecleanup: file "/postgres/13/data/pg_wal//0000000300000000000000CB" would be removed
/postgres/13/data/pg_wal//0000000300000000000000CC
pg_archivecleanup: file "/postgres/13/data/pg_wal//0000000300000000000000CC" would be removed
/postgres/13/data/pg_wal//0000000300000000000000CD
pg_archivecleanup: file "/postgres/13/data/pg_wal//0000000300000000000000CD" would be removed
/postgres/13/data/pg_wal//0000000300000000000000CE
pg_archivecleanup: file "/postgres/13/data/pg_wal//0000000300000000000000CE" would be removed
/postgres/13/data/pg_wal//0000000300000000000000CF
pg_archivecleanup: file "/postgres/13/data/pg_wal//0000000300000000000000CF" would be removed

Then the command to delete:

[postgres@server ~]$ pg_archivecleanup -d /postgres/13/data/pg_wal/ 0000000300000000000000D0
pg_archivecleanup: keeping WAL file "/postgres/13/data/pg_wal//0000000300000000000000D0" and later
pg_archivecleanup: removing file "/postgres/13/data/pg_wal//0000000300000000000000B9"
pg_archivecleanup: removing file "/postgres/13/data/pg_wal//0000000300000000000000BA"
pg_archivecleanup: removing file "/postgres/13/data/pg_wal//0000000300000000000000BB"
pg_archivecleanup: removing file "/postgres/13/data/pg_wal//0000000300000000000000BC"
pg_archivecleanup: removing file "/postgres/13/data/pg_wal//0000000300000000000000BD"
pg_archivecleanup: removing file "/postgres/13/data/pg_wal//0000000300000000000000BE"
pg_archivecleanup: removing file "/postgres/13/data/pg_wal//0000000300000000000000BF"
pg_archivecleanup: removing file "/postgres/13/data/pg_wal//0000000300000000000000C0"
pg_archivecleanup: removing file "/postgres/13/data/pg_wal//0000000300000000000000C1"
pg_archivecleanup: removing file "/postgres/13/data/pg_wal//0000000300000000000000C2"
pg_archivecleanup: removing file "/postgres/13/data/pg_wal//0000000300000000000000C3"
pg_archivecleanup: removing file "/postgres/13/data/pg_wal//0000000300000000000000C4"
pg_archivecleanup: removing file "/postgres/13/data/pg_wal//0000000300000000000000C5"
pg_archivecleanup: removing file "/postgres/13/data/pg_wal//0000000300000000000000C6"
pg_archivecleanup: removing file "/postgres/13/data/pg_wal//0000000300000000000000C7"
pg_archivecleanup: removing file "/postgres/13/data/pg_wal//0000000300000000000000C8"
pg_archivecleanup: removing file "/postgres/13/data/pg_wal//0000000300000000000000C9"
pg_archivecleanup: removing file "/postgres/13/data/pg_wal//0000000300000000000000CA"
pg_archivecleanup: removing file "/postgres/13/data/pg_wal//0000000300000000000000CB"
pg_archivecleanup: removing file "/postgres/13/data/pg_wal//0000000300000000000000CC"
pg_archivecleanup: removing file "/postgres/13/data/pg_wal//0000000300000000000000CD"
pg_archivecleanup: removing file "/postgres/13/data/pg_wal//0000000300000000000000CE"
pg_archivecleanup: removing file "/postgres/13/data/pg_wal//0000000300000000000000CF"

Finally issue a benchmark using:

pgbench --host=server2.domain.com --port=5433 --user=postgres --client=10 --jobs=2 --transactions=10000 pgbenchdb

We can see that pgAdmin is providing interesting graphs of your server or individual database performance:

pgadmin01
pgadmin01

Pgadmin is also a neat graphical query editor:

pgadmin02
pgadmin02

Overall nice monitoring tools even if a bit complex to navigate in menu and options. The chart part could be more exhaustive and you cannot add your own charts…

Postgres Enterprise Manager

This tool is not free and you need an EnterpriseDB subscription to use it. I have created a trial account and you can test the tool for 60 days. For this I have used two virtual machines running Oracle Linux 8. One for the Postgres Enterprise Manager (PEM) repository and one for the client PostgreSQL Instance (deployed with a PEM agent).

Postgres Enterprise Manager Server

My PEM Server will be my first virtual machine called server1.domain.com (192.168.56.101). I have started by creating the PostgreSQL 13 repository database in /postgres/13/data directory and created a service with below, taken from official documentation, startup file:

[root@server1 ~]# cat /etc/systemd/system/postgresql.service
[Unit]
Description=PostgreSQL database server
Documentation=man:postgres(1)

[Service]
Type=notify
User=postgres
ExecStart=/usr/pgsql-13/bin/postgres -D /postgres/13/data
ExecReload=/bin/kill -HUP $MAINPID
KillMode=mixed
KillSignal=SIGINT
TimeoutSec=0

[Install]
WantedBy=multi-user.target

Reload systemd daemon with:

systemctl daemon-reload

Now you can use systemctl stopstart/status postgresql (PEM Server will do it too):

[root@server1 ~]# systemctl status postgresql
● postgresql.service - PostgreSQL database server
   Loaded: loaded (/etc/systemd/system/postgresql.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2021-07-05 12:43:56 CEST; 12min ago
     Docs: man:postgres(1)
 Main PID: 9781 (postgres)
    Tasks: 17 (limit: 49502)
   Memory: 144.9M
   CGroup: /system.slice/postgresql.service
           ├─ 9781 /usr/pgsql-13/bin/postgres -D /postgres/13/data
           ├─ 9782 postgres: logger
           ├─ 9784 postgres: checkpointer
           ├─ 9785 postgres: background writer
           ├─ 9786 postgres: walwriter
           ├─ 9787 postgres: autovacuum launcher
           ├─ 9788 postgres: stats collector
           ├─ 9789 postgres: logical replication launcher
           ├─ 9850 postgres: agent1 pem 127.0.0.1(60830) idle
           ├─ 9871 postgres: agent1 pem 127.0.0.1(60832) idle
           ├─ 9898 postgres: agent1 pem 127.0.0.1(60836) idle
           ├─ 9904 postgres: postgres postgres 127.0.0.1(60838) idle
           ├─ 9910 postgres: agent1 pem 127.0.0.1(60840) idle
           ├─ 9919 postgres: agent1 pem 127.0.0.1(60842) idle
           ├─10358 postgres: postgres pem 127.0.0.1(60944) idle
           ├─10359 postgres: postgres pem 127.0.0.1(60946) idle
           └─10360 postgres: postgres pem 127.0.0.1(60948) idle

Jul 05 12:43:56 server1.domain.com systemd[1]: Starting PostgreSQL database server...
Jul 05 12:43:56 server1.domain.com postgres[9781]: 2021-07-05 12:43:56.615 CEST [9781] LOG:  redirecting log output to logging collector process
Jul 05 12:43:56 server1.domain.com postgres[9781]: 2021-07-05 12:43:56.615 CEST [9781] HINT:  Future log output will appear in directory "log".
Jul 05 12:43:56 server1.domain.com systemd[1]: Started PostgreSQL database server.

To be able to connect locally (PEM server access to 127.0.0.1) and from remote agent I have changed in postgresql.conf:

listen_addresses = 'localhost,server1.domain.com'

And in pg_hba.conf:

host    all             postgres             0.0.0.0/0            trust

As instructed install the EDB repository with:

dnf -y install https://yum.enterprisedb.com/edbrepos/edb-repo-latest.noarch.rpm

And change in /etc/yum.repos.d/edb.repo username and password with provided EDB information. Actually this is not the account you use to connect to EDB website that has to be used but the one provided when you go in your profile (I did that stupid mistake so sharing). Click on the eye icon to read the password:

pem01
pem01

Install PEM server with:

dnf install edb-pem

The interactive configuration tool is /usr/edb/pem/bin/configure-pem-server.sh. If for any reason you want to restart the configuration from scratch the configuration file is (not documented):

[root@server1 ~]# cat /usr/edb/pem/share/.install-config
PEM_INSTALLATION_TYPE=1
PG_INSTALL_PATH=/usr/pgsql-13
SUPERUSER=postgres
HOST=127.0.0.1
PORT=5432
AGENT_CERTIFICATE_PATH=/root/.pem/
PEM_PYTHON=python3
PEM_APP_HOST=
WEB_PEM_CONFIG=/usr/edb/pem/web/config_setup.py
CIDR_ADDR=0.0.0.0/0
DB_UNIT_FILE=postgresql
PEM_SERVER_SSL_PORT=8443

Once you have answered to the few question the PEM server is configured. So far I have not understood on how to stop/start/status the PEM server process and each time I use the configure-pem-server.sh script to start it. Not very convenient…

Postgres Enterprise Manager Agent

My PEM Agent server will be my second virtual machine called server2.domain.com (192.168.56.102). Same as the server part you need to configure the EDB repository and insert inside the repository file your EDB account and password. The only package you have to install is the PEM Agent:

dnf install edb-pem-agent

I have also configure on this client server a prostgreSQL instance where I have installed pgbench repository to generate some workload…

On this client node register the agent with the PostgreSQL repository instance. This is why I had to configure the PostgreSQL repository instance to accept connection from remote clients:

/usr/edb/pem/agent/bin/pemworker --register-agent

Finally you control PEM Agent with systemd:

systemctl start pemagent

Then this is where it is not crystal clear to me on how you need to add PEM Agent and PEM client servers to your repository. For me PEM Agent should be added automatically and when you must add them manually:

pem02
pem02

You do not see the databases list information and you need to add the server a second time (I added it in a specific group I created upfront) not using the PEM Agent sheet but the Connection one… Most probably I’m doing somethign wrong:

pem03
pem03

The tool has also a graphical query interface:

pem04
pem04

Overall the tool is quite complete with plenty of dashboard, alerts and probes (a check) that you can also customize and create on your own. If you have a magical query you can put everything in place to create a dashboard or an alert based on it.

Remark:
One point, same as me, that might not come immediately is the dashboard menu that is changing related to where you point on left menu (server, database, schema).

This is clearly one step beyond PgAdmin but it also takes the bad point of its ancestor. To be honest, overall, I have not been really impressed by the tool (also taking into account that the tool is not free). The tool is not bad but I was clearly expecting something much modern and easy to use for the UI. One nice added feature versus PgAdmin is the custom probes, alert and charts !! I would need to go deeper as I might have not understood it well…

OmniDB

Download the server package for your operating system:

omnidb01
omnidb01

And for my Oracle Linux 8 virtual machine installation has been as simple as:

dnf install omnidb-server-3.0.3b_linux_x86_64.rpm

To connect remotely I had to change the listen address at /root/.omnidb/omnidb-server/config.py configuration file. Execute the omniDB server web interface with:

[root@server1 ~]# omnidb-server
Running database migrations...
Operations to perform:
  Apply all migrations: OmniDB_app, admin, auth, contenttypes, sessions, social_django
Running migrations:
  No migrations to apply.
Starting OmniDB server...
Checking port availability...
Starting server OmniDB 3.0.3b at 192.168.56.101:8000.
Open OmniDB in your favorite browser
Press Ctrl+C to exit

In the web interface (default account is admin/admin) add a new PostgreSQL server with (other database flavor are available):

omnidb02
omnidb02

The tool has a graphical query part (all have more or less):

omnidb03
omnidb03

And a monitoring chart part:

omnidb04
omnidb04

Charts are neat (I really like them: really modern !) and the look and feel is really Web 2.0 but you cannot add your own custom charts while PEM has this capability…

Percona Monitoring and Management

This free multi-databases monitoring tool is a free offer from Percona. It is based on the traditional server/agent model and based on Grafana. To have already played a bit with it I can already say that the look and feel is pretty neat.

Server

For the server part of Percona Monitoring and Management (PMM) you have to download a provided Docker image. On my Oracle Linux 8 virtual box I have decided to use Podman (similar to Docker but daemon less, open source and Linux native tool). The nice thing to transition to Podman is that commands are all the same…

Download Docker PMM server image with podman pull command. I had to configure my corporate proxy by setting HTTPS_PROXY environment variable and had also to add my proxy certificate. I am not re-entering into details as we have seen this already with docker and it’s almost the same with Podamn (link):

[root@server1 ~]# podman pull percona/pmm-server:2
✔ docker.io/percona/pmm-server:2
Trying to pull docker.io/percona/pmm-server:2...
  Get "https://registry-1.docker.io/v2/": x509: certificate signed by unknown authority
Error: Error initializing source docker://percona/pmm-server:2: error pinging docker registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": x509: certificate signed by unknown authority

With proxy and certificates configured:

[root@server1 tmp]# podman pull percona/pmm-server:2
✔ docker.io/percona/pmm-server:2
Trying to pull docker.io/percona/pmm-server:2...
Getting image source signatures
Copying blob 178efec65a21 done
Copying blob 2d473b07cdd5 done
Copying config 82d29be43d done
Writing manifest to image destination
Storing signatures
82d29be43d66377922dcb3b1cabe8e2cb5716a3b9a76bab8791736e465ba50be

Once PMM is installed create a volume with

[root@server1 tmp]# podman create --volume /srv --name pmm-data percona/pmm-server:2 /bin/true
4dcc7c55603d01f4842ed524f5b7d983a67e16ff3d5a42dc84691d70c27eeba4

Run the container with:

[root@server1 tmp]# podman run --detach --restart always --publish 443:443 --volumes-from pmm-data --name pmm-server percona/pmm-server:2
5f5c4a6468f389951d1a44bfbf0f5492050c375d1bddcd2ab7fea25bcc791f45
[root@server1 tmp]# podman container list
CONTAINER ID  IMAGE                           COMMAND               CREATED         STATUS             PORTS                 NAMES
5f5c4a6468f3  docker.io/percona/pmm-server:2  /opt/entrypoint.s...  27 seconds ago  Up 25 seconds ago  0.0.0.0:443->443/tcp  pmm-server

Then you can access to https://192.168.56.101 (for me as I’m accessing my virtual server from my desktop), default login is admin/admin and you will immediately pormpted to change it.

Client

For the client I have chosen the rpm installation as it is by far the simplest to use. Download the rpm on your client server and install it with:

[root@server2 tmp]# dnf install pmm2-client-2.19.0-6.el8.x86_64.rpm
Last metadata expiration check: 0:45:32 ago on Mon 12 Jul 2021 11:05:40 AM CEST.
Dependencies resolved.
=========================================================================================================================================================================================================================================
 Package                                                  Architecture                                        Version                                                    Repository                                                 Size
=========================================================================================================================================================================================================================================
Installing:
 pmm2-client                                              x86_64                                              2.19.0-6.el8                                               @commandline                                               43 M

Transaction Summary
=========================================================================================================================================================================================================================================
Install  1 Package

Total size: 43 M
Installed size: 154 M
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                 1/1
  Running scriptlet: pmm2-client-2.19.0-6.el8.x86_64                                                                                                                                                                                 1/1
  Installing       : pmm2-client-2.19.0-6.el8.x86_64                                                                                                                                                                                 1/1
  Running scriptlet: pmm2-client-2.19.0-6.el8.x86_64                                                                                                                                                                                 1/1
  Verifying        : pmm2-client-2.19.0-6.el8.x86_64                                                                                                                                                                                 1/1

Installed:
  pmm2-client-2.19.0-6.el8.x86_64

Complete!
[root@server2 tmp]# pmm-admin --version
ProjectName: pmm-admin
Version: 2.19.0
PMMVersion: 2.19.0
Timestamp: 2021-06-30 11:31:50 (UTC)
FullCommit: 33d4f4a11ec6c46204d58e6ff6e08ad5742c8ae2

Register with the server repository with:

[root@server2 ~]# pmm-admin config --server-insecure-tls --server-url=https://admin:admin@192.168.56.101:443
Checking local pmm-agent status...
pmm-agent is running.
Registering pmm-agent on PMM Server...
Registered.
Configuration file /usr/local/percona/pmm2/config/pmm-agent.yaml updated.
Reloading pmm-agent configuration...
Configuration reloaded.
Checking local pmm-agent status...
pmm-agent is running.

On your client PostgreSQL instance create a pmm account (chose a strong password, not like me):

postgres=# CREATE USER pmm WITH SUPERUSER ENCRYPTED PASSWORD 'pmm';
CREATE ROLE

And update pg_hba.conf file to be able to connect with pmm account specifying a password:

[postgres@server2 data]$ grep local pg_hba.conf | grep -v "^#"
local   all             pmm                                     md5
local   replication     all                                     trust
local   all             all                                     trust

I have chosen to use pg_stat_statements for the monitoring extension, installed:

[root@server2 ~]# dnf install -y postgresql13-contrib.x86_64

Restart your PostgreSQL instance, check you can connect with pmm account and create extension with:

[postgres@server2 data]$ psql postgres pmm -c "\conninfo"
Password for user pmm:
You are connected to database "postgres" as user "pmm" via socket in "/var/run/postgresql" at port "5432".
[postgres@server1 data]$ psql
psql (13.3)
Type "help" for help.

postgres=# CREATE EXTENSION pg_stat_statements SCHEMA public;
CREATE EXTENSION

To add my client PostgreSQL instance with -postgresql as service name I have used:

[root@server2 ~]# pmm-admin add postgresql --username=pmm --password=pmm --server-url=https://admin:admin@192.168.56.101:443 --server-insecure-tls
PostgreSQL Service added.
Service ID  : /service_id/2516642c-3237-4ef3-810f-6c2ecb6ddd6c
Service name: server2.domain.com-postgresql
[root@server2 tmp]# pmm-admin inventory list services
Services list.

Service type           Service name         Address and Port  Service ID
PostgreSQL             server2.domain.com-postgresql 127.0.0.1:5432    /service_id/2516642c-3237-4ef3-810f-6c2ecb6ddd6c
PostgreSQL             pmm-server-postgresql 127.0.0.1:5432    /service_id/f7112f05-20e9-4933-8591-441fc93662f1

For a free product the look and the displayed informations are just awesome. Of course Grafana neat default look and feel helps but Percona have added a big bunch of cool features:

pmm01
pmm01
pmm02
pmm02
pmm03
pmm03

And obviously as it is Grafana there is zero limit in customization you can make…

PGWatch

I have obviously chosen the container installation and decided to use Podman that comes by default in my Oracle Linux distribution. I expected the installation to be seamless but at then end I have lost a couple of hours fighting with non working container. I have tried pgwatch2 and pgwatch2-postgres container but non of them worked and I had plenty of error like:

  • ERROR 209 name ‘requests’ is not defined
  • influxdb.exceptions.InfluxDBClientError: database not found: pgwatch2

I have decided to give a last try with image pull of pgwatch2-nonroot with:

[root@server1 ~]# podman  pull cybertec/pgwatch2-nonroot
✔ docker.io/cybertec/pgwatch2-nonroot:latest
Trying to pull docker.io/cybertec/pgwatch2-nonroot:latest...
Getting image source signatures
Copying blob 350caab5f3b5 skipped: already exists
Copying blob 49ac0bbe6c8e skipped: already exists
Copying blob 3386e6af03b0 skipped: already exists
Copying blob 1a0f3a523f04 skipped: already exists
Copying blob d1983a67e104 skipped: already exists
Copying blob 91056c4070cb skipped: already exists
Copying blob b23f24e6b1dd skipped: already exists
Copying blob 1ed2f1c72460 skipped: already exists
Copying blob effdfc7f950c skipped: already exists
Copying blob 9a055164fb69 skipped: already exists
Copying blob be763b7af1a3 skipped: already exists
Copying blob 70fa32c9c857 done
Copying blob 174f5722e61d done
Copying blob 8be6b6bc9759 done
Copying blob 7dea3ad5b533 done
Copying blob c7f6ad956dfc done
Copying blob 00e2d15bc136 done
Copying blob fe00b1e59788 done
Copying blob 40a688173fcd done
Copying config 196f099da1 done
Writing manifest to image destination
Storing signatures
196f099da17eb6bb328ed274a7435371969ec73e8c99599a47e8686f22c6f1cc

And run it with:

[root@server1 ~]# podman run -d --restart=unless-stopped --name pw2 -p 3000:3000 -p 8080:8080 -p 127.0.0.1:5432:5432 -e PW2_TESTDB=true cybertec/pgwatch2-nonroot:latest
4bd4150e3cb8991b3f9c4b24c2cc97973f5868bbf5dcbffa203e9e7c473fb465
[root@server1 ~]# podman container list -a
CONTAINER ID  IMAGE                               COMMAND               CREATED        STATUS            PORTS                                           NAMES
4bd4150e3cb8  docker.io/cybertec/pgwatch2:latest  /pgwatch2/docker-...  5 seconds ago  Up 3 seconds ago  0.0.0.0:3000->3000/tcp, 0.0.0.0:8080->8080/tcp  pw2

With chosen option you see the backend PostgreSQL instance of Pgwatch2 (port 3000):

pgwatch201
pgwatch201

On the instance you plan to monitor create a monitoring account with:

CREATE ROLE pgwatch2 WITH LOGIN PASSWORD 'secret';
ALTER ROLE pgwatch2 CONNECTION LIMIT 3;
GRANT pg_monitor TO pgwatch2;
GRANT CONNECT ON DATABASE pgbenchdb TO pgwatch2;
GRANT USAGE ON SCHEMA public TO pgwatch2;
GRANT EXECUTE ON FUNCTION pg_stat_file(text) to pgwatch2;

postgres=# show shared_preload_libraries;
 shared_preload_libraries
--------------------------
 pg_stat_statements
(1 row)

postgres=# show track_io_timing;
 track_io_timing
-----------------
 on
(1 row)

If required modify the pg_hba.conf file to allow remote connection using pgwatch2 account…

Then add the database in admin interface (port 8080) of Pgwatch2 (pg_stat_statements extention loaded in this database):

pgwatch202
pgwatch202

After a while you should see it in the dashboard:

pgwatch203
pgwatch203
pgwatch204
pgwatch204
pgwatch205
pgwatch205
pgwatch206
pgwatch206

Again, as based on Grafana, the web UI is really really neat. Same as Percona’s product customization is limitless. The product is agentless (for good or bad) so even simplify installation. On top of this the product is free and open source: what can you ask more ?

To be continued…

List of potential interesting candidates:

References

The post PostgreSQL graphical monitoring tools comparison appeared first on IT World.

]]>
https://blog.yannickjaquier.com/postgresql/postgresql-graphical-monitoring-tools-comparison.html/feed 0