Red Hat / Oracle Linux I/O scheduler tuning for Oracle databases

Preamble

Starting with Red Hat 4 you can choose between four I/O scheduler which have all their pros and cons versus what you run on your server:

  • noop
  • anticipatory
  • deadline
  • cfq

In Red Hat 6 the anticipatory I/O scheduler has disappeared…

There are multiple documents stating that deadline I/O scheduler is the preferred choice when running Oracle databases. cfq scheduler is the default one on Red Hat edition while Oracle has chosen to activate deadline scheduler by default on its Unbreakable Enterprise Kernel (uek) kernel branch. Which tend to say that it is the one to use when running Oracle databases.

You can change it, dynamically per disk device, by modifying below file:

[root@server1 ~]# cat /sys/block/sda/queue/scheduler
noop anticipatory deadline [cfq]
[root@server1 ~]# echo deadline > /sys/block/sda/queue/scheduler
[root@server1 ~]# cat /sys/block/sda/queue/scheduler
noop anticipatory [deadline] cfq

A bit fastidious, it is anyway possible to change it globally by changing in /etc/grub.conf the below block:

title Red Hat Enterprise Linux Server (2.6.18-274.3.1.el5)
        root (hd0,0)
        kernel /vmlinuz-2.6.18-274.3.1.el5 ro root=/dev/vg00/lvol1 crashkernel=128M@16M
        initrd /initrd-2.6.18-274.3.1.el5.img

By

        kernel /vmlinuz-2.6.18-274.3.1.el5 ro root=/dev/vg00/lvol1 crashkernel=128M@16M elevator=deadline

And then reboot your server…

To avoid this reboot Red Hat is proposing a tool called ktune in Red Hat 4 and 5 and tuned starting with Red Hat 6.

This blog post has been done using Red Hat Enterprise Linux Server release 5.8 (Tikanga) (Oracle Linux Server release 5.8) and Red Hat Enterprise Linux Server release 6.3 (Santiago) (Oracle Linux Server release 6.3) created as Virtual Machine using VirtualBox.

To update packages I have used Oracle Public Yum Server.

ktune (Red Hat 5)

From 5.3 Release Notes:

This release includes ktune (from the ktune package), a service that sets several kernel tuning parameters to values suitable for specific system profiles. Currently, ktune only provides a profile for large-memory systems running disk-intensive and network-intensive applications.
The settings provides by ktune do not override those set in /etc/sysctl.conf or through the kernel command line. ktune may not be suitable on some systems and workloads; as such, you should test it comprehensively before deploying to production.
You can disable any configuration set by ktune and revert to your normal settings by simply stopping the ktune service using service ktune stop (as root).

So clearly running databases and in my case running Oracle databases, unfortunately there is only one profile so if you are not doing databases on your Linux box you could be disappointed (this has been corrected in Red Hat 6, see next chapter):

One command installation:

[root@server1 ~]# yum install ktune
Loaded plugins: rhnplugin, security
This system is not registered with ULN.
ULN support will be disabled.
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package ktune.noarch 0:0.2-6.el5 set to be updated
--> Finished Dependency Resolution
 
Dependencies Resolved
 
===============================================================================================================================================================================================================
 Package                                        Arch                                            Version                                              Repository                                           Size
===============================================================================================================================================================================================================
Installing:
 ktune                                          noarch                                          0.2-6.el5                                            el5_latest                                           17 k
 
Transaction Summary
===============================================================================================================================================================================================================
Install       1 Package(s)
Upgrade       0 Package(s)
 
Total download size: 17 k
Is this ok [y/N]: y
Downloading Packages:
ktune-0.2-6.el5.noarch.rpm                                                                                                                                                              |  17 kB     00:00
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing     : ktune                                                                                                                                                                                   1/1
 
Installed:
  ktune.noarch 0:0.2-6.el5
 
Complete!

By looking in /etc/init.d/ktune file you can then dig in the two configuration files /etc/sysctl.ktune and /etc/sysconfig/ktune:

[root@server1 ~]# cat /etc/sysconfig/ktune
# ktune service configuration
 
# This is the ktune sysctl file.  You can comment this out to prevent ktune
# from applying its sysctl settings.
SYSCTL="/etc/sysctl.ktune"
 
# Use *.conf files in the ktune configuration directory /etc/ktune.d.
#   Value: yes|no,  default: yes
# It is useful if you want to load settings from additional files. Set this to
# no if you to prevent ktune from using these additional files.
USE_KTUNE_D="yes"
 
# This is the custom sysctl configuration file.  Any settings in this file will
# be applied after the ktune settings, overriding them.  Comment this out to
# use only the ktune settings.
SYSCTL_POST="/etc/sysctl.conf"
 
# This is the I/O scheduler ktune will use.  This will *not* override anything
# explicitly set on the kernel command line, nor will it change the scheduler
# for any block device that is using a non-default scheduler when ktune starts.
# You should probably leave this on "deadline", but "as", "cfq", and "noop" are
# also legal values.  Comment this out to prevent ktune from changing I/O
# scheduler settings.
ELEVATOR="deadline"
 
# These are the devices, that should be tuned with the ELEVATOR
ELEVATOR_TUNE_DEVS="/sys/block/{sd,cciss}*/queue/scheduler"
[root@server1 ~]# grep -v ^# /etc/sysctl.ktune
 
net.core.rmem_default = 262144
net.core.wmem_default = 262144
 
net.core.rmem_max = 8388608
net.core.wmem_max = 8388608
 
net.core.netdev_max_backlog = 10000
 
net.ipv4.tcp_rmem = 8192 87380 8388608
net.ipv4.tcp_wmem = 8192 65536 8388608
 
net.ipv4.udp_rmem_min = 16384
net.ipv4.udp_wmem_min = 16384
 
net.ipv4.tcp_mem = 8388608 12582912 16777216
net.ipv4.udp_mem = 8388608 12582912 16777216
 
vm.swappiness = 30
 
vm.dirty_ratio = 50
 
vm.pagecache = 90

Configure startup and start it with:

[root@server1 ~]# chkconfig --list ktune
ktune           0:off   1:off   2:off   3:off   4:off   5:off   6:off
[root@server1 ~]# service ktune start
Applying ktune sysctl settings:
/etc/sysctl.ktune:                                         [FAILED]
  error: "vm.pagecache" is an unknown key
 
Applying sysctl settings from /etc/sysctl.conf:            [  OK  ]
Applying deadline elevator: sda                            [  OK  ]

Strange to receive this alert as vm.pagecache as been replaced by vm.swappiness since apparently Kernel 2.4, and for sure in Kernel 2.6:

[root@server1 ~]# ll /proc/sys/vm/pagecaches
ls: /proc/sys/vm/pagecaches: No such file or directory
[root@server1 ~]# ll /proc/sys/vm/swappiness
-rw-r--r-- 1 root root 0 Jul 18 15:12 /proc/sys/vm/swappiness

Edit and comment-out last line of /etc/sysctl.ktune then stop and restart ktune daemon:

[root@server1 ~]# service ktune stop
Reverting to saved sysctl settings:                        [  OK  ]
Reverting to cfq elevator: sda                             [  OK  ]
[root@server1 ~]# service ktune start
Applying ktune sysctl settings:
/etc/sysctl.ktune:                                         [  OK  ]
Applying sysctl settings from /etc/sysctl.conf:            [  OK  ]
Applying deadline elevator: sda                            [  OK  ]

Get its status with:

[root@server1 ~]# service ktune status
Current ktune sysctl settings:
net.core.rmem_default = 262144
net.core.wmem_default = 262144
net.core.rmem_max = 8388608
net.core.wmem_max = 8388608
net.core.netdev_max_backlog = 10000
net.ipv4.tcp_rmem = 8192        87380   8388608
net.ipv4.tcp_wmem = 8192        65536   8388608
net.ipv4.udp_rmem_min = 16384
net.ipv4.udp_wmem_min = 16384
net.ipv4.tcp_mem = 8388608      12582912        16777216
net.ipv4.udp_mem = 8388608      12582912        16777216
vm.swappiness = 30
vm.dirty_ratio = 50
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
 
Current elevator settings:
/sys/block/sda/queue/scheduler: noop anticipatory [deadline] cfq

Remark:
Notice the deadline scheduler applied to all your disk devices…

tuned (Red Hat 6)

In this edition Red Hat has drastically improved ktune and created tuned utility which has now multiple profiles to adapt your system to your workload. One command installation is:

[root@server1 ~]# yum install tuned
Loaded plugins: refresh-packagekit, security
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package tuned.noarch 0:0.2.19-7.el6 will be installed
--> Finished Dependency Resolution
 
Dependencies Resolved
 
===============================================================================================================================================================================================================
 Package                                       Arch                                           Version                                                 Repository                                          Size
===============================================================================================================================================================================================================
Installing:
 tuned                                         noarch                                         0.2.19-7.el6                                            ol6_latest                                          86 k
 
Transaction Summary
===============================================================================================================================================================================================================
Install       1 Package(s)
 
Total download size: 86 k
Installed size: 208 k
Is this ok [y/N]: y
Downloading Packages:
tuned-0.2.19-7.el6.noarch.rpm                                                                                                                                                           |  86 kB     00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : tuned-0.2.19-7.el6.noarch                                                                                                                                                                   1/1
  Verifying  : tuned-0.2.19-7.el6.noarch                                                                                                                                                                   1/1
 
Installed:
  tuned.noarch 0:0.2.19-7.el6
 
Complete!

Configure startup and start it:

[root@server1 ~]# chkconfig --list tuned
tuned           0:off   1:off   2:off   3:off   4:off   5:off   6:off
[root@server1 ~]# service tuned start
Starting tuned:                                            [  OK  ]

List available profiles:

[root@server1 ~]# tuned-adm list
Available profiles:
- server-powersave
- desktop-powersave
- latency-performance
- enterprise-storage
- laptop-ac-powersave
- virtual-guest
- spindown-disk
- throughput-performance
- virtual-host
- laptop-battery-powersave
- default
Current active profile: default

To see which one is activated and its configuration files:

[root@server1 ~]# tuned-adm active
Current active profile: default
Service tuned: disabled, running
Service ktune: disabled, stopped
[root@server1 ~]# cat /etc/tune-profiles/active-profile
default
[root@server1 ~]# ll /etc/tune-profiles/default/
total 12
-rw-r--r-- 1 root root 1179 Aug 18  2010 ktune.sysconfig
-rw-r--r-- 1 root root   15 Aug 18  2010 sysctl.ktune
-rw-r--r-- 1 root root  970 Aug 18  2010 tuned.conf

Red Hat suggest to use enterprise-storage when running Oracle databases:

[root@server1 ~]# tuned-adm profile enterprise-storage
Stopping tuned:                                            [  OK  ]
Switching to profile 'enterprise-storage'
Applying ktune sysctl settings:
/etc/ktune.d/tunedadm.conf:                                [  OK  ]
Calling '/etc/ktune.d/tunedadm.sh start':                  [  OK  ]
Applying sysctl settings from /etc/sysctl.conf
Applying deadline elevator: dm-0 dm-1 dm-2 dm-3 dm-4 dm-5 d[  OK  ]
Starting tuned:                                            [  OK  ]
[root@server1 ~]# tuned-adm active
Current active profile: enterprise-storage
Service tuned: enabled, running
Service ktune: enabled, running

The list of available profile are the ones of /etc/tune-profiles/ directory. To create your own profile you can copy one of the pre-existing and customize it.

References

About Post Author

Share the knowledge!

2 thoughts on “Red Hat / Oracle Linux I/O scheduler tuning for Oracle databases

  1. Is it possible to create custom profile from tuned-adm profile enterprise-storage with transparent hugepages disabled.
    According some documentation the enterprise-storage profile enable transparent hugepages – and this should be disabled (recommended by oracle) ?

    thx, jochen

    • Yes accordingly to:
      ALERT: Disable Transparent HugePages on SLES11, RHEL6, OL6 and UEK2 Kernels (Doc ID 1557478.1)

      It is recommended to disable Tranaparent HugePages if running an Oracle database.

      I don’t know if it is the cleanest method but if you modify ktune.sh in /etc/tune-profiles/ then you can change any kernel parameter you like…

      Thanks, Yannick.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes:

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>