Non-Uniform Memory Access (NUMA) architecture with Oracle – part 2

Posted on , updated on by Yannick Jaquier
HPUX, Linux, Oracle

Table of contents

Preamble

After first blog post on Non-Uniform Memory Access (NUMA) I have been shared by teammates few interesting articles (see references) and so wanted to go a bit deeper on this subject before definitively closing it (you will see in conclusion below why).

I have been deeper in NUMA details on both Itanium 11iv2 (11.23) and Linux RedHat 5.5 (Tikanga). The Oracle database running on Itanium is a 10.2.0.4 while my Linux box run multiple releases (10.2.0.4, 10.2.0.3, 11.2.0.1, 11.2.0.2, …).

NUMA on HPUX Itanium

The server on which I’m working is the same a previous NUMA post:/

server1{root}# model
ia64 hp server rx8640
server1{root}# mpsched -s
System Configuration
=====================
 
Locality Domain Count: 2
Processor Count      : 4
 
Domain    Processors
------    ----------
   0        0
   1        7   8   9

This server is in fact an nPAR of a bigger physical server:

server1{root}# parstatus -w
The local partition number is 1.
server1{root}# parstatus -V -p 1
[Partition]
Partition Number       : 1
Partition Name         : server2 server1
Status                 : Active
IP Address             :
Primary Boot Path      : 3/0/2/1/0/4/1.0.0
Alternate Boot Path    : 3/0/4/1/0/4/1.0.0
HA Alternate Boot Path :
PDC Revision           : 9.48
IODCH Version          : ffff
Cell Architecture      : Itanium(R)-based
CPU Compatibility      : CDH-640
CPU Speed              : 1598 MHz
Core Cell              : cab0,cell2
Core Cell Choice [0]   : cab0,cell2
Total Good Memory Size : 64.0 GB
Total Interleave Memory: 64.0 GB
Total Requested CLM    : 0.0 GB
Total Allocated CLM    : 0.0 GB
Hyperthreading Enabled : no
 
 
[Cell]
                        CPU     Memory                                Use
                        OK/     (GB)                          Core    On
Hardware   Actual       Deconf/ OK/                           Cell    Next Par
Location   Usage        Max     Deconf    Connected To        Capable Boot Num
========== ============ ======= ========= =================== ======= ==== ===
cab0,cell2 Active Core  6/0/8   32.0/0.0  cab8,bay0,chassis0  yes     yes  1
cab0,cell3 Active Base  4/0/8   32.0/0.0  cab8,bay0,chassis1  yes     yes  1
 
Notes: * = Cell has no interleaved memory.
 
 
[Chassis]
                                 Core Connected  Par
Hardware Location   Usage        IO   To         Num
=================== ============ ==== ========== ===
cab8,bay0,chassis0  Active       yes  cab0,cell2 1
 
 
[Chassis]
                                 Core Connected  Par
Hardware Location   Usage        IO   To         Num
=================== ============ ==== ========== ===
cab8,bay0,chassis1  Active       yes  cab0,cell3 1

On top of the fact that my cells are not CPU equally balanced no Cell Local Memory (CLM) has been configured (default on older HP servers is: 100% ILM, 0% CLM so behave of default configuration). The generic HP advice is to configure 7/8 of available memory as CLM.

Then using the following shell commands you are able to see to which processor are attached the Oracle background and slave processes:

server1{root}# top -f /tmp/top_output -n 2000
server1{root}# grep ora\_ /tmp/top_output

Then HP states that parallel slaves and database writer should be evenly distributed on cells. They are all attached to cell 0 (and so to unique CPU 0) in my case so another *strange* behavior… Behavior I have not seen on antoher Itanium box where each cell was running an database writer and archive process.

Then reading further HP clarify it once for all:

For Oracle Database version 10.2.0.4, the optimizations are not implemented correctly and should never be used: _enable_NUMA_optimization should always be set to false (and the server should thus be configured ILM-heavy) for Oracle 10.2.0.4

Notice that you can choose on which cell (or processor, option mutually exclusive with locality-domain one) you want to run your process. Even if in my small example it has no interest HP suggest to investigate the LL (Least Loaded) policy for Oracle listener:

server1{root}# mpsched -c 9 -O LL sleep 60 &
[1]     28198
server1{root}# Pid 28198: bound to processor 9 using the least loaded process launch policy

Then check it is using the CPU you expected:

server1{root}# top -f /tmp/top_output -n 2000
server1{root}# grep 28198 /tmp/top_output
 9 pts/6 28198 root     168 24  3724K   264K sleep    0:00  0.00  0.00 sleep

Then what about Oracle ? Here below what I get with my 10.2.0.4 instance running on Itanium (_enable_NUMA_optimization set to true by default):

SQL> SET lines 150 pages 100
SQL> col description FOR a70
SQL> col  PARAMETER FOR a30
SQL> col VALUE FOR a15
SQL> SELECT
  a.ksppinm AS parameter,
  c.ksppstvl AS VALUE,
  a.ksppdesc AS description,
  b.ksppstdf AS "Default?"
FROM x$ksppi a, x$ksppcv b, x$ksppsv c
WHERE a.indx = b.indx
AND a.indx = c.indx
AND LOWER(a.ksppinm) LIKE '%numa%'
ORDER BY a.ksppinm;
 
PARAMETER                      VALUE           DESCRIPTION                                                            DEFAULT?
------------------------------ --------------- ---------------------------------------------------------------------- ---------
_NUMA_instance_mapping         NOT specified   SET OF nodes that this instance should run ON                          TRUE
_NUMA_pool_size                NOT specified   aggregate SIZE IN bytes OF NUMA pool                                   TRUE
_db_block_numa                 2               NUMBER OF NUMA nodes                                                   TRUE
_enable_NUMA_optimization      TRUE            Enable NUMA specific optimizations                                     TRUE
_rm_numa_sched_enable          FALSE           IS RESOURCE Manager (RM) related NUMA scheduled policy enabled         TRUE
_rm_numa_simulation_cpus       0               NUMBER OF cpus per PG FOR numa simulation IN RESOURCE manager          TRUE
_rm_numa_simulation_pgs        0               NUMBER OF PGs FOR numa simulation IN RESOURCE manager                  TRUE
 
7 ROWS selected.

The display in alert.log file is also difficult to catch:

.
Opening with internal Resource Manager plan
where NUMA PG = 2, CPUs = 1
.

NUMA on Linux

As a reminder use numactl command to display hardware information (the server behind is an HP BL620c G7 with 2 CPUs (Intel(R) Xeon(R) CPU X7560 @ 2.27GHz) of 8 cores each, so 32 threads if you activate hyper-threading):

[root@server1 ~]# numactl --hardware
available: 2 nodes (0-1)
node 0 size: 32289 MB
node 0 free: 2595 MB
node 1 size: 32320 MB
node 1 free: 267 MB
node distances:
node   0   1
  0:  10  20
  1:  20  10

And how NUMA settings:

[root@server1 ~]# numactl --show
policy: default
preferred node: current
physcpubind: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
cpubind: 0 1
nodebind: 0 1
membind: 0 1

In my previous post I was wondering how NUMA got implemented with x86 technology. In short it is coming from Intel/AMD technologies. Intel QuickPath Interconnect (QuickPath, QPI), previously call Common System Interface (CSI) for Intel. HyperTransport (HT), earlier Lightning Data Transport (LDT) for AMD. Refer to Wikipedia for more information.

On Linux the top command does not print the CPU id where is running the process, fortunately the ps command is enhanced and you can have it using a custom display (psr column). The example database below is a 11.2.0.2 release (where NUMA is off by default):

[root@server1 ~]$ ps -eo psr,sgi_p,pid,user,rtprio,ni,time,%cpu,cmd | egrep 'PSR|oratlste'
PSR P   PID USER     RTPRIO  NI     TIME %CPU CMD
  1 *  6223 oratlste      -   0 00:00:00  0.0 ora_w000_tlste
  1 * 12773 oratlste      -   0 00:00:02  0.0 ora_pmon_tlste
 11 * 12775 oratlste      -   0 00:00:00  0.0 ora_psp0_tlste
 18 * 12782 oratlste      -   0 00:00:00  0.0 ora_vktm_tlste
 16 * 12786 oratlste      -   0 00:00:00  0.0 ora_gen0_tlste
  0 * 12788 oratlste      -   0 00:00:00  0.0 ora_diag_tlste
 10 * 12790 oratlste      -   0 00:00:00  0.0 ora_dbrm_tlste
 23 * 12792 oratlste      -   0 00:01:19  0.0 ora_dia0_tlste
 15 * 12794 oratlste      -   0 00:00:00  0.0 ora_mman_tlste
 18 * 12796 oratlste      -   0 00:00:01  0.0 ora_dbw0_tlste
 18 * 12798 oratlste      -   0 00:00:01  0.0 ora_dbw1_tlste
 17 * 12800 oratlste      -   0 00:00:00  0.0 ora_dbw2_tlste
 18 * 12802 oratlste      -   0 00:00:00  0.0 ora_dbw3_tlste
  2 * 12804 oratlste      -   0 00:00:02  0.0 ora_lgwr_tlste
  3 * 12807 oratlste      -   0 00:00:16  0.0 ora_ckpt_tlste
 15 * 12809 oratlste      -   0 00:00:09  0.0 ora_smon_tlste
 26 * 12811 oratlste      -   0 00:00:00  0.0 ora_reco_tlste
  3 * 12813 oratlste      -   0 00:00:08  0.0 ora_mmon_tlste
  7 * 12815 oratlste      -   0 00:00:52  0.0 ora_mmnl_tlste
 24 * 12841 oratlste      -   0 00:00:00  0.0 ora_qmnc_tlste
  2 * 12917 oratlste      -   0 00:00:00  0.0 /ora_tlste/software/bin/tnslsnr LISTENER_tlste -inherit
  6 * 13018 oratlste      -   0 00:00:01  0.0 ora_q000_tlste
 10 * 13020 oratlste      -   0 00:00:00  0.0 ora_q001_tlste
  9 * 29779 oratlste      -   0 00:00:00  0.0 ora_smco_tlste
.
.

The cell affinity can be displayed using /proc/pid/numa_maps file (default affinity by default 🙂 ):

[root@server1 ~]# echo $$
20518
[root@server1 ~]# cat /proc/20518/numa_maps
00400000 default file=/bin/bash mapped=118 mapmax=38 N0=96 N1=22
006b2000 default file=/bin/bash anon=10 dirty=10 active=9 N0=10
006bc000 default anon=5 dirty=5 active=3 N0=5
008bb000 default file=/bin/bash mapped=2 mapmax=33 N0=2
06116000 default heap anon=65 dirty=65 active=58 N0=65
320dc00000 default file=/lib64/ld-2.5.so mapped=25 mapmax=868 N0=25
320de1c000 default file=/lib64/ld-2.5.so anon=1 dirty=1 N0=1
320de1d000 default file=/lib64/ld-2.5.so anon=1 dirty=1 N0=1
320e000000 default file=/lib64/libc-2.5.so mapped=123 mapmax=877 N0=123
320e14e000 default file=/lib64/libc-2.5.so
320e34e000 default file=/lib64/libc-2.5.so anon=1 dirty=1 mapped=4 mapmax=839 N0=4
320e352000 default file=/lib64/libc-2.5.so anon=1 dirty=1 N0=1
320e353000 default anon=5 dirty=5 N0=5
320e400000 default file=/lib64/libdl-2.5.so mapped=2 mapmax=841 N0=2
320e402000 default file=/lib64/libdl-2.5.so
320e602000 default file=/lib64/libdl-2.5.so anon=1 dirty=1 N0=1
320e603000 default file=/lib64/libdl-2.5.so anon=1 dirty=1 N0=1
320e800000 default file=/lib64/libtermcap.so.2.0.8 mapped=3 mapmax=35 N0=3
320e803000 default file=/lib64/libtermcap.so.2.0.8
320ea02000 default file=/lib64/libtermcap.so.2.0.8 anon=1 dirty=1 N0=1
2af022372000 default anon=2 dirty=2 N0=2
2af022384000 default anon=2 dirty=2 N0=2
2af022386000 default file=/lib64/libnss_files-2.5.so mapped=5 mapmax=810 N0=5
2af022390000 default file=/lib64/libnss_files-2.5.so
2af02258f000 default file=/lib64/libnss_files-2.5.so anon=1 dirty=1 N0=1
2af022590000 default file=/lib64/libnss_files-2.5.so anon=1 dirty=1 N0=1
2af022591000 default file=/usr/lib/locale/locale-archive mapped=15 mapmax=321 N0=15
2af025b5d000 default file=/usr/lib64/gconv/gconv-modules.cache mapped=6 mapmax=112 N0=6
2af025b64000 default anon=2 dirty=2 N0=2
7fffb4eb2000 default stack anon=7 dirty=7 N0=7
7fffb4f67000 default mapped=1 mapmax=821 active=0 N0=1

Remark:
As clearly stated by RedHat it is not recommended to display this file on production system because of possible side effects…

An example using memory on node 1 (whenever possible):

[root@server1 ~]# numactl --preferred=1 sleep 30 &
[1] 17819
[root@server1 ~]# cat /proc/17819/numa_maps
00400000 prefer=1 file=/bin/sleep mapped=4 N0=4
00604000 prefer=1 file=/bin/sleep anon=1 dirty=1 N1=1
0ddad000 prefer=1 heap anon=2 dirty=2 N1=2
320dc00000 prefer=1 file=/lib64/ld-2.5.so mapped=23 mapmax=872 N0=23
320de1c000 prefer=1 file=/lib64/ld-2.5.so anon=1 dirty=1 N1=1
320de1d000 prefer=1 file=/lib64/ld-2.5.so anon=1 dirty=1 N1=1
320e000000 prefer=1 file=/lib64/libc-2.5.so mapped=73 mapmax=880 N0=73
320e14e000 prefer=1 file=/lib64/libc-2.5.so
320e34e000 prefer=1 file=/lib64/libc-2.5.so anon=1 dirty=1 mapped=4 mapmax=846 N0=3 N1=1
320e352000 prefer=1 file=/lib64/libc-2.5.so anon=1 dirty=1 N1=1
320e353000 prefer=1 anon=4 dirty=4 N1=4
2b4cc8f50000 prefer=1 anon=2 dirty=2 N1=2
2b4cc8f62000 prefer=1 anon=1 dirty=1 N1=1
2b4cc8f63000 prefer=1 file=/usr/lib/locale/locale-archive mapped=12 mapmax=318 N0=12
7fff644be000 prefer=1 stack anon=3 dirty=3 N1=3
7fff6451f000 prefer=1 mapped=1 mapmax=826 active=0 N0=1

N{x} is the node where memory has been allocated. And number is size in number of pages (4KB).

Using migratepages command you can move pages from one node to another. First let’s cerate a process using node 1

[root@server1 ~]# numactl --preferred=1 sleep 300 &
[1] 26021

We can see that 2 pages have been allocated on node 1 (refer=1 stack anon=2 dirty=2 N1=2):

[root@server1 ~]# cat /proc/26021/numa_maps
00400000 prefer=1 file=/bin/sleep mapped=4 N1=4
00604000 prefer=1 file=/bin/sleep anon=1 dirty=1 N1=1
146f0000 prefer=1 heap anon=2 dirty=2 N1=2
35bec00000 prefer=1 file=/lib64/ld-2.5.so mapped=23 mapmax=51 N0=23
35bee1c000 prefer=1 file=/lib64/ld-2.5.so anon=1 dirty=1 N1=1
35bee1d000 prefer=1 file=/lib64/ld-2.5.so anon=1 dirty=1 N1=1
35bf000000 prefer=1 file=/lib64/libc-2.5.so mapped=73 mapmax=58 N0=73
35bf14e000 prefer=1 file=/lib64/libc-2.5.so
35bf34e000 prefer=1 file=/lib64/libc-2.5.so anon=1 dirty=1 mapped=4 mapmax=39 N0=3 N1=1
35bf352000 prefer=1 file=/lib64/libc-2.5.so anon=1 dirty=1 N1=1
35bf353000 prefer=1 anon=4 dirty=4 N1=4
2abc9ff58000 prefer=1 anon=2 dirty=2 N1=2
2abc9ff6c000 prefer=1 anon=1 dirty=1 N1=1
2abc9ff6d000 prefer=1 file=/usr/lib/locale/locale-archive mapped=11 mapmax=38 N0=11
7fff9689f000 prefer=1 stack anon=2 dirty=2 N1=2
7fff9698d000 prefer=1 mapped=1 mapmax=41 active=0 N1=1

Let’s migrate the process to node 0:

[root@server1 ~]# migratepages 26021 1 0

The memory pages are now located on node 0 (prefer=1 stack anon=2 dirty=2 N0=2):

[root@server1 ~]# cat /proc/26021/numa_maps
00400000 prefer=1 file=/bin/sleep mapped=4 N0=4
00604000 prefer=1 file=/bin/sleep anon=1 dirty=1 N0=1
146f0000 prefer=1 heap anon=2 dirty=2 N0=2
35bec00000 prefer=1 file=/lib64/ld-2.5.so mapped=23 mapmax=51 N0=23
35bee1c000 prefer=1 file=/lib64/ld-2.5.so anon=1 dirty=1 N0=1
35bee1d000 prefer=1 file=/lib64/ld-2.5.so anon=1 dirty=1 N0=1
35bf000000 prefer=1 file=/lib64/libc-2.5.so mapped=73 mapmax=58 N0=73
35bf14e000 prefer=1 file=/lib64/libc-2.5.so
35bf34e000 prefer=1 file=/lib64/libc-2.5.so anon=1 dirty=1 mapped=4 mapmax=39 N0=4
35bf352000 prefer=1 file=/lib64/libc-2.5.so anon=1 dirty=1 N0=1
35bf353000 prefer=1 anon=4 dirty=4 N0=4
2abc9ff58000 prefer=1 anon=2 dirty=2 N0=2
2abc9ff6c000 prefer=1 anon=1 dirty=1 N0=1
2abc9ff6d000 prefer=1 file=/usr/lib/locale/locale-archive mapped=11 mapmax=38 N0=11
7fff9689f000 prefer=1 stack anon=2 dirty=2 N0=2
7fff9698d000 prefer=1 mapped=1 mapmax=41 active=0 N1=1

For my Oracle 11.2.0.2 test database, if I set _enable_NUMA_support to true (in 11gR2 _enable_NUMA_optimization is deprecated and you must use _enable_NUMA_support):

SQL> SET lines 150 pages 100
SQL> col description FOR a70
SQL> col  PARAMETER FOR a30
SQL> col VALUE FOR a15
SQL> SELECT
  a.ksppinm AS parameter,
  c.ksppstvl AS VALUE,
  a.ksppdesc AS description,
  b.ksppstdf AS "Default?"
FROM x$ksppi a, x$ksppcv b, x$ksppsv c
WHERE a.indx = b.indx
AND a.indx = c.indx
AND LOWER(a.ksppinm) LIKE '%numa%'
ORDER BY a.ksppinm;
 
PARAMETER                      VALUE           DESCRIPTION                                                            DEFAULT?
------------------------------ --------------- ---------------------------------------------------------------------- ---------
_NUMA_instance_mapping         NOT specified   SET OF nodes that this instance should run ON                          TRUE
_NUMA_pool_size                NOT specified   aggregate SIZE IN bytes OF NUMA pool                                   TRUE
_db_block_numa                 2               NUMBER OF NUMA nodes                                                   TRUE
_enable_NUMA_interleave        TRUE            Enable NUMA interleave MODE                                            TRUE
_enable_NUMA_optimization      FALSE           Enable NUMA specific optimizations                                     TRUE
_enable_NUMA_support           TRUE            Enable NUMA support AND optimizations                                  FALSE
_numa_buffer_cache_stats       0               Configure NUMA buffer cache stats                                      TRUE
_numa_trace_level              0               numa trace event                                                       TRUE
_rm_numa_sched_enable          TRUE            IS RESOURCE Manager (RM) related NUMA scheduled policy enabled         TRUE
_rm_numa_simulation_cpus       0               NUMBER OF cpus FOR each pg FOR numa simulation IN RESOURCE manager     TRUE
_rm_numa_simulation_pgs        0               NUMBER OF PGs FOR numa simulation IN RESOURCE manager                  TRUE
 
11 ROWS selected.

The display in alert.log file is more verbose in 11gR2 and it i clearly showing that NUMA is in action (please note I also use HugePages):

.
Numa pool size adjusted
****************** Huge Pages Information *****************
Huge Pages memory pool detected (total: 10395 free: 7995)
DFLT Huge Pages allocation successful (allocated: 146)
NUMA Huge Pages allocation on node (1) (allocated: 186)
NUMA Huge Pages allocation on node (0) (allocated: 182)
DFLT Huge Pages allocation successful (allocated: 1)
***********************************************************
SGA Local memory support enabled
.
.
NUMA system found and support enabled (2 domains - 16,16)
.

As expected the SGA is now split in multiple part:

[root@server1 ~]# ipcs -ma | grep oratlste
0x00000000 1614250006 oratlste  660        306184192  22
0x00000000 1614282778 oratlste  660        390070272  22
0x00000000 1614315548 oratlste  660        381681664  22
0x2c40136c 1614348317 oratlste  660        2097152    22
0x780fe068 630849560  oratlste  660        204

I have tried with a 10.2.0.3 database but even setting explicitly in init.ora below parameters:

_db_block_numa                 2               Number of NUMA nodes                                                   FALSE
_enable_NUMA_optimization      TRUE            Enable NUMA specific optimizations                                     FALSE

The alert.log file does not display anything so really difficult, not to say impossible, to know if NUMA is active or not… I finally got my answer checking the memory segments (so NUMA not in use and I can’t say why):

[root@server1 ~]# ipcs -ma |grep oratrema
0x323c247c 1643937813 oratrema  600        1075838976 18
0xdadb2270 700547097  oratrema  640        154

Additionally the numastat command display few interesting NUMA related counters:

[root@server1 ~]# numastat
                           node0           node1
numa_hit              7498337258      7809234537
numa_miss              123022896       106050915
numa_foreign           106050915       123022896
interleave_hit          16454618        16108121
local_node            7487872474      7801578924
other_node             133487680       113706528

Another funny command is numademo that apparently aim at providing performance of your NUMA system:

[root@server1 ~]# numademo 10m random
2 nodes available
memory with no policy random              Avg 473.38 MB/s Min 698.12 MB/s Max 256.07 MB/s
local memory random                       Avg 682.44 MB/s Min 694.56 MB/s Max 601.49 MB/s
memory interleaved on all nodes random    Avg 677.94 MB/s Min 695.71 MB/s Max 560.02 MB/s
memory on node 0 random                   Avg 680.93 MB/s Min 694.15 MB/s Max 594.03 MB/s
memory on node 1 random                   Avg 667.67 MB/s Min 689.99 MB/s Max 544.57 MB/s
memory interleaved on 0 1 random          Avg 675.53 MB/s Min 692.36 MB/s Max 565.09 MB/s
setting preferred node to 0
memory without policy random              Avg 672.54 MB/s Min 692.04 MB/s Max 553.75 MB/s
setting preferred node to 1
memory without policy random              Avg 678.79 MB/s Min 698.35 MB/s Max 550.35 MB/s
manual interleaving to all nodes random   Avg 680.77 MB/s Min 697.01 MB/s Max 570.59 MB/s
manual interleaving on node 0/1 random    Avg 681.65 MB/s Min 697.24 MB/s Max 578.78 MB/s
current interleave node 0
running on node 0, preferred node 0
local memory random                       Avg 674.03 MB/s Min 688.13 MB/s Max 582.70 MB/s
memory interleaved on all nodes random    Avg 682.52 MB/s Min 696.40 MB/s Max 587.21 MB/s
memory interleaved on node 0/1 random     Avg 681.68 MB/s Min 694.01 MB/s Max 595.24 MB/s
alloc on node 1 random                    Avg 677.95 MB/s Min 694.47 MB/s Max 565.39 MB/s
local allocation random                   Avg 676.42 MB/s Min 692.72 MB/s Max 565.00 MB/s
setting wrong preferred node random       Avg 679.50 MB/s Min 696.17 MB/s Max 578.14 MB/s
setting correct preferred node random     Avg 674.12 MB/s Min 692.68 MB/s Max 550.14 MB/s
running on node 1, preferred node 0
local memory random                       Avg 613.30 MB/s Min 697.84 MB/s Max 301.31 MB/s
memory interleaved on all nodes random    Avg 680.04 MB/s Min 695.39 MB/s Max 574.03 MB/s
memory interleaved on node 0/1 random     Avg 683.59 MB/s Min 696.45 MB/s Max 598.16 MB/s
alloc on node 0 random                    Avg 674.49 MB/s Min 695.76 MB/s Max 548.22 MB/s
local allocation random                   Avg 674.20 MB/s Min 696.64 MB/s Max 552.78 MB/s
setting wrong preferred node random       Avg 672.26 MB/s Min 694.84 MB/s Max 525.68 MB/s
setting correct preferred node random     Avg 679.85 MB/s Min 696.77 MB/s Max 567.47 MB/s

Conclusion

Then all these for what ? Googling around the gain you may see having a fully compliant chain of software and hardware NUMA aware components may bring you up to 10% improvement or none or even a drawback. So clearly as Oracle corporation state do perform exhaustive test before activating it. Then I tend to say that generic advice is to deactivate it at hardware level.

If we take 11gR2 example where it is not activated by default is there a drawback letting it activated at hardware level ?

On Itanium relying to default value is worst situation and I don’t know if you can deactivate it at hardware level.

Finally even Oracle database does not look too much mature on this feature in 10gR2 (no obsolete by the way) so better play with it in 11gR2.

I can’t resist quoting excellent blog post from Jeremy Cole on MySQL (not too far from Oracle behavior), where he is suggesting to interleave memory for commands starting MySQL…

References

About Post Author

Yannick Jaquier

administrator

Find more about me on social media.

See author's posts

Share the knowledge!

Related Posts:

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes:

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>