After first blog post on Non-Uniform Memory Access (NUMA) I have been shared by teammates few interesting articles (see references) and so wanted to go a bit deeper on this subject before definitively closing it (you will see in conclusion below why).
I have been deeper in NUMA details on both Itanium 11iv2 (11.23) and Linux RedHat 5.5 (Tikanga). The Oracle database running on Itanium is a 10.2.0.4 while my Linux box run multiple releases (10.2.0.4, 10.2.0.3, 11.2.0.1, 11.2.0.2, …).
NUMA on HPUX Itanium
The server on which I’m working is the same a previous NUMA post:/
server1{root}# model
ia64 hp server rx8640
server1{root}# mpsched -s
System Configuration
=====================
Locality Domain Count: 2
Processor Count : 4
Domain Processors
----------------001789
server1{root}# model
ia64 hp server rx8640
server1{root}# mpsched -s
System Configuration
=====================Locality Domain Count: 2
Processor Count : 4Domain Processors
------ ----------
0 0
1 7 8 9
This server is in fact an nPAR of a bigger physical server:
server1{root}# parstatus -w
The local partition number is 1.
server1{root}# parstatus -w
The local partition number is 1.
server1{root}# parstatus -V -p 1[Partition]
Partition Number : 1
Partition Name : server2 server1
Status : Active
IP Address :
Primary Boot Path : 3/0/2/1/0/4/1.0.0
Alternate Boot Path : 3/0/4/1/0/4/1.0.0
HA Alternate Boot Path :
PDC Revision : 9.48
IODCH Version : ffff
Cell Architecture : Itanium(R)-based
CPU Compatibility : CDH-640
CPU Speed : 1598 MHz
Core Cell : cab0,cell2
Core Cell Choice [0] : cab0,cell2
Total Good Memory Size : 64.0 GB
Total Interleave Memory: 64.0 GB
Total Requested CLM : 0.0 GB
Total Allocated CLM : 0.0 GB
Hyperthreading Enabled : no
[Cell]
CPU Memory Use
OK/(GB) Core On
Hardware Actual Deconf/ OK/ Cell Next Par
Location Usage Max Deconf Connected To Capable Boot Num
========== ============ ======= ========= =================== ======= ==== ===
cab0,cell2 Active Core 6/0/832.0/0.0 cab8,bay0,chassis0 yesyes1
cab0,cell3 Active Base 4/0/832.0/0.0 cab8,bay0,chassis1 yesyes1
Notes: * = Cell has no interleaved memory.
[Chassis]
Core Connected Par
Hardware Location Usage IO To Num
=================== ============ ==== ========== ===
cab8,bay0,chassis0 Active yes cab0,cell2 1[Chassis]
Core Connected Par
Hardware Location Usage IO To Num
=================== ============ ==== ========== ===
cab8,bay0,chassis1 Active yes cab0,cell3 1
server1{root}# parstatus -V -p 1
[Partition]
Partition Number : 1
Partition Name : server2 server1
Status : Active
IP Address :
Primary Boot Path : 3/0/2/1/0/4/1.0.0
Alternate Boot Path : 3/0/4/1/0/4/1.0.0
HA Alternate Boot Path :
PDC Revision : 9.48
IODCH Version : ffff
Cell Architecture : Itanium(R)-based
CPU Compatibility : CDH-640
CPU Speed : 1598 MHz
Core Cell : cab0,cell2
Core Cell Choice [0] : cab0,cell2
Total Good Memory Size : 64.0 GB
Total Interleave Memory: 64.0 GB
Total Requested CLM : 0.0 GB
Total Allocated CLM : 0.0 GB
Hyperthreading Enabled : no[Cell]
CPU Memory Use
OK/ (GB) Core On
Hardware Actual Deconf/ OK/ Cell Next Par
Location Usage Max Deconf Connected To Capable Boot Num
========== ============ ======= ========= =================== ======= ==== ===
cab0,cell2 Active Core 6/0/8 32.0/0.0 cab8,bay0,chassis0 yes yes 1
cab0,cell3 Active Base 4/0/8 32.0/0.0 cab8,bay0,chassis1 yes yes 1Notes: * = Cell has no interleaved memory.[Chassis]
Core Connected Par
Hardware Location Usage IO To Num
=================== ============ ==== ========== ===
cab8,bay0,chassis0 Active yes cab0,cell2 1[Chassis]
Core Connected Par
Hardware Location Usage IO To Num
=================== ============ ==== ========== ===
cab8,bay0,chassis1 Active yes cab0,cell3 1
On top of the fact that my cells are not CPU equally balanced no Cell Local Memory (CLM) has been configured (default on older HP servers is: 100% ILM, 0% CLM so behave of default configuration). The generic HP advice is to configure 7/8 of available memory as CLM.
Then using the following shell commands you are able to see to which processor are attached the Oracle background and slave processes:
server1{root}# top -f /tmp/top_output -n 2000
server1{root}# grep ora\_ /tmp/top_output
server1{root}# top -f /tmp/top_output -n 2000
server1{root}# grep ora\_ /tmp/top_output
Then HP states that parallel slaves and database writer should be evenly distributed on cells. They are all attached to cell 0 (and so to unique CPU 0) in my case so another *strange* behavior… Behavior I have not seen on antoher Itanium box where each cell was running an database writer and archive process.
Then reading further HP clarify it once for all:
For Oracle Database version 10.2.0.4, the optimizations are not implemented correctly and should never be used: _enable_NUMA_optimization should always be set to false (and the server should thus be configured ILM-heavy) for Oracle 10.2.0.4
Notice that you can choose on which cell (or processor, option mutually exclusive with locality-domain one) you want to run your process. Even if in my small example it has no interest HP suggest to investigate the LL (Least Loaded) policy for Oracle listener:
server1{root}# mpsched -c 9 -O LL sleep 60 &[1]28198
server1{root}# Pid 28198: bound to processor 9 using the least loaded process launch policy
server1{root}# mpsched -c 9 -O LL sleep 60 &
[1] 28198
server1{root}# Pid 28198: bound to processor 9 using the least loaded process launch policy
Then what about Oracle ? Here below what I get with my 10.2.0.4 instance running on Itanium (_enable_NUMA_optimization set to true by default):
SQL>SET lines 150 pages 100
SQL> col description FOR a70
SQL> col PARAMETER FOR a30
SQL> col VALUEFOR a15
SQL>SELECT
a.ksppinm AS parameter,
c.ksppstvl ASVALUE,
a.ksppdesc AS description,
b.ksppstdf AS"Default?"FROM x$ksppi a, x$ksppcv b, x$ksppsv c
WHERE a.indx = b.indx
AND a.indx = c.indx
ANDLOWER(a.ksppinm)LIKE'%numa%'ORDERBY a.ksppinm;
PARAMETER VALUE DESCRIPTION DEFAULT?
------------------------------ --------------- ---------------------------------------------------------------------- ---------
_NUMA_instance_mapping NOT specified SETOF nodes that this instance should run ONTRUE
_NUMA_pool_size NOT specified aggregate SIZEIN bytes OF NUMA pool TRUE
_db_block_numa 2NUMBEROF NUMA nodes TRUE
_enable_NUMA_optimization TRUE Enable NUMA specific optimizations TRUE
_rm_numa_sched_enable FALSEISRESOURCE Manager (RM) related NUMA scheduled policy enabled TRUE
_rm_numa_simulation_cpus 0NUMBEROF cpus per PG FOR numa simulation INRESOURCE manager TRUE
_rm_numa_simulation_pgs 0NUMBEROF PGs FOR numa simulation INRESOURCE manager TRUE7ROWS selected.
SQL> SET lines 150 pages 100
SQL> col description FOR a70
SQL> col PARAMETER FOR a30
SQL> col VALUE FOR a15
SQL> SELECT
a.ksppinm AS parameter,
c.ksppstvl AS VALUE,
a.ksppdesc AS description,
b.ksppstdf AS "Default?"
FROM x$ksppi a, x$ksppcv b, x$ksppsv c
WHERE a.indx = b.indx
AND a.indx = c.indx
AND LOWER(a.ksppinm) LIKE '%numa%'
ORDER BY a.ksppinm;PARAMETER VALUE DESCRIPTION Default?
------------------------------ --------------- ---------------------------------------------------------------------- ---------
_NUMA_instance_mapping Not specified Set of nodes that this instance should run on TRUE
_NUMA_pool_size Not specified aggregate size in bytes of NUMA pool TRUE
_db_block_numa 2 Number of NUMA nodes TRUE
_enable_NUMA_optimization TRUE Enable NUMA specific optimizations TRUE
_rm_numa_sched_enable FALSE Is Resource Manager (RM) related NUMA scheduled policy enabled TRUE
_rm_numa_simulation_cpus 0 number of cpus per PG for numa simulation in resource manager TRUE
_rm_numa_simulation_pgs 0 number of PGs for numa simulation in resource manager TRUE7 rows selected.
The display in alert.log file is also difficult to catch:
.
Opening with internal Resource Manager plan
where NUMA PG = 2, CPUs = 1
.
.
Opening with internal Resource Manager plan
where NUMA PG = 2, CPUs = 1
.
NUMA on Linux
As a reminder use numactl command to display hardware information (the server behind is an HP BL620c G7 with 2 CPUs (Intel(R) Xeon(R) CPU X7560 @ 2.27GHz) of 8 cores each, so 32 threads if you activate hyper-threading):
In my previous post I was wondering how NUMA got implemented with x86 technology. In short it is coming from Intel/AMD technologies. Intel QuickPath Interconnect (QuickPath, QPI), previously call Common System Interface (CSI) for Intel. HyperTransport (HT), earlier Lightning Data Transport (LDT) for AMD. Refer to Wikipedia for more information.
On Linux the top command does not print the CPU id where is running the process, fortunately the ps command is enhanced and you can have it using a custom display (psr column). The example database below is a 11.2.0.2 release (where NUMA is off by default):
For my Oracle 11.2.0.2 test database, if I set _enable_NUMA_support to true (in 11gR2 _enable_NUMA_optimization is deprecated and you must use _enable_NUMA_support):
SQL>SET lines 150 pages 100
SQL> col description FOR a70
SQL> col PARAMETER FOR a30
SQL> col VALUEFOR a15
SQL>SELECT
a.ksppinm AS parameter,
c.ksppstvl ASVALUE,
a.ksppdesc AS description,
b.ksppstdf AS"Default?"FROM x$ksppi a, x$ksppcv b, x$ksppsv c
WHERE a.indx = b.indx
AND a.indx = c.indx
ANDLOWER(a.ksppinm)LIKE'%numa%'ORDERBY a.ksppinm;
PARAMETER VALUE DESCRIPTION DEFAULT?
------------------------------ --------------- ---------------------------------------------------------------------- ---------
_NUMA_instance_mapping NOT specified SETOF nodes that this instance should run ONTRUE
_NUMA_pool_size NOT specified aggregate SIZEIN bytes OF NUMA pool TRUE
_db_block_numa 2NUMBEROF NUMA nodes TRUE
_enable_NUMA_interleave TRUE Enable NUMA interleave MODETRUE
_enable_NUMA_optimization FALSE Enable NUMA specific optimizations TRUE
_enable_NUMA_support TRUE Enable NUMA support AND optimizations FALSE
_numa_buffer_cache_stats 0 Configure NUMA buffer cache stats TRUE
_numa_trace_level 0 numa trace event TRUE
_rm_numa_sched_enable TRUEISRESOURCE Manager (RM) related NUMA scheduled policy enabled TRUE
_rm_numa_simulation_cpus 0NUMBEROF cpus FOR each pg FOR numa simulation INRESOURCE manager TRUE
_rm_numa_simulation_pgs 0NUMBEROF PGs FOR numa simulation INRESOURCE manager TRUE11ROWS selected.
SQL> SET lines 150 pages 100
SQL> col description FOR a70
SQL> col PARAMETER FOR a30
SQL> col VALUE FOR a15
SQL> SELECT
a.ksppinm AS parameter,
c.ksppstvl AS VALUE,
a.ksppdesc AS description,
b.ksppstdf AS "Default?"
FROM x$ksppi a, x$ksppcv b, x$ksppsv c
WHERE a.indx = b.indx
AND a.indx = c.indx
AND LOWER(a.ksppinm) LIKE '%numa%'
ORDER BY a.ksppinm;PARAMETER VALUE DESCRIPTION Default?
------------------------------ --------------- ---------------------------------------------------------------------- ---------
_NUMA_instance_mapping Not specified Set of nodes that this instance should run on TRUE
_NUMA_pool_size Not specified aggregate size in bytes of NUMA pool TRUE
_db_block_numa 2 Number of NUMA nodes TRUE
_enable_NUMA_interleave TRUE Enable NUMA interleave mode TRUE
_enable_NUMA_optimization FALSE Enable NUMA specific optimizations TRUE
_enable_NUMA_support TRUE Enable NUMA support and optimizations FALSE
_numa_buffer_cache_stats 0 Configure NUMA buffer cache stats TRUE
_numa_trace_level 0 numa trace event TRUE
_rm_numa_sched_enable TRUE Is Resource Manager (RM) related NUMA scheduled policy enabled TRUE
_rm_numa_simulation_cpus 0 number of cpus for each pg for numa simulation in resource manager TRUE
_rm_numa_simulation_pgs 0 number of PGs for numa simulation in resource manager TRUE11 rows selected.
The display in alert.log file is more verbose in 11gR2 and it i clearly showing that NUMA is in action (please note I also use HugePages):
.
Numa pool size adjusted
****************** Huge Pages Information *****************
Huge Pages memory pool detected (total: 10395 free: 7995)
DFLT Huge Pages allocation successful (allocated: 146)
NUMA Huge Pages allocation on node (1)(allocated: 186)
NUMA Huge Pages allocation on node (0)(allocated: 182)
DFLT Huge Pages allocation successful (allocated: 1)***********************************************************
SGA Local memory support enabled
.
.
NUMA system found and support enabled (2 domains - 16,16)
.
.
Numa pool size adjusted
****************** Huge Pages Information *****************
Huge Pages memory pool detected (total: 10395 free: 7995)
DFLT Huge Pages allocation successful (allocated: 146)
NUMA Huge Pages allocation on node (1) (allocated: 186)
NUMA Huge Pages allocation on node (0) (allocated: 182)
DFLT Huge Pages allocation successful (allocated: 1)
***********************************************************
SGA Local memory support enabled
.
.
NUMA system found and support enabled (2 domains - 16,16)
.
As expected the SGA is now split in multiple part:
I have tried with a 10.2.0.3 database but even setting explicitly in init.ora below parameters:
_db_block_numa 2 Number of NUMA nodes FALSE
_enable_NUMA_optimization TRUE Enable NUMA specific optimizations FALSE
_db_block_numa 2 Number of NUMA nodes FALSE
_enable_NUMA_optimization TRUE Enable NUMA specific optimizations FALSE
The alert.log file does not display anything so really difficult, not to say impossible, to know if NUMA is active or not… I finally got my answer checking the memory segments (so NUMA not in use and I can’t say why):
Another funny command is numademo that apparently aim at providing performance of your NUMA system:
[root@server1 ~]# numademo 10m random2 nodes available
memory with no policy random Avg 473.38 MB/s Min 698.12 MB/s Max 256.07 MB/s
local memory random Avg 682.44 MB/s Min 694.56 MB/s Max 601.49 MB/s
memory interleaved on all nodes random Avg 677.94 MB/s Min 695.71 MB/s Max 560.02 MB/s
memory on node 0 random Avg 680.93 MB/s Min 694.15 MB/s Max 594.03 MB/s
memory on node 1 random Avg 667.67 MB/s Min 689.99 MB/s Max 544.57 MB/s
memory interleaved on 01 random Avg 675.53 MB/s Min 692.36 MB/s Max 565.09 MB/s
setting preferred node to 0
memory without policy random Avg 672.54 MB/s Min 692.04 MB/s Max 553.75 MB/s
setting preferred node to 1
memory without policy random Avg 678.79 MB/s Min 698.35 MB/s Max 550.35 MB/s
manual interleaving to all nodes random Avg 680.77 MB/s Min 697.01 MB/s Max 570.59 MB/s
manual interleaving on node 0/1 random Avg 681.65 MB/s Min 697.24 MB/s Max 578.78 MB/s
current interleave node 0
running on node 0, preferred node 0local memory random Avg 674.03 MB/s Min 688.13 MB/s Max 582.70 MB/s
memory interleaved on all nodes random Avg 682.52 MB/s Min 696.40 MB/s Max 587.21 MB/s
memory interleaved on node 0/1 random Avg 681.68 MB/s Min 694.01 MB/s Max 595.24 MB/s
alloc on node 1 random Avg 677.95 MB/s Min 694.47 MB/s Max 565.39 MB/s
local allocation random Avg 676.42 MB/s Min 692.72 MB/s Max 565.00 MB/s
setting wrong preferred node random Avg 679.50 MB/s Min 696.17 MB/s Max 578.14 MB/s
setting correct preferred node random Avg 674.12 MB/s Min 692.68 MB/s Max 550.14 MB/s
running on node 1, preferred node 0local memory random Avg 613.30 MB/s Min 697.84 MB/s Max 301.31 MB/s
memory interleaved on all nodes random Avg 680.04 MB/s Min 695.39 MB/s Max 574.03 MB/s
memory interleaved on node 0/1 random Avg 683.59 MB/s Min 696.45 MB/s Max 598.16 MB/s
alloc on node 0 random Avg 674.49 MB/s Min 695.76 MB/s Max 548.22 MB/s
local allocation random Avg 674.20 MB/s Min 696.64 MB/s Max 552.78 MB/s
setting wrong preferred node random Avg 672.26 MB/s Min 694.84 MB/s Max 525.68 MB/s
setting correct preferred node random Avg 679.85 MB/s Min 696.77 MB/s Max 567.47 MB/s
[root@server1 ~]# numademo 10m random
2 nodes available
memory with no policy random Avg 473.38 MB/s Min 698.12 MB/s Max 256.07 MB/s
local memory random Avg 682.44 MB/s Min 694.56 MB/s Max 601.49 MB/s
memory interleaved on all nodes random Avg 677.94 MB/s Min 695.71 MB/s Max 560.02 MB/s
memory on node 0 random Avg 680.93 MB/s Min 694.15 MB/s Max 594.03 MB/s
memory on node 1 random Avg 667.67 MB/s Min 689.99 MB/s Max 544.57 MB/s
memory interleaved on 0 1 random Avg 675.53 MB/s Min 692.36 MB/s Max 565.09 MB/s
setting preferred node to 0
memory without policy random Avg 672.54 MB/s Min 692.04 MB/s Max 553.75 MB/s
setting preferred node to 1
memory without policy random Avg 678.79 MB/s Min 698.35 MB/s Max 550.35 MB/s
manual interleaving to all nodes random Avg 680.77 MB/s Min 697.01 MB/s Max 570.59 MB/s
manual interleaving on node 0/1 random Avg 681.65 MB/s Min 697.24 MB/s Max 578.78 MB/s
current interleave node 0
running on node 0, preferred node 0
local memory random Avg 674.03 MB/s Min 688.13 MB/s Max 582.70 MB/s
memory interleaved on all nodes random Avg 682.52 MB/s Min 696.40 MB/s Max 587.21 MB/s
memory interleaved on node 0/1 random Avg 681.68 MB/s Min 694.01 MB/s Max 595.24 MB/s
alloc on node 1 random Avg 677.95 MB/s Min 694.47 MB/s Max 565.39 MB/s
local allocation random Avg 676.42 MB/s Min 692.72 MB/s Max 565.00 MB/s
setting wrong preferred node random Avg 679.50 MB/s Min 696.17 MB/s Max 578.14 MB/s
setting correct preferred node random Avg 674.12 MB/s Min 692.68 MB/s Max 550.14 MB/s
running on node 1, preferred node 0
local memory random Avg 613.30 MB/s Min 697.84 MB/s Max 301.31 MB/s
memory interleaved on all nodes random Avg 680.04 MB/s Min 695.39 MB/s Max 574.03 MB/s
memory interleaved on node 0/1 random Avg 683.59 MB/s Min 696.45 MB/s Max 598.16 MB/s
alloc on node 0 random Avg 674.49 MB/s Min 695.76 MB/s Max 548.22 MB/s
local allocation random Avg 674.20 MB/s Min 696.64 MB/s Max 552.78 MB/s
setting wrong preferred node random Avg 672.26 MB/s Min 694.84 MB/s Max 525.68 MB/s
setting correct preferred node random Avg 679.85 MB/s Min 696.77 MB/s Max 567.47 MB/s
Conclusion
Then all these for what ? Googling around the gain you may see having a fully compliant chain of software and hardware NUMA aware components may bring you up to 10% improvement or none or even a drawback. So clearly as Oracle corporation state do perform exhaustive test before activating it. Then I tend to say that generic advice is to deactivate it at hardware level.
If we take 11gR2 example where it is not activated by default is there a drawback letting it activated at hardware level ?
On Itanium relying to default value is worst situation and I don’t know if you can deactivate it at hardware level.
Finally even Oracle database does not look too much mature on this feature in 10gR2 (no obsolete by the way) so better play with it in 11gR2.
I can’t resist quoting excellent blog post from Jeremy Cole on MySQL (not too far from Oracle behavior), where he is suggesting to interleave memory for commands starting MySQL…
References
The Oracle Database on HP Integrity servers (HP document)
Dynamic server resource allocation with Oracle Database 10g or 11g on an HP-UX ccNUMA-based server (HP document)
Running Oracle Database 10g or 11g on an HP-UX ccNUMA-based server (HP document)
