HP Process Resource Manager (PRM)

Posted on , updated on by Yannick Jaquier
HPUX ,

Table of contents

Preamble

HP Process Resource Manager (HP PRM) is a resource management tool used to control the amount of resources that processes use during peak system load. For the one who knows Oracle Resource Manager it is exactly the same philosophy but at OS level. The Oracle product can now be seen with a deeper granularity. PRM allows to split available resources between multiple databases while Oracle Resource Manager allows to share resources between applications running in the same databases.

HP PRM can manage allocation of the following resources:

  • CPU: Ensures a minimum allocation of CPU; can also simultaneously cap CPU resource usage for each group in the configuration. (Starting with HP-UX 11i v3, per-group capping is available.)
  • Memory: Ensures a minimum allocation of private real memory; can also cap private memory usage. Allows you to allocate an exact amount of shared memory.
  • Disk bandwidth: Ensures a minimum allocation of disk bandwidth.

Remark:
Apparently there is a boring limitation with latest release of the product (3.05) as “PRM can manage disk bandwidth for disks that are mounted and under the control of VERITAS Volume Manager. However, PRM’ s management does not work with VERITAS File System 4.0 and later”.

Product is free, but starting with HPUX 11iv3…

Resource sharing with PRM can be done using two different approaches:

  • Partial resource isolation (Fair Share Scheduler (FSS) PRM groups): CPU and private memory resources can be borrowed from other PRM groups if it not used.
  • Total resource isolation (Processor Set (PSET) PRM groups): Complete isolation of both CPU and the private memory resources.

Remark:
Oracle Resource Manager is not compatible with FSS PRM groups as it will think that the whole server is allowed to the instance and so will not see how you managed your PRM groups. leading to strange behavior in prioritizing resources.

Over time HP has corrected lots of bugs in their product so make sure you have the latest release (not the case in my example):

server1{root}# prmconfig -V
HP PRM C.03.02 (20060711_162001) hpux_11.00 prmconfig

Configuration

First check with prmavail command what’s available on your server:

server1{orahdte}# prmavail -p
2 Cores - 0, 1
64 FSS groups available
72863 real memory pages or 284 MB available (PRM estimate)
4 volume groups
        /dev/vg00
        /dev/vgb92
        /dev/vgb92a
        /dev/vgc92

The prmanalyze command can be used to create the first initial configuration based on current usage of the server:

server1{root}# touch /tmp/test
server1{root}# prmanalyze -s command -r cpu -p -t summary -1 /tmp/test
summary CPU report by command name : 124 records processed
unique id   processes    ave secs   peak secs    total secs  % total
disk_em             1     1551.00     1551.00       1551.00     1.53
dm_TL_ad            1    55045.00    55045.00      55045.00    54.22
midaemon            1     3471.00     3471.00       3471.00     3.42
monitors            1     2506.00     2506.00       2506.00     2.47
opcmona             1     1070.00     1070.00       1070.00     1.05
prm2d               1     1402.00     1402.00       1402.00     1.38
statdaem            1     2597.00     2597.00       2597.00     2.56
swapper             1     7235.00     7235.00       7235.00     7.13
syncer              1     1196.00     1196.00       1196.00     1.18
utild               1     1708.00     1708.00       1708.00     1.68
vxfsd               1     5831.00     5831.00       5831.00     5.74
server1{root}# prmanalyze -s command -r mem -p -t summary -1 /tmp/test
summary memory report by command name : 272 records processed
unique id   processes      ave KB     peak KB    KB minutes  % total
avcsvcd             1      607.00      607.00  111258608200.00     1.66
cimprova            5     1893.36     9082.00  1833308800500.00    27.29
cimserve            3      477.95     1480.00  287782198200.00     4.28
coda                1     3148.00     3148.00  355916972400.00     5.30
dced                1      278.00      278.00  108318418800.00     1.61
httpd               2      522.50     1001.00  183516638800.00     2.73
midaemon            1     2150.00     2150.00  243082655000.00     3.62
opcmona             1      746.00      746.00  84272188400.00     1.25
opcmsga             1      638.00      638.00  72072116600.00     1.07
opcmsgi             1      634.00      634.00  71620253800.00     1.07
oracle             68      249.66     1205.00  123703547400.00     1.84
ovbbccb             1     1414.00     1414.00  159869102400.00     2.38
ovcd                1     2828.00     2828.00  319738770400.00     4.76
ovconfd             1      745.00      745.00  84160117000.00     1.25
perfalar            1      692.00      692.00  78238696400.00     1.16
scopeux             1     2474.00     2474.00  279715140600.00     4.16
storapid            1      482.00      482.00  108893970200.00     1.62
utild               1      174.00      174.00  67790521800.00     1.01
vxfsd               1     2008.00     2008.00  782417601600.00    11.64
vxsvc               1      263.00      263.00  102472164000.00     1.53
server1{root}# prmanalyze -s command -r disk -p -t summary -1 /tmp/test
summary disk report by command name : 238 records processed
unique id   processes     ave Bps    peak Bps      total KB  % total
disk_em             1     7842.67     7842.67  1806676938.77     1.99
dm_TL_ad            1   361810.41   361810.41  83348045319.73    91.82
dm_chass            1     5549.20     5549.20  1278330733.69     1.41

Use prmconfig to create the initial configuration file (/etc/prmconf firl and -i option), check the mofication you have done (-s), enable PRM (-e [manager]), disable it (-d [manager]).

server1{root}# prmconfig -i
 
PRM configured from file:  /etc/prmconf
File last modified:        Thu Jun 17 11:30:24 2010
 
PRM CPU scheduler state:  Enabled
 
                                                  CPU  Upper  LCPU
PRM Group                          PRMID  Entitlement  Bound  Attr
__________________________________________________________________
ORACLE/orahdqa                         2       48.00%
ORACLE/orahdte                         3       32.00%
OTHERS                                 1       20.00%
 
PRM memory manager state:  Enabled  (polling interval: 10 seconds)
PRM memory manager logging state:  Disabled
 
                                               Memory  Upper
PRM Group                          PRMID  Entitlement  Bound
____________________________________________________________
ORACLE/orahdqa                         2       48.00%
ORACLE/orahdte                         3       32.00%
OTHERS                                 1       20.00%
 
PRM User                       Initial Group                  Alternate Group(s)
________________________________________________________________________________
adm                            (NONE)
anujd                          (NONE)
bin                            (NONE)
daemon                         (NONE)
dasusr1                        (NONE)
db2fenc1                       (NONE)
db2inst1                       (NONE)
ftp                            (NONE)
hp                             (NONE)
hpsmdb                         (NONE)
hpsmh                          (NONE)
iwww                           (NONE)
jboss                          (NONE)
lp                             (NONE)
maesb92                        OTHERS
mksingla                       (NONE)
nobody                         (NONE)
opc_op                         (NONE)
orahdqa                        ORACLE/orahdqa
orahdte                        ORACLE/orahdte
ovo                            (NONE)
owww                           (NONE)
remedy                         (NONE)
root                           (PRM_SYS)
sanjana                        (NONE)
sfmdb                          (NONE)
smbnull                        (NONE)
sshd                           (NONE)
ssinghal                       (NONE)
sys                            (NONE)
www                            (NONE)
 
PRM application manager state:  Enabled  (polling interval: 30 seconds)
PRM application manager logging state:  Disabled
 
Disk manager state:  Disabled
 
Unix group                     Default PRM Group
________________________________________________
adm                            (NONE)
admin                          (NONE)
bin                            (NONE)
contractors                    (NONE)
dasadm1                        (NONE)
db2fgrp1                       (NONE)
db2grp1                        (NONE)
dba                            (NONE)
ftp                            (NONE)
hpsmc                          (NONE)
hpsmh                          (NONE)
jboss                          (NONE)
lp                             (NONE)
mail                           (NONE)
nogroup                        (NONE)
opcgrp                         (NONE)
other                          (NONE)
root                           (NONE)
smbnull                        (NONE)
sshd                           (NONE)
sys                            (NONE)
tftp                           (NONE)
tty                            (NONE)
unison                         (NONE)
users                          (NONE)

So our configuration is (for CPU/private memory):

  • OTHERS 20% (All accounts that have no PRM group specify go into this PRM group)
  • ORACLE 80%
    • ORACLE/orahdqa 60% (orahdqa account)
    • ORACLE/orahdte 40% (orahdte account)

CPU testing

Testing CPU usage is quite easy you just need a small script that uses CPU.

One must simpler (maybe too simpler to be understandable by a normal human being) I have found on Internet is (2 is your actual number of cores):

server1{orahdte}# /opt/STMtools/STMperl/bin/perl -e 'while (--$ARGV[0] and fork) {}; while () {}' 2

I would have written something like:

#!/opt/STMtools/STMperl/bin/perl
 
$x = 0.000001;
while (TRUE) {
$y = sin($x)/cos($x);
$x = $x + 0.000001;
}

So by doing a fork of child processes equal to the number of cores (ease to overload the server):

#!/opt/STMtools/STMperl/bin/perl
 
print("Running for ",$ARGV[0]," CPU(s)n");
 
for ($i=1; $i<=$ARGV[0]; $i++) {
  $pid=fork();
  if ($pid == 0) {
    # In the child
    $x = 0.000001;
    while (TRUE) {
      $y = sin($x)/cos($x);
      $x = $x + 0.000001;
    }
  }
}

For the state of the art the C version:

#include <unistd.h>
#include <math.h>
#define FALSE  0
#define TRUE   1
 
/* cc eatcpu4.c -o eatcpu4 -lm */
int main(int argc, char *argv[])
{
	pid_t pid;
	int i;
	double x,y;
 
  if (argc <= 1) {
	  printf("Please provide number of CPUn");
    exit();
  }
  else
    printf("%s running for %d CPU(s)n",argv[0],atoi(argv[1]));
 
  for (i=1; i<=atoi(argv[1]); i++) {
    pid=fork();
    if (pid == 0) {
  	  printf("Creating child %dn",i);
      x = 0.000001;
      while (TRUE) {
        y = tan(x);
        x = x + 0.000001;
      }
      exit(0);
    }
  }
  return 0;
}

The tool to monitor this is prmmonitor, interesting option are:

  • -h Include parent hierarchical groups in output
  • -w Print the PRM group names in wide-column format, exceeding the 30-column default if necessary to avoid clipping any names.
  • -t Terse output. Print statistics without table headers.
  • -s Include PRMID 0 in the statistics. By default, PRMID 0 is not included.

When orahdte only is using CPU I got:

                                              CPU  Upper      CPU   LCPU
PRM Group                          PRMID  Entitle  Bound     Used  State
________________________________________________________________________
(PRM_SYS)                              0                    0.90%
ORACLE                              HIER   80.00%          99.10%
ORACLE/orahdqa                         2   48.00%           0.00%
ORACLE/orahdte                         3   32.00%          99.10%
OTHERS                                 1   20.00%           0.00%

So as expected orahdte is taking 100% CPU (FSS groups so shared resources when not used by other groups).

Remark
48% and 32% are respectively 60% and 40% of 80%…

When both orahdte & orahdqa are using CPU I got:

                                              CPU  Upper      CPU   LCPU
PRM Group                          PRMID  Entitle  Bound     Used  State
________________________________________________________________________
(PRM_SYS)                              0                    0.00%
ORACLE                              HIER   80.00%         100.00%
ORACLE/orahdqa                         2   48.00%          59.80%
ORACLE/orahdte                         3   32.00%          40.20%
OTHERS                                 1   20.00%           0.00%

As expected orahdte & orahdqa are taking 100% CPU with their respective 60% and 40% allocated.

When orahdte, orahdqa and maesb92 are using CPU I got:

                                              CPU  Upper      CPU   LCPU
PRM Group                          PRMID  Entitle  Bound     Used  State
________________________________________________________________________
(PRM_SYS)                              0                    0.00%
ORACLE                              HIER   80.00%          80.04%
ORACLE/orahdqa                         2   48.00%          47.90%
ORACLE/orahdte                         3   32.00%          32.14%
OTHERS                                 1   20.00%          19.96%

We again see it is working well, 60% (48%) and 40% (32%) respectively for Oracle accounts and OTHERS group taking 20%.

Then I finally tested when root is using CPU (PRM_SYS default group), and as expected it has maximum priority over the other FSS groups but anyway not taking 100% CPU (50% around from what I observed):

                                              CPU  Upper      CPU   LCPU
PRM Group                          PRMID  Entitle  Bound     Used  State
________________________________________________________________________
(PRM_SYS)                              0                   49.27%
ORACLE                              HIER   80.00%          40.98%
ORACLE/orahdqa                         2   48.00%          24.39%
ORACLE/orahdte                         3   32.00%          16.59%
OTHERS                                 1   20.00%           9.76%

Memory testing

Private memory

As on every Unix, available memory you see is made of real physical memory and disk (swap). So when PRM group can borrow memory from other group, trying to get it back by the group to which you stole it will tell HPUX to swapout from memory to disk the memory space initially allocated to the first group.

The first concept to understand is that PRM memory capping is only acting on the remaining memory on the server, to see what you have issue:

server1{root}# prmavail MEM
253726 real memory pages or 991 MB available (PRM estimate)

Then after to monitor memory usage you can either use glance (there is a PRM menu, P in uppercase for direct access) or MRG option of prmmonitor. The default display is not in MB and so difficult to understand, this MRG option is key for understanding.

Remark:
To monitor the usage of one single process you can also use glance, then press s, enter the spid of the process and then you can explicitly monitor it (CPU, MEM, I/O, …).

First let’s use the following C program to use private memory for Unix account of various PRM groups:

#include <stdlib.h>
#include <stdio.h>
#define FALSE  0
#define TRUE   1
 
/* cc -Aa -fast eatmem1.c -o eatmem1 */
int main(int argc, char *argv[])
{
  void *pointer, *temp_pointer;
  char c;
  long int memory_size;
 
 
  if (argc <= 1) {
          printf("Please provide amount of memory to allocate (in Megabyte)n");
    exit(0);
  }
  else
    printf("%s running for %d Megabyten",argv[0],atoi(argv[1]));
 
  memory_size=atoi(argv[1])*1024*1024;
  pointer = calloc(atoi(argv[1]),1024*1024);
 
  if (pointer != NULL) {
    printf("Allocation done...n");
    printf("Size allocated %u bytesn",memory_size);
    sleep(600); // 10 minutes
    free(pointer);
    exit(0);
  }
  else {
    printf("Allocation impossible...n");
    exit(1);
  }
 
}

As you can see this simple C program accept one argument that is the total megabytes of memory you request for submitter.

What’s available before allocating anything for any PRM group:

server1{root}# prmavail MEM;prmmonitor MRG -hs
78393 real memory pages or 306 MB available (PRM estimate)
 
PRM configured from file:  /etc/prmconf
File last modified:        Thu Jun 17 11:30:24 2010
 
HP-UX server1 B.11.11 U 9000/800    06/28/10
 
Mon Jun 28 14:13:54 2010    Sample:  1 second
Memory manager state:  Enabled    Paging:  No
 
                                         MB        MB     MB      Total   Pages
PRM Group                     PRMID   Entitled    Max   Locked   MB Used  VM IO
_______________________________________________________________________________
(PRM_SYS)                         0    3481.04           13.45   2063.84      1
ORACLE                         HIER     244.98            0.00     73.10      0
ORACLE/orahdqa                    2     146.98            0.00     43.26      0
ORACLE/orahdte                    3      97.99            0.00     29.84      0
OTHERS                            1      61.25            0.00      5.84      0

All PRM groups using 73.10+5.84 i.e. 78.94MB of real physical memory.

Let’s allocate 800MB for orahdte account (wait a while before :

server1{root}# prmavail MEM;prmmonitor MRG -hs
77028 real memory pages or 300 MB available (PRM estimate)
 
PRM configured from file:  /etc/prmconf
File last modified:        Thu Jun 17 11:30:24 2010
 
HP-UX server1 B.11.11 U 9000/800    06/28/10
 
Mon Jun 28 14:21:37 2010    Sample:  1 second
Memory manager state:  Enabled    Paging:  No
 
                                         MB        MB     MB      Total   Pages
PRM Group                     PRMID   Entitled    Max   Locked   MB Used  VM IO
_______________________________________________________________________________
(PRM_SYS)                         0    3485.66           13.45   2063.18      1
ORACLE                         HIER     204.59            0.00    229.04      0
ORACLE/orahdqa                    2     122.75            0.00     42.46      0
ORACLE/orahdte                    3      81.84            0.00    186.58      0
OTHERS                            1      97.01            0.00      5.79      0

All PRM groups using 229.04+5.79 i.e. 234.83MB of physical memory. ORACLE/orahdte PRM group using 62% (186.58 / 300) of memory. So more than its 32% initial percent allocated.

Let’s additionally allocate 800MB for orahdqa account:

server1{root}# prmavail MEM;prmmonitor MRG -hs
78498 real memory pages or 306 MB available (PRM estimate)
 
PRM configured from file:  /etc/prmconf
File last modified:        Thu Jun 17 11:30:24 2010
 
HP-UX server1 B.11.11 U 9000/800    06/28/10
 
Mon Jun 28 14:36:21 2010    Sample:  1 second
Memory manager state:  Enabled    Paging:  No
 
                                         MB        MB     MB      Total   Pages
PRM Group                     PRMID   Entitled    Max   Locked   MB Used  VM IO
_______________________________________________________________________________
(PRM_SYS)                         0    3480.63           13.45   2063.22      1
ORACLE                         HIER     245.30            0.00    254.10      0
ORACLE/orahdqa                    2     147.18            0.00    181.88      0
ORACLE/orahdte                    3      98.12            0.00     72.23      0
OTHERS                            1      61.33            0.00      5.80      0

Let’s additionally allocate 800MB for maesb92 account (OTHERS PRM group):

server1{root}# prmavail MEM;prmmonitor MRG -hs
202630 real memory pages or 791 MB available (PRM estimate)
 
PRM configured from file:  /etc/prmconf
File last modified:        Thu Jun 17 11:30:24 2010
 
HP-UX server1 B.11.11 U 9000/800    06/28/10
 
Mon Jun 28 14:45:55 2010    Sample:  1 second
Memory manager state:  Enabled    Paging:  No
 
                                         MB        MB     MB      Total   Pages
PRM Group                     PRMID   Entitled    Max   Locked   MB Used  VM IO
_______________________________________________________________________________
(PRM_SYS)                         0    2992.66           13.45   1947.13     43
ORACLE                         HIER     635.68            0.00    563.85      0
ORACLE/orahdqa                    2     381.41            0.00    361.89      0
ORACLE/orahdte                    3     254.27            0.00    201.96      0
OTHERS                            1     158.92            0.00    154.65      0

In this final testing the memory allocated to each groups respect the initial percentages set in PRM configuration file. You have also seen that available memory available onto the server is frequently changing over time…

Apparently when the memory pool has been allocated, freeing memory in other pool will not make PRM to swap in the already allocated pools. In below example we would expect PRM to increase the memory used by OTHERS group…:

server1{root}# prmavail MEM;prmmonitor MRG -hs
198757 real memory pages or 776 MB available (PRM estimate)
 
PRM configured from file:  /etc/prmconf
File last modified:        Thu Jun 17 11:30:24 2010
 
HP-UX server1 B.11.11 U 9000/800    06/28/10
 
Mon Jun 28 14:58:34 2010    Sample:  1 second
Memory manager state:  Enabled    Paging:  No
 
                                         MB        MB     MB      Total   Pages
PRM Group                     PRMID   Entitled    Max   Locked   MB Used  VM IO
_______________________________________________________________________________
(PRM_SYS)                         0    3010.87           13.45   1949.79      1
ORACLE                         HIER     621.11            0.00     33.07      0
ORACLE/orahdqa                    2     372.67            0.00     17.12      0
ORACLE/orahdte                    3     248.45            0.00     15.95      0
OTHERS                            1     155.28            0.00    175.36      0

The remaining memory would be used by OTHERS group for new allocation and if you allocate more than its predefined percentage then what will be swap-out will not be swap-in when freeing memory…

Shared memory

Shared memory is linked to Oracle SGA (sga_max_size parameter) and allow you to define a minimum of shared memory for PRM groups (capping not available). This feature is available starting with HP-UX 11i v2 Update 2 and PRM C.03.01 so not possible on my test machine… To be continued…

You can allocate a shared memory segment using the following C program:

#include <stdio.h>
#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/shm.h>
#define FALSE  0
#define TRUE   1
 
/* cc eatmem2.c -o eatmem2 */
int main(int argc, char *argv[])
{
  key_t  key; /* key to be passed to shmget() */
  int  shmflg;   /* shmflg to be passed to shmget() */
  int  shmid;   /* return value from shmget() */
  char c;
 
 
  if (argc <= 1) {
          printf("Please provide amount of shared memory to allocate (in Megabyte)n");
    exit(0);
  }
  else
    printf("%s running for %d Megabyten",argv[0],atoi(argv[1]));
 
  srand(time(NULL));
  key=rand();
 
  /*printf("key = %dn",key);*/
 
  if ((shmid = shmget (key, atoi(argv[1])*1024*1024, 0600 | IPC_CREAT | IPC_EXCL)) == -1) {
    printf("Allocation impossible...n");
    exit(1);
   }
  else {
    printf("Allocation done...n");
    printf("shmget: shmget returned %dn", shmid);
    printf("Press Enter to release memory:");
    c=getchar();
    shmctl(shmid, IPC_RMID, NULL);
    exit(0);
  }
 
}

To monitor shared memory segment use ipcs command. PRM_MEM is the PRM keyword for the configuration…

Disk testing

The disk I/O bandwidth limitation is at volume group (VG) level. So clearly to use it you must use HP’s Logical Volume Manager (LVM) or Veritas Volume Manager (VxVM). Since ages STMicroelectronics kept the habit to have distinct VG per database and distinct LV per database component (Software, redo log, dump, data, indexes…) so making non-interesting this functionality…

As a matter of test, you should add in your configuration something like:

/dev/vgb92a:MAESTRO:98::
/dev/vgb92a:OTHERS:2::
/dev/vgb92a:ORACLE:0::
/dev/vgb92a:ORACLE/orahdqa:0::
/dev/vgb92a:ORACLE/orahdte:0::

So MAESTRO PRM group has 98% of I/O priority on Volume Group vgb92a, OTHERS PRM group is taking the last 2% remaining.

Check it is activated with:

server1{root}# prmavail
2 Cores
64 FSS groups available
22058 real memory pages or  86 MB available (PRM estimate)
4 volume groups
        /dev/vg00
        /dev/vgb92
        /dev/vgb92a
        /dev/vgc92
server1{root}# prmavail -f DISK
4 volume groups
        /dev/vg00
        /dev/vgb92
        /dev/vgb92a
        /dev/vgc92
In-kernel memory controls available
Disk Bandwidth available

Then the idea is to create a small C program that is generating a big file (eatdisk.c):

#include <stdio.h>
#define FALSE  0
#define TRUE   1
 
/* cc -Aa -fast eatdisk.c -o eatdisk */
int main(int argc, char *argv[])
{
        int i,result;
        FILE *filein;
        char sentence[1024] = "stringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstringof1024bytesstri";
 
  if (argc <= 2) {
          printf("Please provide a filename and a file size (MB) to be createdn");
    exit();
  }
  else
    printf("%s running to create %s with %d MB(s)n",argv[0],argv[1],atoi(argv[2]));
 
  filein=fopen(argv[1],"w");
 
  for (i=1; i<=1024*atoi(argv[2]); i++) {
        result=fputs(sentence, filein);
        if (result==0)
          break;
  }
 
  printf("File closed (<>0 if not)? %dn",fclose(filein));
  printf("File deleted (<>0 if not) ? %dn",remove(argv[1]));
 
  return 0;
}

Then create two Logical Volume and mount them in Direct I/O to bypass the Operating System buffer cache:

server1{root}# bdf | grep vgb92a
/dev/vgb92a/lvol1   458752   89424  366480   20% /opt/maesb92
/dev/vgb92a/lvol3   524288    5208  515088    1% /opt/maesb92/maestro/stdlist
/dev/vgb92a/lvol2   262144    2432  257688    1% /opt/maesb92/maestro/job
/dev/vgb92a/lvol99 7208960    2868 6755718    0% /yannick2
/dev/vgb92a/lvol98 7208960    2868 6755718    0% /yannick1
server1{root}# mount | grep vgb92a
/opt/maesb92 on /dev/vgb92a/lvol1 delaylog,largefiles on Fri May 14 14:38:41 2010
/opt/maesb92/maestro/stdlist on /dev/vgb92a/lvol3 delaylog,largefiles on Fri May 14 14:38:42 2010
/opt/maesb92/maestro/job on /dev/vgb92a/lvol2 delaylog,largefiles on Fri May 14 14:38:42 2010
/yannick2 on /dev/vgb92a/lvol99 delaylog,nodatainlog,largefiles,mincache=direct,convosync=direct on Mon Jul 19 17:49:20 2010
/yannick1 on /dev/vgb92a/lvol98 delaylog,nodatainlog,largefiles,mincache=direct,convosync=direct on Mon Jul 19 17:49:20 2010

Then execute at least six times the program per account in OTHERS and MAESTRO PRM groups to create 12 files of 1GB. This is mandatory to really request more I/O than the disk is capable to serve (also activate the trace of this test with accton):

server1{root}#  rm -rf /var/adm/pacct
server1{root}# /usr/sbin/acct/accton
server1{root}# /usr/sbin/acct/accton /var/adm/pacct
server1{root}# ps -ef | grep eat
 orahdte 22940 20754 11 14:53:15 pts/0     0:05 /home/orahdte/yannick/eatdisk /yannick1/file2 1024
 maesb92 23098 20807 14 14:54:01 pts/3     0:03 /home/orahdte/yannick/eatdisk /yannick2/file2 1024
 orahdte 22966 20754 15 14:53:21 pts/0     0:05 /home/orahdte/yannick/eatdisk /yannick1/file3 1024
 orahdte 23008 20754 16 14:53:37 pts/0     0:04 /home/orahdte/yannick/eatdisk /yannick1/file6 1024
 orahdte 23007 20754 14 14:53:32 pts/0     0:04 /home/orahdte/yannick/eatdisk /yannick1/file5 1024
 maesb92 23102 20807 13 14:54:24 pts/3     0:02 /home/orahdte/yannick/eatdisk /yannick2/file6 1024
 maesb92 23100 20807 12 14:54:15 pts/3     0:02 /home/orahdte/yannick/eatdisk /yannick2/file4 1024
 orahdte 22754 20754 16 14:53:07 pts/0     0:05 /home/orahdte/yannick/eatdisk /yannick1/file1 1024
    root 23626 20282  1 14:57:12 pts/1     0:00 grep eat
 orahdte 23005 20754 14 14:53:27 pts/0     0:04 /home/orahdte/yannick/eatdisk /yannick1/file4 1024
 maesb92 23099 20807 13 14:54:07 pts/3     0:03 /home/orahdte/yannick/eatdisk /yannick2/file3 1024
 maesb92 23101 20807 12 14:54:20 pts/3     0:02 /home/orahdte/yannick/eatdisk /yannick2/file5 1024
 maesb92 23094 20788  2 14:53:50 pts/2     0:03 /home/orahdte/yannick/eatdisk /yannick2/file1 1024

Then monitor the usage:

server1{root}# prmmonitor /dev/vgb92a -hs
 
PRM configured from file:  /etc/prmconf
File last modified:        Fri Jul  9 13:34:08 2010
 
HP-UX server1 B.11.11 U 9000/800    07/26/10
 
Mon Jul 26 14:57:34 2010    Sample:  1 second
Disk manager state:  Enabled
Device: /dev/vgb92a
                                         %            %         KB
PRM Group                      PRMID  entitled    achieved  transferred
_______________________________________________________________________
(PRM_SYS)                          0                  6.24         1666
MAESTRO                            4     98.00       47.94        12808
ORACLE                          HIER      0.20       45.82        12240
ORACLE/orahdqa                     2      0.10        0.00            0
ORACLE/orahdte                     3      0.10       45.82        12240
OTHERS                             1      2.00        0.00            0
 
server1{root}# prmanalyze -r d
summary disk report by command name : 509 records processed
unique id   processes     ave Bps    peak Bps      total KB  % total
awk               113    32850.13   170666.67      12800.00     0.38
bdf                 6    49605.54    76800.00      14000.00     0.41
cHmem_ut            1     2178.72     2178.72        200.00     0.01
cat                 4   102400.00   102400.00        500.00     0.01
cimprova            1        3.37        3.37        100.00     0.00
cimprovi            1     2438.10     2438.10        100.00     0.00
cimserve            3     9600.00    14628.57        600.00     0.02
col                 1    23630.77    23630.77        300.00     0.01
conman              7     5461.33    27927.27       1200.00     0.04
cp                 85  7790871.05  17028740.74    3127000.00    91.75
cut                 1   102400.00   102400.00        100.00     0.00
date                9    35446.15   102400.00        900.00     0.03
df                  4    39302.35    62720.00      14700.00     0.43
diff                6     3432.40   102400.00        600.00     0.02
dm_ql_ad            1        9.56        9.56        300.00     0.01
eatdisk             1   307200.00   307200.00        300.00     0.01
extract             1      398.44      398.44        100.00     0.00
fc-check            1     3828.04     3828.04        400.00     0.01
file                1   102400.00   102400.00        200.00     0.01
glance              1     2766.28     2766.28       5800.00     0.17
grep              113    39040.00   204800.00      12200.00     0.36
ha_disk_            1      156.69      156.69        200.00     0.01
ksh                22       83.31    15515.15       5300.00     0.16
ll                  5   167563.64   307200.00       1800.00     0.05
lpstat              1   204800.00   204800.00        400.00     0.01
ls                  2   153600.00   204800.00        300.00     0.01
lsnrctl             1      562.64      562.64        100.00     0.00
mail                3   102400.00   204800.00        600.00     0.02
man                 1     2322.87     2322.87       2400.00     0.07
more                2      990.17     4326.76       1200.00     0.04
neqn                1   153600.00   153600.00        600.00     0.02
nroff               1   283569.23   283569.23       3600.00     0.11
opcmon              8     3150.77     4266.67       1600.00     0.05
opcmsg              1     3592.98     3592.98        200.00     0.01
or-chkar            3      892.16     2133.33        900.00     0.03
or-chkde            1      615.63      615.63        300.00     0.01
or-chkpr            3      664.17      859.30       1700.00     0.05
oracle             13    49663.30    65375.59     142200.00     4.17
perl                4     5769.01    15864.79       3200.00     0.09
prmanaly            1   163840.00   163840.00        800.00     0.02
prmmonit            1     6762.26     6762.26        700.00     0.02
ps                 10    38684.44   163840.00       3400.00     0.10
rad                 2    56888.89    61440.00        500.00     0.01
registra            9      281.53    25600.00       1100.00     0.03
rm                  2   102400.00   153600.00        600.00     0.02
sed                 4   179200.00   204800.00        700.00     0.02
sendmail            1     6400.00     6400.00        100.00     0.00
sftp-ser            1    22755.56    22755.56       1000.00     0.03
sh                 16     2281.04     6400.00       3700.00     0.11
sort                2     7964.44    68266.67        700.00     0.02
sqlplus            14    23767.46    92160.00      32100.00     0.94
sshd                1     4452.17     4452.17        200.00     0.01
sy-check            2      806.30     3375.82       1000.00     0.03
tbl                 1   273066.67   273066.67        800.00     0.02
uncompre            1   256000.00   256000.00        500.00     0.01
who                 4   187733.33   256000.00       1100.00     0.03
whoami              3    61440.00   102400.00        300.00     0.01

Then stop the tracing with:

server1{root}# /usr/sbin/acct/accton

And you see…. nothing… It does not work, I/Os are 50-50% split per PRM groups and PRM is absolutely not playing its role. Opened a call with HP support and they just quoted the official documentation:

PRM does not support disk bandwidth management on VERITAS File System (VxFS)
Issue PRM does not support disk bandwidth control on VxFS. The reason for this limitation is that VxFS does not support the implementation of I/O disk bandwidth that PRM relies on. The Unix File System (HFS) supported this feature, but when HP moved to VERITAS File System 4.1, the daemon invalidated this feature for all the current HP-UX versions.
Workaround There is no immediate fix or workaround known for this issue.

So until next PRM release there is no option to limit I/O with VxFS, filesystems that is used by at least 99% of HPUX customers…

Useful commands

server1{root}# prmavail MEM
85500 real memory pages or 333 MB available (PRM estimate)
server1{root}# ps -efP
     UID        PRMID    PID  PPID  C    STIME TTY       TIME COMMAND
    root    (PRM_SYS)      0     0  0  Apr 28  ?         9:47 swapper
    root    (PRM_SYS)      1     0  0  Apr 28  ?         8:27 init
    root    (PRM_SYS)      8     0  0  Apr 28  ?         0:00 kmemdaemon
    root    (PRM_SYS)      9     0  0  Apr 28  ?         0:01 ioconfigd
    root    (PRM_SYS)     10     0  0  Apr 28  ?         0:00 ObjectThreadPool
    root    (PRM_SYS)     11     0  0  Apr 28  ?        12:31 nfsktcpd
    root    (PRM_SYS)     12     0  0  Apr 28  ?         0:21 autofskd
    root    (PRM_SYS)     13     0  0  Apr 28  ?         4:09 lvmkd
    root    (PRM_SYS)     14     0  0  Apr 28  ?         5:25 lvmkd

References

About Post Author

Yannick Jaquier

administrator

Find more about me on social media.

See author's posts

Share the knowledge!

Related Posts:

3 thoughts on “HP Process Resource Manager (PRM)”

  1. Shantanu bhardwaj says:

    Hello,

    Could you please help me to know how to stop kmemdeamon on the HP-UX 11.31?
    I have a server where kmemdaemon is constantly consuming high CPU (90 – 150%) and this server is facing high performance issue. I want to disable the kmemdaemon to check how server behaves after that?

    Best Regards

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes:

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>