Performance Monitoring and Tunig
Performance monitoring
Performance Tuning
Monitoring Commands
nmon vmstat iostat sar topas nmon svmon filemon mpstat rmss netpmon
nmon
http://www-128.ibm.com/developerworks/eserver/articles/analyze_aix/index.html
vmstat
The vmstat command is useful for obtaining an overall picture of CPU, paging, and memory usage. The following is a sample report produced by the vmstat command:
- vmstat 5 2
kthr memory page faults cpu ----- ----------- ------------------------ ------------ ----------- r b avm fre re pi po fr sr cy in sy cs us sy id wa 1 1 197167 477552 0 0 0 7 21 0 106 1114 451 0 0 99 0 0 0 197178 477541 0 0 0 0 0 0 443 1123 442 0 0 99 0
Remember that the first report from the vmstat command displays cumulative activity since the last system boot. The second report shows activity for the first 5-second interval.
The iostat command is the fastest way to get a first impression, whether or not the system has a disk I/O-bound performance problem. This tool also reports CPU statistics.
Flags
-a Specifies adapter throughput report.
-d Specifies drive report only.
-m Specifies statistics for paths.
-t Specifies tty/cpu report only.
-z Resets the disk input/output statistics
# iostat 2 2
tty: tin tout avg-cpu: % user % sys % idle % iowait
0.0 0.8 8.4 2.6 88.5 0.5
0.0 80.2 4.5 3.0 92.1 0.5
Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk3 0.0 0.3 0.0 258032 224266
hdisk2 0.1 1.1 0.0 258088 1658678
The svmon command provides a more in-depth analysis of memory usage. It is more informative, but also more intrusive, than the vmstat and ps commands. The svmon command captures a snapshot of the current state of memory.
The memory consumption is reported using the inuse, free, pin, virtual and paging space counters.
- The inuse counter represents the number of used frames.
- The free counter represents the number of free frames from all memory pools.
- The pin counter represents the number of pinned frames, that is, frames that cannot be swapped.
- The virtual counter represents the number of pages allocated in the system virtual space.
- The paging space counter represents the number of pages reserved or used on paging spaces.
Flags
-G Global report -U User report -P Process report -i To define intervel and number of intervels. eg. -i 1 5
To find out the total memory/swap and free memory/swap available in an AIX system
# svmon -G
size inuse free pin virtual
memory 3932160 3914793 17367 444363 1609451
pg space 1048576 6622
work pers clnt
pin 444363 0 0
in use 1609451 0 2305342
PageSize PoolSize inuse pgsp pin virtual
s 4 KB - 3787625 6622 370027 1482283
m 64 KB - 7948 0 4646 7948
# pagesize
4096
So, the above system have almost 16GB physical Memory and 4 GB swap
A memory leak can be detected with the svmon command, by looking for processes whose working segment continually grows. A leak in a kernel segment can be caused by an mbuf leak or by a device driver, kernel extension, or even the kernel. To determine if a segment is growing, use the svmon command with the -i option to look at a process or a group of processes and see if any segment continues to grow.
# svmon -P 13548 -i 1 2 Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd LPage 13548 pacman 8535 2178 847 8533 N N N Vsid Esid Type Description LPage Inuse Pin Pgsp Virtual 0 0 work kernel seg - 4375 2176 847 4375 48412 2 work process private - 2357 2 0 2357 6c01b d work shared library text - 1790 0 0 1790 4c413 f work shared library data - 11 0 0 11 3040c 1 pers code,/dev/prodlv:4097 - 2 0 - - ginger :svmon -P 13548 -i 1 3 Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd LPage 13548 pacman 8589 2178 847 8587 N N N Vsid Esid Type Description LPage Inuse Pin Pgsp Virtual 0 0 work kernel seg - 4375 2176 847 4375 48412 2 work process private - 2411 2 0 2411 6c01b d work shared library text - 1790 0 0 1790 4c413 f work shared library data - 11 0 0 11 3040c 1 pers code,/dev/prodlv:4097 - 2 0 - -
filemon
The filemon command monitor a trace for file system and IO system events and reports performance statistics for files, virtual memory segments, logical volumes and physical volumes. filemon is useful to those whose applications are believed to be disk-bound and want to know where and why.
filemon command shows the load on different disks, logical volumes and files in a great detail.
trcstop command is used to stop the filemon monitoring.
The syntax of filemon command is
filemon [-o output_file] [-O levels] [-u] [-v]
-O [lv | pv | vm | If | all ]
(If - Logical file level, vm - Virtual memory level, lv - lv level)
-u Reports on files that were opened prior to the start of the trace daemon
If the output file is not specified, the output is sent to standard output.
To start the filemon monitoring for 1 min.
# filemon -uo filemon.out -O all ; sleep 60; trcstop
To find out the most active Logical Volumes
# awk '/Most Active Logical Volumes/,/^$/' filemon.out Most Active Logical Volumes ------------------------------------------------------------------------ util #rblk #wblk KB/s volume description ------------------------------------------------------------------------ 0.04 4208 0 34.9 /dev/paging00 paging 0.04 4000 0 33.2 /dev/hd6 paging 0.01 1680 11408 108.6 /dev/oralvr32 /oracle/R32 0.00 0 264 2.2 /dev/hd8 jfs2log
To find out most active Files
# awk '/Most Active Files/,/^$/' filemon.out
To find out most active physical Volumes
# awk '/Most Active Physical Volumes/,/^$/' filemon.out
rmss
The rmss command provides you with a means to simulate different sizes of real memory that are smaller than your actual machine, without having to extract and replace memory boards or reconfigure memory using logical partitions.
To change the memory size to 500 MB,
# rmss -c 500 Simulated memory size changed to 500 Mb.
To reset the memory size to the real memory size of the machine, enter:
# rmss -r
Tuning Commands
vmo ioo no nice and renice vmtune defragfs
The /etc/tunables commands
To manage its files in the /etc/tunables directory, new commands have been added to AIX. They are as follows:
tuncheck: This command validates a file either to be applied immediately or at reboot time (-r flag). It checks the ranges, dependencies, and prompts to run bosboot if required. Run this command if you copy a file to a new system, or edit it with an editor such as vi.
tunsave: This command saves all current values to a file, including optionally the
nextboot file.
tunrestore: This command applies values from a file, either immediately, or at the next reboot (-r flag). With the -r flag, it validates and copies the file over the current nextboot file.
tundefault: This command resets all parameters to their default value. It can be applied at the next reboot with the -r flag.
ioo, vmo and no commands:
These commands are used to set or display current or next boot values of different tuning parameters.
- ioo for IO tuning parameters
- vmo for Virtual Memory Manager parameters
- no for network tuning parameters
These commands can also make permanent changes or defer changes until the next reboot. When a permanent change or nextboot value is changed using these commands, the /etc/tunables/nextboot file is updated automatically with the new values (if the new value is different from the default value).
The following flags are common for ioo, vmo and no commands.
-L [tunable] List the characteristics of one or all tunables -d tunable - Resets 'tunable' to default value -o [tunable] - Display the current value of 'tunable' -o tunable=<value> - Set the 'tuneble' to new value -D - Resets all tunables to their default value -p - Changes apply to both current and reboot values (/etc/tunables/nextboot file updated) -r - Changes apply to only reboot values (/etc/tunables/nextboot file updated)
Examples:
# vmo -p -o minfree=1200 -o maxfree=1280 # ioo -r -o maxpgahead=64 -o j2_minPageReadAhead=8 # no -r -o rfc1323=1 -o tcp_recvspace=262144 -o tcp_sendspace=262144
A sample /etc/tunables/nextboot file.
cat /etc/tunables/nextboot
vmo:
minfree = "1200"
maxfree = "1280"
minperm% = "10"
maxperm% = "40"
maxclient% = "40"
ioo:
j2_nBufferPerPagerDevice = "1024"
no:
tcp_recvspace = "65536"
tcp_sendspace = "65536"
tcp_pmtu_discover = "0"
udp_pmtu_discover = "0
minfree Minimum acceptable number of real-memory page frames in the free list. When the size of the free list falls below this number, the VMM begins stealing pages. It continues stealing pages until the size of the free list reaches maxfree.
maxfree Maximum size to which the free list will grow by VMM page-stealing. The size of the free list may exceed this number as a result of processes terminating and freeing their working-segment pages or the deletion of files that have pages in memory.
minperm If the percentage of real memory occupied by file pages falls below this level, the page-replacement algorithm steals both file and computational pages, regardless of repage rates.
maxperm' If the percentage of real memory occupied by file pages rises above this level, the page-replacement algorithm steals only file pages.
maxclient If the percentage of real memory occupied by file pages is above this level, the page-replacement algorithm steals only client pages.
Tuning Asynchronous IO parameters
# smit aio
or
# chdev -l aio0 -P -a minservers=100 -a maxservers=100 -a maxreqs=9152
-a autoconfig=available
minserver: Minimum number of kernel processes dedicated to asynchronous I/O processing
masservers: Maxiimum number of kernel processes dedicated to AIO processing
maxreqs: maximum number of asynchronous I/O requests that can be outstanding at one time
autoconfig: The state to which AIO is to be configured during system initialization. The possible values are "defined", which means that AIO cannot be used, and "available"
Disk IO pacing (High water-mark and Low Water-mark)
Disk-I/O pacing is intended to prevent programs that generate very large amounts of output from saturating the system’s I/O facilities and causing the response times of less-demanding programs to deteriorate.
When a process tries to write to a file that already has high-water mark pending writes, the process is put to sleep until enough I/Os have completed to make the number of pending writes less than or equal to the low-water mark. The logic of I/O-request handling does not change. The output from high-volume processes is slowed down somewhat.
The maxpout parameter specifies the number of pages that can be scheduled in the I/O state to a file before the threads are suspended. The minpout parameter specifies the minimum number of scheduled pages at which the threads are woken up from the suspended state. The default value for both the maxpout and minpout parameters is 0, which means that the I/O pacing feature is disabled. Changes to the system-wide values of the maxpout and minpout parameters take effect immediately without rebooting the system.
Changing the values for the maxpout and minpout parameters overwrites the system-wide settings. You can exclude a file system from system-wide I/O pacing by mounting the file system and setting the values for the maxpout and minpout parameters explicitly to 0. The following command is an example: mount -o minpout=0,maxpout=0 /<file system>
To change the high water-mark level
# chdev -a maxpout=20 -l sys0
netpmon Monitors activity and reports statistics on network I/O and network-related CPU usage