CPU monitoring in Unix Systems using command line tools

UNIX

Linux and Solaris also provide command line tools to monitor CPU utilization. These command line tools are useful when you want to keep a running textual history of CPU utilization or keep a log of CPU utilization. Linux and Solaris have vmstat, which shows combined CPU utilization across all virtual processors. Both versions of vmstat optionally take a reporting interval, in seconds, as a command line argument. If no reporting interval is given to vmstat, the reported output is a summary of all CPU utilization data collected since the system has last been booted. When a reporting interval is specified, the first row of statistics is a summary of all data collected since the system was last booted. As a result, the first row of data from vmstat is most often ignored. The display format of vmstat for Linux and Solaris is similar. For example, the following shows vmstat from Linux. The columns of interest for monitoring CPU utilization are shown in bold.

vmstat for Linux and Solaris

The “us” column shows the percentage of user CPU utilization. The “sy” column shows the percentage of kernel or system CPU utilization. The “id” column shows the percentage of idle or available CPU. The sum of the “us” column and “sy” column should be equal to 100 minus the value in the “id” column, that is, 100 – (“id” column value).

The vmstat output from Solaris, shown in the following example, has three columns of CPU utilization interest and has column headings of “us,” “sy,” and “id” that show user, kernel or system, and idle CPU utilization, respectively.

vmstat output from Solaris

Solaris and Linux also offer a tabular view of CPU utilization for each virtual processor using the command line tool mpstat.

Using mpstat to observe per virtual processor CPU utilization can be useful in identifying whether an application has threads that tend to consume larger percentages of CPU cycles than other threads or whether application threads tend to utilize the same percentage of CPU cycles. The latter observed behavior usually suggests an application that may scale better. CPU utilization in Solaris mpstat, as shown in the following example, is reported in the columns “usr,” “sys,” “wt,” and “idl,” where usr is the percentage of CPU time spent executing user code, sys is the percentage of CPU time spent executing kernel code, wt is the percentage of I/O wait time (no longer calculated and always reports 0), and idl is percentage of time the CPU was idle.

mpstat to observe per virtual processor CPU utilization

 

 

If no reporting interval is given to mpstat, the reported output is a summary of all mpstat data collected since the system was last booted. When a reporting interval is given, the first row of statistics is a summary of all data collected since the system was last booted.

Other popular alternatives to vmstat on Solaris and Linux can be used to monitor CPU utilization. A couple of the more common ones are prstat for Solaris and top for Linux.

Linux top reports not only CPU utilization but also process statistics and memory utilization. Its display, shown in the following example, has two major parts. The upper section of the display reports overall system statistics, while the lower section reports process level statistics that, by default, are ordered in highest to lowest CPU utilization.

 

CPU utilization but also process statistics and memory utilization

 

Solaris prstat shows similar information to Linux top. The following example is the default output for prstat.

per process level statistics that are ordered, by default, from highest to lowest CPU utilization

Solaris prstat does not show an overall system summary section like top. But, like top, it does report per process level statistics that are ordered, by default, from highest to lowest CPU utilization. Both prstat and top are good tools for providing a high level view of CPU utilization at a per process level. But as the need arises to focus more on per process and per lightweight process CPU utilization, Solaris prstat has additional capabilities such as reporting both user and kernel or system CPU utilization along with other microstate information using the prstat -m and -L options. The -m option prints microstate information, and -L prints statistics on per lightweight process.

Using the -m and -L options can be useful when you want to isolate CPU utilization per lightweight process and Java thread. A Java process showing high CPU utilization with prstat -mL can be mapped to a Java process and Java thread(s) on Solaris through a sequence of steps using prstat, pstack, and Java 6’s jstack command line tool. The following example illustrates how to do this. The output in the following example, gathered with prstat -mL 5, shows process id 3897 has three lightweight process ids consuming about 5% of kernel or system CPU. LWPID 2 is consuming the most at 5.7%.

 

PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROC/LWPID
3897 techpaste 6.0 5.7 0.1 0.0 0.0 2.6 8.2 78 9K 8K 64K 0 java/2
3897 techpaste 4.9 4.8 0.0 0.0 0.0 59 0.0 31 6K 6K 76K 0 java/13
3897 techpaste 4.7 4.6 0.0 0.0 0.0 56 0.0 35 5K 6K 72K 0 java/14
3917 techpaste 7.4 1.5 0.0 0.0 0.0 3.8 53 34 5K 887 16K 0 java/28
...

In the absence of using a profiler, there is a quick way to isolate which Java thread, along with which Java method, is consuming large amounts of CPU as reported by prstat, either USR or SYS. A Java, or JVM, process thread stack dump at the Solaris level can be generated using the Solaris command line tool pstack and the process id 3897. The pstack output in the following example, produced using the command pstack 3897/2, shows the lightweight process (lwp) id and thread id that matches LWPID 2 from prstat.

----------------- lwp# 2 / thread# 2 --------------------
fef085c7 _lwp_cond_signal (81f4200) + 7
feb45f04 __1cNObjectMonitorKExitEpilog6MpnGThread_pnMObjectWaiter__v_
(829f2d4, 806f800, e990d710) + 64
fe6e7e26 __1cNObjectMonitorEexit6MpnGThread__v_ (829f2d4, 806f800) + 4fe
fe6cabcb __1cSObjectSynchronizerJfast_exit6FpnHoopDesc_pnJBasicLock_
pnGThread__v_ (ee802108, fe45bb10, 806f800) + 6b

 

If you convert the thread id value to hexadecimal and use the JDK’s jstack command you can find the Java thread that corresponds to Solaris thread# 2 by searching for a “nid” label. The thread number, 2 in decimal, is also 2 in hexadecimal. The following output from the JDK’s jstack command is trimmed but shows that a Java thread with a 0x2 is the “main” Java thread. According to the stack trace produced by jstack, the Java thread corresponding to Solaris pstack’s LWPID 2 and prstat’s LWPID 2 is executing a Java NIO Selector.select() method.

”main” prio=3 tid=0x0806f800 nid=0x2 runnable [0xfe45b000..0xfe45bd38]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:164)
at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0xee809778> (a sun.nio.ch.Util$1)
- locked <0xee809768> (a java.util.Collections$UnmodifiableSet)
- locked <0xee802440> (a sun.nio.ch.DevPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at com.sun.grizzly.SelectorThread.doSelect(SelectorThread.java:1276)

 

Once a Java thread has been identified and with the stack trace readily available, you can begin to investigate in more detail the methods shown in the stack trace for possible candidates of high kernel or system CPU utilization through a more thorough profiling activity.

 

In case of any ©Copyright or missing credits issue please check CopyRights page for faster resolutions.

Leave a Reply