Monitor JVM Memory Usage

Java

A Java application or JVM that is swapping or utilizing virtual memory experiences pronounced performance issues. Swapping occurs when there is more memory being consumed by applications running on the system than there is physical memory available. To deal with this potential situation, a system is usually configured with an area called swap space. Swap space is typically allocated on a disk in a distinct disk partition. When the amount of physical memory is exhausted by the applications running on the system, the operating system swaps out a portion of an application to swap space on disk. Usually the operating system swaps out a portion of an application that is executing the least frequently so as to not impact the applications or the portions of applications that are the busiest. When a portion of an application is accessed that has been swapped out, that portion of the application must be paged in from the swap space on disk to memory. Swapping in from disk to memory can have a significant impact on an application’s responsiveness and throughput.


ADVERTISEMENT

 

A JVM’s garbage collector performs poorly on systems that are swapping because a large portion of memory is traversed by the garbage collector to reclaim space from objects that are unreachable. If part of the Java heap has been swapped out it must be paged into memory so its contents can be scanned for live objects by the garbage collector. The time it takes to page in any portion of the Java heap into memory can dramatically increase the duration of a garbage collection. If the garbage collection is a “stop the world” type of operation, one that stops all application threads from executing, a system that is swapping during a garbage collection is likely to experience lengthy JVM induced pause times.

If you observe lengthy garbage collections, it is a possibility that the system is swapping. To prove whether the lengthy garbage collection pauses are caused by swapping, you must monitor the system for swapping activity.

Monitoring Memory Utilization on Windows

On Windows systems that include the Performance Monitor, monitoring memory pages per second (\Memory\Pages / second) and available memory bytes (\Memory\Available MBytes), can identify whether the system is swapping. When the available memory, as reported by the \Memory\Available MBytes counter, is low and you observe paging activity, as reported by the \Memory\Pages / Second counter, the system is likely swapping.
It is easiest to illustrate a Windows system that is swapping using the output from the typeperf command. The following is a typeperf command to report available memory and paging activity at 5 second intervals (the -si specifies the reporting interval).

typeperf -si 5 ”\Memory\Available Mbytes” ”\Memory\Pages/sec”


The following output from typeperf is taken from a system that is swapping. The first column of data is the date and time stamp. The second column is the available memory, and the third column is the pages per second.

”02/15/2011 15:28:11.737”,”150.000000”,”0.941208”
”02/15/2011 15:28:16.799”,”149.000000”,”1.857361”
”02/15/2011 15:28:21.815”,”149.000000”,”2.996049”
”02/15/2011 15:28:26.831”,”149.000000”,”17.687691”
”02/15/2011 15:28:31.909”,”149.000000”,”0.929074”
”02/15/2011 15:28:36.940”,”149.000000”,”1.919541”
”02/15/2011 15:28:41.956”,”149.000000”,”0.991037”
”02/15/2011 15:28:46.971”,”149.000000”,”1.977258”
”02/15/2011 15:28:51.002”,”149.000000”,”0.969558”
”02/15/2011 15:28:56.065”,”149.000000”,”14.120284”
”02/15/2011 15:29:01.127”,”150.000000”,”8.470692”
”02/15/2011 15:29:06.174”,”152.000000”,”9.552139”
”02/15/2011 15:29:11.174”,”151.000000”,”2.000104”
”02/15/2011 15:29:16.174”,”152.000000”,”1.999969”
”02/15/2011 15:29:21.174”,”153.000000”,”0.999945”

 

Notice the amount of memory available is staying fairly constant around 150 megabytes yet there is consistent paging activity. Since the amount of available memory is staying fairly constant, it is reasonable to assume no new applications are being launched. When an application launches, the amount of available memory is expected to drop, and it is expected to see paging activity since the application must be paged into memory. Therefore, if the system is using a fairly consistent amount of memory and no new applications are launching, yet there is paging activity, it is likely the system is swapping.

It is important to note that a system can report little available memory and report no paging activity. In such a situation, the system is not swapping. It just simply is utilizing most of the physical RAM available on the system. Likewise, a system may be experiencing paging activity, yet have sufficient memory available and as a result not be swapping. The paging activity could be the result of an application being launched.

Monitoring Memory Utilization on Solaris

On Solaris, when available memory becomes low, the kernel’s page scanner begins looking for memory pages no longer in use by an application so they can be made available for other applications and processes. If the page scanner is unable to find the memory demanded by the applications and no additional physical memory is available, it begins to swap out the least recently used memory pages to a swap space on disk. The lower the amount of available memory, the higher the page scan rate.

In other words, as lower memory is available, the page scanner gets more aggressive with trying to find available memory pages it can reclaim. Since the page scanner becomes more aggressive as available memory becomes low, identifying a Solaris system that is experiencing swapping requires monitoring a combination of the amount of free memory and page scanner activity. Both available free memory and page scanner activity are reported in Solaris vmstat columns labeled “free” and “sr.”

When vmstat, cpubar, or any other Solaris monitoring tool reports a scan rate of 0, regardless of the reported available free memory, no swapping is occurring. However, if the scan rate is nonzero and the trend of reported free memory is decreasing, then swapping is likely occurring. The following example output from Solaris vmstat illustrates a system currently using most of its available physical memory; about 100 megabytes are free, as shown in the “free” column, but it is not swapping since its scan rate, the “sr” column, is 0.

 

kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr cd f0 s0 - in sy cs us sy id
0 0 0 1641936 861222 106 2591 0 3 3 0 0 0 0 0 0 4930 24959 10371 60 10 30
0 0 0 1594944 116940 37 1718 8 0 0 0 0 8 0 0 0 4169 17820 10111 52 5 43
0 0 0 1579952 103208 24 521 0 0 0 0 0 1 0 0 0 2948 14274 6814 67 4 29
0 0 0 1556244 107408 97 1116 3 0 0 0 0 11 0 0 0 1336 7662 1576 45 3 52

 

In contrast, the following example illustrates a system that is experiencing a shortage of available physical memory, dropping pretty rapidly from about 150 Mbytes to 44 Mbytes, and by the time it reaches 17 Mbytes, the scan rate, the “sr” column, is reporting significant activity. Observing this kind of pattern with vmstat is an indication the system may be swapping and its performance will begin to become sluggish if it is not already.

kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr cd f0 s0 -- in sy cs us sy id
1 0 0 499792 154720 1 1697 0 0 0 0 0 0 0 0 12 811 612 1761 90 7 4
1 0 0 498856 44052 1 3214 0 0 0 0 0 0 0 0 12 1290 2185 3078 66 18 15
3 0 0 501188 17212 1 1400 2 2092 4911 0 37694 0 53 0 12 5262 3387 1485 52 27 21
1 0 0 500696 20344 26 2562 13 4265 7553 0 9220 0 66 0 12 1192 3007 2733 71 17 12
1 0 0 499976 20108 3 3146 24 3032 10009 0 10971 0 63 0 6 1346 1317 3358 78 15 7
1 0 0 743664 25908 61 1706 70 8882 10017 0 19866 0 78 0 52 1213 595 688 70 12 18

Notice in the example, paying attention only to either the “free” or “swap” columns can be misleading and alone do not provide obvious clues that a system may be swapping.

Monitoring Memory Utilization on Linux

On Linux, monitoring for swapping activity can be done using vmstat and observing the free column. There are other ways to monitor for swap activity on Linux such as using the top command or observing the contents of the file /proc/meminfo. Monitoring for swapping activity using Linux vmstat is shown here. The columns in Linux vmstat to monitor are the “si” and “so” columns, which represent the amount of memory paged-in and the amount of memory paged-out. In addition, the “free” column reports the amount of available free memory. The actual units are not as important as observing whether the amount of free memory is low and high paging activity is occurring at the same time. Observing the pattern just described in these statistics is an indication that the system maybe experiencing swapping activity. The following is an example of a system that is experiencing no swapping activity; since there is no paging activity as shown in the “si” and “so” columns and the amount of free memory is not very low.

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 0 0 9383948 265684 1879740 0 0 0 0 1 1 0 0 100 0 0
3 0 0 9383948 265684 1879740 0 0 0 11 1012 529 14 0 86 0 0
3 0 0 9383916 265684 1879740 0 0 0 0 1021 5105 20 0 80 0 0
3 0 0 9383932 265684 1879740 0 0 0 13 1014 259 19 0 81 0 0
3 0 0 9383932 265684 1879740 0 0 0 7 1018 4952 20 0 80 0 0

 

However, the following vmstat output from a Linux system illustrates a system that is experiencing swapping.

procs ------------memory----------- ----swap--- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 0 9500784 265744 1879752 0 0 0 0 1015 228 0 6 94 0 0
1 0 0 8750540 265744 1879752 0 0 0 2 1011 216 0 6 94 0 0
1 0 0 2999792 265744 1879752 0 0 0 2 1012 218 0 6 94 0 0
2 0 0 155964 185204 1370300 0 0 0 0 1009 215 0 9 90 0 0
2 0 9816 155636 24160 815332 0 1963 0 2000 1040 238 0 13 87 0 0
0 2 218420 165152 384 18964 0 41490 0 41498 1247 286 0 6 88 5 0
0 6 494504 157028 396 18280 45 55217 67 55219 1363 278 0 1 79 21 0
0 7 799972 159508 408 18356 70 61094 145 61095 1585 337 0 1 72 27 0
0 8 1084136 155592 416 18512 65 56833 90 56836 1359 292 0 1 75 24 0
0 3 1248428 174292 500 23420 563 32858 1689 32869 1391 550 0 0 83 17 0
1 1 1287616 163312 624 28800 13901 7838 15010 7838 2710 6765 1 0 93 6 0
1 0 1407744 163508 648 29688 18218 24026 18358 24054 3154 2465 1 1 92 6 0
0 2 1467764 159484 648 28380 19386 12053 19395 12118 2893 2746 2 1 91 5 0

 

Notice the pattern in this example. Where free memory initially decreases, there is little paging activity shown in either the “si” column or “so” column. But as free memory reaches values in the 155,000 – 175,000 range, page-out activity picks up as shown in the “so” column. Once the page-out activity begins to plateau, the page-in activity begins and increases rather quickly as shown in the “si” column. In general what is happening is the system has an application, or set of applications, that placed significant memory allocation and/or memory access pressure on the system. As the amount of physical memory started to become exhausted, the system began to page-out to virtual memory the least recently used pages in memory. As the applications on the system began to demand pages from memory, page-in activity began to occur. As the paging activity increased, the amount of free memory remained about the same. In other words, the system is swapping in pages nearly as quickly as  it is paging them out while the amount of free memory remained rather small. This is a typical pattern that can be observed in Linux vmstat when a Linux system is experiencing swapping.

 

In case of any ©Copyright or missing credits issue please check CopyRights page for faster resolutions.

Leave a Reply