Steps to investigate Java Low on Memory in HP Service Manager(SM)

HP Service Manager Low memory

Log report:
2120( 2232) 02/04/2014 13:25:02 JRTE I Starting Memory Monitoring thread to check for memory every 15 seconds.
2120( 5980) 02/04/2014 13:25:02 JRTE D JavaMemory Max(123928576) Used(1011872) %Used(0.0)
2120( 5980) 02/04/2014 13:25:02 JRTE D NativeMemory Max(2147352576) Used(358121472) %Used(16.0)
2120( 5980) 02/04/2014 13:28:32 JRTE W Process Low on Java Memory. Max(123928576) Used(119603800) PercentUsed(96.0)
2120 02/04/2009 13:30:14 JRTE W Send error response: Server is running low on memory try again.
2120 02/04/2009 13:30:16 JRTE W Send error response: Server is running low on memory try again.
2120( 5980) 02/04/2014 13:30:47 RTE I Process Java Heap Memory is back to normal range.
2120( 5980) 02/04/2014 13:30:47 JRTE I Process Java Memory. Max(123928576) Used(4256640) PercentUsed(3.0)

Response:
When a Java Virtual Machine (JVM) is started in Service Manager process, the Service Manager process defines the minimum and the maximum amount of memory that is allocated for JVM.

The minimum and maximum vary for different kinds of Service Manager processes.

* For servlet container processes that needs more java processing, the minimum and maximum defined are 256M. For processes that are shortlived and do not need much java processing, the minimum and maximum defined are 64M and 96M.

* You can override these default values by using the JVMOption parameters.
For example, you could set JVMOption0:-Xms128M, JVMOption1:-Xmx256M, to provide minimum and maximum values of 128M and 256M for java heap memory.

* The JVM internally divides the allocated heap into 4 different memory pools. The Service Manager process monitor for the low memory condition on “Tenured Gen(Old Generation)” only. Therefore, the low memory condition still can occur on other memory pools like “Eden Space (Young Generation)” or “Survivor Space”. For detailed explanation of Garbage collection and Memory Pools, refer to Sun Microsystem’s Memory Management whitepaper”.

Reasons for Low Memory:
1. Soap Request from temip is having large response.
2. Huge Attachment with ticket
3. Single Ticket with many activity records.
4. Web services requesting all keys.

You need to monitor the memory usage on servlets on regular basis.
– Check if servlet is low on Native or JVM memory
– Identify the frequency of low memory occurrence

Check following configuration on servers where servlets reporting low on memory:
* Shared memory
* threads per process
* Apps threads per session (configured in sm.ini) checks if many tabs are open, max tabs are 5-10
* JVM min,max

You can turn on detailed messages of the memory monitoring thread by adding the following parameter to the sm.ini file:

log4jdebug:com.hp.ov.sm.common.oom.LowMemoryHandler

Messages like these will be sent to the sm.log file when the parameter is turned on
.
Warning messages are written to the log whether detailed messages are turned on or not
2120( 2232) 02/04/2014 13:25:02 JRTE I Starting Memory Monitoring thread to check for memory every 15 seconds.
2120( 5980) 02/04/2014 13:25:02 JRTE D JavaMemory Max(123928576) Used(1011872) %Used(0.0)
2120( 5980) 02/04/2014 13:25:02 JRTE D NativeMemory Max(2147352576) Used(358121472) %Used(16.0)
2120( 5980) 02/04/2014 13:28:32 JRTE W Process Low on Java Memory. Max(123928576) Used(119603800) PercentUsed(96.0)
2120 02/04/2009 13:30:14 JRTE W Send error response: Server is running low on memory try again.
2120 02/04/2009 13:30:16 JRTE W Send error response: Server is running low on memory try again.
2120( 5980) 02/04/2014 13:30:47 RTE I Process Java Heap Memory is back to normal range.
2120( 5980) 02/04/2014 13:30:47 JRTE I Process Java Memory. Max(123928576) Used(4256640) PercentUsed(3.0)

Look for the following messages in the logs while you monitor the servlets:

1. –Memory and if memory is consuming more memory. S( ) O( ) Max( 100M )
2. Large Response
3. If the above two are not seen in logs then most probably memory leak

When troubleshooting memory leak issue you need to configure java.lang.OutOfMemoryError messages, please ensure that you enable following JVM option in sm.ini/sm.cfg

This allows the JVM running SC/SM servlet to generate a diagnostic Java heap dump (HPROF or PHD (on AIX)) file which can be analyzed by our teams to determine root cause. This is a standard Java parameter useful in troubleshooting these errors and has no significant performance overhead associated with leaving it enabled indefinitely.

JVMOptionN:-XX:+HeapDumpOnOutOfMemoryError

Where N = 1 or higher depending on pre-existing options.

As a hard and fast rule, never set Xmx values greater than 512MB. As such, even 512MB is likely too large a value. These large values take away from the native (non Java related) memory required by SM to run correctly. In addition, as a secondary rule, customers should be on the latest patch available as there are continuous server resiliency, performance, and memory improvements.

The threadsperprocess values specified in sm.ini/sc.ini files has a direct impact on memory usage as well. Please be aware that the default value of 50 is typically sufficient in Windows platform deployments. On unix and Windows 64-bit platforms, the default value can be set to 70 without any cause for concern. This is because the memory management models on these platforms allows for larger values without negative impact to SM/SC performance.

Please note that threadsperprocess can also be set higher than 50 for SM/SC servlets that are dedicated to hosting ONLY Web Services transactions. This is because WS transactions do not have the typical overhead associated with SM GUI clients

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s