Server Crashing

 
Author
Message
SysAider
33
 
Hi,

I'm getting this error in my tomcat log and then the server hangs.

$ more localhost.2010-08-03.log

Aug 3, 2010 12:36:38 PM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet HelpDesk_jsp threw exception
java.lang.OutOfMemoryError: PermGen space
Aug 3, 2010 12:38:17 PM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet HelpDesk_jsp threw exception
java.lang.OutOfMemoryError: PermGen space
Aug 3, 2010 12:39:28 PM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet EndUserSRView_jsp threw exception
java.lang.OutOfMemoryError: PermGen space


$ more catalina.2010-08-03.log

Aug 3, 2010 12:36:45 PM org.apache.coyote.http11.Http11Processor process
SEVERE: Error processing request
java.lang.OutOfMemoryError: PermGen space
Aug 3, 2010 12:36:48 PM org.apache.catalina.connector.CoyoteAdapter service
SEVERE: An exception or error occurred in the container during the request processing
java.lang.OutOfMemoryError: PermGen space
Aug 3, 2010 12:36:50 PM org.apache.coyote.http11.Http11Processor process
SEVERE: Error finishing response
java.lang.OutOfMemoryError: PermGen space
Aug 3, 2010 12:36:57 PM org.apache.catalina.connector.CoyoteAdapter service
SEVERE: An exception or error occurred in the container during the request processing
java.lang.OutOfMemoryError: PermGen space
Aug 3, 2010 12:36:57 PM org.apache.catalina.connector.CoyoteAdapter service
SEVERE: An exception or error occurred in the container during the request processing
java.lang.OutOfMemoryError: PermGen space
Aug 3, 2010 12:37:01 PM org.apache.catalina.connector.CoyoteAdapter service
SEVERE: An exception or error occurred in the container during the request processing
java.lang.OutOfMemoryError: PermGen space
Aug 3, 2010 12:37:04 PM org.apache.coyote.http11.Http11Processor process
SEVERE: Error finishing response
java.lang.OutOfMemoryError: PermGen space
Aug 3, 2010 12:37:19 PM org.apache.catalina.connector.CoyoteAdapter service
SEVERE: An exception or error occurred in the container during the request processing
java.lang.OutOfMemoryError: PermGen space
Aug 3, 2010 12:37:19 PM org.apache.coyote.http11.Http11Processor process
SEVERE: Error processing request
java.lang.OutOfMemoryError: PermGen space
Aug 3, 2010 12:37:23 PM org.apache.coyote.http11.Http11Processor process
SEVERE: Error processing request
java.lang.OutOfMemoryError: PermGen space
Aug 3, 2010 12:37:20 PM org.apache.catalina.connector.CoyoteAdapter service
SEVERE: An exception or error occurred in the container during the request processing
java.lang.OutOfMemoryError: PermGen space
Aug 3, 2010 12:37:28 PM org.apache.catalina.connector.CoyoteAdapter service
SEVERE: An exception or error occurred in the container during the request processing
java.lang.OutOfMemoryError: PermGen space
Aug 3, 2010 12:37:29 PM org.apache.coyote.http11.Http11Processor process
SEVERE: Error finishing response
java.lang.OutOfMemoryError: PermGen space
Aug 3, 2010 12:38:06 PM org.apache.catalina.connector.CoyoteAdapter service
SEVERE: An exception or error occurred in the container during the request processing
java.lang.OutOfMemoryError: PermGen space
Aug 3, 2010 12:38:06 PM org.apache.catalina.connector.CoyoteAdapter service
SEVERE: An exception or error occurred in the container during the request processing
java.lang.OutOfMemoryError: PermGen space

$ more sysaid.log

2010-08-03 12:31:47,105 [http-80-7] ERROR com.sysaid - Exception while set default to status
java.lang.NumberFormatException: For input string: "New"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:4
at java.lang.Integer.parseInt(Integer.java:449)
at java.lang.Integer.parseInt(Integer.java:499)
at com.sysaid.server.ai.readControl(Unknown Source)
at com.sysaid.util.FormHandler.setDefaults(Unknown Source)
at com.sysaid.jsp.SREdit_jsp._jspService(Unknown Source)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at jcifs.http.NtlmHttpFilter.doFilter(NtlmHttpFilter.java:11
at com.sysaid.server.LoginFilter.doFilter(Unknown Source)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:29
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:58
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:619)
2010-08-03 12:34:23,923 [http-80-7] ERROR com.sysaid - Error while processing SysAid request.
2010-08-03 12:34:40,779 [http-80-26] ERROR com.sysaid - Error while processing SysAid request.
2010-08-03 12:35:05,644 [http-80-23] ERROR com.sysaid - Exception in displayNews: java.lang.OutOfMemoryError: PermGen space
2010-08-03 12:35:11,154 [http-80-5] ERROR com.sysaid - Error while processing SysAid request.
2010-08-03 12:35:29,652 [http-80-1] ERROR com.sysaid - Error while processing SysAid request.
2010-08-03 12:35:32,496 [http-80-18] ERROR com.sysaid - Error while processing SysAid request.
2010-08-03 12:36:03,286 [http-80-15] ERROR com.sysaid - Error while processing SysAid request.
2010-08-03 12:36:07,592 [http-80-24] ERROR com.sysaid - Error while processing SysAid request.
2010-08-03 12:36:20,144 [http-80-2] ERROR com.sysaid - Error while processing SysAid request.

Anyone seen this before?

I noticed that java process was hitting one of the cores and was fairly constant at 100% and mysql was doing very little during this period.

Server is Tomcat 6 with 8gig of ram running linux 64bit. Tomcat is the standalone version downloaded from tomcat website as per recommendations I've seen in forum and java is jdk1.6.0_21.

I tried to copy the main tomcat server.xml file from the windows server we migrated from but tomcat would not deploy the app using it as I think it is quite dated.

Restarting the tomcat server returned everything to normal but looking at the errors above makes me think I'm missing some config settings for tomcat.

Do i need to implement any of the following settings?

-XX:MaxPermSize=??m;
-XXermSize=??m;
-XX:+CMSPermGenSweepingEnabled
-XX:+CMSClassUnloadingEnabled


Cheers

Steve

This message was edited 1 time. Last update was at Aug. 03, 2010 02:00 AM

SysAid Customer Relations
1092
 
Hi Steve,

Please send us email to helpdesk@sysaid.com or open a SR in https://helpdesk.sysaid.com regarding this issue with short description, link to the post and the support team will contact you.

Please restart the SysAid Server service and send us the updated logs folders and files:
...\SysAidServer\root\WEB-INF\logs folder (zipped)
...\SysAidServer\root\WEB-INF\conf\serverConf.xml
...\SysAidServer\tomcat\logs
...\SysAidServer\logs\wrapper.log
Best Regards,
Itay
SysAider
33
 
Hi itayH,

We had previously logged an SR for this and are still waiting a response: Service Request #311,996.

We had the server crash twice today which gave the opportunity to do some debugging of the JVM environment and particularly the (java.lang.OutOfMemoryError: PermGen space) error we were getting.

We noticed that the PermGen was hitting 100% utilised a fair bit so we have reconfigured the JVM to have a larger min and max heap size and also increased the PermSize and MaxPermSize.

As the PS Perm Generation memory utilisation was sitting at 99% and the garbage collection was holding it between that and 96%.

It seems the default value for MaxPermSize is 84mb.

We have created a setenv.sh script which get's called by the catalina start up script which contains the following entry.



export CATALINA_OPTS="-Xms1024m -Xmx2048m -XXermSize=128m -XX:MaxPermSize=256m -XX:+CMSClassUnloadingEnabled"

We restarted the tomcat and as you can see below are sitting inside the allocated thresholds now.



Attaching to process ID 8195, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 17.0-b16

using thread-local object allocation.
Parallel GC with 4 thread(s)

Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 2147483648 (2048.0MB)
NewSize = 1310720 (1.25MB)
MaxNewSize = 17592186044415 MB
OldSize = 5439488 (5.1875MB)
NewRatio = 2
SurvivorRatio = 8
PermSize = 134217728 (128.0MB)
MaxPermSize = 268435456 (256.0MB)

Heap Usage:
PS Young Generation
Eden Space:
capacity = 326369280 (311.25MB)
used = 113782672 (108.51161193847656MB)
free = 212586608 (202.73838806152344MB)
34.86316849428966% used
From Space:
capacity = 15204352 (14.5MB)
used = 0 (0.0MB)
free = 15204352 (14.5MB)
0.0% used
To Space:
capacity = 14483456 (13.8125MB)
used = 0 (0.0MB)
free = 14483456 (13.8125MB)
0.0% used
PS Old Generation
capacity = 715849728 (682.6875MB)
used = 95043408 (90.64045715332031MB)
free = 620806320 (592.0470428466797MB)
13.277005533764763% used
PS Perm Generation
capacity = 134217728 (128.0MB)
used = 78440848 (74.80702209472656MB)
free = 55776880 (53.19297790527344MB)
58.44298601150513% used
SysAider
33
 
We have had a resonse back from the SR which suggest same as above.
SysAid Customer Relations
1092
 
Hi Steve,

This is what I wanted to tell you but we don't give this kind of information in the forum in order to prevent other issues

I hope the SysAid is working fine now.
Best Regards,
Itay