java.net.SocketException: Connection reset by peer: socket write error

Oracle Weblogic Server

During a moderate load below error appears in weblogic logs : Connection reset by peer: socket write error

Environment:

They are running Weblogic in a windows 2003 ESX vmware environment.

The configuration starts with a RHEL 5.1 running apache, cluster of 6 servers with the weblogic plugin.
The plugin points to a cisco load balancer which will forward the requests to a weblogic tier servers.
The weblogic tier has a cluster of 4 servers and another 5 managed servers.

ERROR

####<Nov 26, 2011 12:52:31 PM AST> <Error> <HTTP> <TechPaste> <PRE-WL3> <ExecuteThread: '18' for queue: 'weblogic.kernel.Default'> <<WLS Kernel>> <BEA-101017> <http://ServletContext(id=24499159,name=spWeb.war,context-path=) Root cause of ServletException.

javax.servlet.jsp.JspException: ServletException in:/games/confirm.jsp Input/output error: java.net.SocketException: Connection reset by peer: socket write error'

at org.apache.struts.taglib.tiles.InsertTag$InsertHandler.processException(InsertTag.java:875)

at org.apache.struts.taglib.tiles.InsertTag$InsertHandler.doEndTag(InsertTag.java:835)

at org.apache.struts.taglib.tiles.InsertTag.doEndTag(InsertTag.java:466)

at jsp_servlet._templates.__main_template._jspService(__main_template.java:655)

at weblogic.servlet.jsp.JspBase.service(JspBase.java:33)

at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)

at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)

at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:348)

at weblogic.servlet.internal.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:336)

at org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java:1058)

at org.apache.struts.tiles.TilesRequestProcessor.doForward(TilesRequestProcessor.java:269)

at org.apache.struts.tiles.TilesRequestProcessor.processTilesDefinition(TilesRequestProcessor.java:249)

at org.apache.struts.tiles.TilesRequestProcessor.processForwardConfig(TilesRequestProcessor.java:303)

at org.apache.struts.action.RequestProcessor.processActionForward(RequestProcessor.java:401)

at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:279)

at car.lce.plays.ui.struts.PsRequestProcessor.process(PsRequestProcessor.java:391)

at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1420)

at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:520)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:760)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)

at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)

at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)

at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:348)

at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:7058)

at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)

at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)

at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3902)

at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2756)

at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)

at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)

 

Below are the logs seen in cisco load balancer probes while probing for server

Nov 27 2011 13:40:25 : %ACE-3-251010: Health probe failed for server 10.136.12.243 on port 443, graceful disconnect timeout (no FIN ACK)

Nov 27 2011 13:40:55 : %ACE-3-251010: Health probe failed for server 10.136.12.243 on port 443, graceful disconnect timeout (no FIN ACK)

Nov 27 2011 13:41:06 : %ACE-3-251010: Health probe failed for server 10.136.21.421 on port 443, graceful disconnect timeout (no FIN ACK)

Nov 27 2011 13:41:11 : %ACE-3-251010: Health probe failed for server 10.136.12.243 on port 443, graceful disconnect timeout (no FIN ACK)

Nov 27 2011 13:41:11 : %ACE-4-442002: Health probe TCP-443 detected RS-EWAPPS_PRE-WL3 (interface vlan619) in serverfarm SF_WLSAPPS-443 changed state to DOWN
The server cannot write to the socket (OS level).

There are bridge errors detected by the apache web server. The user gets a strange error

Failure of server APACHE bridge:
_____

No backend server available for connection: timed out after 12 seconds or idempotent set to OFF.
_____

CAUSE:

As the load increased, the Cisco load balancer was configured to probe the weblogic cluster managed servers at the port.
If the response from the port 443 probe exceeded a value (number of probes and timeout) the load balancer would set the server status as down, thus getting the connection reset by peer error in the weblogic server logs and a bridge error in the apache logs / end users screen.

SOLUTION:

Turning the port probe off or setting the probe values larger would impact the users in a positive way and would not reset the connection at all or not as quickly as the case may be.

 

In case of any ┬ęCopyright or missing credits issue please check CopyRights page for faster resolutions.

Leave a Reply