Solving java.net.SocketException: Write failed: Broken pipe Errors | Weblogic

Oracle Weblogic Server

During high load we usually get very low performance usually in navigating and executing any queries from application server to the back end database. In server log files we usually get broken pipe errors like below.

 

java.net.SocketException: Write failed: Broken pipe
        at jrockit.net.SocketNativeIO.writeBytesPinned(Native Method)
        at jrockit.net.SocketNativeIO.socketWrite(SocketNativeIO.java:46)
        at java.net.SocketOutputStream.socketWrite0(SocketOutputStream.java)
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
        at weblogic.servlet.internal.ChunkOutput.writeChunkTransfer(ChunkOutput.java:568)
        at weblogic.servlet.internal.ChunkOutput.writeChunks(ChunkOutput.java:539)
        at weblogic.servlet.internal.ChunkOutput.flush(ChunkOutput.java:427)
        at weblogic.servlet.internal.CharsetChunkOutput.flush(CharsetChunkOutput.java:298)
        at weblogic.servlet.internal.ChunkOutputWrapper.flush(ChunkOutputWrapper.java:188)
        at weblogic.servlet.internal.ServletOutputStreamImpl.flush(ServletOutputStreamImpl.java:121)
        at weblogic.servlet.internal.ServletResponseImpl.flushBuffer(ServletResponseImpl.java:197)
        at javax.servlet.ServletResponseWrapper.flushBuffer(ServletResponseWrapper.java:176)
        at javax.servlet.ServletResponseWrapper.flushBuffer(ServletResponseWrapper.java:176)

 

Or below kind of errors if service is failing:

javax.xml.ws.soap.SOAPFaultException: java.lang.NullPointerException
        at oracle.j2ee.ws.client.jaxws.DispatchImpl.throwJAXWSSoapFaultException(DispatchImpl.java:1028)
        at oracle.j2ee.ws.client.jaxws.DispatchImpl.invoke(DispatchImpl.java:812)
        at oracle.j2ee.ws.client.jaxws.OracleDispatchImpl.synchronousInvocationWithRetry(OracleDispatchImpl.java:235)
        at oracle.j2ee.ws.client.jaxws.OracleDispatchImpl.invoke(OracleDispatchImpl.java:106) 
        at oracle.j2ee.ws.client.jaxws.WsClientProxyInvocationHandler.invoke(WsClientProxyInvocationHandler.java:225)
        at $Proxy453.getDeliveryServiceDefinition(Unknown Source)
        at oracle.xdo.service.client.ReportService.getDeliveryServiceDefinition(ReportService.java:822)
        at oracle.apps.fnd.applcp.srs.model.srsService.applicationModule.ScheduleAMImpl.getDestinationServerList(ScheduleAMImpl.java:4279)
        at oracle.apps.fnd.applcp.srs.view.backing.ScheduleRequest.getDestinationServerList(ScheduleRequest.java:2398)
        at oracle.apps.fnd.applcp.srs.view.backing.OutputHandler.getPrinterList(OutputHandler.java:680)

OR

javax.xml.ws.WebServiceException: java.io.InterruptedIOException
        at com.sun.xml.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:306)
        at com.sun.xml.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:124)
        at com.sun.xml.ws.transport.DeferredTransportPipe.processRequest(DeferredTransportPipe.java:121)
        at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:866)
        at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:815)
        at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:778)
        at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:680)

Note: “SOAPFaultException: java.lang.NullPointerException” & “javax.xml.ws.WebServiceException: java.io.InterruptedIOException” can be ignored as in my case we saw this after a huge number of broken pipes exceptions and at the time when the server load and heap free was way too below threshold. Here we are not investigated above 2 errors assuming the server was heavily loaded and not able to process request but in your case you do take it into consideration.

Usually this socket exception and broken pipe comes because of database connection downtime or overload or long pending idle connections which never got released jamming the con. pool. This(SocketException: Write failed: Broken pipe) is caused by writing to a connection when the other end has already closed it.So you have a poorly defined or implemented application protocol.

Broken pipe simply means that the connection has failed. It is reasonable to assume that this is unrecoverable, and to then perform any required cleanup actions (closing connections, etc). If you are using non-blocking mode then the SocketChannel.connect method will return false, and you will need to use the isConnectionPending and finishConnect methods to insure that the connection is complete.

Usually once the server becomes way too slow you will notice IO related errors in all other working servers too. So instead of getting confused first check from where all started. Usually for quick fix one bounce is enough but its always good to get DB analysis with server IO checks for supresing the error.

Even sometimes uploading a bigger sized file can cause broken pipe errors,If server receives message even with in the limit, but while unmarshalling data, application exceeds heap size, then server will drop the connection established with client abruptly with OutOfMemoryException. Since connection got closed then we do get IOException as above.

Note:Broken pipe message comes for anytime stream could not able to write/read data from file(socket). In general, heart beat will be delivered by server/client in stipulated format. If this max number of attempts cross then application assumes that socket is closed by other party and throws this exception.

To solve broken pipe errors while uploading a bigger sized attachment we can follow below steps for weblogic:

In weblogic server console or config xml, we can set maximum message size only to the http protocol or t3 or all the protocols.

Environment > servers > soa_server > protocols > General > change Maximum Message Size as below

<max-message-size>100000000</max-message-size>
<max-t3-message-size>100000000</max-t3-message-size>
<max-http-message-size>100000000</max-http-message-size>

The above code set 100mb for these properties. These configuration helps to intimate the server that maximum message of receiving capacity. There is no configuration required to be configured to send message from server.

What about client side, to receive 100 mb message?

Yes, we have to set weblogic.MaxMessageSize config property in command line.

-Dweblogic.MaxMessageSize=100000000

In case of any ©Copyright or missing credits issue please check CopyRights page for faster resolutions.

7 Responses

  1. org.terracotta.ehcachedx.monitor.util.RestProxy – java.net.SocketException: Socket Closed………. on weblogic 10…

    • Usually socket closed errors come when Your application has closed either the socket or its input or output stream.Possibly creating multiple stream connected to the same input stream is causing this. check your code snippet with developer… it will be more helpful if you can put the full stack trace here

    • 2012-07-15 20:21:20,168 WARN [Thread-15] org.terracotta.ehcachedx.monitor.util.RestProxy – Error connecting to the server at address 'localhost' and port '9889', retrying in 30 seconds …
      2012-07-15 20:21:20,168 DEBUG [Thread-15] org.terracotta.ehcachedx.monitor.util.RestProxy – java.net.SocketException: Socket Closed
      at java.net.PlainSocketImpl.setOption(PlainSocketImpl.java:219)
      at java.net.Socket.setTcpNoDelay(Socket.java:864)
      at weblogic.net.http.HttpClient.openServer(HttpClient.java:338)
      at weblogic.net.http.HttpClient.openServer(HttpClient.java:425)
      at weblogic.net.http.HttpClient.New(HttpClient.java:252)
      at weblogic.net.http.HttpURLConnection.connect(HttpURLConnection.java:189)
      at org.terracotta.ehcachedx.monitor.util.HttpUtils.performHttpConnectionRequest(HttpUtils.java:82)
      at org.terracotta.ehcachedx.monitor.util.HttpUtils.getHttpConnectionContent(HttpUtils.java:40)
      at org.terracotta.ehcachedx.monitor.util.RestProxy.registerProbe(RestProxy.java:45)
      at org.terracotta.ehcachedx.monitor.probe.ProbeDxService$RegisterWithServer.run(ProbeDxService.java:384)
      at java.lang.Thread.run(Thread.java:662)

    • Vinod Kumbar Hi,

      Few things to do:

      — check whether the telnet to the port works: telnet localhost 9889
      Note: If the port 9889 is open, It should connect and go into telnet prompt, if it exits immediately then you might need to check the service is running correctly or not

      — Change the configuration so that app will try to connect through the real hostname(FQDN)/IP address instead of localhost

      — In many cases this issue can be ignored safely in weblogic(It's a WARNING message not an ERROR) as it might be caused de to below things too:

      Weblogic WARN messages can come due to below things:

      1. The handle http connections are output when the user closes the IE/browser
      2. The server the number of concurrent connections has exceeded its carrying capacity, the server will kill some long waiting connection Down
      3. The customers turned off the browser and the server is still trying to send data to the client
      4. Browser-side client Stopped downloading, clicked Cancel can cause direct closure of socket and server-side it comes as a exception which is normal.

      This is a normal phenomenon, generally no problem can be safely ignored.
      If you dont want this WARN message to fill out your weblogic/app logs then you can set the log lebel to ERROR on weblogic admin console.

    • Thanks a ton !!! it has solved my part of the problem, yet…

  2. SIddharth says:

    <java.io.IOException:
    java.net.SocketException: Broken pipe
    at java.net.SocketOutputStream.socketWrite0(Native Method)
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
    at weblogic.servlet.internal.ChunkOutput.writeChunkNoTransfer(ChunkOutput.java:591)
    at weblogic.servlet.internal.ChunkOutput.writeChunks(ChunkOutput.java:540)
    at weblogic.servlet.internal.ChunkOutput.flush(ChunkOutput.java:427)
    at weblogic.servlet.internal.CharsetChunkOutput.flush(CharsetChunkOutput.java:298)
    at weblogic.servlet.internal.ChunkOutput$2.checkForFlush(ChunkOutput.java:648)
    at weblogic.servlet.internal.CharsetChunkOutput.write(CharsetChunkOutput.java:200)
    at weblogic.servlet.internal.ChunkOutputWrapper.write(ChunkOutputWrapper.java:148)
    at weblogic.servlet.internal.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:148)
    at org.apache.axis.utils.ByteArray.writeTo(ByteArray.java:375)
    at org.apache.axis.SOAPPart.writeTo(SOAPPart.java:265)
    at org.apache.axis.Message.writeTo(Message.java:539)
    at org.apache.axis.transport.http.AxisServlet.sendResponse(AxisServlet.java:902)
    at org.apache.axis.transport.http.AxisServlet.doPost(AxisServlet.java:777)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
    at org.apache.axis.transport.http.AxisServletBase.service(AxisServletBase.java:351)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
    at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
    at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
    at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:301)
    at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
    at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
    at oracle.security.wls.filter.SSOSessionSynchronizationFilter.doFilter(SSOSessionSynchronizationFilter.java:291)
    at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
    at oracle.xdo.servlet.metadata.track.MostRecentFilter.doFilter(MostRecentFilter.java:64)
    at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
    at oracle.xdo.servlet.init.InitCheckingFilter.doFilter(InitCheckingFilter.java:63)
    at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
    at oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
    at java.security.AccessController.doPrivileged(Native Method)
    at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:315)
    at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:442)
    at oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
    at oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:171)
    at oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:71)
    at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
    at oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:139)
    at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
    at weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27)
    at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
    at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3730)
    at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696)
    at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
    at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
    at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2273)
    at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2179)
    at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1490)
    at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
    at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
    =================

    Getting above error in my OBIEE server logs unable to download particular pdf's from my web application..

    Please suggest,,,,,

    Thankyou

    Regards
    Siddharth

    • admin says:

      Hi Siddharth,

      Broken pipe while downloading pdf from webapp might come for many reasons.I will outline few areas which you can check to find the rootcause.

      1. It so happen that the thread which responds to the download is nomore available. You can bounce the managed server once and check if that fixes the issue.
      2. If you are using CLOB/BLOB in db which is having issue.
      3. Check if you have some downloading limit set in your webapp or in server. Like UPLOAD_MAX_FILE_SIZE or UPLOAD_MAX_DISK_SPACE kind of settings in web.xml
      4. Check the size of pdf and availability of pdf file, it might so happen that the pdf is already deleted which you are trying to download.

      If you can add more logs prior to the broken pipe it will be more helpful.

      Thanks

Leave a Reply