Here we will discuss how managed server works in weblogic and how the health status of weblogic managed servers are reported and what does it mean. Server health information is returned by the class ServerRuntimeMBean.HealthState.
Health state is reported by the server’s self-health monitoring which is default each 60 seconds.Things like memory conditions, work managers capacity and threads status among others gets evaluated.
Usually Server health is closely related to the stuck thread definition.
WebLogic Server automatically detects when a thread in an execute queue becomes “stuck”: if it is continually working (not in idle state) for a set period of time(Default 600secs=10mins).
Because a stuck thread cannot complete its current work or accept new work, the server logs a message each time it diagnoses a stuck thread.
What are the server states in weblogic?
State by default (healthy). All subsystem are working as expected.
System could have problems in the future. If all threads in weblogic.admin.HTTP, weblogic.admin.RMI, or an user-defined execute queue become stuck, the server changes its health state to warning.
Example Log Entry:
<2/10/2009 03:17:17 PM IST> <Warning> <WebLogicServer> <BEA-000337> <ExecuteThread: ’14’ for queue: ‘weblogic.kernel.Default’ has been busy for “652” seconds working on the request “Http Request: /GeamWeb/jsp/createConfLoc1.jsp”, which is more than the configured time (StuckThreadMaxTime) of “600” seconds.>
A subsystem is in critical state when a part of it is malfunctioning (e.g stuck threads.) and something has to be done to prevent service failure.
Example log Entry:
<aug 13, 2010 2:49:51 PM BST> <Critical> <WebLogicServer> <BEA-000386><Server subsystem failed. Reason: java.lang.NumberFormatException: null
Truncated. see log file for complete stacktrace
If an individual subsystem determines that it can no longer operate in a consistent and reliable manner, it registers its health state as failed with the host server and must be restarted. If a server finds one or more critical subsystems have reached this state, the server marks its own health state as failed.
<Sep 8, 2011 5:41:10 PM IST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>
<Sep 8, 2011 5:41:10 PM IST> <Error> <WebLogicServer> <BEA-000383> <Sep 8, 2011 5:41:10 PM IST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>
When system capacity is reached, when Work Manager capacity is exceeded or as a result of low memory.
Upon entering the overloaded state, server instances start rejecting requests from the Work Manager queue (if a Work Manager is configured), HTTP requests return a 503 Error (Service Unavailable), and RMI requests fail over to another server if clustered, otherwise, a remote exception is returned to the client.
Example Log Entry:
Internal error: Cannot obtain XAConnection weblogic.common.resourcepool.ResourceLimitException: No resources currently available in pool AppsQAE to allocate to applications, please increase the size of the pool and retry..[[