Weblogic server hangs at startup

Oracle Weblogic Server

While starting weblogic admin server, weblogic server not starting or weblogic server hangs at startup at one point with out any further logs on stdout. This might come due to many reasons, we use to frequently see these errors on old weblogic setups and finally started keeping record of solutions which we tried to fix them. Please follow below solutions and see which one works for your environment for issues like weblogic server not starting or weblogic server hangs at startup.

Solution 1

Usually it used to get stuck at starting with below logs:

<Oct 22, 2013 3:42:23 AM PDT> <Info> <Security> <BEA-090905> <Disabling CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true>
<Oct 22, 2013 3:42:23 AM PDT> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true>
<Oct 22, 2013 3:42:23 AM PDT> <Notice> <WebLogicServer> <BEA-000395> <Following extensions directory contents added to the end of the classpath:
/mydomain/WLS/user_projects/domains/wls_mydomain/lib/mbeantypes/csp-id-asserter.jar:/mydomain/WLS/user_projects/domains/wls_mydomain/lib/mbeantypes/jps-wls-trustprovider.jar>
<Oct 22, 2013 3:42:23 AM PDT> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with Oracle JRockit(R) Version R28.2.5-50-153520-1.6.0_37-20121220-0843-linux-x86_64 from Oracle Corporation>
<Oct 22, 2013 3:42:24 AM PDT> <Info> <Management> <BEA-141107> <Version: WebLogic Server Temporary Patch for BUG16362974 Tue Feb 19 23:49:30 CST 2013
WebLogic Server Temporary Patch for BUG16104758 Thu Jan 17 00:49:16 CST 2013
WebLogic Server Temporary Patch for BUG16180221 Wed Jun 05 12:46:15 MDT 2013
WebLogic Server Temporary Patch for BUG14595754 Wed Oct 03 10:10:13 CDT 2012
WebLogic Server 10.3.6.0 Tue Nov 15 08:52:36 PST 2011 1441050 >
<Oct 22, 2013 3:42:26 AM PDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>
<Oct 22, 2013 3:42:26 AM PDT> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool>
<Oct 22, 2013 3:42:26 AM PDT> <Notice> <Log Management> <BEA-170019> <The server log file /mydomain/WLS/user_projects/domains/wls_mydomain/servers/AdminServer/logs/AdminServer.log is opened. All server side log events will be written to this file.>


weblogic server hangs at startup and used to get stuck at above point and will not even log anything. After many tries we came to know the issue was with data folder inside /mydomain/WLS/user_projects/domains/wls_mydomain/servers/AdminServer/data folder, so to check if its really issue with data folder we renamed/moved the data folder to data_backup and tried to start it again. We saw the server recreated the data folder by itself and started up fine without any hangs.

So we came to know the issue is with the data folders some file corruption, so we extracted the old backups which gets stored as EmbeddedLDAPBackup.1.zip in /mydomain/WLS/user_projects/domains/wls_mydomain/servers/AdminServer/data/ldap/backup folder and replaced all existing EmbeddedLDAP.data, EmbeddedLDAP.index, EmbeddedLDAP.delete, EmbeddedLDAP.tran, EmbeddedLDAP.twpos files with the backed up file and restarted the Admin Server. this time it came up fine, once its up, to avoid this situation again follow steps mentioned here: (i.e: Login to WebLogic console, click on domain name on left pane –> go to security tab –> Embedded LDAP tab –> click on “Refresh replica at startup” option.)

http://www.techpaste.com/2013/04/steps-resolve-embeddedldap-error-java-lang-nullpointerexception-weblogic/

 Solution 2

If you are getting below error while starting the managed server try to remove the edit.lok file, it will fix the issue or regenerate the boot.properties file.

<Oct 19, 2012 8:01:41 PM MMT> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>
<Oct 19, 2012 8:01:41 PM MMT> <Critical> <Security> <BEA-090403> <Authentication for user denied>
<Oct 19, 2012 8:01:41 PM MMT> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: Authentication for user denied
weblogic.security.SecurityInitializationException: Authentication for user denied
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.doBootAuthorization(CommonSecurityServiceManagerDelegateImpl.java:965)
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1050)
at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:873)
at weblogic.security.SecurityService.start(SecurityService.java:141)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
Truncated. see log file for complete stacktrace
Caused By: javax.security.auth.login.FailedLoginException: [Security:090304]Authentication Failed: User javax.security.auth.login.LoginException: [Security:090301]Password Not Supplied
at weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.login(LDAPAtnLoginModuleImpl.java:261)
at com.bea.common.security.internal.service.LoginModuleWrapper$1.run(LoginModuleWrapper.java:110)
at java.security.AccessController.doPrivileged(Native Method)
at com.bea.common.security.internal.service.LoginModuleWrapper.login(LoginModuleWrapper.java:106)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Truncated. see log file for complete stacktrace

Solution 3

If you are getting below error before starting the server or just after starting the server and weblogic server not starting then please execute below mentioned steps to fix it.

Delete all the corrupted DAT files under /opt/domain/server_name/servers/AdminServer/data/store/diagnostics/*.DAT and try to restart the server again.

####<Oct 25, 2006 11:03:15 AM MEST> <Notice> <WebLogicServer> <mynew-srv> <AdminServer> <main> <<WLS Kernel>> <> <> <1161766995356> <BEA-000360> <Server started in RUNNING mode>
####<Oct 25, 2006 11:07:29 AM MEST> <Error> <Diagnostics> <mynew-srv> <AdminServer> <[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>
<> <26be653e45def710:-52c11f7a:10e7eb579cc:-7fe1-0000000000000001><1161767249271> <BEA-320103> <There was an error creating the DiagnosticDataRuntimeMBean instance with the given logical name EventsDataArchive.
weblogic.diagnostics.accessor.DataAccessServiceCreateException: weblogic.store.PersistentStoreException: [Store:280020]There was an error while reading from the log file
at weblogic.diagnostics.accessor.DiagnosticDataAccessServiceFactory.createDiagnosticDataAccessService(DiagnosticDataAccessServiceFactory.java:76)
at weblogic.diagnostics.accessor.DiagnosticDataAccessRuntime.createDiagnosticDataAccessService (DiagnosticDataAccessRuntime.java:164)
at weblogic.diagnostics.accessor.DiagnosticDataAccessRuntime.createDiagnosticDataAccessRuntime(DiagnosticDataAccessRuntime.java:100)
at weblogic.diagnostics.accessor.DiagnosticAccessRuntime$1.run(DiagnosticAccessRuntime.java:144)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:147)
at weblogic.diagnostics.accessor.DiagnosticAccessRuntime.lookupWLDFDataAccessRuntime (DiagnosticAccessRuntime.java:140)
at weblogic.diagnostics.accessor.DiagnosticAccessRuntime.lookupWLDFDataAccessRuntime(DiagnosticAccessRuntime.java:94)
at weblogic.diagnostics.instrumentation.EventQueue.getArchive(EventQueue.java:338)
at weblogic.diagnostics.instrumentation.EventQueue.access$200(EventQueue.java:46)
at weblogic.diagnostics.instrumentation.EventQueue$ArchiveEventsAction.run(EventQueue.java:371)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
at weblogic.diagnostics.instrumentation.EventQueue.timerExpired(EventQueue.java:316)
at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:259)
at weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:179)
weblogic.store.PersistentStoreException: [Store:280020]There was an error while reading from the log file
at weblogic.store.io.file.Heap.getNextRecoveryFile(Heap.java:794)
at weblogic.store.io.file.Heap.open(Heap.java:181)
at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:83)
at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:332)
at weblogic.store.PersistentStoreManager.createFileStore(PersistentStoreManager.java:201)
at weblogic.diagnostics.archive.DiagnosticStoreRepository.getStore(DiagnosticStoreRepository.java:54)
at weblogic.diagnostics.archive.wlstore.PersistentStoreDataArchive.getArchiveSchema(PersistentStoreDataArchive.java:309)
at weblogic.diagnostics.archive.wlstore.PersistentStoreDataArchive.<init>(PersistentStoreDataArchive.java:268)
at weblogic.diagnostics.archive.wlstore.EventsPersistentStoreDataArchive.<init>(EventsPersistentStoreDataArchive.java:53)
at weblogic.diagnostics.accessor.DiagnosticDataAccessServiceFactory.createEventsDataAccessService(DiagnosticDataAccessServiceFactory.java:174)
at weblogic.diagnostics.accessor.DiagnosticDataAccessServiceFactory.createDiagnosticDataAccessService(DiagnosticDataAccessServiceFactory.java:61)
at weblogic.diagnostics.accessor.DiagnosticDataAccessRuntime.createDiagnosticDataAccessService(DiagnosticDataAccessRuntime.java:164)
at weblogic.diagnostics.accessor.DiagnosticDataAccessRuntime.createDiagnosticDataAccessRuntime(DiagnosticDataAccessRuntime.java:100)
at weblogic.diagnostics.accessor.DiagnosticAccessRuntime$1.run(DiagnosticAccessRuntime.java:144)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:147)
at weblogic.diagnostics.accessor.DiagnosticAccessRuntime.lookupWLDFDataAccessRuntime(DiagnosticAccessRuntime.java:140)
at weblogic.diagnostics.accessor.DiagnosticAccessRuntime.lookupWLDFDataAccessRuntime(DiagnosticAccessRuntime.java:94)
at weblogic.diagnostics.instrumentation.EventQueue.getArchive(EventQueue.java:338)
at weblogic.diagnostics.instrumentation.EventQueue.access$200(EventQueue.java:46)
at weblogic.diagnostics.instrumentation.EventQueue$ArchiveEventsAction.run(EventQueue.java:371)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
at weblogic.diagnostics.instrumentation.EventQueue.timerExpired(EventQueue.java:316)
at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:259)
at weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:179)
java.io.IOException: Error reading from file, No such file or directory, errno=2
at weblogic.store.io.file.direct.DirectIONative.read(Native Method)
at weblogic.store.io.file.direct.DirectFileChannel.read(DirectFileChannel.java:133)
at weblogic.store.io.file.StoreFile.read(StoreFile.java:275)
at weblogic.store.io.file.Heap.getNextRecoveryFile(Heap.java:792)
at weblogic.store.io.file.Heap.open(Heap.java:181)
at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:83)
at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:332)
at weblogic.store.PersistentStoreManager.createFileStore(PersistentStoreManager.java:201)
at weblogic.diagnostics.archive.DiagnosticStoreRepository.getStore(DiagnosticStoreRepository.java:54)
at weblogic.diagnostics.archive.wlstore.PersistentStoreDataArchive.getArchiveSchema(PersistentStoreDataArchive.java:309)

Solution 4

If you are getting below error while starting the server and weblogic server not starting then follow below mentioned steps to fix this issue.

Note: Always keep backup of files before removing it.
1: Edit the file, use plain text username and password instead of encrypted like
Username: weblogic
password: Weblogic1
After start the server from command line, it will encrypt again automitically.
2: Remove the ManagedServerDir/data/ldap directory completely or restore it from the automated backups taken @ /mydomain/WLS/user_projects/domains/wls_mydomain/servers/AdminServer/data/ldap/backup folder.
3: Now try to start the server from the command prompt and provide the username/password used to login to the Admin Console.

####<May 18, 2011 12:05:53 PM IST> <Critical> <Security> <CALTP8BB14> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1275028553535> <BEA-090402> <Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.>
####<May 18, 2011 12:05:53 PM IST> <Critical> <WebLogicServer> <CALTP8BB14> <AdminServer> <main> <<WLS Kernel>> <> <> <1275028553535> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.
weblogic.security.SecurityInitializationException: Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.doBootAuthorization(CommonSecurityServiceManagerDelegateImpl.java:959)
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1050)
at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:875)
at weblogic.security.SecurityService.start(SecurityService.java:141)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
javax.security.auth.login.FailedLoginException: [Security:090303]Authentication Failed: User weblogic weblogic.security.providers.authentication.LDAPAtnDelegateException: [Security:090295]caught unexpected exception
at weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.login(LDAPAtnLoginModuleImpl.java:243)
at com.bea.common.security.internal.service.LoginModuleWrapper$1.run(LoginModuleWrapper.java:110)
at java.security.AccessController.doPrivileged(Native Method)
at com.bea.common.security.internal.service.LoginModuleWrapper.login(LoginModuleWrapper.java:106)

 

These  are few solutions which we implement to make things work when you have issues like weblogic server not starting or weblogic server hangs at startup . If you have few other solutions which is commonly used by you then please post in comments it will help others to debug there issue.

In case of any ©Copyright or missing credits issue please check CopyRights page for faster resolutions.

12 Responses

  1. reddym says:

    Hi ,

    I am trying to create user with Read only permissions for development team to have access to weblogic console , I know the process is simple but for some reason after creating the user i am not able login using the Read only user.

    Below steps to create read only user:

    1. Login into Weblogic Admin Console as weblogic user (administrator) ,Then => security realms -> myrealm => users&groups =>New “User” created “appdev” user with password “password01”.

    2. Once user is created, selected username => selected Groups Tab => Selected group by name “Monitors”. Saved the changes made.

    3. Logged out and closed the browser.

    4. Restarted the server and Opened a new browser window.

    5. Attempting to login with the new user is returning access denied.

    We are using weblogic 9.2 verison , Please let me know if i am missing something here..

    Thanks
    ReddyM

    • admin says:

      Hi,

      All the steps seems correct. an you paste the logs whcih you are getting while trying login with readonly user, it will be very helpful in finding the issue.

      One thing you can try out is below listed step and let us know incase that works:

      – Enable the “Refresh Replica At StartUp” from weblogic admin console and bounce the admin server again and check. Below is the path of the “Refresh Replica At StartUp”

      Navigate to weblogic admin console ->Domain->Security->Embedded LDAP->Refresh Replica At StartUp

      Since Security data (e.g., password) is stored in the Embedded LDAP by default and it is replicated from the admin server to the managed servers. If there is an inconsistency of the security data between the admin and managed servers, the error will occur.

      Hence you have to refresh all replicated data at boot time, in order to do this you need to set ‘Refresh Replica At Startup’ from the Admin Console. This option specifies whether a Managed Server should refresh all replicated data at boot time.

      Thanks,
      TechPaste.Com

  2. htamuly says:

    Hi,

    Thanks a loy for sharing such information. It helped me a lot. I was also having a similiar problem. The 1st solution worked for me. It saved me a lot of time as this was a time critical issue.

    Keep up the good work. Kuddos to you.

    Thanks and Regards
    htamuly

  3. Thank you so much. Solution 1 was so useful for me.

  4. Thank you so much. Solution 1 was so useful for me.

  5. Serene says:

    Your Post is really good!

  6. Vee says:

    Thank you; Solution 1 was very helpful for me.

  7. I had two of these errors in different opportunities. This post helped me a lot!

    Thanks!

Leave a Reply