A cluster with 2 nodes each in a different machine there were issues with one of the servers, OOM, unhealthy persistent store and after this we tried to recycle the server,but were unable to do so with below error in log.
<Oct 11, 2011 12:37:39 PM PDT> <Critical> <Health> <BEA-310006> <Critical Subsystem DatabaseLessLeasing has failed. Setting server state to FAILED. Reason: Server is not in the majority cluster partition>
Usually this error shows up when there are only 2 managed servers in once cluster.
To solve this kind of issues, create another server; since the cluster is only of 2 nodes, any server will fall out of the majority cluster partition if it loses connectivity/drops cluster broadcast messages. In this scenario, there are no other servers part of the cluster.
For Consensus Leasing, it is always recommended to create a cluster with at-least 3 nodes; that way you can ensure some stability.
In that scenario, even if one server falls out of the cluster, the other two still function correctly as they remain in the majority cluster partition The third one will rejoin the cluster, or will be eventually restarted.
In a scenario where you have only 2 servers as part of the cluster, one falling out from the cluster will result in both the servers being restarted, as they are not a part of the majority cluster partition; this would ultimately result in a very unstable environment.