After network recovery reconnected Space instance was stuck verifying last primary state

Description

Thread was stuck on the call to com.gigaspaces.internal.server.space.ZookeeperLastPrimaryHandler.isLastPrimary(ZookeeperLastPrimaryHandler.java:87)

full stack trace:

"Curator-ConnectionStateManager-0" Id=85 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at java.lang.Thread.sleep(Thread.java:340)
at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:386)
at org.apache.curator.RetryLoop$1.sleepFor(RetryLoop.java:75)
at org.apache.curator.retry.SleepingRetry.allowRetry(SleepingRetry.java:46)
at org.apache.curator.retry.RetryUntilElapsed.allowRetry(RetryUntilElapsed.java:41)
at org.apache.curator.RetryLoop.takeException(RetryLoop.java:174)
at org.apache.curator.connection.StandardConnectionHandlingPolicy.callWithRetry(StandardConnectionHandlingPolicy.java:70)
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:100)
at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:313)
at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:304)
at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:35)
at org.openspaces.zookeeper.attribute_store.ZooKeeperAttributeStore.get(ZooKeeperAttributeStore.java:88)
at com.gigaspaces.internal.server.space.ZookeeperLastPrimaryHandler.getLastPrimaryName(ZookeeperLastPrimaryHandler.java:95)
at com.gigaspaces.internal.server.space.ZookeeperLastPrimaryHandler.isLastPrimary(ZookeeperLastPrimaryHandler.java:87)
at org.openspaces.zookeeper.leader_selector.ZooKeeperBasedLeaderSelectorHandler$Adapter.onReconnected(ZooKeeperBasedLeaderSelectorHandler.java:462)
at org.openspaces.zookeeper.curator.XapLeaderSelector$InternalListener.stateChanged(XapLeaderSelector.java:219)

  • locked java.lang.Object@1471870119
    at org.apache.curator.framework.recipes.leader.LeaderSelector$WrappedListener.stateChanged(LeaderSelector.java:567)
    at org.apache.curator.framework.state.ConnectionStateManager$2.apply(ConnectionStateManager.java:274)
    at org.apache.curator.framework.state.ConnectionStateManager$2.apply(ConnectionStateManager.java:270)
    at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:93)
    at org.apache.curator.shaded.com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:435)
    at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:85)
    at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:268)
    at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:44)
    at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:120)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

Workaround

None

Acceptance Test

None

Assignee

Unassigned

Reporter

Meron Avigdor

Labels

None

Priority

Major

SalesForce Case ID

None

Fix versions

None

Commitment Version/s

None

Due date

None

Product

None

Edition

Premium

Platform

All
Configure