We're updating the issue view to help you get more done. 

GS Agent components don't terminate when parent process is no longer alive

Description

GS-Agent spawns processes and monitors them. These processes also monitor the liveness of the GSA process. When the GSA is not found, the process terminates itself.

When running "./gs.sh demo" and killing the GSA we noticed that some of the GSC processes were not shutdown properly. The stack trace below shows a remote call to the ZooKeeperAttributeStore waiting for a response. Hence the GSC won't continue its termination sequence until the wait time is over.

 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 State: WAITING on org.apache.zookeeper.ClientCnxn$Packet@2185bcb Total blocked: 47 Total waited: 100 Stack trace: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1411) org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1219) org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:327) org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:316) org.apache.curator.connection.StandardConnectionHandlingPolicy.callWithRetry(StandardConnectionHandlingPolicy.java:64) org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:100) org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:313) org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:304) org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:35) org.openspaces.zookeeper.attribute_store.ZooKeeperAttributeStore.get(ZooKeeperAttributeStore.java:87) com.gigaspaces.internal.server.space.ZookeeperLastPrimaryHandler.getLastPrimaryName(ZookeeperLastPrimaryHandler.java:79) com.gigaspaces.internal.server.space.ZookeeperLastPrimaryHandler.isLastPrimary(ZookeeperLastPrimaryHandler.java:71) org.openspaces.zookeeper.leader_selector.ZooKeeperBasedLeaderSelectorHandler.terminate(ZooKeeperBasedLeaderSelectorHandler.java:79) com.gigaspaces.internal.server.space.SpaceImpl.destroy(SpaceImpl.java:752) com.gigaspaces.internal.server.space.SpaceImpl.shutdown(SpaceImpl.java:1316) com.j_spaces.core.JSpaceContainerImpl.shutdownInternal(JSpaceContainerImpl.java:998) - locked java.lang.Object@2cc5980 com.j_spaces.core.JSpaceContainerImpl.shutdown(JSpaceContainerImpl.java:982) com.j_spaces.core.LRMIJSpaceContainer.shutdown(LRMIJSpaceContainer.java:89) com.gigaspaces.internal.client.spaceproxy.SpaceProxyImpl.shutdown(SpaceProxyImpl.java:299) org.openspaces.core.space.AbstractSpaceFactoryBean.close(AbstractSpaceFactoryBean.java:271) - locked org.openspaces.core.space.EmbeddedSpaceFactoryBean@43766629 org.openspaces.core.space.AbstractSpaceFactoryBean.destroy(AbstractSpaceFactoryBean.java:234) org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:256) org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:571) org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:543) org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:1034) org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:504) org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:1027) org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1057) org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1026) org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:975) - locked java.lang.Object@1b8734d8 org.openspaces.pu.container.integrated.IntegratedProcessingUnitContainer.close(IntegratedProcessingUnitContainer.java:80) org.openspaces.pu.container.servicegrid.PUServiceBeanImpl.stopPU(PUServiceBeanImpl.java:1176) org.openspaces.pu.container.servicegrid.PUServiceBeanImpl.stop(PUServiceBeanImpl.java:307) org.jini.rio.jsb.ServiceBeanAdapter.destroy(ServiceBeanAdapter.java:1067) org.jini.rio.jsb.ServiceBeanAdapter.destroy(ServiceBeanAdapter.java:998) org.jini.rio.resources.servicecore.ServiceAdminImpl.destroy(ServiceAdminImpl.java:153) org.jini.rio.resources.servicecore.ServiceAdminProxy.destroy(ServiceAdminProxy.java:229) org.jini.rio.cybernode.JSBDelegate.terminate(JSBDelegate.java:432) org.jini.rio.cybernode.JSBContainer.terminateServices(JSBContainer.java:142) org.jini.rio.cybernode.JSBContainer.terminate(JSBContainer.java:125) org.jini.rio.cybernode.CybernodeImpl.destroy(CybernodeImpl.java:230) com.gigaspaces.grid.gsc.GSCImpl.destroy(GSCImpl.java:337) org.jini.rio.jsb.ServiceBeanAdapter.destroy(ServiceBeanAdapter.java:998) org.jini.rio.resources.servicecore.ServiceAdminImpl.destroy(ServiceAdminImpl.java:153) org.jini.rio.resources.servicecore.ServiceAdminProxy.destroy(ServiceAdminProxy.java:229) org.jini.rio.jsb.ServiceBeanActivation$LifeCycleManager.terminate(ServiceBeanActivation.java:423) org.jini.rio.jsb.ServiceBeanActivation$ShutdownHook.run(ServiceBeanActivation.java:480)

 

 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Name: ShutdownHook State: WAITING on java.util.concurrent.FutureTask@798d00f Total blocked: 0 Total waited: 1 Stack trace: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429) java.util.concurrent.FutureTask.get(FutureTask.java:191) org.jini.rio.cybernode.JSBContainer.terminateServices(JSBContainer.java:150) org.jini.rio.cybernode.JSBContainer.terminate(JSBContainer.java:125) org.jini.rio.cybernode.CybernodeImpl.destroy(CybernodeImpl.java:230) com.gigaspaces.grid.gsc.GSCImpl.destroy(GSCImpl.java:337) org.jini.rio.jsb.ServiceBeanAdapter.destroy(ServiceBeanAdapter.java:998) org.jini.rio.resources.servicecore.ServiceAdminImpl.destroy(ServiceAdminImpl.java:153) org.jini.rio.resources.servicecore.ServiceAdminProxy.destroy(ServiceAdminProxy.java:229) org.jini.rio.jsb.ServiceBeanActivation$LifeCycleManager.terminate(ServiceBeanActivation.java:423) org.jini.rio.jsb.ServiceBeanActivation$ShutdownHook.run(ServiceBeanActivation.java:480)

 

Workaround

None

Acceptance Test

manual

Status

Assignee

Meron Avigdor

Reporter

Meron Avigdor

Labels

None

Priority

Medium

SalesForce Case ID

None

Fix versions

Commitment Version/s

None

Due date

None

Product

XAP

Edition

Premium

Platform

All