Observed at QA environment. During automated shutdown sequence, the exception was thrown:
$ bin/gdi-qa-com1.sh stop JBOSS_CMD_STOP = java -classpath /opt/jboss/jboss-as/bin/shutdown.jar:/opt/jboss/jboss-as/client/jnet.jar org.jboss.Shutdown --shutdown --user=admin --password=admin -s jnp://0.0.0.0:1099 Exception in thread "main" javax.naming.CommunicationException: Could not obtain connection to any of these urls: 0.0.0.0:1099 [Root exception is javax.naming.CommunicationException: Failed to connect to server /0.0.0.0:1099 [Root exception is javax.naming.ServiceUnavailableException: Failed to connect to server /0.0.0.0:1099 [Root exception is java.net.ConnectException: Connection refused]]] at org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1763) at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:693) at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:686) at javax.naming.InitialContext.lookup(InitialContext.java:392) at org.jboss.Shutdown.main(Shutdown.java:219) Caused by: javax.naming.CommunicationException: Failed to connect to server /0.0.0.0:1099 [Root exception is javax.naming.ServiceUnavailableException: Failed to connect to server /0.0.0.0:1099 [Root exception is java.net.ConnectException: Connection refused]] at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:335) at org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1734) ... 4 more Caused by: javax.naming.ServiceUnavailableException: Failed to connect to server /0.0.0.0:1099 [Root exception is java.net.ConnectException: Connection refused] at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:305) ... 5 more Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:211) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at org.jnp.interfaces.TimedSocketFactory.createSocket(TimedSocketFactory.java:97) at org.jnp.interfaces.TimedSocketFactory.createSocket(TimedSocketFactory.java:82) at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:301) ... 5 more To watch the log: tail -f /var/log/atg/gdi-qa-com1.log The conflict of 1099 between Endeca and ATG was resolved according Fixing port conflict on 1099 between out-of-the-box Endeca and JBOSS - the actual port was 11199
Symptoms # restarting after kill -9 of CSC instance does not succeed. The server never starts, the WAR files are not deployed properly, socket 8080 is open but nobody answers (not even with wget on localhost).
The exception: # "main" prio=10 tid=0x0000000051a39000 nid=0x30fb in Object.wait() [0x00002b5df9d6c000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000007e36ae9d0> (a atg.service.lockmanager.ClientLockEntry) at atg.service.lockmanager.ClientLockEntry.acquireWriteLock(ClientLockEntry.java:399) - locked <0x00000007e36ae9d0> (a atg.service.lockmanager.ClientLockEntry) at atg.service.lockmanager.ClientLockManager.acquireWriteLock(ClientLockManager.java:1137) at atg.repository.search.indexing.ConfigStatePersister.acquireLock(ConfigStatePersister.java:1844) at atg.repository.search.indexing.ConfigStatePersister$IndexingReadLockedTransactionAction.doWithTransaction(ConfigStatePersister.java:2083) at atg.repository.search.indexing.ConfigStatePersister.releaseAllConfigurations(ConfigStatePersister.java:1545) at atg.repository.search.indexing.ConfigStatePersister.performAction(ConfigStatePersister.java:1775) at atg.nucleus.Nucleus.performPostStartupActions(Nucleus.java:4336) at atg.nucleus.Nucleus.performPostStartupActions(Nucleus.java:4323) at atg.nucleus.Nucleus.<init>(Nucleus.java:954) at atg.nucleus.Nucleus.<init>(Nucleus.java:717) at atg.nucleus.Nucleus.startNucleusCheckLicense(Nucleus.java:4144) at atg.nucleus.servlet.NucleusServlet.initBigEarNucleus(NucleusServlet.java:1234) at atg.nucleus.servlet.NucleusServlet.init(NucleusServlet.java:443) at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1048) - locked <0x0000000781734f48> (a org.apache.catalina.core.StandardWrapper) at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:950) - locked <0x0000000781734f48> (a org.apache.catalina.core.StandardWrapper) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:4122) at org.apache.catalina.core.StandardContext.start(StandardContext.java:4417) - locked <0x0000000781733e18> (a org.apache.catalina.core.StandardContext) at org.jboss.web.tomcat.service.deployers.TomcatDeployment.performDeployInternal(TomcatDeployment.java:321) at org.jboss.web.tomcat.service.deployers.TomcatDeployment.performDeploy(TomcatDeployment.java:145) at org.jboss.web.deployers.AbstractWarDeployment.start(AbstractWarDeployment.java:461) - locked <0x0000000780878058> (a org.jboss.web.tomcat.service.deployers.TomcatDeployment) at org.jboss.web.deployers.WebModule.startModule(WebModule.java:118) - locked <0x0000000780878038> (a org.jboss.web.deployers.WebModule) at org.jboss.web.deployers.WebModule.start(WebModule.java:97) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96) at org.jboss.mx.server.Invocation.invoke(Invocation.java:88) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668) at org.jboss.system.microcontainer.ServiceProxy.invoke(ServiceProxy.java:206) at $Proxy38.start(Unknown Source) at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:42) at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:37) at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62) at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71) at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51) at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348) at org.jboss.system.microcontainer.ServiceControllerContext.install(ServiceControllerContext.java:297) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1652) at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:938) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:988) at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:826) at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:556) at org.jboss.system.ServiceController.doChange(ServiceController.java:688) at org.jboss.system.ServiceController.start(ServiceController.java:460) at org.jboss.system.deployers.ServiceDeployer.start(ServiceDeployer.java:163) at org.jboss.system.deployers.ServiceDeployer.deploy(ServiceDeployer.java:99) at org.jboss.system.deployers.ServiceDeployer.deploy(ServiceDeployer.java:46) at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:62) at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:55) at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:179) at org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(DeployersImpl.java:1454) at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1172) at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1193) at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1225) at org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:1113) at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1652) at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:938) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:988) at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:826) at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:556) at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:789) at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:699) at org.jboss.system.server.profileservice.repository.MainDeployerAdapter.process(MainDeployerAdapter.java:117) at org.jboss.system.server.profileservice.repository.ProfileDeployAction.install(ProfileDeployAction.java:70) at org.jboss.system.server.profileservice.repository.AbstractProfileAction.install(AbstractProfileAction.java:53) at org.jboss.system.server.profileservice.repository.AbstractProfileService.install(AbstractProfileService.java:403) at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1652) at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:938) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:988) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:778) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:543) at org.jboss.system.server.profileservice.repository.AbstractProfileService.registerProfile(AbstractProfileService.java:308) at org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(ProfileServiceBootstrap.java:256) at org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:461) at org.jboss.Main.boot(Main.java:221) at org.jboss.Main$1.run(Main.java:556) at java.lang.Thread.run(Thread.java:662) What is going on: # In the run of deployment, JBoss is starting context, hands over control to NucleusServlet init. Nucleus is trying to get write access lock and hangs there forever, while WAR deployment and all subsequent deployment never completes.