Fixing the shutdown problem with JBoss

Sat, 11 Apr 2015 11:22:48 +0800

Observed at QA environment. During automated shutdown sequence, the exception was thrown:

$ bin/gdi-qa-com1.sh stop
JBOSS_CMD_STOP = java -classpath /opt/jboss/jboss-as/bin/shutdown.jar:/opt/jboss/jboss-as/client/jnet.jar org.jboss.Shutdown --shutdown --user=admin --password=admin -s jnp://0.0.0.0:1099
Exception in thread "main" javax.naming.CommunicationException: Could not obtain connection to any of these urls: 0.0.0.0:1099 [Root exception is javax.naming.CommunicationException: Failed to connect to server /0.0.0.0:1099 [Root exception is javax.naming.ServiceUnavailableException: Failed to connect to server /0.0.0.0:1099 [Root exception is java.net.ConnectException: Connection refused]]]
    at org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1763)
    at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:693)
    at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:686)
    at javax.naming.InitialContext.lookup(InitialContext.java:392)
    at org.jboss.Shutdown.main(Shutdown.java:219)
Caused by: javax.naming.CommunicationException: Failed to connect to server /0.0.0.0:1099 [Root exception is javax.naming.ServiceUnavailableException: Failed to connect to server /0.0.0.0:1099 [Root exception is java.net.ConnectException: Connection refused]]
    at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:335)
    at org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1734)
    ... 4 more
Caused by: javax.naming.ServiceUnavailableException: Failed to connect to server /0.0.0.0:1099 [Root exception is java.net.ConnectException: Connection refused]
    at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:305)
    ... 5 more
Caused by: java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
    at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:211)
    at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
    at java.net.Socket.connect(Socket.java:529)
    at org.jnp.interfaces.TimedSocketFactory.createSocket(TimedSocketFactory.java:97)
    at org.jnp.interfaces.TimedSocketFactory.createSocket(TimedSocketFactory.java:82)
    at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:301)
    ... 5 more
To watch the log: tail -f /var/log/atg/gdi-qa-com1.log

The conflict of 1099 between Endeca and ATG was resolved according Fixing port conflict on 1099 between out-of-the-box Endeca and JBOSS - the actual port was 11199

ATG + JBoss rant

Wed, 03 Sep 2014 11:22:48 +0800

Symptoms
#

restarting after kill -9 of CSC instance does not succeed. The server never starts, the WAR files are not deployed properly, socket 8080 is open but nobody answers (not even with wget on localhost).

The exception:
#

"main" prio=10 tid=0x0000000051a39000 nid=0x30fb in Object.wait() [0x00002b5df9d6c000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000007e36ae9d0> (a atg.service.lockmanager.ClientLockEntry)
    at atg.service.lockmanager.ClientLockEntry.acquireWriteLock(ClientLockEntry.java:399)
    - locked <0x00000007e36ae9d0> (a atg.service.lockmanager.ClientLockEntry)
    at atg.service.lockmanager.ClientLockManager.acquireWriteLock(ClientLockManager.java:1137)
    at atg.repository.search.indexing.ConfigStatePersister.acquireLock(ConfigStatePersister.java:1844)
    at atg.repository.search.indexing.ConfigStatePersister$IndexingReadLockedTransactionAction.doWithTransaction(ConfigStatePersister.java:2083)
    at atg.repository.search.indexing.ConfigStatePersister.releaseAllConfigurations(ConfigStatePersister.java:1545)
    at atg.repository.search.indexing.ConfigStatePersister.performAction(ConfigStatePersister.java:1775)
    at atg.nucleus.Nucleus.performPostStartupActions(Nucleus.java:4336)
    at atg.nucleus.Nucleus.performPostStartupActions(Nucleus.java:4323)
    at atg.nucleus.Nucleus.<init>(Nucleus.java:954)
    at atg.nucleus.Nucleus.<init>(Nucleus.java:717)
    at atg.nucleus.Nucleus.startNucleusCheckLicense(Nucleus.java:4144)
    at atg.nucleus.servlet.NucleusServlet.initBigEarNucleus(NucleusServlet.java:1234)
    at atg.nucleus.servlet.NucleusServlet.init(NucleusServlet.java:443)
    at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1048)
    - locked <0x0000000781734f48> (a org.apache.catalina.core.StandardWrapper)
    at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:950)
    - locked <0x0000000781734f48> (a org.apache.catalina.core.StandardWrapper)
    at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:4122)
    at org.apache.catalina.core.StandardContext.start(StandardContext.java:4417)
    - locked <0x0000000781733e18> (a org.apache.catalina.core.StandardContext)
    at org.jboss.web.tomcat.service.deployers.TomcatDeployment.performDeployInternal(TomcatDeployment.java:321)
    at org.jboss.web.tomcat.service.deployers.TomcatDeployment.performDeploy(TomcatDeployment.java:145)
    at org.jboss.web.deployers.AbstractWarDeployment.start(AbstractWarDeployment.java:461)
    - locked <0x0000000780878058> (a org.jboss.web.tomcat.service.deployers.TomcatDeployment)
    at org.jboss.web.deployers.WebModule.startModule(WebModule.java:118)
    - locked <0x0000000780878038> (a org.jboss.web.deployers.WebModule)
    at org.jboss.web.deployers.WebModule.start(WebModule.java:97)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)
    at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
    at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
    at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
    at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
    at org.jboss.system.microcontainer.ServiceProxy.invoke(ServiceProxy.java:206)
    at $Proxy38.start(Unknown Source)
    at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:42)
    at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:37)
    at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62)
    at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71)
    at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51)
    at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)
    at org.jboss.system.microcontainer.ServiceControllerContext.install(ServiceControllerContext.java:297)
    at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1652)
    at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:938)
    at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)
    at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:988)
    at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:826)
    at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:556)
    at org.jboss.system.ServiceController.doChange(ServiceController.java:688)
    at org.jboss.system.ServiceController.start(ServiceController.java:460)
    at org.jboss.system.deployers.ServiceDeployer.start(ServiceDeployer.java:163)
    at org.jboss.system.deployers.ServiceDeployer.deploy(ServiceDeployer.java:99)
    at org.jboss.system.deployers.ServiceDeployer.deploy(ServiceDeployer.java:46)
    at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:62)
    at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:55)
    at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:179)
    at org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(DeployersImpl.java:1454)
    at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1172)
    at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1193)
    at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1225)
    at org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:1113)
    at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)
    at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1652)
    at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:938)
    at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)
    at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:988)
    at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:826)
    at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:556)
    at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:789)
    at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:699)
    at org.jboss.system.server.profileservice.repository.MainDeployerAdapter.process(MainDeployerAdapter.java:117)
    at org.jboss.system.server.profileservice.repository.ProfileDeployAction.install(ProfileDeployAction.java:70)
    at org.jboss.system.server.profileservice.repository.AbstractProfileAction.install(AbstractProfileAction.java:53)
    at org.jboss.system.server.profileservice.repository.AbstractProfileService.install(AbstractProfileService.java:403)
    at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)
    at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1652)
    at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:938)
    at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)
    at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:988)
    at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:778)
    at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:543)
    at org.jboss.system.server.profileservice.repository.AbstractProfileService.registerProfile(AbstractProfileService.java:308)
    at org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(ProfileServiceBootstrap.java:256)
    at org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:461)
    at org.jboss.Main.boot(Main.java:221)
    at org.jboss.Main$1.run(Main.java:556)
    at java.lang.Thread.run(Thread.java:662)

What is going on:
#

In the run of deployment, JBoss is starting context, hands over control to NucleusServlet init. Nucleus is trying to get write access lock and hangs there forever, while WAR deployment and all subsequent deployment never completes.

Jboss on Miro's World

Fixing the shutdown problem with JBoss

ATG + JBoss rant

Symptoms #

The exception: #

What is going on: #

Symptoms
#

The exception:
#

What is going on:
#