Devops

How to get information about all ATG Nucleus components # Dyn Admin has useful feature - Configuration reporter: http://localhost:8080/dyn/admin/atg/dynamo/admin/en/config-reporter-property-representation1.jhtml There are two variations of the report: Property representation report - PRR (shorter version) and Bean representation report (BRR). The PRR does not report properties that have default values, while BRR is reporting current values of all properties. Compare the sizes

Symptoms: # build fails cannot checkout / connect to GIT Validate this is the case # ssh to jenkins2 assume identity of jenkins user (VERY IMPORTANT) go to /home/projects/workspace/PROJECTNAME directory This directory is repository clone. Try ‘git status’ in that directory. If you see something like

After re-creating the SSH key, I got the above error from Gitolite. Full trace: ➜ TWC ssh -v thinkwrap@pensieve.thinkwrap.com OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013 debug1: Reading configuration data /etc/ssh/ssh_config debug1: Applying options for * debug1: Connecting to pensieve.thinkwrap.com [24.137.198.18] port 22. debug1: Connection established. debug1: identity file /home/thinkwrap/.ssh/identity type -1 debug1: identity file /home/thinkwrap/.ssh/identity-cert type -1 debug1: identity file /home/thinkwrap/.ssh/id_rsa type 1 debug1: identity file /home/thinkwrap/.ssh/id_rsa-cert type -1 debug1: identity file /home/thinkwrap/.ssh/id_dsa type -1 debug1: identity file /home/thinkwrap/.ssh/id_dsa-cert type -1 debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3 debug1: match: OpenSSH_5.3 pat OpenSSH* debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_5.3 debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: server->client aes128-ctr hmac-md5 none debug1: kex: client->server aes128-ctr hmac-md5 none debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP debug1: SSH2_MSG_KEX_DH_GEX_INIT sent debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY debug1: Host 'pensieve.thinkwrap.com' is known and matches the RSA host key. debug1: Found key in /home/thinkwrap/.ssh/known_hosts:2 debug1: ssh_rsa_verify: signature correct debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: SSH2_MSG_SERVICE_REQUEST sent debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password debug1: Next authentication method: gssapi-keyex debug1: No valid Key exchange context debug1: Next authentication method: gssapi-with-mic debug1: Unspecified GSS failure. Minor code may provide more information Credentials cache file '/tmp/krb5cc_500' not found debug1: Unspecified GSS failure. Minor code may provide more information Credentials cache file '/tmp/krb5cc_500' not found debug1: Unspecified GSS failure. Minor code may provide more information debug1: Unspecified GSS failure. Minor code may provide more information Credentials cache file '/tmp/krb5cc_500' not found debug1: Next authentication method: publickey debug1: Offering public key: /home/thinkwrap/.ssh/id_rsa debug1: Remote: Forced command: /home/thinkwrap/gitolite/src/gitolite-shell miro.adamy+ATG11-TrainingVM debug1: Remote: Port forwarding disabled. debug1: Remote: X11 forwarding disabled. debug1: Remote: Agent forwarding disabled. debug1: Remote: Pty allocation disabled. debug1: Server accepts key: pkalg ssh-rsa blen 277 Agent admitted failure to sign using the key. debug1: Trying private key: /home/thinkwrap/.ssh/identity debug1: Trying private key: /home/thinkwrap/.ssh/id_dsa debug1: Next authentication method: password thinkwrap@pensieve.thinkwrap.com's password: Solution # log-out, log on

Observed at QA environment. During automated shutdown sequence, the exception was thrown: $ bin/gdi-qa-com1.sh stop JBOSS_CMD_STOP = java -classpath /opt/jboss/jboss-as/bin/shutdown.jar:/opt/jboss/jboss-as/client/jnet.jar org.jboss.Shutdown --shutdown --user=admin --password=admin -s jnp://0.0.0.0:1099 Exception in thread "main" javax.naming.CommunicationException: Could not obtain connection to any of these urls: 0.0.0.0:1099 [Root exception is javax.naming.CommunicationException: Failed to connect to server /0.0.0.0:1099 [Root exception is javax.naming.ServiceUnavailableException: Failed to connect to server /0.0.0.0:1099 [Root exception is java.net.ConnectException: Connection refused]]] at org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1763) at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:693) at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:686) at javax.naming.InitialContext.lookup(InitialContext.java:392) at org.jboss.Shutdown.main(Shutdown.java:219) Caused by: javax.naming.CommunicationException: Failed to connect to server /0.0.0.0:1099 [Root exception is javax.naming.ServiceUnavailableException: Failed to connect to server /0.0.0.0:1099 [Root exception is java.net.ConnectException: Connection refused]] at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:335) at org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1734) ... 4 more Caused by: javax.naming.ServiceUnavailableException: Failed to connect to server /0.0.0.0:1099 [Root exception is java.net.ConnectException: Connection refused] at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:305) ... 5 more Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:211) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at org.jnp.interfaces.TimedSocketFactory.createSocket(TimedSocketFactory.java:97) at org.jnp.interfaces.TimedSocketFactory.createSocket(TimedSocketFactory.java:82) at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:301) ... 5 more To watch the log: tail -f /var/log/atg/gdi-qa-com1.log The conflict of 1099 between Endeca and ATG was resolved according Fixing port conflict on 1099 between out-of-the-box Endeca and JBOSS - the actual port was 11199

The groovy script does parse the XML files (scraped from dyn-admin, or output of the XML combine) and produces file in the syntax of Plant UML - http://plantuml.sourceforge.net if (args.size() == 0) { println "Usage: groovy extractItems.groovy XMLFILE-IN-GSA-FORMAT.xml" System.exit(1) } def gsaFile = new XmlSlurper().parseText(new File(args[0]).text) def desc = gsaFile.'item-descriptor' def processItemDescriptor(myDesc) { def info = [:] def tables = [] def props = [:] def descriptors = [:] for (table in myDesc.table) { String tableName = table.@name.text() tables += tableName // collect table's properties for (prop in table.property) { String name = "${prop.@name.text()}(${tableName})".replaceAll(/[ -]/, '_') props[name] = [:] if ("${prop.@'required'.text()}" != "") { props[name]['required'] = "${prop.@'required'.text()}" } if ("${prop.@'hidden'.text()}" != "") { props[name]['hidden'] = "${prop.@'hidden'.text()}" } if ("${prop.@'item-type'.text()}" != "") { props[name]['item-type'] = "${prop.@'item-type'.text()}" descriptors["${prop.@name.text()}"] = "${prop.@'item-type'.text().replaceAll(/[ -]/, '_')}" } } } info['tables'] = tables // scan for properties outside of tables for (prop in myDesc.property) { String name = "${prop.@name.text()}".replaceAll(/[ -]/, '_') props[name] = [:] if ("${prop.@'item-type'.text()}" != "") { props[name]['item-type'] = "${prop.@'item-type'.text()}" } } info['properties'] = props info['descriptors'] = descriptors return info } println "Got ${desc.size()} item-descriptors" println "@startuml" for (idesc in desc) { def info = processItemDescriptor(idesc) def name = "${idesc.@name.text().replaceAll(/[ -]/, '_')}" println "\nclass ${name} {" for (p in info.tables) println "\t"+p println ".." for (p in info.properties) println "\t"+p println "}" for (p in info.descriptors?.keySet()) println "\t ${name} *-- ${info.descriptors[p]} : ${p}" } println "@enduml" The generation (requires installed dot tools)

When trying to start DBeaver (http://dbeaver.jkiss.org) I got this I do have Java 7 installed: ➜ Contents java -version java version "1.7.0_55" Java(TM) SE Runtime Environment (build 1.7.0_55-b13) Java HotSpot(TM) 64-Bit Server VM (build 24.55-b03, mixed mode) ➜ Contents which java java is /Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/bin/java java is /usr/bin/java which should be a superset. But it is not. Solution # Oracle seems to screw up the PLIST capabilities in JAva7 install

With emails (Unix only) # ➜ dev1 git:(miro-ui) ✗ git log --raw | grep "^Author: " | sort | uniq -c 9 Author: Brendan Smith <Brendan.Smith@thinkwrap.com> 22 Author: Kai Cheng <kai.cheng@thinkwrap.com> 97 Author: Miro Adamy <miro.adamy@thinkwrap.com> 49 Author: Training VM <training-vm@thinkwrap.com> 24 Author: Your Name <your.name@thinkwrap.com> Pure git # ➜ dev1 git:(miro-ui) ✗ git shortlog -sen 97 Miro Adamy <miro.adamy@thinkwrap.com> 49 Training VM <training-vm@thinkwrap.com> 24 Your Name <your.name@thinkwrap.com> 22 Kai Cheng <kai.cheng@thinkwrap.com> 9 Brendan Smith <Brendan.Smith@thinkwrap.com> Mailmap option to consolidate the variety of authors # Before (no mailmap) # ... 609 milos hanzel <milos.hanzel@thinkwrap.com> 361 brant maahs <brant.maahs@thinkwrap.com> 69 Sullivan Tseng <sullivantseng@gmail.com> 42 sullivan tseng <sullivan.tseng@thinkwrap.com> 17 milos.hanzel <milos.hanzel@thinkwrap.com> 2 Miro Adamy <miro.adamy@thinkwrap.com> ... Generating the mailman start

Make sure the VBOx extensions are installed. Go here for details: Add share # In VirtualBox, Device add transient shared folder: This one has made available path /Users/Shared/ATG-11 under the name ATG-11 Mount it # sudo mount -t vboxsf ATG-11 /media $ ll /media/Unix/ total 19M drwxr-xr-x. 1 root 408 Sep 12 16:02 ATG11-1 drwxr-xr-x. 1 root 272 Sep 12 16:02 ATG11-docs -rw-rw-r--. 1 root 675 May 13 23:46 atg-platform-mac.userlibraries -rw-rw-r--. 1 root 787 May 13 23:17 atg-platform-vm.userlibraries -rwxr--r--. 1 root 19M Feb 15 2013 jrebel-5.1.3-nosetup.zip -rw-r--r--. 1 root 1020 May 15 15:36 jrebel.lic -rw-r--r--. 1 root 582 May 15 14:58 rebel.xml drwxr-xr-x. 1 root 272 Apr 6 01:33 security drwxr-xr-x. 1 root 306 Jun 9 22:34 servers -rwxr--r--. 1 root 349 Dec 17 2013 users.txt drwxr-xr-x. 1 root 340 Sep 3 16:10 xrebel

I have been trying to make this work for about 1.5 hr. Looks like there is open bug - https://logstash.jira.com/browse/LOGSTASH-703 Helpful link: https://groups.google.com/forum/#!topic/logstash-users/sZM03po7HJE What should work: grok { match => ["message", "regex to parse severity"], match => ["message", "regex to parse server IP"], match => ["message", "regex to parse user"] } What needs to be done instead You _should_ be able to do exactly what you listed at the bottom of your email, except that you'd need `break_on_match => false` so that it would parse each snippet for each message instead of just parsing the first one that matches. Unfortunately, due to a bug (https://logstash.jira.com/browse/LOGSTASH-703), this doesn't work when you're matching against the same field in each match expression ("message" in your case). I was hoping to take a stab at fixing this bug (as several others have mentioned an interest in doing), but got distracted and haven't done it yet. It shouldn't be terribly hard to fix, just needs some time. As a work-around, the following should work, but unfortunately the way that your tag_on_failure will end up working will be different because you have multiple Grok filters that could fail independently. It's probably slightly less efficient to do it this way because the event has to pass from one filter to the next through the pipeline, but my guess (based on absolutely no empirical data) is that it isn't significantly slower because the same work would need to be done by Regex either way, there's just more LogStash in the mix this way. grok { match => ["message", "regex to parse severity"] } grok { match => ["message", "regex to parse server IP"] } grok { match => ["message", "regex to parse user"] } ~Greg Mefford

Graphviz # This time using brew ➜ DPS brew install graphviz ==> Downloading https://downloads.sf.net/project/machomebrew/Bottles/graphviz-2.38.0.mavericks.bottle.tar.gz ######################################################################## 100.0% ==> Pouring graphviz-2.38.0.mavericks.bottle.tar.gz 🍺 /usr/local/Cellar/graphviz/2.38.0: 469 files, 68M ➜ DPS dot -V dot - graphviz version 2.38.0 (20140413.2041) ➜ DPS Schema Spy # From http://sourceforge.net/projects/schemaspy/files/schemaspy/SchemaSpy%205.0.0/ To /opt/diagram Run: diagram java -jar ./schemaSpy_5.0.0.jar -cp .:/Users/miro/lib/jdbc/mysql-connector-java-5.1.26-bin.jar -t mysql -o library -host localhost -u training -db training_dev01 -p training