↓ Skip to main content

Elk

logstash - known error

·288 words·2 mins
I have been trying to make this work for about 1.5 hr. Looks like there is open bug - https://logstash.jira.com/browse/LOGSTASH-703 Helpful link: https://groups.google.com/forum/#!topic/logstash-users/sZM03po7HJE What should work: grok { match => ["message", "regex to parse severity"], match => ["message", "regex to parse server IP"], match => ["message", "regex to parse user"] } What needs to be done instead You _should_ be able to do exactly what you listed at the bottom of your email, except that you'd need `break_on_match => false` so that it would parse each snippet for each message instead of just parsing the first one that matches. Unfortunately, due to a bug (https://logstash.jira.com/browse/LOGSTASH-703), this doesn't work when you're matching against the same field in each match expression ("message" in your case). I was hoping to take a stab at fixing this bug (as several others have mentioned an interest in doing), but got distracted and haven't done it yet. It shouldn't be terribly hard to fix, just needs some time. As a work-around, the following should work, but unfortunately the way that your tag_on_failure will end up working will be different because you have multiple Grok filters that could fail independently. It's probably slightly less efficient to do it this way because the event has to pass from one filter to the next through the pipeline, but my guess (based on absolutely no empirical data) is that it isn't significantly slower because the same work would need to be done by Regex either way, there's just more LogStash in the mix this way. grok { match => ["message", "regex to parse severity"] } grok { match => ["message", "regex to parse server IP"] } grok { match => ["message", "regex to parse user"] } ~Greg Mefford

Elastic search and Kibana - getting started

·604 words·3 mins
#Installation: See curl -L -O http://download.elasticsearch.org/PATH/TO/VERSION.zip unzip elasticsearch-$VERSION.zip cd elasticsearch-$VERSION Marvel is management plugin - console about cluster. Install: ./bin/plugin -i elasticsearch/marvel/latest # disable data collection for local cluster echo 'marvel.agent.enabled: false' >> ./config/elasticsearch.yml Test # Startup log: ➜ elasticsearch-1.3.2 bin/elasticsearch [2014-08-27 11:17:08,325][INFO ][node ] [Joe Fixit] version[1.3.2], pid[59610], build[dee175d/2014-08-13T14:29:30Z] [2014-08-27 11:17:08,325][INFO ][node ] [Joe Fixit] initializing ... [2014-08-27 11:17:08,338][INFO ][plugins ] [Joe Fixit] loaded [marvel], sites [marvel, kopf] [2014-08-27 11:17:10,905][INFO ][marvel.agent ] [Joe Fixit] collecting disabled by settings [2014-08-27 11:17:11,020][INFO ][node ] [Joe Fixit] initialized [2014-08-27 11:17:11,021][INFO ][node ] [Joe Fixit] starting ... [2014-08-27 11:17:11,131][INFO ][transport ] [Joe Fixit] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/192.168.179.121:9300]} [2014-08-27 11:17:11,148][INFO ][discovery ] [Joe Fixit] elasticsearch/nNgRXlHARHCPutTq7dZXWg [2014-08-27 11:17:14,160][INFO ][cluster.service ] [Joe Fixit] new_master [Joe Fixit][nNgRXlHARHCPutTq7dZXWg][Miros-MacBook-Pro-2.local][inet[/192.168.179.121:9300]], reason: zen-disco-join (elected_as_master) [2014-08-27 11:17:14,181][INFO ][http ] [Joe Fixit] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/192.168.179.121:9200]} [2014-08-27 11:17:14,181][INFO ][node ] [Joe Fixit] started [2014-08-27 11:17:14,726][INFO ][gateway ] [Joe Fixit] recovered [4] indices into cluster_state [2014-08-27 11:18:00,491][INFO ][cluster.service ] [Joe Fixit] added {[logstash-Miros-MacBook-Pro-2.local-59618-4086][Eh5YjUFxSIWIn4xK45xu0w][Miros-MacBook-Pro-2.local][inet[/192.168.179.121:9301]]{client=true, data=false},}, reason: zen-disco-receive(join from node[[logstash-Miros-MacBook-Pro-2.local-59618-4086][Eh5YjUFxSIWIn4xK45xu0w][Miros-MacBook-Pro-2.local][inet[/192.168.179.121:9301]]{client=true, data=false}]) Installed some pluggins - marvel, kopf ... ➜ ~ curl 'http://localhost:9200/?pretty' { "status" : 200, "name" : "Sea Urchin", "version" : { "number" : "1.3.2", "build_hash" : "dee175dbe2f254f3f26992f5d7591939aaefd12f", "build_timestamp" : "2014-08-13T14:29:30Z", "build_snapshot" : false, "lucene_version" : "4.9" }, "tagline" : "You Know, for Search" } The sense console allows to avoid using curl and do it from browser: - <http://localhost:9200/_plugin/marvel/sense/ >