logstash - known error

Tue, 16 Sep 2014 11:22:48 +0800

I have been trying to make this work for about 1.5 hr. Looks like there is open bug - https://logstash.jira.com/browse/LOGSTASH-703

Helpful link: https://groups.google.com/forum/#!topic/logstash-users/sZM03po7HJE

What should work:

grok {
   match => ["message", "regex to parse severity"],
   match => ["message", "regex to parse server IP"],
   match => ["message", "regex to parse user"]
}

What needs to be done instead

You _should_ be able to do exactly what you listed at the bottom of your email, 
except that you'd need `break_on_match => false` so that it would parse each snippet for each message 
instead of just parsing the first one that matches. 
Unfortunately, due to a bug (https://logstash.jira.com/browse/LOGSTASH-703), 
this doesn't work when you're matching against the same field in each match expression ("message" in your case). 
I was hoping to take a stab at fixing this bug (as several others have mentioned an interest in doing), 
but got distracted and haven't done it yet. It shouldn't be terribly hard to fix, just needs some time.
 
As a work-around, the following should work, but unfortunately the way that your tag_on_failure 
will end up working will be different 
because you have multiple Grok filters that could fail independently. 
It's probably slightly less efficient to do it this way because the event has to pass 
from one filter to the next through the pipeline, 
but my guess (based on absolutely no empirical data) is that it isn't significantly slower 
because the same work would need to be done by Regex either way, 

there's just more LogStash in the mix this way.
 
 
grok {
   match => ["message", "regex to parse severity"]
 
}
grok {
   match => ["message", "regex to parse server IP"]
}
grok {
   match => ["message", "regex to parse user"]
}
 
 
 
~Greg Mefford

Elastic search and Kibana - getting started

Wed, 27 Aug 2014 11:22:48 +0800

#Installation:

See

curl -L -O http://download.elasticsearch.org/PATH/TO/VERSION.zip
unzip elasticsearch-$VERSION.zip
cd  elasticsearch-$VERSION

Marvel is management plugin - console about cluster. Install:

./bin/plugin -i elasticsearch/marvel/latest
# disable data collection for local cluster

echo 'marvel.agent.enabled: false' >> ./config/elasticsearch.yml

Test
#

Startup log:

➜  elasticsearch-1.3.2  bin/elasticsearch
[2014-08-27 11:17:08,325][INFO ][node                     ] [Joe Fixit] version[1.3.2], pid[59610], build[dee175d/2014-08-13T14:29:30Z]
[2014-08-27 11:17:08,325][INFO ][node                     ] [Joe Fixit] initializing ...
[2014-08-27 11:17:08,338][INFO ][plugins                  ] [Joe Fixit] loaded [marvel], sites [marvel, kopf]
[2014-08-27 11:17:10,905][INFO ][marvel.agent             ] [Joe Fixit] collecting disabled by settings
[2014-08-27 11:17:11,020][INFO ][node                     ] [Joe Fixit] initialized
[2014-08-27 11:17:11,021][INFO ][node                     ] [Joe Fixit] starting ...
[2014-08-27 11:17:11,131][INFO ][transport                ] [Joe Fixit] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/192.168.179.121:9300]}
[2014-08-27 11:17:11,148][INFO ][discovery                ] [Joe Fixit] elasticsearch/nNgRXlHARHCPutTq7dZXWg
[2014-08-27 11:17:14,160][INFO ][cluster.service          ] [Joe Fixit] new_master [Joe Fixit][nNgRXlHARHCPutTq7dZXWg][Miros-MacBook-Pro-2.local][inet[/192.168.179.121:9300]], reason: zen-disco-join (elected_as_master)
[2014-08-27 11:17:14,181][INFO ][http                     ] [Joe Fixit] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/192.168.179.121:9200]}
[2014-08-27 11:17:14,181][INFO ][node                     ] [Joe Fixit] started
[2014-08-27 11:17:14,726][INFO ][gateway                  ] [Joe Fixit] recovered [4] indices into cluster_state
[2014-08-27 11:18:00,491][INFO ][cluster.service          ] [Joe Fixit] added {[logstash-Miros-MacBook-Pro-2.local-59618-4086][Eh5YjUFxSIWIn4xK45xu0w][Miros-MacBook-Pro-2.local][inet[/192.168.179.121:9301]]{client=true, data=false},}, reason: zen-disco-receive(join from node[[logstash-Miros-MacBook-Pro-2.local-59618-4086][Eh5YjUFxSIWIn4xK45xu0w][Miros-MacBook-Pro-2.local][inet[/192.168.179.121:9301]]{client=true, data=false}])
Installed some pluggins - marvel, kopf ...

 

➜  ~  curl 'http://localhost:9200/?pretty'
{
  "status" : 200,
  "name" : "Sea Urchin",
  "version" : {
    "number" : "1.3.2",
    "build_hash" : "dee175dbe2f254f3f26992f5d7591939aaefd12f",
    "build_timestamp" : "2014-08-13T14:29:30Z",
    "build_snapshot" : false,
    "lucene_version" : "4.9"
  },
  "tagline" : "You Know, for Search"
}

The sense console allows to avoid using curl and do it from browser: - <http://localhost:9200/_plugin/marvel/sense/ >

Elk on Miro's World

logstash - known error

Elastic search and Kibana - getting started

Test #

Test
#