Kibana + ElasticSearch + Logstash + Redis on RHEL 6

システムの各部分の応用紹介:

Kibana:オープンソースweb

を表示

Elasticsearch:オープンソースの検索エンジンフレームワークlogstash部分は、マルチデータクラスタを行うことができ、効率を高め、redisからデータを読み出し、Kibana

に転送することができる.

Logstash:システムロゴ収集、転載のツール、同時に各種類のログプラグインを集積し、ログ検索と分析の効率を著しく向上させる

Logstash shipper:logを収集し、logをredisストレージ

に転送する.

Logstash indexer:redisからデータを読み出しelasticsearch

に転送する

Redis:dbで、logstash shipperはlogをredisデータベースに転送して

を格納します.
配置:
1、jdk:
path:/usr/local/jdk7
cat/etc/profile
export JAVA_HOME=/usr/local/jdk7    export PATH=$JAVA_HOME/bin:$PATH
export REDIS_HOME=/usr/local/redis-2.6.12
export ES_HOME=/usr/local/elasticsearch
export ES_CLASSPATH=$ES_HOME/config
2、ElasticSearch

wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.20.2.tar.gz
tar xvf elasticsearch-0.20.2.tar.gz
mv elasticsearch-0.20.2 elasticsearch

cd /usr/local/elasticsearch/config

vim elasticsearch.yml

cluster.name: elasticsearch

node.name: "litong"

path.conf: /usr/local/elasticsearch

path.data: /usr/local/elasticsearch/data

path.work: /usr/local/elasticsearch/tmp

path.logs: /usr/local/elasticsearch/logs

bootstrap.mlockall: true

 
 
mkdir -p /usr/local/elasticsearch/data /usr/local/elasticsearch/tmp  /usr/local/elasticsearch/logs
  
3、Configure Java Service Wrapper
  
Get the service wrapper
  
wget http://github.com/elasticsearch/elasticsearch-servicewrapper/archive/master.zip
unzip master.zip
mv elasticsearch-servicewrapper-master/service/ /usr/local/elasticsearch/bin/
rm -rf elasticsearch-servicewrapper-master/
  
 
 
vim service/elasticsearch.conf
  
set.default.ES_HOME=/usr/local/elasticsearch
set.default.ES_HEAP_SIZE=1024                         #memory
wrapper.java.additional.10=-Des.max-open-files=true
wrapper.logfile.maxsize=5m
wrapper.logfile.maxfiles=5
  
 
 
service：/etc/init.d/elasticsearch
  
bin/service/elasticsearch install
  
service elasticsearch start
  
 
 
4、ElasticSearch Head
 
 
bin/plugin -install mobz/elasticsearch-head
  
5、Redis
  
Install Redis server 
wget http://redis.googlecode.com/files/redis-2.6.12.tar.gz
tar xzf redis-2.6.12.tar.gz

mv redis-2.6.12 /usr/local/redis
cd /usr/local/redis
make
  
make install
  
5、Configure Redis �C ‘cp redis.conf 6379.conf’  
vim  6379.conf 
daemonize yes
pidfile /var/run/redis/redis_6379.pid
port 6379
timeout 300
tcp-keepalive 60
logfile /var/log/redis/redis_6379.log
  
Add REDIS home to root user’s ‘.bash_profile’ 
# Redis
export REDIS_HOME=/usr/local/redis
  
Copy Redis init script 
cp utils/redis_init_script /etc/init.d/redis_6379
  
Configure Redis init script 
# chkconfig:   - 85 15
# description: Redis is a persistent key-value database
# processname: redis

REDISPORT=6379
EXEC=/usr/local/redis/src/redis-server
CLIEXEC=/usr/local/redis/src/redis-cli
PIDFILE=/var/run/redis/redis_6379.pid
CONF="/usr/local/redis/6379.conf"
  
Activate Redis service 
mkdir /var/run/redis /var/log/redis
cd /etc/init.d
chkconfig --add redis
  
Start 
service redis start
  
6、Logstash
  
mkdir /usr/local/logstash
cd /usr/local/logstash
wget https://logstash.objects.dreamhost.com/release/logstash-1.1.9-monolithic.jar
  
Indexer configuration �C indexer.conf：
  
input {     redis {       host => "192.168.0.235"      port => "6379"      type => "redis-input"      data_type => "list"      key => "logstash"      format => "json_event"    }   }  
output {     stdout { debug => true debug_format => "json"}  
 elasticsearch {       host => "192.168.0.235"      port => "9300"      cluster => "elasticsearch"    }   } 
Shipper configuration �C shipper.conf:
  
input {     file {       type => "nginx"      path => ["/usr/local/nginx/logs/*.log"] 
   exclude => ["*.gz"]  
   tags => ["nginx"]     }   }  
output {     stdout { debug => true debug_format => "json"}  
 redis {       host => "192.168.0.235"      data_type => "list"      key => "logstash"    }   } 
java -jar logstash-1.1.9-monolithic.jar agent -f indexer.conf &

java -jar logstash-1.1.9-monolithic.jar agent -f shipper.conf &
  
7、Kibana 
Setup Ruby 
yum install ruby ruby-devel ruby-ri ruby-rdoc rubygems
  
wget http://production.cf.rubygems.org/rubygems/rubygems-2.0.3.zip
unzip rubygems-2.0.3.zip
ruby rubygems-2.0.3/setup.rb
  
Get Kibana 
wget https://github.com/rashidkpc/Kibana/archive/v0.2.0.zip
unzip v0.2.0.zip
cd Kibana-0.2.0
gem install bundler
bundle install
  
Configure KibanaConfig.rb:
  
Elasticsearch = "192.168.0.235:9200"
KibanaPort = 80
KibanaHost = '192.168.0.235'
  
Run Kibana 
bundle exec ruby kibana.rb