Elasticsearch 6.0クラスタのインストールと構成

事前の要件
ESクラスタのインストールには最低3台のマシンが必要です.ubuntu 16.04のマシン3台(192.168.71.181~183)を用意しています.ESバージョンは6.0です.
国内ソースの交換

$ sudo cp /etc/apt/sources.list /etc/apt/sources.list.bak
$ sudo vim /etc/apt/sources.list
"""
# deb cdrom:[Ubuntu 16.04 LTS _Xenial Xerus_ - Release amd64 (20160420.1)]/ xenial main restricted
deb-src http://archive.ubuntu.com/ubuntu xenial main restricted 
#Added by software-properties
deb http://mirrors.aliyun.com/ubuntu/ xenial main restricted
deb-src http://mirrors.aliyun.com/ubuntu/ xenial main restricted multiverse universe 
#Added by software-properties
deb http://mirrors.aliyun.com/ubuntu/ xenial-updates main restricted
deb-src http://mirrors.aliyun.com/ubuntu/ xenial-updates main restricted multiverse universe 
#Added by software-properties
deb http://mirrors.aliyun.com/ubuntu/ xenial universe
deb http://mirrors.aliyun.com/ubuntu/ xenial-updates universe
deb http://mirrors.aliyun.com/ubuntu/ xenial multiverse
deb http://mirrors.aliyun.com/ubuntu/ xenial-updates multiverse
deb http://mirrors.aliyun.com/ubuntu/ xenial-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ xenial-backports main restricted universe multiverse 
#Added by software-properties
deb http://archive.canonical.com/ubuntu xenial partner
deb-src http://archive.canonical.com/ubuntu xenial partner
deb http://mirrors.aliyun.com/ubuntu/ xenial-security main restricted
deb-src http://mirrors.aliyun.com/ubuntu/ xenial-security main restricted multiverse universe 
#Added by software-properties
deb http://mirrors.aliyun.com/ubuntu/ xenial-security universe
deb http://mirrors.aliyun.com/ubuntu/ xenial-security multiverse
"""

$ sudo apt-get update 
$ sudo apt-get update --fix-missing

JAVAのインストール

$ sudo apt-get install openjdk-8-jdk
$ sudo apt-get install apt-transport-https

ESとKIBANAのインストール

#      deb 
$ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.3.deb
$ wget https://artifacts.elastic.co/downloads/kibana/kibana-6.2.3-amd64.deb
$ wget https://artifacts.elastic.co/downloads/logstash/logstash-6.2.3.deb

#   
$ sudo dpkg -i elasticsearch-6.2.3.deb
$ sudo dpkg -i kibana-6.2.3-amd64.deb 
$ sudo dpkg -i logstash-6.2.3.deb 

"""
    ：
   ：/usr/share/elasticsearch
   ：/etc/elasticsearch
    ：/etc/default/elasticsearch
  ：/var/lib/elasticsearch
  ：/var/log/elasticsearch

   ：/usr/share/kibana
   ：/etc/kibana
  ：/var/lib/kibana
"""

#       
$ sudo systemctl daemon-reload
$ sudo systemctl enable elasticsearch.service
$ sudo systemctl enable kibana.service
$ sudo systemctl enable logstash.service 

#   
$ sudo systemctl start elasticsearch.service
$ sudo systemctl start kibana.service
$ sudo systemctl start logstash.service

コンフィギュレーション
すべての3台の機器は以下のような構成が必要で、自身の対応IPを修正するだけでよい.
181サーバプライマリノードの構成

"""
#      
cluster.name: ccnu-resource-cluster

node.name: node-181
node.master: true
node.data: false
node.ingest: false

network.host: 0.0.0.0

http.port: 9200
http.cors.enabled: true
http.cors.allow-origin: "*"

discovery.zen.ping.unicast.hosts: ["192.168.71.181"]
"""

サーバをデータノードと前処理ノードとして構成する182

"""
#      
cluster.name: ccnu-resource-cluster

node.name: node-182
node.master: false
node.data: true
node.ingest: true

network.host: 0.0.0.0

http.port: 9200
http.cors.enabled: true
http.cors.allow-origin: "*"

discovery.zen.ping.unicast.hosts: ["192.168.71.181"]
"""

183サーバをデータノードと前処理ノードとして構成する

"""
#      
cluster.name: ccnu-resource-cluster

node.name: node-183
node.master: false
node.data: true
node.ingest: true

network.host: 0.0.0.0

http.port: 9200
http.cors.enabled: true
http.cors.allow-origin: "*"

discovery.zen.ping.unicast.hosts: ["192.168.71.181"]
"""

名詞の解釈

cluster.name:クラスタ内のクラスタ名が一致する必要がある

node.マスター:clusterを制御するプライマリサーバ

を決定する

node.data:スライスされたCRUDなどのデータ関連操作と、検索と統合操作を担当します.これらの動作は、CPU、メモリ、I/Oリソース

を比較的消費する.

discovery.zen.ping.unicast.hosts:gossipアルゴリズムの伝播方式を使用して最終的な一貫性を保証し、unicast hostのリストが合法的なプライマリノードサーバであることを提案します.

KIBANAの設定

$ sudo vim /etc/kibana/kibana.yml
"""
server.host: "192.168.71.181"
"""

再起動

$ sudo systemctl restart elasticsearch.service
$ sudo systemctl restart kibana.service

メンテナンス
クラスタのステータスの表示

curl localhost:9200/_cluster/health?pretty
curl localhost:9200/_nodes/http?pretty
curl localhost:9200/_nodes/jvm?pretty
curl localhost:9200/_nodes/network?pretty
curl localhost:9200/_nodes/os?pretty
curl localhost:9200/_nodes/plugins?pretty
curl localhost:9200/_nodes/process?pretty
curl localhost:9200/_nodes/settings?pretty
curl localhost:9200/_nodes/thread_pool?pretty
curl localhost:9200/_nodes/transport?pretty

Trouble Shotting
failed to send join request to master, with the same id but is a different node instance
解決策:Ok so the issue was copying the elasticsearch folder from one node to another over scp.Elasticsearch saves the node id in elasticsearch/data/folder. Deleted the data folder on one node and restarted it. The cluster is up and running.

$ sudo chmod -R 777 /var/lib/elasticsearch/
$ sudo rm -rf /var/lib/elasticsearch/*
$ sudo systemctl restart elasticsearch.service