Overview of ElasticSearch

3441 ワード
elasticsearch テキストリンク

1. ElasticSearch


open source analytics & search engine
  • Search
  • Application Performance Management
  • Send events
  • Analyze data (ex. predict sale, anomality detection)

    • 2. How ElasticSearch Works


  • data stored as documents

  • document : json object = row in RDB
    • {
    "field name": "field value"
}
  • query through REST api
  • written in java

    • 3. Elastic Stack Overview



    3-1. Kibana : analytics & visualization platform

  • also with machine learning
  • web interface for elasticsearch
  • provides dashboards

    • 3-2. Logstash : data processing pipeline

  • data received as events, process and send to others
  • input plugins : files, http, sql, etc
  • filter plugins : csv, xml, json, data enrichment, etc
  • output plugins : elasticsearch, email, http, etc
    • ex) process access log from web server

  • Logstash reads logs as events

  • process logs w/something like regular expression

  • send processed logs to ElasticSearch

    • 3-3. X-Pack : additional features to elasticsearch & kibana

  • Security : authentication and authorization
  • Monitoring : CPU usage, disk space
  • Alerting : configurable using other features
  • Reporting : export Kibana visualizations to pdf, csv, etc
  • Machine Learning : for Elasticsearch & Kibana
  • Graph : analyze relationships in data
  • SQL : query elasticsearch with sql

    • 3-4. Beats : collect data and send to ElasticSearch & Logstash


  • Filebeat : collect log files

  • Metricbeat : collect system and service metrics

    • 4. Common ElasticSearch Architectures


    4-1. E-commerce Application


    Data in a RDB, wants to Improve search to be full-text search
  • Send user search to elasticsearch through web app server
  • Duplicate data on RDB and ElasticSearch
  • Simplest architecture of elasticsearch

    • 4-2. Visualize Data

  • add Kibana to communicate with ElasticSearch

    • 4-3. Monitor Server Metrics

  • install metircbeats on web app server
  • Kibana config is stored within ElasticSearch, so metricbeats data sent to ElasticSearch can easily be put on dashboard

    • 4-4. Monitor Acess and Error Logs


  • how long it takes to process each input? (monitor endpoints)

  • how to keep bugs to minimum?
    --> use Filebeat

    • 4-5. More Advanced Event Processing


  • Doing it within web app decreases maintainability
    --> use Logstash
  • In a perfect world, all events go through Logstash
    • 「Patternの設計」ステータス・アレイ
    byte配列とStringの相互回転