Elasticsearch with Docker から Kibana へ (NetFlowの可視化)
Fluentd で取得した NetFlow v5 のデータを Kibana で表示するための環境を Docker を使って構築します。
1. 前段として、dockerとdocker−composeは入れておく
2. メモリマップの上限値を上げておく
vm.max_map_count=262144
3. docker-compose.yml を以下のように記述。
fluentd:
build: fluentd
ports:
- 24224:24224
- 24224:24224/udp
volumes:
- ./fluentd:/fluentd/etc
- ./fluentd/plugins:/fluentd/plugins
links:
- elasticsearch
elasticsearch:
image: elasticsearch:5.6.4
ports:
- 9200:9200
- 9300:9300
expose:
- 9300
kibana:
image: kibana:5.6.4
ports:
- 5601:5601
environment:
- ELASTICSEARCH_URL=http://172.17.0.2:9200
# - ELASTICSEARCH_URL=http://elasticsearch:9200
links:
- elasticsearch
4. fluentd から elastic search へ送る設定を作る
FROM fluent/fluentd:latest-onbuild
#USER fluent
WORKDIR ~/netflow-compose/fluentd
ENV PATH ~/.gem/ruby/2.3.0/bin:$PATH
RUN apk add --no-cache --update --virtual=build-dependencies ruby-dev build-base && \
gem install fluent-plugin-rewrite-tag-filter && \
apk del build-dependencies && \
rm -rf /tmp/* /var/tmp/* /var/cache/apt/*
RUN gem install fluent-plugin-netflow
RUN gem install fluent-plugin-secure-forward
RUN gem install fluent-plugin-elasticsearch # Elasticsearch連携
#EXPOSE 24284
EXPOSE 24224
CMD fluentd -c /fluentd/etc/fluent.conf -p /fluentd/plugins -vv
<source>
type netflow
tag netflow.event
port 24224
</source>
<match netflow.event>
@type copy
<store>
@type file
path /fluentd/log/data.*.log
symlink_path /fluentd/log/data.log
append true
</store>
<store>
@type elasticsearch
# host localhost
# host elasticsearch
host 172.17.0.2
port 9200
type_name netflow
logstash_format true
logstash_prefix flow
logstash_dateformat %Y%m%d
</store>
</match>
5. あとは、docker-compose up する
$ sudo docker-compose up
参考リンク
【Docker】FluentdでとElasticsearchとRe:dashでログ管理
https://qiita.com/narumi_/items/59237554ea1c0e713dd9
・Docker 公式
https://www.elastic.co/guide/en/elasticsearch/reference/5.6/docker.html
http://www.elastic.co/guide/en/elasticsearch/reference/5.6/docker.html#docker-prod-cluster-composefile
・enqueueing all chunks in bufferについて
https://qiita.com/tatsu-yam/items/bd7006e483f3b3c64309
・fluentdのRuby導入についてはこのへんを参考に、
http://cross-black777.hatenablog.com/entry/2017/11/15/190000
Author And Source
この問題について(Elasticsearch with Docker から Kibana へ (NetFlowの可視化)), 我々は、より多くの情報をここで見つけました https://qiita.com/Yohichi_Hayashi/items/f1c9c890f2736eb9429a著者帰属:元の著者の情報は、元のURLに含まれています。著作権は原作者に属する。
Content is automatically searched and collected through network algorithms . If there is a violation . Please contact us . We will adjust (correct author information ,or delete content ) as soon as possible .