Flume1.6.0 ES2.3.5 sink

1.flume 1を削除する.6.0/libの中のguava-jar and jackson-core-.jar
2.コピーes 2.3.5/lib里es 2.3.5.JArパッケージ、flume 1.6.0/lib
3.コピーelasticsearch-sink 2-1.0.JAr,flume 1.6.0/lib
4.flume 1を編集する.6.0/conf/es.conf
5.スタートes
6.テストファイルを作成し、flumeを起動する

3.コピーelasticsearch-sink 2-1.0.JAr,flume 1.6.0/lib

buildでjarパッケージを出すこともできますhttps://github.com/lucidfrontier45/ElasticsearchSink2

ElasticsearchSink2

This is Flume-NG Sink for Elasticsearch >= 2.0. I developed this because the official version does not support Elasticsearch >= 2.0 due to API changes. Hope Flume dev team will fix this soon.

Requirements

Flume-NG >= 1.6
Elasticsearch >= 2.0
Build

Build standard jar by the following command

$ ./gradlew build
Build fat jar which contains elasticsearch dependencies

$ ./gradlew assembly
Jar will be generated in build/libs

Usage

Append the built jar to Flume's classpath
remove guava-*.jar and jackson-core-*.jar in flume's default libs dir. They are outdated and newer version are included in Elasticsearch.
set sink type to com.frontier45.flume.sink.elasticsearch2.ElasticSearchSink in flume.conf
start flume agent

4.flume 1を編集する.6.0/conf/es.conf

agent.sources = tail
agent.channels = memoryChannel
agent.channels.memoryChannel.type = memory
agent.sources.tail.channels = memoryChannel
agent.sources.tail.type = exec
agent.sources.tail.command = tail -F /tmp/ttt

# Describe the sink ES
agent.sinks = elasticsearch
agent.sinks.elasticsearch.type = com.frontier45.flume.sink.elasticsearch2.ElasticSearchSink
agent.sinks.elasticsearch.hostNames = 127.0.0.1:9300
agent.sinks.elasticsearch.indexType = logs
agent.sinks.elasticsearch.indexName = flume
agent.sinks.elasticsearch.clusterName = elas
agent.sinks.elasticsearch.batchSize = 500
agent.sinks.elasticsearch.ttl = 5d
agent.sinks.elasticsearch.serializer = com.frontier45.flume.sink.elasticsearch2.ElasticSearchDynamicSerializer
agent.sinks.elasticsearch.indexNameBuilder = com.frontier45.flume.sink.elasticsearch2.TimeBasedIndexNameBuilder
agent.sinks.elasticsearch.channel = memoryChannel

5.スタートes

bin/elasticsearch

6.テストファイルを作成し、flumeを起動する

touch/tmp/ttt echo "test">>/tmp/ttt
bin/flume-ng agent -c . -f conf/es.conf -n agent -Dflume.root.logger=INFO,console