flumeレプリケーションとスプリットの実装

今日、お客様からのお问い合わせを受けました.既存のデータを分流しなければなりません.flumeが実现しても大丈夫だと思いますが、心が落ち着かないので、何かあったら颜を出して、それから小さなテストをしました.テストの结果も私の考えを検証しました.くだらないことは言わないで、次のように直接配置します.

# ：flume-test.conf
# list the sources, sinks and channels in the agent
a1.sources= source_1
a1.sinks= fs_sink_local_file_1 fs_sink_local_file_2
a1.channels= fs_channel_memory_1 fs_channel_memory_2

# source from  spooldir
a1.sources.source_1.type = spooldir
a1.sources.source_1.spoolDir = /home/users/flume/data
a1.sources.source_1.fileHeader = true


#channel for source configure
a1.sources.source_1.channels= fs_channel_memory_1 fs_channel_memory_2

##sinks
#sink1
a1.sinks.fs_sink_local_file_1.channel = fs_channel_memory_1
a1.sinks.fs_sink_local_file_1.type = file_roll
a1.sinks.fs_sink_local_file_1.sink.directory = /home/users/flume/dataA
#sink2
a1.sinks.fs_sink_local_file_2.channel = fs_channel_memory_2
a1.sinks.fs_sink_local_file_2.type = file_roll
a1.sinks.fs_sink_local_file_2.sink.directory = /home/users/flume/dataB

###channels
#channels1
a1.channels.fs_channel_memory_1.type = memory
a1.channels.fs_channel_memory_1.capacity = 5000000
a1.channels.fs_channel_memory_1.transactionCapacity = 5000000
a1.channels.fs_channel_memory_1.byteCapacityBufferPercentage = 40
a1.channels.fs_channel_memory_1.byteCapacity = 8000000000
a1.channels.fs_channel_memory_1.keep-alive = 30

#channels2
a1.channels.fs_channel_memory_2.type = memory
a1.channels.fs_channel_memory_2.capacity = 5000000
a1.channels.fs_channel_memory_2.transactionCapacity = 5000000
a1.channels.fs_channel_memory_2.byteCapacityBufferPercentage = 40
a1.channels.fs_channel_memory_2.byteCapacity = 8000000000
a1.channels.fs_channel_memory_2.keep-alive = 30

そのまま貼れば走れるような配置で、起動命令は以下の通りです.

flume-ng agent -n a1 -f /path/to/conf/flume-test.conf -Xmx100M

以上のflume分流レプリケーションを実現する機能は、/home/users/flume/dataディレクトリの下のファイルを/home/users/flume/dataaと/home/users/flume/dataBにそれぞれコピーする機能です.