Elasticsearchシリーズ-logstashインポートデータ
7916 ワード
Elasticsearchシリーズ-logstashインポートデータ
インストールの準備
コンフィギュレーション
logstashファイルを解凍しconfigディレクトリの下に入ると、logstash-sampleが表示されます.confファイル、コピーしてlogstash-moviesと名付けます.conf.
プロファイルは自分で勝手に名前を付けることができます.自分で正しく見つけて認識すればいいです.
ファイルの編集:# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.
#
input {
file {
#
path => "D:/Tool/ml-25m/movies.csv"
start_position => "beginning"
}
}
#
filter {
# csv
csv {
#
separator => ","
# csv
columns => ["id","content","genre"]
}
# genre
mutate {
split => {
"genre" => "|" }
remove_field => ["path", "host","@timestamp","message"]
}
# content
mutate {
split => ["content", "("]
# title
add_field => {
"title" => "%{[content][0]}"}
# year
add_field => {
"year" => "%{[content][1]}"}
}
mutate {
# year
convert => {
"year" => "integer"
}
strip => ["title"]
#
remove_field => ["path", "host","@timestamp","message","content"]
}
}
#
output {
# es
elasticsearch {
hosts => "http://localhost:9200"
index => "movies"
document_id => "%{id}"
#user => "elastic"
#password => "changeme"
}
stdout {
}
}
binディレクトリに入り、logstashを起動します:-fパラメータでプロファイルを指定して起動します.\logstash.bat -f D:\Tool\logstash-7.12.0-windows-x86_64\logstash-7.12.0\config\logstash-movies.conf
私がここで使っているのはローカルテストで、Windowsシステムを使っているので、使います.bat実行.
に質問
# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.
#
input {
file {
#
path => "D:/Tool/ml-25m/movies.csv"
start_position => "beginning"
}
}
#
filter {
# csv
csv {
#
separator => ","
# csv
columns => ["id","content","genre"]
}
# genre
mutate {
split => {
"genre" => "|" }
remove_field => ["path", "host","@timestamp","message"]
}
# content
mutate {
split => ["content", "("]
# title
add_field => {
"title" => "%{[content][0]}"}
# year
add_field => {
"year" => "%{[content][1]}"}
}
mutate {
# year
convert => {
"year" => "integer"
}
strip => ["title"]
#
remove_field => ["path", "host","@timestamp","message","content"]
}
}
#
output {
# es
elasticsearch {
hosts => "http://localhost:9200"
index => "movies"
document_id => "%{id}"
#user => "elastic"
#password => "changeme"
}
stdout {
}
}
.\logstash.bat -f D:\Tool\logstash-7.12.0-windows-x86_64\logstash-7.12.0\config\logstash-movies.conf
input {
file {
# "/" "\" ，
# path => "D:\Tool\ml-25m\movies.csv"
path => "D:/Tool/ml-25m/movies.csv"
start_position => "beginning"
}
}