fluentd理解
概要
さくらのナレッジの説明が一番わかりやすかったです。
http://knowledge.sakura.ad.jp/tech/1336/
環境
- CentOS 6.7
- fluentd 0.12.20
Install+設定
こんな感じでansibleでinstallしました。
webserverに立てたapachのlogをlogserverで受けています。
logを送る側もlogを受ける側もどちらもtd-agentという同じserviceが動いています。
webserverが複数台になった時にlogserverにlogが集約されるので、あとはElastic searchを何か入れれば良いと思います。
- td-agentというuserで実行しようとしてpermission deniedが発生していたので、rootで実行するように設定fileを書き換えている。
ansible-galaxyから次のroleを使わせていただきました。
- geerlingguy.apache
- williamyeh.fluentd
# webserver playbook
- name: configure the web server
hosts: webserver
vars:
- apache_listen_port: 10443
roles:
- geerlingguy.apache
tasks:
- name: copy init files
copy: src=roles/tomcat8/files/httpd-proxy.conf dest=/etc/httpd/conf/httpd-proxy.conf owner=root group=wheel mode=0644
- name: deploy setting file
lineinfile: dest=/etc/httpd/conf/httpd.conf line='Include /etc/httpd/conf/httpd-proxy.conf'
- name: restart service and auto startup setting
service: name=httpd state=restarted
- name: configure fluentd
hosts: webserver
vars:
- tdagent_conf_template: "roles/td-agent/templates/web-td-agent.conf.j2"
- aggregator_ip: "{{ logserver.ip }}"
- tdagent_port: "{{ logserver.tdagent_port }}"
roles:
- williamyeh.fluentd
tasks:
- name: change start user to root
lineinfile: >
dest=/etc/init.d/td-agent
regexp='TD_AGENT_USER'
line='TD_AGENT_USER=root'
notify: td-agent restart
- name: change start group to root
lineinfile: >
dest=/etc/init.d/td-agent
regexp='TD_AGENT_GROUP'
line='TD_AGENT_GROUP=root'
notify: td-agent restart
handlers:
- name: td-agent restart
service: name=td-agent state=restarted
# logserver playbook
- name: configure fluentd
hosts: logserver
vars:
- tdagent_conf_template: "roles/td-agent/templates/log-td-agent.conf.j2"
- tdagent_port: 24224
roles:
- williamyeh.fluentd
####
## Source descriptions:
##
#
# Apache Log
#
## access
<source>
type tail
path /var/log/httpd/access_log
tag apache.access
pos_file /var/log/td-agent/httpd-access_log.pos
format apache2
</source>
## error
<source>
type tail
path /var/log/httpd/error_log
tag apache.error
pos_file /var/log/td-agent/httpd-error_log.pos
format apache_error
</source>
####
## Output descriptions:
##
#
# Apache Log
#
<match apache.**>
type forward
<server>
host {{ aggregator_ip }}
port {{ tdagent_port }}
</server>
</match>
####
## Source descriptions:
##
#
# Recieve Logs
#
<source>
type forward
port {{ tdagent_port }}
</source>
####
## Output descriptions:
##
<match apache.access>
type file
path /var/log/td-agent/httpd/access.log
time_slice_format %Y%m%d
time_slice-wait 10m
compress gzip
</match>
<match apache.error>
type file
path /var/log/td-agent/httpd/error.log
time_slice_format %Y%m%d
time_slice-wait 10m
compress gzip
</match>
構成はこんな感じ
結果
webserverのapacheのlog
192.168.1.1 - - [07/Jul/2016:19:26:31 +0900] "GET / HTTP/1.1" 403 4961 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"
192.168.1.1 - - [07/Jul/2016:19:26:40 +0900] "GET / HTTP/1.1" 403 4961 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"
192.168.2.3 - - [07/Jul/2016:19:55:28 +0900] "GET / HTTP/1.1" 403 4961 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"
192.168.2.3 - - [07/Jul/2016:19:55:28 +0900] "GET /icons/apache_pb.gif HTTP/1.1" 304 - "http://164.70.6.213:10443/" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"
logserverに送られたlog
2016-07-07T19:26:31+09:00 apache.access {"host":"192.168.1.1","user":null,"method":"GET","path":"/","code":403,"size":4961,"referer":null,"agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"}
2016-07-07T19:26:40+09:00 apache.access {"host":"192.168.1.1","user":null,"method":"GET","path":"/","code":403,"size":4961,"referer":null,"agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"}
2016-07-07T19:55:28+09:00 apache.access {"host":"192.168.2.3","user":null,"method":"GET","path":"/","code":403,"size":4961,"referer":null,"agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"}
2016-07-07T19:55:28+09:00 apache.access {"host":"192.168.2.3","user":null,"method":"GET","path":"/icons/apache_pb.gif","code":304,"size":null,"referer":"http://164.70.6.213:10443/","agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"}
便利ツール
- logの正規表現を書く時に役立ちそうなサイト
Author And Source
この問題について(fluentd理解), 我々は、より多くの情報をここで見つけました https://qiita.com/clutter/items/49880a5aaa68ce977326著者帰属:元の著者の情報は、元のURLに含まれています。著作権は原作者に属する。
Content is automatically searched and collected through network algorithms . If there is a violation . Please contact us . We will adjust (correct author information ,or delete content ) as soon as possible .