over 7 years ago
因為要收集所有系統log,但公司是用萬惡的Windows,所以要多繞點路用NXLog才有辦法傳到Fluentd再存到Elasticsearch
Prerequisites 環境準備
Windows 8
Ubuntu 14.01 -> 目前最新就是支援到這版,不要鐵齒不然Fluentd Plugin裝不上去
Installing and Configuring Elasticsearch In Ubuntu
sudo ufw disable
sudo apt-get remove --purge openjdk*
sudo apt-get update
sudo apt-get -y upgrade
sudo apt-get clean
sudo apt-get install ntp ntpdate
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | sudo /usr/bin/debconf-set-selections
sudo apt-get install oracle-java8-installer
wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.6.0.deb
sudo dpkg -i elasticsearch-1.6.0.deb
sudo service elasticsearch start
Install Fluentd In Ubuntu
這邊有提供對應的版本安裝指令安裝看我
Ubuntu Trusty 則執行
curl -L https://td-toolbelt.herokuapp.com/sh/install-ubuntu-trusty-td-agent2.sh | sh
Installing Fluentd Plugins
sudo apt-get install make libcurl4-gnutls-dev --yes
sudo /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-elasticsearch
sudo /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-record-reformer
Edit td-agent.conf
sudo vi /etc/td-agent/td-agent.conf
<source>
type tcp
format json
port 5140
tag windowslog
</source>
<match windowslog>
type elasticsearch
</match>
About fluent-plugin-elasticsearch see here GitHub
Starting Fluentd
sudo service td-agent start
#watch
tail -f /var/log/td-agent/td-agent.log
INstall NXlog in Windows
NXlog下載
安裝很簡單一直下一步就完成了
安裝完成後開啟設定檔 C:\Program Files (x86)\nxlog\conf\nxlog.conf
define ROOT C:\Program Files (x86)\nxlog
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
#選了兩個模組載入備用
<Extension json>
Module xm_json
</Extension>
<Extension syslog>
Module xm_syslog
</Extension>
#設定資料來源
<Input in>
Module im_msvistalog
# ReadFromLast FALSE
# SavePos FALSE
# Exec to_syslog_bsd();
# Exec to_syslog_ietf();
# 最後是採用to_json()的資料格式,因為ES本身就是存Json
Exec to_json();
</Input>
#這邊設定輸出,到時候Fluentd預設是聽5140
<Output out>
Module om_tcp
Host 192.168.56.103
Port 5140
</Output>
#這邊輸出到檔案是我為了方便觀察使用
<Output outfile>
Module om_file
File "D:\logs\log"
</Output>
#設定你的資料流
<Route r1>
Path in => out
</Route>
<Route r2>
Path in => outfile
</Route>
測試可以使用指令叫起
cd C:\Program Files (x86)\nxlog
nxlog.exe -f -c ./conf/nxlog.conf
如果測試都OK可以到服務中設定隨機啟動
完成後可以在ES看到Windows的系統log啦
Windows全部事件每秒可多達700筆,會影響到效能,這邊有找到一組設定,此設定只輸出主機稽核、物件存取、帳
戶管理等 eventlog,過濾大部分雜訊,對系統影響較小
<Input in_eventlog>
# For windows 2008/vista/7/8/2012/2012R2 and latter use the following:
Module im_msvistalog
ReadFromLast TRUE
SavePos TRUE
Query <QueryList> \
<Query Id="0"> \
<Select Path="Security">*[System[(EventID=4768)]]</Select> \
<Select Path="Security">*[System[(EventID=4769)]]</Select> \
<Select Path="Security">*[System[(EventID=4771)]]</Select> \
<Select Path="Security">*[System[(EventID=4624)]]</Select> \
<Select Path="Security">*[System[(EventID=4625)]]</Select> \
<Select Path="Security">*[System[(EventID=4634)]]</Select> \
<Select Path="Security">*[System[(EventID=4647)]]</Select> \
<Select Path="Security">*[System[(EventID=4648)]]</Select> \
<Select Path="Security">*[System[(EventID=4656)]]</Select> \
<Select Path="Security">*[System[(EventID=4719)]]</Select> \
<Select Path="Security">*[System[(EventID=4720)]]</Select> \
<Select Path="Security">*[System[(EventID=4722)]]</Select> \
<Select Path="Security">*[System[(EventID=4723)]]</Select> \
<Select Path="Security">*[System[(EventID=4724)]]</Select> \
<Select Path="Security">*[System[(EventID=4725)]]</Select> \
<Select Path="Security">*[System[(EventID=4726)]]</Select> \
<Select Path="Security">*[System[(EventID=4727)]]</Select> \
<Select Path="Security">*[System[(EventID=4728)]]</Select> \
<Select Path="Security">*[System[(EventID=4729)]]</Select> \
<Select Path="Security">*[System[(EventID=4730)]]</Select> \
<Select Path="Security">*[System[(EventID=4731)]]</Select> \
<Select Path="Security">*[System[(EventID=4732)]]</Select> \
<Select Path="Security">*[System[(EventID=4733)]]</Select> \
<Select Path="Security">*[System[(EventID=4734)]]</Select> \
<Select Path="Security">*[System[(EventID=4735)]]</Select> \
<Select Path="Security">*[System[(EventID=4737)]]</Select> \
<Select Path="Security">*[System[(EventID=4738)]]</Select> \
<Select Path="Security">*[System[(EventID=4739)]]</Select> \
<Select Path="Security">*[System[(EventID=4741)]]</Select> \
<Select Path="Security">*[System[(EventID=4742)]]</Select> \
<Select Path="Security">*[System[(EventID=4743)]]</Select> \
<Select Path="System">*[System[(EventID=7036)]]</Select> \
<Select Path="Application">*[System[(EventID=18454)]]</Select> \
<Select Path="Application">*[System[(EventID=18456)]]</Select> \
</Query> \
</QueryList>
</Input>
補充一下如果你要收集Windows上Tomcat的log可以使用logstash
設定
input {
file {
path => ["D:/SimulationServer/apache-tomcat-8.0.21/logs/*.log"]
}
}
filter {
}
output {
elasticsearch {
host => "192.168.56.103"
port => "9200"
protocol => "http"
}
}
執行方式
SET PATH=D:\Logserver\logstash-1.5.2\bin;%PATH%
SET JAVA_HOME=C:\Program Files\Java\jre8
logstash agent -f agent.conf