Collecting Log Data from Windows Use Nxlog Fluentd to Elasticsearch

about 2 years ago

Collecting Log Data from Windows Use Nxlog Fluentd to Elasticsearch

因為要收集所有系統log，但公司是用~~萬惡的~~Windows，所以要多繞點路用NXLog才有辦法傳到Fluentd再存到Elasticsearch

Prerequisites 環境準備

Windows 8
Ubuntu 14.01 -> 目前最新就是支援到這版，不要鐵齒不然Fluentd Plugin裝不上去

Installing and Configuring Elasticsearch In Ubuntu

Update Ubuntu

sudo ufw disable
sudo apt-get remove --purge openjdk*
sudo apt-get update
sudo apt-get -y upgrade
sudo apt-get clean
sudo apt-get install ntp ntpdate

Install JDK

sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | sudo /usr/bin/debconf-set-selections
sudo apt-get install oracle-java8-installer

Install Elasticsearch

wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.6.0.deb
sudo dpkg -i elasticsearch-1.6.0.deb
sudo service elasticsearch start

Install Fluentd In Ubuntu

這邊有提供對應的版本安裝指令安裝看我

Ubuntu Trusty 則執行

curl -L https://td-toolbelt.herokuapp.com/sh/install-ubuntu-trusty-td-agent2.sh | sh

Installing Fluentd Plugins

sudo apt-get install make libcurl4-gnutls-dev --yes
sudo /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-elasticsearch
sudo /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-record-reformer

Edit td-agent.conf

sudo vi /etc/td-agent/td-agent.conf

td-agent.conf

<source>
  type tcp
  format json
  port 5140
  tag windowslog
</source>
<match windowslog>
  type elasticsearch
</match>

About fluent-plugin-elasticsearch see here GitHub

Starting Fluentd

sudo service td-agent start
#watch

tail -f /var/log/td-agent/td-agent.log

INstall NXlog in Windows

NXlog下載
安裝很簡單一直下一步就完成了

安裝完成後開啟設定檔 C:\Program Files (x86)\nxlog\conf\nxlog.conf

nxlog.conf

define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

#選了兩個模組載入備用
<Extension json>
 Module xm_json
</Extension>
 
<Extension syslog>
 Module xm_syslog
</Extension>

#設定資料來源
<Input in>
  Module im_msvistalog
#  ReadFromLast FALSE
#  SavePos     FALSE
#  Exec to_syslog_bsd();
#  Exec to_syslog_ietf();
# 最後是採用to_json()的資料格式，因為ES本身就是存Json
  Exec to_json(); 
</Input>

#這邊設定輸出，到時候Fluentd預設是聽5140
<Output out>
  Module om_tcp
  Host 192.168.56.103
  Port 5140
</Output> 
#這邊輸出到檔案是我為了方便觀察使用
<Output outfile>
  Module om_file
  File   "D:\logs\log"
</Output>

#設定你的資料流
<Route r1>
  Path in => out
</Route>
<Route r2>
  Path in => outfile
</Route>

測試可以使用指令叫起

cd C:\Program Files (x86)\nxlog
nxlog.exe -f -c ./conf/nxlog.conf

如果測試都OK可以到服務中設定隨機啟動

完成後可以在ES看到Windows的系統log啦

Windows全部事件每秒可多達700筆，會影響到效能，這邊有找到一組設定，此設定只輸出主機稽核、物件存取、帳
戶管理等 eventlog，過濾大部分雜訊，對系統影響較小

<Input in_eventlog>
# For windows 2008/vista/7/8/2012/2012R2 and latter use the following:
  Module      im_msvistalog
  ReadFromLast TRUE
  SavePos     TRUE
  Query     <QueryList> \
    <Query Id="0"> \
      <Select Path="Security">*[System[(EventID=4768)]]</Select> \
      <Select Path="Security">*[System[(EventID=4769)]]</Select> \
      <Select Path="Security">*[System[(EventID=4771)]]</Select> \
      <Select Path="Security">*[System[(EventID=4624)]]</Select> \
      <Select Path="Security">*[System[(EventID=4625)]]</Select> \
      <Select Path="Security">*[System[(EventID=4634)]]</Select> \
      <Select Path="Security">*[System[(EventID=4647)]]</Select> \
      <Select Path="Security">*[System[(EventID=4648)]]</Select> \
      <Select Path="Security">*[System[(EventID=4656)]]</Select> \
      <Select Path="Security">*[System[(EventID=4719)]]</Select> \
      <Select Path="Security">*[System[(EventID=4720)]]</Select> \
      <Select Path="Security">*[System[(EventID=4722)]]</Select> \
      <Select Path="Security">*[System[(EventID=4723)]]</Select> \
      <Select Path="Security">*[System[(EventID=4724)]]</Select> \
      <Select Path="Security">*[System[(EventID=4725)]]</Select> \
      <Select Path="Security">*[System[(EventID=4726)]]</Select> \
      <Select Path="Security">*[System[(EventID=4727)]]</Select> \
      <Select Path="Security">*[System[(EventID=4728)]]</Select> \
      <Select Path="Security">*[System[(EventID=4729)]]</Select> \
      <Select Path="Security">*[System[(EventID=4730)]]</Select> \
      <Select Path="Security">*[System[(EventID=4731)]]</Select> \
      <Select Path="Security">*[System[(EventID=4732)]]</Select> \
      <Select Path="Security">*[System[(EventID=4733)]]</Select> \
      <Select Path="Security">*[System[(EventID=4734)]]</Select> \
      <Select Path="Security">*[System[(EventID=4735)]]</Select> \
      <Select Path="Security">*[System[(EventID=4737)]]</Select> \
      <Select Path="Security">*[System[(EventID=4738)]]</Select> \
      <Select Path="Security">*[System[(EventID=4739)]]</Select> \
      <Select Path="Security">*[System[(EventID=4741)]]</Select> \
      <Select Path="Security">*[System[(EventID=4742)]]</Select> \
      <Select Path="Security">*[System[(EventID=4743)]]</Select> \
      <Select Path="System">*[System[(EventID=7036)]]</Select> \
      <Select Path="Application">*[System[(EventID=18454)]]</Select> \
      <Select Path="Application">*[System[(EventID=18456)]]</Select> \
    </Query> \
  </QueryList> 
</Input>

補充一下如果你要收集Windows上Tomcat的log可以使用logstash
設定

agent.conf

input {
    file {
        path => ["D:/SimulationServer/apache-tomcat-8.0.21/logs/*.log"]
    }
}

filter {
}

output {
    elasticsearch {
        host => "192.168.56.103"
        port => "9200"
        protocol => "http"
    }
}

執行方式

SET PATH=D:\Logserver\logstash-1.5.2\bin;%PATH%
SET JAVA_HOME=C:\Program Files\Java\jre8
logstash agent -f agent.conf

July 02, 2015 18:42
Permalink
Comments