????????〇、抓包與批量轉(zhuǎn)換cap文件
? ? ? ?1. Network Monitor 抓包
? ? ? ? 我們?cè)贑ENTO OS上的網(wǎng)絡(luò)安全工具（十七）搭建Cascade的Docker開(kāi)發(fā)環(huán)境中捎帶腳介紹了以下windows下的抓包軟件。大意就是微軟又一款不錯(cuò)的抓包分析軟件，名曰nmcap，可在Download Microsoft Network Monitor 3.4 (archive) from Official Microsoft Download Center下載
??????????而且，這個(gè)軟件有個(gè)不錯(cuò)的不間斷抓包功能，使用如下命令可對(duì)所有網(wǎng)卡（如果對(duì)指定網(wǎng)卡，可以先用displaynetwork命令查出網(wǎng)卡序號(hào)進(jìn)行指定）抓包，并根據(jù)指定大小將文件編號(hào)存儲(chǔ)在給定的路徑下：
PS C:\Users\lhyzw> nmcap /DisplayNetwork
Network Monitor Command Line Capture (nmcap) 3.4.2350.0
0. vEthernet (WSL) (Hyper-V Virtual Ethernet Adapter)
1. WLAN (MediaTek Wi-Fi 6 MT7921 Wireless LAN Card)
2. * 3 (Microsoft Wi-Fi Direct Virtual Adapter #3)
3. * 4 (Microsoft Wi-Fi Direct Virtual Adapter #4)
4. ?(Realtek PCIe GbE Family Controller)
PS C:\Users\lhyzw> nmcap /Network * /Capture /File d:\downlaod\test.chn:2MB
Network Monitor Command Line Capture (nmcap) 3.4.2350.0
Saving info to: d:\downlaod\test.cap - using chain captures of size 2.00 MB.
ATTENTION: Conversations Disabled: Some filters require conversations and will not work correctly (see Help for details)
ATTENTION: Process Tracking Disabled: Use /CaptureProcesses to enable (see Help for details)
Note: Process Filtering Disabled.
Exit by Ctrl+C
Capturing ? | Received: 16165 Pending: 0 Saved: 16165 Dropped: 0 | Time: 57 seconds.? ? ? ? 2. 批量轉(zhuǎn)換cap文件
? ? ? ? 不幸的是network monitor抓取的文件采用nmcap格式存放，yaf并不支持這個(gè)格式。好在windows下，還有一個(gè)著名的wireshark軟件可以完成這個(gè)轉(zhuǎn)換工作。
? ? ? ? wireshark軟件的安裝目錄下自帶名為editcap.exe的文件，可以用來(lái)轉(zhuǎn)換：
C:\Program Files\Wireshark>editcap -F pcap -T ether g:\pcap\testmine(2).cap g:\t\testmine(2).pcap? ? ? ? 其中，-F標(biāo)識(shí)轉(zhuǎn)換后的格式，-T標(biāo)識(shí)轉(zhuǎn)換前的格式。使用-F或者-T攜帶空的參數(shù)，可以打印出editcap所支持的所有格式。然后指定輸入文件和輸出文件，即可以實(shí)現(xiàn)轉(zhuǎn)換。
? ? ? ? 不幸在于editcap僅支持一分多、多合一和一對(duì)一，我并沒(méi)有找到可批量轉(zhuǎn)換的參數(shù)。最后逼不得已只能上批處理了。在cap文件所在目錄下vim（因?yàn)榍懊鏋榱司幾ghadoop我裝了個(gè)gitbash嘛）一個(gè)test.bat文件如下：????????
for /r . %%i in (*.cap) do C:\Progra~1\Wireshark\editcap -F pcap -T ether %%i g:\t\%%~ni.pcap? ? ? ? 就一行，含義是對(duì)當(dāng)前目錄下的所有后綴為cap的文件，使用editcap（注意絕對(duì)路徑，因?yàn)椴](méi)有將其加入PATH環(huán)境變量中）進(jìn)行轉(zhuǎn)換，轉(zhuǎn)換后文件名，使用%~ni，即提取%i變量所對(duì)應(yīng)的文件名進(jìn)行構(gòu)造。
? ? ? ? 然后執(zhí)行test.bat就可以實(shí)現(xiàn)批量轉(zhuǎn)換了：
G:\pcap>test.bat
G:\pcap>for /R . %i in (*.cap) do C:\Progra~1\Wireshark\editcap -F pcap -T ether %i g:\t\%~ni.pcap
G:\pcap>C:\Progra~1\Wireshark\editcap -F pcap -T ether G:\pcap\testmine(103).cap g:\t\testmine(103).pcap
G:\pcap>C:\Progra~1\Wireshark\editcap -F pcap -T ether G:\pcap\testmine(104).cap g:\t\testmine(104).pcap
G:\pcap>C:\Progra~1\Wireshark\editcap -F pcap -T ether G:\pcap\testmine(105).cap g:\t\testmine(105).pcap
G:\pcap>C:\Progra~1\Wireshark\editcap -F pcap -T ether G:\pcap\testmine(106).cap g:\t\testmine(106).pcap
G:\pcap>C:\Progra~1\Wireshark\editcap -F pcap -T ether G:\pcap\testmine(107).cap g:\t\testmine(107).pcap
………………
…………
……? ? ? ? 一、編譯YAF3
? ? ? ? 首先聲明，為了偷懶，并沒(méi)有在centos7環(huán)境下做下面的事情。所以下面的編譯操作是在centos stream 8的環(huán)境下進(jìn)行的。
????????1. 編譯環(huán)境安裝
????????按照官網(wǎng)的指南，首先安裝編譯環(huán)境
[root@bogon share]# yum install gcc gcc-c++ make pkgconfig -y
????????或者直接使用群組安裝方式，將所有的開(kāi)發(fā)工具安裝上去，大約需下載170MB左右
[root@bogon share]# yum -y group install "Development Tools"
上次元數(shù)據(jù)過(guò)期檢查：0:01:24 前，執(zhí)行于 2023年06月24日 星期六 20時(shí)32分35秒。
依賴關(guān)系解決。
===================================================================================================================================================================================================================
軟件包 架構(gòu) 版本 倉(cāng)庫(kù) 大小
===================================================================================================================================================================================================================
升級(jí):
binutils x86_64 2.30-121.el8 baseos 5.9 M
elfutils-libelf x86_64 0.189-2.el8 baseos 232 k
elfutils-libs x86_64 0.189-2.el8 baseos 302 k
glibc x86_64 2.28-228.el8 baseos 2.2 M
……………………
…………
kernel-devel x86_64 4.18.0-497.el8 baseos 27 M
openssl-devel x86_64 1:1.1.1k-6.el8 baseos 2.3 M
啟用模塊流:
javapackages-runtime 201801
安裝組:
Development Tools
事務(wù)概要
===================================================================================================================================================================================================================
安裝 97 軟件包
升級(jí) 19 軟件包
????????2. 抓包依賴庫(kù)安裝
????????libpcap依賴庫(kù)很容易安裝，在centos stream 8下，只需要yum install就行了
[root@bogon share]# yum install libpcap -y
????????但是libpcap-devel庫(kù)在centos stream 8下沒(méi)有，不安裝的話會(huì)在編譯時(shí)產(chǎn)生找不到pcap.h頭文件的錯(cuò)誤。因此需要手工下載安裝。
????????直接到pkgs.org網(wǎng)站https://pkgs.org/download/libpcap-devel上下載：
????????或者啟動(dòng)Powertools庫(kù)后安裝：
[root@bogon share]# dnf config-manager --set-enabled powertools
? ? ? ? 或者直接在參數(shù)中指定powertools庫(kù)。需要注意的是，按照官方的說(shuō)法，這個(gè)庫(kù)在某些版本的Centos中叫做“PowerTools”……?
[root@bogon share]# dnf --enablerepo=powertools install libpcap-devel
CentOS Stream 8 - PowerTools 3.7 MB/s | 6.0 MB 00:01
上次元數(shù)據(jù)過(guò)期檢查：0:00:02 前，執(zhí)行于 2023年06月24日 星期六 20時(shí)47分46秒。
依賴關(guān)系解決。
===================================================================================================================================================================================================================
軟件包 架構(gòu) 版本 倉(cāng)庫(kù) 大小
===================================================================================================================================================================================================================
安裝:
libpcap-devel x86_64 14:1.9.1-5.el8 powertools 144 k
事務(wù)概要
===================================================================================================================================================================================================================
安裝 1 軟件包
總下載：144 k
安裝大?。?27 k
確定嗎？[y/N]： y
下載軟件包：
libpcap-devel-1.9.1-5.el8.x86_64.rpm 534 kB/s | 144 kB 00:00
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
總計(jì) 164 kB/s | 144 kB 00:00
運(yùn)行事務(wù)檢查
事務(wù)檢查成功。
運(yùn)行事務(wù)測(cè)試
事務(wù)測(cè)試成功。
運(yùn)行事務(wù)
準(zhǔn)備中 : 1/1
安裝 : libpcap-devel-14:1.9.1-5.el8.x86_64 1/1
運(yùn)行腳本: libpcap-devel-14:1.9.1-5.el8.x86_64 1/1
驗(yàn)證 : libpcap-devel-14:1.9.1-5.el8.x86_64 1/1
已安裝:
libpcap-devel-14:1.9.1-5.el8.x86_64
完畢！
[root@bogon share]#
????????3. 其它依賴庫(kù)安裝
? ? ? ? （1）GLib-2.0
? ? ? ? 按照官方的說(shuō)法，GLib實(shí)際被包含在了大多數(shù)操作系統(tǒng)的基礎(chǔ)環(huán)境中，我們只需要檢查一下安裝了沒(méi)有：
[root@bogon share]# rpm -qa|grep glib
avahi-glib-0.7-20.el8.x86_64
ModemManager-glib-1.18.2-1.el8.x86_64
glibmm24-2.56.0-2.el8.x86_64
geocode-glib-3.26.0-3.el8.x86_64
spice-glib-0.38-6.el8.x86_64
glibc-langpack-zh-2.28-228.el8.x86_64
pulseaudio-libs-glib2-14.0-2.el8.x86_64
libappstream-glib-0.7.14-3.el8.x86_64
glib2-2.56.4-158.el8.x86_64
json-glib-1.4.4-1.el8.x86_64
glibc-common-2.28-228.el8.x86_64
glibc-2.28-228.el8.x86_64
glibc-devel-2.28-228.el8.x86_64
PackageKit-glib-1.1.12-6.el8.x86_64
dbus-glib-0.110-2.el8.x86_64
taglib-1.11.1-8.el8.x86_64
glib-networking-2.56.1-1.1.el8.x86_64
glibc-gconv-extra-2.28-228.el8.x86_64
libvirt-glib-3.0.0-1.el8.x86_64
glibc-headers-2.28-228.el8.x86_64
glibc-langpack-en-2.28-228.el8.x86_64
poppler-glib-20.11.0-4.el8.x86_64
glibc-all-langpacks-2.28-228.el8.x86_64
????????貌似是都裝了，但是一個(gè)大坑在這：
[root@bogon libfixbuf-3.0.0.alpha2]# yum install glib2-devel -y
?????????如果沒(méi)有裝這個(gè)開(kāi)發(fā)版本的話，執(zhí)行./configure時(shí)仍然會(huì)報(bào)錯(cuò)，而且只告訴你你的glib版本不高于2.18……，所以找起來(lái)那是相當(dāng)生氣。
????????（2）libfixbuf
????????YAF3對(duì)應(yīng)3.0版本的libfixbuf，這個(gè)需要我們預(yù)先編譯號(hào)。源代碼包在此處fixbuf — Latest Downloads (cert.org)下載
? ? ? ? ?1）準(zhǔn)備源代碼
[root@bogon ~]# tar zxvf libfixbuf-3.0.0.alpha2.tar.gz
[root@bogon ~]# cd libfixbuf-3.0.0.alpha2/
[root@bogon libfixbuf-3.0.0.alpha2]#
? ? ? ? 2）安裝依賴
? ? ? ? 除了glib2和glib2-devel以外，如果需要加入openssl支持（--with-openssl），則系統(tǒng)也需要安裝openssl和openssl-devel。
? ? ? ? 如果需要SCTP支持（--with-sctp），也需要實(shí)現(xiàn)使內(nèi)核支持SCTP協(xié)議。
? ? ? ? 為了確保我們編譯的庫(kù)最大可能的兼容性，此處我們還是能減則減了。
????????3）配置libfixbuf
? ? ? ? 由于僅需要更新libfixbuf庫(kù)，所以使用了--disable-tools參數(shù)
[root@bogon libfixbuf-3.0.0.alpha2]# ./configure --disable-tools
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
………………
…………
……
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating src/Makefile
config.status: creating src/infomodel/Makefile
config.status: creating include/Makefile
config.status: creating include/fixbuf/version.h
config.status: creating libfixbuf.pc
config.status: creating libfixbuf.spec
config.status: creating Doxyfile
config.status: creating include/fixbuf/config.h
config.status: executing depfiles commands
config.status: executing libtool commands
config.status: executing print-config commands
* Configured package: libfixbuf 3.0.0.alpha2
* Host type: x86_64-pc-linux-gnu
* Source files ($top_srcdir): .
* Install directory: NONE
* Build command-line tools: NO
* pkg-config path:
* GLIB: -lgthread-2.0 -pthread -lglib-2.0
* OpenSSL Support: YES
* DTLS Support: YES
* SCTP Support: NO
* Compiler (CC): gcc
* Compiler flags (CFLAGS): -I. -I$(top_srcdir)/include -Wall -Wextra -Wshadow -Wpointer-arith -Wformat=2 -Wunused -Wundef -Wduplicated-cond -Wwrite-strings -Wmissing-prototypes -Wstrict-prototypes -DNDEBUG -DG_DISABLE_ASSERT -g -O2
* Linker flags (LDFLAGS):
* Libraries (LIBS): -lssl -lcrypto -lpthread
? ? ? ? 4）make && make install libfixbuf
? ? ? ? 由于我們沒(méi)有在configure時(shí)指定--prefix和--exe-prefix參數(shù)，所以使用了默認(rèn)目錄安裝，后期可能會(huì)設(shè)計(jì)更改LIBDIR環(huán)境變量。
[root@bogon libfixbuf-3.0.0.alpha2]# make && make install
Making all in src
make[1]: 進(jìn)入目錄“/root/libfixbuf-3.0.0.alpha2/src”
srcdir='' ; test -f ./make-infomodel || srcdir=./ ; /usr/bin/perl "${srcdir}make-infomodel" --package libfixbuf cert ipfix netflowv9 || { rm -f infomodel.c infomodel.h ; exit 1 ; }
make all-recursive
………………
…………
……
/usr/bin/mkdir -p '/usr/local/lib'
/bin/sh ../libtool --mode=install /usr/bin/install -c libfixbuf.la '/usr/local/lib'
libtool: install: /usr/bin/install -c .libs/libfixbuf.so.10.0.0 /usr/local/lib/libfixbuf.so.10.0.0
libtool: install: (cd /usr/local/lib && { ln -s -f libfixbuf.so.10.0.0 libfixbuf.so.10 || { rm -f libfixbuf.so.10 && ln -s libfixbuf.so.10.0.0 libfixbuf.so.10; }; })
libtool: install: (cd /usr/local/lib && { ln -s -f libfixbuf.so.10.0.0 libfixbuf.so || { rm -f libfixbuf.so && ln -s libfixbuf.so.10.0.0 libfixbuf.so; }; })
libtool: install: /usr/bin/install -c .libs/libfixbuf.lai /usr/local/lib/libfixbuf.la
libtool: install: /usr/bin/install -c .libs/libfixbuf.a /usr/local/lib/libfixbuf.a
libtool: install: chmod 644 /usr/local/lib/libfixbuf.a
libtool: install: ranlib /usr/local/lib/libfixbuf.a
libtool: finish: PATH="/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/root/bin:/sbin" ldconfig -n /usr/local/lib
----------------------------------------------------------------------
Libraries have been installed in:
/usr/local/lib
If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the '-LLIBDIR'
flag during linking and do at least one of the following:
- add LIBDIR to the 'LD_LIBRARY_PATH' environment variable
during execution
- add LIBDIR to the 'LD_RUN_PATH' environment variable
during linking
- use the '-Wl,-rpath -Wl,LIBDIR' linker flag
- have your system administrator add LIBDIR to '/etc/ld.so.conf'
See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------
……………………
…………
[root@bogon libfixbuf-3.0.0.alpha2]#
????????4. 編譯安裝
? ? ? ? （1）準(zhǔn)備YAF3源碼
? ? ? ? 下載并解壓源碼：
? ? ? ? ?（2）configure YAF3
[root@bogon ~]# cd yaf-3.0.0.alpha2/
[root@bogon yaf-3.0.0.alpha2]# ls
acinclude.m4 airframe autoconf configure.ac etc infomodel libyaf.pc.in lua Makefile.am make-infomodel README scripts xml2fixbuf.xslt
aclocal.m4 AUTHORS configure doc include libltdl LICENSE.txt m4 Makefile.in NEWS README.in src yaf.spec.in
[root@bogon yaf-3.0.0.alpha2]# ./configure --enable-plugins --enable-applabel --enable-dpi --enable-entropy --enable-zlib
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
………………
…………
……
config.status: executing depfiles commands
config.status: executing libtool commands
config.status: executing yaf_summary commands
* Configured package: yaf 3.0.0.alpha2
* pkg-config path: /usr/local/lib/pkgconfig
* Host type: x86_64-pc-linux-gnu
* OS: linux-gnu
* Source files ($top_srcdir): .
* Install directory: /usr/local
* GLIB: -lglib-2.0
* Timezone support: UTC
* Libfixbuf version: 3.0.0.alpha2
* DAG support: NO
* NAPATECH support: NO
* PFRING support: NO
* NETRONOME support: NO
* BIVIO support: NO
* Compact IPv4 support: YES
* Plugin support: YES
* PCRE support: YES
* Application Labeling: YES
* Deep Packet Inspection: YES
* nDPI Support: NO
* Payload Processing Support: YES
* Entropy Support: YES
* Fingerprint Export Support: NO
* OpenSSL Support: YES (-lssl -lcrypto)
* P0F Support: NO
* MPLS NetFlow Enabled: NO
* Non-IP Flow Enabled: NO
* IE/Template Metadata Export: YES
* GCC Atomic Builtin functions: NO
* Compiler (CC): gcc
* Compiler flags (CFLAGS): -I$(top_srcdir)/include -I$(top_srcdir)/airframe/include -Wall -Wextra -Wshadow -Wpointer-arith -Wformat=2 -Wunused -Wundef -Wduplicated-cond -Wwrite-strings -Wmissing-prototypes -Wstrict-prototypes -Wno-unused-parameter -g -O2
* Linker flags (LDFLAGS): -lpcre
* Libraries (LIBS): -lpcap -lm -lz
? ? ? ? 為了避免復(fù)雜性，很多可選的支持我們都沒(méi)有選，要實(shí)現(xiàn)DPI支持所必須的--enable-applabel和--enable-dpi是不能丟的，另外捎帶著我們也選了plugin、entropy和zlib，看看后面有沒(méi)有用上的機(jī)會(huì)。
? ? ? ? （3）make && make install YAF3
? ? ? make && make install以后，同樣，需要到默認(rèn)目錄下去尋找編譯結(jié)果
/usr/bin/mkdir -p '/usr/local/lib'
/bin/sh ../libtool --mode=install /usr/bin/install -c libyaf.la '/usr/local/lib'
libtool: install: /usr/bin/install -c .libs/libyaf-3.0.0.alpha2.so.4.0.0 /usr/local/lib/libyaf-3.0.0.alpha2.so.4.0.0
libtool: install: (cd /usr/local/lib && { ln -s -f libyaf-3.0.0.alpha2.so.4.0.0 libyaf-3.0.0.alpha2.so.4 || { rm -f libyaf-3.0.0.alpha2.so.4 && ln -s libyaf-3.0.0.alpha2.so.4.0.0 libyaf-3.0.0.alpha2.so.4; }; })
libtool: install: (cd /usr/local/lib && { ln -s -f libyaf-3.0.0.alpha2.so.4.0.0 libyaf.so || { rm -f libyaf.so && ln -s libyaf-3.0.0.alpha2.so.4.0.0 libyaf.so; }; })
libtool: install: /usr/bin/install -c .libs/libyaf.lai /usr/local/lib/libyaf.la
libtool: finish: PATH="/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/root/bin:/sbin" ldconfig -n /usr/local/lib
----------------------------------------------------------------------
Libraries have been installed in:
/usr/local/lib
If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the '-LLIBDIR'
flag during linking and do at least one of the following:
- add LIBDIR to the 'LD_LIBRARY_PATH' environment variable
during execution
- add LIBDIR to the 'LD_RUN_PATH' environment variable
during linking
- use the '-Wl,-rpath -Wl,LIBDIR' linker flag
- have your system administrator add LIBDIR to '/etc/ld.so.conf'
See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------
? ? ? ? ? （4）測(cè)試執(zhí)行：
[root@bogon yaf-3.0.0.alpha2]# yaf --version
yaf version 3.0.0.alpha2 Build Configuration:
* Timezone support: UTC
* Fixbuf version: 3.0.0.alpha2
* DAG support: NO
* Napatech support: NO
* Netronome support: NO
* Bivio support: NO
* PFRING support: NO
* Compact IPv4 support: YES
* Plugin support: YES
* Application Labeling: YES
* Payload Processing Support: YES
* Deep Packet Inspection Support: YES
* Entropy support: YES
* Fingerprint Export Support: NO
* P0F Support: NO
* MPLS Support: NO
* Non-IP Support: NO
* Separate Interface Support: NO
* nDPI Support: NO
* IE/Template Metadata Export: YES
(c) 2000-2023 Carnegie Mellon University.
GNU General Public License (GPL) Rights pursuant to Version 2, June 1991
Some included library code covered by LGPL 2.1; see source for details.
Send bug reports, feature requests, and comments to netsa-help@cert.org.
[root@bogon yaf-3.0.0.alpha2]#
? ? ? ? 二、編譯super mediator
[root@bogon ~]# tar zxvf super_mediator-2.0.0.alpha2.tar.gz
[root@bogon ~]# cd super_mediator-2.0.0.alpha2/
[root@bogon super_mediator-2.0.0.alpha2]# ./configure --with-mysql --with-zlib
[root@bogon super_mediator-2.0.0.alpha2]# make && make install
…………………………
make[4]: 進(jìn)入目錄“/root/super_mediator-2.0.0.alpha2/src”
/usr/bin/mkdir -p '/usr/local/bin'
/usr/bin/install -c super_mediator '/usr/local/bin'
/usr/bin/mkdir -p '/usr/local/share/man/man1'
/usr/bin/install -c -m 644 super_mediator.1 super_mediator.conf.1 '/usr/local/share/man/man1'
make[4]: 離開(kāi)目錄“/root/super_mediator-2.0.0.alpha2/src”
make[3]: 離開(kāi)目錄“/root/super_mediator-2.0.0.alpha2/src”
make[2]: 離開(kāi)目錄“/root/super_mediator-2.0.0.alpha2/src”
make[1]: 離開(kāi)目錄“/root/super_mediator-2.0.0.alpha2/src”
Making install in etc
make[1]: 進(jìn)入目錄“/root/super_mediator-2.0.0.alpha2/etc”
make[2]: 進(jìn)入目錄“/root/super_mediator-2.0.0.alpha2/etc”
/usr/bin/mkdir -p '/usr/local/etc'
/usr/bin/install -c -m 644 super_mediator.conf '/usr/local/etc'
make[2]: 對(duì)“install-data-am”無(wú)需做任何事。
make[2]: 離開(kāi)目錄“/root/super_mediator-2.0.0.alpha2/etc”
make[1]: 離開(kāi)目錄“/root/super_mediator-2.0.0.alpha2/etc”
Making install in doc
make[1]: 進(jìn)入目錄“/root/super_mediator-2.0.0.alpha2/doc”
…………………………? ? ? ? 同樣，我們選擇平平無(wú)奇的安裝，不過(guò)雖然這里我們選擇了--with-mysql，后面檢查版本的時(shí)候任然被報(bào)沒(méi)有mysql支持，可能是選項(xiàng)沒(méi)選夠的原因吧。
? ? ? ? 如果此時(shí)直接執(zhí)行測(cè)試，大概率會(huì)找不到libfixbuf.so文件：
[root@bogon local]# super_mediator --version
super_mediator: error while loading shared libraries: libfixbuf.so.10: cannot open shared object file: No such file or directory
? ? ? ? 其主要原因其實(shí)已經(jīng)在前面的編譯結(jié)果中了。也就是?環(huán)境變量'LD_LIBRARY_PATH'必須被設(shè)置到/usr/local/lib上：
? ? ? ? 在~/.bashrc文件中添加環(huán)境變量，并且source一下就好了。
# .bashrc
# User specific aliases and functions
alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'
# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi
export LTDL_LIBRARY_PATH=/usr/local/lib/yaf
export LD_LIBRARY_PATH=/usr/local/lib
? ? ? ? ?此時(shí)，再執(zhí)行super_mediator --version就沒(méi)問(wèn)題了：
[root@bogon local]# super_mediator --version
super_mediator version 2.0.0.alpha2
Build Configuration:
* Fixbuf version: 3.0.0.alpha2
* MySQL support: NO
* OpenSSL support: NO
* SiLK IPSet support: NO
Copyright (C) 2012-2023 Carnegie Mellon University
GNU General Public License (GPL) Rights pursuant to Version 2, June 1991
Send bug reports, feature requests, and comments to netsa-help@cert.org.
? ? ? ? 之所以要編譯super-mediator，原因就在這里，因?yàn)閥af3需要libfixbuf3支持，libfixbuf3對(duì)應(yīng)的super mediator版本至少應(yīng)該是2。?
? ? ? ? 三、測(cè)試使用YAF3+Super_mediator
? ? ? ? 既然是測(cè)試，我們還是從一開(kāi)始就整最簡(jiǎn)單的用法。看看自找麻煩的編譯后是不是獲得了DPI的加持。
? ? ? ? 簡(jiǎn)單的對(duì)一個(gè)pcap文件進(jìn)行轉(zhuǎn)IPFIX處理，其中設(shè)定applable為[53，80，443]。對(duì)應(yīng)去查內(nèi)置的applabel號(hào)，代表對(duì)DNS、Http和TLS協(xié)議進(jìn)行解析。不過(guò)貌似我的樣例數(shù)據(jù)中沒(méi)有足夠完整的Http數(shù)據(jù)，所以最終只得到了DNS和TLS相關(guān)的數(shù)據(jù)。
? ? ? ? 【這個(gè)命令是錯(cuò)的，文末分解】
[root@bogon pcap]# yaf --in /root/share/pcap/test2.pcap --out /root/test.yaf --dpi --dpi-select=[80,53,443] --applabel --max-payload=2048
[2023-06-25 09:35:14] Rejected 63574 out-of-sequence packets.
?????????然后使用super_mediator進(jìn)行提取轉(zhuǎn)換，也就是簡(jiǎn)單的講IPFIX數(shù)據(jù)轉(zhuǎn)換為TEXT類型輸出。
[root@bogon ~]# super_mediator -o result.txt -m text test.yaf
Initialization Successful, starting...
[2023-06-25 09:39:18] Running as root in --live mode, but not dropping privilege
????????簡(jiǎn)單查看一下數(shù)據(jù)，這個(gè)和官方文件中的描述是一致的。也就是流數(shù)據(jù)以時(shí)間開(kāi)始的一行，和流相關(guān)的元數(shù)據(jù)在流數(shù)據(jù)的下面追加行：
2022-11-15 03:02:22.560|2022-11-15 03:02:23.075|0.515|0.004|6|2.20.192.41|443|5|1521|0|00:00:00:00:00:00|192.168.137.113|62629|3|807|0|00:00:00:00:00:00|AP|AP|AP|AP|1363957954|3883502200|0|0|0|0|0|72|eof|C1|||
2022-11-15 03:02:23.066|2022-11-15 03:02:23.093|0.027|0.006|6|192.168.137.113|62636|4|487|0|00:00:00:00:00:00|42.81.247.1|80|4|1724|0|00:00:00:00:00:00|S|AP|AS|AP|2964962409|62197094|0|0|0|0|80|0|eof|C1|||
2022-11-15 02:55:32.228|2022-11-15 03:02:23.167|410.939|0.011|17|192.168.137.113|65520|8|549|0|00:00:00:00:00:00|192.168.137.1|53|8|1436|0|00:00:00:00:00:00|||||0|0|0|0|0|0|53|0|eof|C1|||
|dns|Q|16475|0|0|0|1|0|dss0.bdstatic.com.|
|dns|R|16475|1|0|0|5|7126|dss0.bdstatic.com.|sslbaiduv6.jomodns.com.
|dns|R|16475|1|0|0|1|25|sslbaiduv6.jomodns.com.|106.38.179.33
2022-11-15 03:02:22.751|2022-11-15 03:02:23.407|0.656|0.215|6|192.168.137.113|62635|5|555|0|00:00:00:00:00:00|2.20.192.41|443|7|6078|0|00:00:00:00:00:00|S|AP|AS|AP|3356565209|283469080|0|0|0|0|443|0|eof|C1|||
tls|187|I|0|0xc030
tls|186|I|0|3
tls|288|I|0|0x0303
2022-11-15 03:02:22.704|2022-11-15 03:02:23.416|0.712|0.229|6|192.168.137.113|62633|5|555|0|00:00:00:00:00:00|2.20.192.41|443|7|6078|0|00:00:00:00:00:00|S|AP|AS|AP|614308801|272455679|0|0|0|0|443|0|eof|C1|||
tls|187|I|0|0xc030
tls|186|I|0|3
tls|288|I|0|0x0303
2022-11-15 02:47:48.069|2022-11-15 03:02:23.484|875.415|0.008|17|192.168.137.113|64236|15|1022|0|00:00:00:00:00:00|192.168.137.1|53|15|2344|0|00:00:00:00:00:00|||||0|0|0|0|0|0|53|0|eof|C1|||
? ? ? ? ?使用grep看一下tls類元數(shù)據(jù)：
? ? ? ? 能夠看出元數(shù)據(jù)采取了“表名“+”information element id“+”data“的記法：
? ? ? ? 表明其實(shí)對(duì)應(yīng)的就是app label，ie id=31，代表cRLDistributionPoints （吊銷證書列表發(fā)布點(diǎn)），32代表CertificatePolicies 證書策略對(duì)象。
[root@bogon ~]# cat result.txt|grep http
tls|32|E|0|http://www.digicert.com/CPS
tls|32|E|0|http://www.digicert.com/CPS
tls|32|E|0|http://www.digicert.com/CPS
tls|31|E|0|http://crl3.digicert.com/SecureSiteCAG2.crl
tls|31|E|0|http://crl4.digicert.com/SecureSiteCAG2.crl
tls|32|E|0|http://www.digicert.com/CPS
tls|31|E|0|http://crl.digicert.cn/GeoTrustRSACNCAG2.crl
tls|32|E|0|http://www.digicert.com/CPS
tls|31|E|0|http://crl3.digicert.com/GeoTrustCNRSACAG1.crl
tls|31|E|0|http://crl4.digicert.com/GeoTrustCNRSACAG1.crl
tls|32|E|0|http://www.digicert.com/CPS
tls|32|E|0|https://sectigo.com/CPS
tls|31|E|0|http://crl.digicert.cn/GeoTrustRSACNCAG2.crl
tls|32|E|0|http://www.digicert.com/CPS
tls|32|E|0|http://www.digicert.com/CPS
tls|31|E|0|http://crl.digicert.cn/TrustAsiaOVTLSProCAG3.crl
tls|32|E|0|http://www.digicert.com/CPS
tls|32|E|0|http://www.digicert.com/CPS
tls|31|E|0|http://crl3.digicert.com/GeoTrustCNRSACAG1.crl
tls|31|E|0|http://crl4.digicert.com/GeoTrustCNRSACAG1.crl
tls|32|E|0|http://www.digicert.com/CPS
tls|31|E|0|http://crl3.digicert.com/SecureSiteCAG2.crl
tls|31|E|0|http://crl4.digicert.com/SecureSiteCAG2.crl
……………………? ? ? ? 使用grep看一下DNS類元數(shù)據(jù)。anzhao guanfang de shuofa ,DNS、SSL/TLS和DNP3采用不同的輸出方法。
[root@bogon ~]# cat result.txt|grep dns
|dns|Q|37404|0|0|0|1|0|pagead2.googlesyndication.com.|
|dns|R|37404|1|0|0|1|254|pagead2.googlesyndication.com.|180.163.150.166
|dns|Q|14041|0|0|0|1|0|SMS_SLP.|
|dns|Q|14041|0|0|0|1|0|SMS_SLP.|
|dns|Q|60145|0|0|0|1|0|hub5pr.v6.phub.sandai.net.|
|dns|R|60145|2|0|0|6|1412|sandai.net.|localhost.
|dns|Q|39275|0|0|0|1|0|pr.x.hub.sandai.net.|
|dns|R|39275|1|0|0|1|342|pr.x.hub.sandai.net.|180.163.56.147
|dns|R|39275|1|0|0|1|342|pr.x.hub.sandai.net.|123.182.51.211
|dns|Q|33362|0|0|0|28|0|hub5pr.v6.phub.sandai.net.|
|dns|R|33362|1|0|0|28|1001|hub5pr.v6.phub.sandai.net.|2408:4004:0100:2e02:2c2c:fbe3:74d9:6d44
|dns|R|33362|1|0|0|28|1001|hub5pr.v6.phub.sandai.net.|2408:4004:0100:2e02:2c2c:fbe3:74d9:6d45
|dns|Q|46802|0|0|0|28|0|pr.x.hub.sandai.net.|
|dns|R|46802|2|0|0|6|1323|sandai.net.|localhost.
|dns|Q|59395|0|0|0|1|0|SMS_SLP.|
|dns|Q|59395|0|0|0|1|0|SMS_SLP.|
|dns|Q|31369|0|0|0|1|0|XTZJ-20211028AW.|
|dns|Q|31369|0|0|0|1|0|XTZJ-20211028AW.|
|dns|Q|47261|0|0|0|28|0|XTZJ-20211028AW.|
|dns|Q|47261|0|0|0|28|0|XTZJ-20211028AW.|
|dns|Q|19998|0|0|0|12|0|126.40.4.11.in-addr.arpa.|
|dns|Q|45562|0|0|0|1|0|wpad.|
|dns|Q|45562|0|0|0|1|0|wpad.|
|dns|Q|49129|0|0|0|28|0|wpad.|
|dns|Q|49129|0|0|0|28|0|wpad.|
|dns|Q|0|0|0|0|1|0|wpad.local.|
|dns|Q|0|0|0|0|1|0|wpad.local.|
|dns|Q|25633|0|0|0|12|0|252.0.0.224.in-addr.arpa.|
|dns|Q|47544|0|0|0|1|0|XTZJ-20211028AW.|
|dns|Q|47544|0|0|0|1|0|XTZJ-20211028AW.|
|dns|Q|29209|0|0|0|28|0|XTZJ-20211028AW.|
|dns|Q|29209|0|0|0|28|0|XTZJ-20211028AW.|
|dns|Q|60062|0|0|0|12|0|3.0.0.0.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.f.f.ip6.arpa.|
????????三、鏡像化YAF3+Super-mediator
? ? ? ? 一致覺(jué)得NetSA的官方支持做得不好，依靠官方的安裝部署指南部署總會(huì)遇到大大小小的問(wèn)題，需要通讀大量說(shuō)明文檔和熟練掌握Linux系統(tǒng)才有可能解決。至少，如上在Centos stream 8上安裝的過(guò)程就有不少問(wèn)題，而安裝官方的指南，在CentOS 7上的安裝會(huì)遇到更多莫名的問(wèn)題。
? ? ? ? 經(jīng)過(guò)一系列磕磕絆絆的嘗試，總算在下面這種情況下配通了，不過(guò)好多我想用的配置項(xiàng)沒(méi)法打開(kāi)，因?yàn)榇蜷_(kāi)就會(huì)報(bào)錯(cuò)，且我還不知道時(shí)什么東西沒(méi)裝齊整，或者什么東西裝錯(cuò)版本了。比如，按照官方的說(shuō)法，只安裝gcc、gcc-c++、pkgconfig、make是無(wú)法完成libfixbuf的編譯的，而整個(gè)“Development Tools”包的安裝就沒(méi)有問(wèn)題……
? ? ? ? Dockerfile文件如下：? ??
FROM centos:centos7
#RUN yum install gcc gcc-c++ make pkgconfig -y
RUN yum group install "Development Tools" -y
RUN yum install libpcap libpcap-devel -y
RUN yum install glib2 glib2-devel -y
RUN yum install zlib zlib-devel -y
ADD src/libfixbuf-3.0.0.alpha2.tar.gz /root/.
ADD src/super_mediator-2.0.0.alpha2.tar.gz /root/.
ADD src/yaf-3.0.0.alpha2.tar.gz /root/.
ADD src/mothra-1.6.0-src.tar.gz /root/.
RUN mv /root/libfixbuf-3.0.0.alpha2 /root/libfixbuf3
RUN mv /root/super_mediator-2.0.0.alpha2 /root/super_mediator2
RUN mv /root/yaf-3.0.0.alpha2 /root/yaf3
RUN mv /root/mothra-1.6.0-src /root/mothra160
RUN cd /root/libfixbuf3 \
&& ./configure --disable-tools \
&& make \
&& make install
RUN cd /root/yaf3 \
&& ./configure --enable-plugins --enable-applabel --enable-dpi --enable-entropy --enable-zlib \
&& make \
&& make install
RUN cd /root/super_mediator2 \
&& ./configure --with-zlib \
&& make \
&& make install
RUN echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib' >> /root/.bashrc
RUN echo 'export LTDL_LIBRARY_PATH=$LTDL_LIBRARY_PATH:/usr/local/lib/yaf' >> /root/.bashrc
CMD ["./init-silk.sh"]
????????四、測(cè)試使用mothra
? ? ? ? 簡(jiǎn)單的mothra使用似乎不涉及從cert網(wǎng)站上下載源碼編譯等等之類的過(guò)程。僅僅需要在搭建好的spark-shell環(huán)境——比如我們前面搭建過(guò)的windows下的spark環(huán)境進(jìn)行操作即可。
PS F:\tmp> spark-shell --packages "org.cert.netsa:mothra_2.12:1.6.0"
:: loading settings :: url = jar:file:/C:/spark/jars/ivy-2.5.1.jar!/org/apache/ivy/core/settings/ivysettings.xml
Ivy Default Cache set to: C:\Users\lhyzw\.ivy2\cache
The jars for the packages stored in: C:\Users\lhyzw\.ivy2\jars
org.cert.netsa#mothra_2.12 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-36498b97-d9d5-492f-81e2-0cbe1ac02c77;1.0
confs: [default]
found org.cert.netsa#mothra_2.12;1.6.0 in central
found org.cert.netsa#mothra-analysis_2.12;1.6.0 in central
found org.cert.netsa#netsa-data_2.12;1.6.0 in central
found org.cert.netsa#netsa-io-silk_2.12;1.6.0 in central
found com.beachape#enumeratum_2.12;1.7.0 in central
found com.beachape#enumeratum-macros_2.12;1.6.1 in central
found org.scala-lang#scala-reflect;2.12.11 in central
found org.anarres.lzo#lzo-core;1.0.6 in central
found com.google.code.findbugs#annotations;2.0.3 in central
found commons-logging#commons-logging;1.1.1 in local-m2-cache
found org.scala-lang.modules#scala-parser-combinators_2.12;1.1.2 in central
found org.xerial.snappy#snappy-java;1.1.8.4 in local-m2-cache
found org.cert.netsa#mothra-datasources_2.12;1.6.0 in central
found org.cert.netsa#mothra-datasources-base_2.12;1.6.0 in central
found org.cert.netsa#mothra-datasources-ipfix_2.12;1.6.0 in central
found com.typesafe.scala-logging#scala-logging_2.12;3.9.4 in central
found org.scala-lang#scala-reflect;2.12.13 in central
found org.slf4j#slf4j-api;1.7.30 in local-m2-cache
found com.github.scopt#scopt_2.12;3.7.1 in central
found org.cert.netsa#netsa-io-ipfix_2.12;1.6.0 in central
found org.apache.commons#commons-text;1.1 in central
found org.apache.commons#commons-lang3;3.5 in local-m2-cache
found org.scala-lang.modules#scala-xml_2.12;1.3.0 in central
found org.cert.netsa#mothra-datasources-silk_2.12;1.6.0 in central
found org.cert.netsa#netsa-util_2.12;1.6.0 in central
found org.cert.netsa#mothra-functions_2.12;1.6.0 in central
downloading https://repo1.maven.org/maven2/org/cert/netsa/mothra_2.12/1.6.0/mothra_2.12-1.6.0.jar ...
[SUCCESSFUL ] org.cert.netsa#mothra_2.12;1.6.0!mothra_2.12.jar (511ms)
downloading https://repo1.maven.org/maven2/org/cert/netsa/mothra-analysis_2.12/1.6.0/mothra-analysis_2.12-1.6.0.jar ...
[SUCCESSFUL ] org.cert.netsa#mothra-analysis_2.12;1.6.0!mothra-analysis_2.12.jar (2321ms)
downloading https://repo1.maven.org/maven2/org/cert/netsa/mothra-datasources_2.12/1.6.0/mothra-datasources_2.12-1.6.0.jar ...
[SUCCESSFUL ] org.cert.netsa#mothra-datasources_2.12;1.6.0!mothra-datasources_2.12.jar (577ms)
downloading https://repo1.maven.org/maven2/org/cert/netsa/mothra-functions_2.12/1.6.0/mothra-functions_2.12-1.6.0.jar ...
[SUCCESSFUL ] org.cert.netsa#mothra-functions_2.12;1.6.0!mothra-functions_2.12.jar (1751ms)
downloading https://repo1.maven.org/maven2/org/cert/netsa/netsa-data_2.12/1.6.0/netsa-data_2.12-1.6.0.jar ...
[SUCCESSFUL ] org.cert.netsa#netsa-data_2.12;1.6.0!netsa-data_2.12.jar (2050ms)
downloading https://repo1.maven.org/maven2/org/cert/netsa/netsa-io-ipfix_2.12/1.6.0/netsa-io-ipfix_2.12-1.6.0.jar ...
[SUCCESSFUL ] org.cert.netsa#netsa-io-ipfix_2.12;1.6.0!netsa-io-ipfix_2.12.jar (5512ms)
downloading https://repo1.maven.org/maven2/org/cert/netsa/netsa-io-silk_2.12/1.6.0/netsa-io-silk_2.12-1.6.0.jar ...
[SUCCESSFUL ] org.cert.netsa#netsa-io-silk_2.12;1.6.0!netsa-io-silk_2.12.jar (4328ms)
downloading https://repo1.maven.org/maven2/org/cert/netsa/netsa-util_2.12/1.6.0/netsa-util_2.12-1.6.0.jar ...
[SUCCESSFUL ] org.cert.netsa#netsa-util_2.12;1.6.0!netsa-util_2.12.jar (632ms)
downloading https://repo1.maven.org/maven2/com/beachape/enumeratum_2.12/1.7.0/enumeratum_2.12-1.7.0.jar ...
[SUCCESSFUL ] com.beachape#enumeratum_2.12;1.7.0!enumeratum_2.12.jar (853ms)
downloading https://repo1.maven.org/maven2/org/anarres/lzo/lzo-core/1.0.6/lzo-core-1.0.6.jar ...
[SUCCESSFUL ] org.anarres.lzo#lzo-core;1.0.6!lzo-core.jar (777ms)
downloading https://repo1.maven.org/maven2/org/scala-lang/modules/scala-parser-combinators_2.12/1.1.2/scala-parser-combinators_2.12-1.1.2.jar ...
[SUCCESSFUL ] org.scala-lang.modules#scala-parser-combinators_2.12;1.1.2!scala-parser-combinators_2.12.jar(bundle) (1531ms)
……………………
………………
…………? ? ? ? 一般來(lái)說(shuō)，這個(gè)過(guò)程總會(huì)有點(diǎn)小坎坷，比如說(shuō)有若干jar包安裝不上，但命名這些包就是在maven repository centra中能找到的，我也不知到為什么：
:: problems summary ::
:::: WARNINGS
[NOT FOUND ] org.xerial.snappy#snappy-java;1.1.8.4!snappy-java.jar(bundle) (2ms)
==== local-m2-cache: tried
file:/C:/Users/lhyzw/.m2/repository/org/xerial/snappy/snappy-java/1.1.8.4/snappy-java-1.1.8.4.jar
[NOT FOUND ] commons-logging#commons-logging;1.1.1!commons-logging.jar (1ms)
==== local-m2-cache: tried
file:/C:/Users/lhyzw/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar
[NOT FOUND ] org.slf4j#slf4j-api;1.7.30!slf4j-api.jar (1ms)
==== local-m2-cache: tried
file:/C:/Users/lhyzw/.m2/repository/org/slf4j/slf4j-api/1.7.30/slf4j-api-1.7.30.jar
::::::::::::::::::::::::::::::::::::::::::::::
:: FAILED DOWNLOADS ::
:: ^ see resolution messages for details ^ ::
::::::::::::::::::::::::::::::::::::::::::::::? ? ? ? ?好在錯(cuò)誤信息指示很明確，去repository中下載就是了
? ? ? ? ?下載完成后放到錯(cuò)誤信息所指定的那個(gè)目錄中：? ?
? ? ? ? ?然后再運(yùn)行就可以成功啟動(dòng)了：
PS C:\Users\lhyzw> spark-shell --packages "org.cert.netsa:mothra_2.12:1.6.0"
:: loading settings :: url = jar:file:/C:/spark/jars/ivy-2.5.1.jar!/org/apache/ivy/core/settings/ivysettings.xml
Ivy Default Cache set to: C:\Users\lhyzw\.ivy2\cache
The jars for the packages stored in: C:\Users\lhyzw\.ivy2\jars
org.cert.netsa#mothra_2.12 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-771cfadc-626d-41ae-9ce3-6976e01a4db7;1.0
confs: [default]
found org.cert.netsa#mothra_2.12;1.6.0 in central
found org.cert.netsa#mothra-analysis_2.12;1.6.0 in central
found org.cert.netsa#netsa-data_2.12;1.6.0 in central
found org.cert.netsa#netsa-io-silk_2.12;1.6.0 in central
found com.beachape#enumeratum_2.12;1.7.0 in central
found com.beachape#enumeratum-macros_2.12;1.6.1 in central
found org.scala-lang#scala-reflect;2.12.11 in central
found org.anarres.lzo#lzo-core;1.0.6 in central
found com.google.code.findbugs#annotations;2.0.3 in central
found commons-logging#commons-logging;1.1.1 in local-m2-cache
found org.scala-lang.modules#scala-parser-combinators_2.12;1.1.2 in central
found org.xerial.snappy#snappy-java;1.1.8.4 in local-m2-cache
found org.cert.netsa#mothra-datasources_2.12;1.6.0 in central
found org.cert.netsa#mothra-datasources-base_2.12;1.6.0 in central
found org.cert.netsa#mothra-datasources-ipfix_2.12;1.6.0 in central
found com.typesafe.scala-logging#scala-logging_2.12;3.9.4 in central
found org.scala-lang#scala-reflect;2.12.13 in central
found org.slf4j#slf4j-api;1.7.30 in local-m2-cache
found com.github.scopt#scopt_2.12;3.7.1 in central
found org.cert.netsa#netsa-io-ipfix_2.12;1.6.0 in central
found org.apache.commons#commons-text;1.1 in central
found org.apache.commons#commons-lang3;3.5 in local-m2-cache
found org.scala-lang.modules#scala-xml_2.12;1.3.0 in central
found org.cert.netsa#mothra-datasources-silk_2.12;1.6.0 in central
found org.cert.netsa#netsa-util_2.12;1.6.0 in central
found org.cert.netsa#mothra-functions_2.12;1.6.0 in central
downloading file:/C:/Users/lhyzw/.m2/repository/org/xerial/snappy/snappy-java/1.1.8.4/snappy-java-1.1.8.4.jar ...
[SUCCESSFUL ] org.xerial.snappy#snappy-java;1.1.8.4!snappy-java.jar(bundle) (9ms)
downloading file:/C:/Users/lhyzw/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar ...
[SUCCESSFUL ] commons-logging#commons-logging;1.1.1!commons-logging.jar (5ms)
downloading file:/C:/Users/lhyzw/.m2/repository/org/slf4j/slf4j-api/1.7.30/slf4j-api-1.7.30.jar ...
[SUCCESSFUL ] org.slf4j#slf4j-api;1.7.30!slf4j-api.jar (4ms)
:: resolution report :: resolve 426ms :: artifacts dl 51ms
:: modules in use:
com.beachape#enumeratum-macros_2.12;1.6.1 from central in [default]
com.beachape#enumeratum_2.12;1.7.0 from central in [default]
com.github.scopt#scopt_2.12;3.7.1 from central in [default]
com.google.code.findbugs#annotations;2.0.3 from central in [default]
com.typesafe.scala-logging#scala-logging_2.12;3.9.4 from central in [default]
commons-logging#commons-logging;1.1.1 from local-m2-cache in [default]
org.anarres.lzo#lzo-core;1.0.6 from central in [default]
org.apache.commons#commons-lang3;3.5 from local-m2-cache in [default]
org.apache.commons#commons-text;1.1 from central in [default]
org.cert.netsa#mothra-analysis_2.12;1.6.0 from central in [default]
org.cert.netsa#mothra-datasources-base_2.12;1.6.0 from central in [default]
org.cert.netsa#mothra-datasources-ipfix_2.12;1.6.0 from central in [default]
org.cert.netsa#mothra-datasources-silk_2.12;1.6.0 from central in [default]
org.cert.netsa#mothra-datasources_2.12;1.6.0 from central in [default]
org.cert.netsa#mothra-functions_2.12;1.6.0 from central in [default]
org.cert.netsa#mothra_2.12;1.6.0 from central in [default]
org.cert.netsa#netsa-data_2.12;1.6.0 from central in [default]
org.cert.netsa#netsa-io-ipfix_2.12;1.6.0 from central in [default]
org.cert.netsa#netsa-io-silk_2.12;1.6.0 from central in [default]
org.cert.netsa#netsa-util_2.12;1.6.0 from central in [default]
org.scala-lang#scala-reflect;2.12.13 from central in [default]
org.scala-lang.modules#scala-parser-combinators_2.12;1.1.2 from central in [default]
org.scala-lang.modules#scala-xml_2.12;1.3.0 from central in [default]
org.slf4j#slf4j-api;1.7.30 from local-m2-cache in [default]
org.xerial.snappy#snappy-java;1.1.8.4 from local-m2-cache in [default]
:: evicted modules:
org.scala-lang#scala-reflect;2.12.11 by [org.scala-lang#scala-reflect;2.12.13] in [default]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 26 | 0 | 0 | 1 || 25 | 3 |
---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent-771cfadc-626d-41ae-9ce3-6976e01a4db7
confs: [default]
25 artifacts copied, 0 already retrieved (9958kB/59ms)
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://192.168.137.14:4040
Spark context available as 'sc' (master = local[*], app id = local-1687751809117).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.4.0
/_/
Using Scala version 2.12.17 (OpenJDK 64-Bit Server VM, Java 1.8.0_302)
Type in expressions to have them evaluated.
Type :help for more information.
scala>? ? ? ? 打印版本號(hào)，導(dǎo)入數(shù)據(jù)源驅(qū)動(dòng)都沒(méi)有問(wèn)題：
scala> org.cert.netsa.util.versionInfo("mothra")
res0: Option[String] = Some(1.6.0)
scala> import org.cert.netsa.mothra.datasources._
import org.cert.netsa.mothra.datasources._????????導(dǎo)入數(shù)據(jù)也木有問(wèn)題：
scala> var df = spark.read.ipfix("f:/tmp/test2.ipfix")
df: org.apache.spark.sql.DataFrame = [startTime: timestamp, endTime: timestamp ... 17 more fields]
scala> df.count
res3: Long = 3418?????????查看數(shù)據(jù)，有點(diǎn)大，就不全打印了：
scala> df.show
+--------------------+--------------------+---------------+----------+--------------------+---------------+------------------+-------------------+------+-------------+------------+-----------+------------------+----------+-----------------+---------------+----------------------+-------------+--------------------+
| startTime| endTime|sourceIPAddress|sourcePort|destinationIPAddress|destinationPort|protocolIdentifier|observationDomainId|vlanId|reverseVlanId|silkAppLabel|packetCount|reversePacketCount|octetCount|reverseOctetCount|initialTCPFlags|reverseInitialTCPFlags|unionTCPFlags|reverseUnionTCPFlags|
+--------------------+--------------------+---------------+----------+--------------------+---------------+------------------+-------------------+------+-------------+------------+-----------+------------------+----------+-----------------+---------------+----------------------+-------------+--------------------+
|2022-11-15 10:34:...|2022-11-15 10:34:...| 47.110.20.149| 443| 192.168.182.76| 62116| 6| 0| 0| 0| 0| 3| 4| 674| 253| 24| 24| 25| 21|
|2022-11-15 10:34:...|2022-11-15 10:34:...| 192.168.182.76| 62126| 40.90.184.73| 443| 6| 0| 0| 0| 443| 11| 10| 3328| 7410| 2| 18| 25| 25|
|2022-11-15 10:34:...|2022-11-15 10:34:...|192.168.137.113| 56138| 40.90.184.73| 443| 6| 0| 0| 0| 443| 11| 9| 3281| 7230| 2| 18| 25| 25|
|2022-11-15 10:34:...|2022-11-15 10:34:...|192.168.137.113| 56145| 183.47.103.43| 36688| 6| 0| 0| 0| 0| 11| 16| 4977| 780| 2| 24| 25| 25|
|2022-11-15 10:34:...|2022-11-15 10:34:...|192.168.137.113| 56065| 220.181.33.6| 443| 6| 0| 0| null| 0| 3| null| 120| null| 17| null| 20| null|
|2022-11-15 10:34:...|2022-11-15 10:34:...|192.168.137.113| 56064| 220.181.33.6| 443| 6| 0| 0| null| 0| 2| null| 80| null| 17| null| 20| null|????????【這一段的說(shuō)法是錯(cuò)的】不過(guò)看起來(lái)只有流數(shù)據(jù)被存在ipfix文件中，DPI數(shù)據(jù)似乎并沒(méi)有被導(dǎo)入進(jìn)來(lái)。其實(shí)折騰mothra的目的就是希望官方驅(qū)動(dòng)源能夠幫助我們將dpi信息導(dǎo)入spark，如果只是流數(shù)據(jù)，依賴SiLK工具集就夠用，完全沒(méi)必要折騰spark?？赡躨pfix的導(dǎo)出格式就不支持dpi信息吧，具體為何我也不知，只能下回分解了。
? ? ? ? 【這是一個(gè)悲傷的故事】后面我們打算再開(kāi)一篇仔細(xì)分解以下如何查看“完整”的DPI信息。然而，即使是把下一篇的問(wèn)題都解決了，按照上面的過(guò)程，也無(wú)法得到http的DPI信息——除了AppLabel=80這種情況下的http元數(shù)據(jù)沒(méi)有被提取出來(lái)，dns和ssl的元數(shù)據(jù)都已經(jīng)被提取出來(lái)了。所以我花了好幾天的時(shí)間來(lái)排查——包括在高鐵上的時(shí)間……最終發(fā)現(xiàn)這個(gè)錯(cuò)誤在于使用--applabel參數(shù)的時(shí)候相當(dāng)然的用[]將applabel框了起來(lái)——而官方的指南中式不需要帶這個(gè)中括號(hào)的。然而結(jié)果就是，yaf程序并不會(huì)報(bào)錯(cuò)，而是將“[80”識(shí)別為一個(gè)applabel——那自然式不存在的，所以從--verbose信息中顯示，它只解析了2個(gè)protocol：
? ? ? ? ?所以，正確的命令應(yīng)該是：
文章來(lái)源：http://www.zghlxwxcb.cn/news/detail-502179.html
? ? ? ? ?如此，在spark中就可以看到完整的http信息了：文章來(lái)源地址http://www.zghlxwxcb.cn/news/detail-502179.html
scala> my-http-dataframe.show(1,0,true)
-RECORD 0----------------------------------------------------------------------------------------
startTime | 2022-11-15 10:34:55.611
endTime | 2022-11-15 10:34:55.708
sourceIPAddress | 192.168.…………
sourcePort | 49845
destinationIPAddress | 180.163.…………
destinationPort | 80
protocolIdentifier | 6
observationDomainId | 0
vlanId | 0
reverseVlanId | 0
silkAppLabel | 80
packetCount | 5
reversePacketCount | 5
octetCount | 533
reverseOctetCount | 379
initialTCPFlags | 2
reverseInitialTCPFlags | 18
unionTCPFlags | 25
reverseUnionTCPFlags | 29
………………
httpAcceptList | []
httpAcceptCharsetList | []
httpAcceptLanguageList | []
httpAgeList | []
httpAllowList | []
httpAuthorizationList | []
httpConnectionList | [Close, close]
httpContentEncodingList | []
httpContentLanguageList | []
httpContentLengthList | [192, 36]
httpContentLocationList | []
httpContentTypeList | [application/octet-stream, application/octet-stream]
httpCookieList | []
httpDNTList | []
httpDateList | []
httpExpectList | []
httpExpiresList | []
httpFromList | []
httpGetList | [POST /]
httpHostList | [pr.x.hub.sandai.net:80]
httpIMEIList | []
httpIMSIList | []
httpLastModifiedList | []
httpLocationList | []
httpMSISDNList | []
httpProxyAuthenticationList | []
httpRefererList | []
httpRefreshList | []
httpResponseList | [200 OK]
httpServerStringList | []
httpSetCookieList | []
httpSubscriberList | []
httpUpgradeList | []
httpUserAgentList | []
httpVersionList | [HTTP/1.1, HTTP/1.1]
httpViaList | []
httpWarningList | []
httpXDeviceIdList | []
httpXForwardedForList | []
httpXForwardedHostList | []
httpXForwardedProtoList | []
httpXForwardedServerList | []
httpXProfileList | []
httpXUaCompatibleList | []
………………
reverseStandardDeviationPayloadLength | null
tcpSequenceNumber | 2466486182
reverseTcpSequenceNumber | 472507334
ndpiL7Protocol | null
ndpiL7SubProtocol | null
mplsTopLabelStackSection | null
mplsLabelStackSection2 | null
mplsLabelStackSection3 | null
yafFlowKeyHash | 3065966287
only showing top 1 row到了這里，關(guān)于CENTOS上的網(wǎng)絡(luò)安全工具（二十六）SPARK+NetSA Security Tools容器化部署(2)的文章就介紹完了。如果您還想了解更多內(nèi)容，請(qǐng)?jiān)谟疑辖撬阉鱐OY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！
