寫到最前

? ? ? ?這段本想寫到最后的，但是哥們?yōu)榱硕帱c粉絲把這個文章設(shè)置成了粉絲可見，所以在最前面這段我想談一下我本人對當前國產(chǎn)信創(chuàng)要求以及ARM架構(gòu)服務(wù)器的看法。

? ? ? ? 總的來說我對國產(chǎn)信創(chuàng)ARM服務(wù)器這條技術(shù)路線持絕對樂觀的態(tài)度。首先從服務(wù)器CPU算力來說目前鯤鵬CPU性能和迭代水平雖然和X86架構(gòu)的CPU還有點差距但是我認為這個差距是段時間可以彌補的，另外從我的運維經(jīng)驗來說大部分客戶場景下其實對CPU的使用率都不高，也就是說現(xiàn)在ARM架構(gòu)的CPU基本完全能滿足大部分客戶的算力需求。從這次Mysql Sysbench測試來看兩顆kunpeng920 48核心CPU的服務(wù)器在10張1000萬數(shù)據(jù)的表下通過sysbench的oltp_read_write.lua來測試結(jié)果tps約1.6萬qps約33萬（最終性能瓶頸CPU，IOSP約5000，內(nèi)存沒跑滿），并且我不是搞數(shù)據(jù)庫的，我認為經(jīng)過專業(yè)DBA調(diào)試后這個數(shù)還可以更高，我在X86的測試環(huán)境中也測試過，基本和這個數(shù)據(jù)差不了多少。

? ? ? ? 目前ARM架構(gòu)下主要的阻力是中間件和應(yīng)用軟件，當前從CPU、虛擬化平臺、操作系統(tǒng)、存儲各個層面對ARM架構(gòu)支持都是到位的，只要業(yè)務(wù)廠商進行軟件適配，這個架構(gòu)絕對可以滿足絕大部分的客戶的需求。

????????我以前做過PowerLinux的推廣，當前的ARM Linux可比當年的PowerLinux的處境好多了，有政策的支持、有廠商的投入、有更好的生態(tài)，所以我認為這條技術(shù)路線絕對是可以落地并發(fā)展的。

????????

1. 背景

????????最近遇到一個客戶想買服務(wù)器，但是在信創(chuàng)的大環(huán)境下底層硬件只有兩條路線選擇，一條是海光芯片為基礎(chǔ)的超融合架構(gòu)，另一種是ARM芯片為基礎(chǔ)的傳統(tǒng)服務(wù)器加存儲架構(gòu)?？蛻粜枰獙煞N架構(gòu)進行壓力測試進行比對，測試的方案定為Sysbench壓力測試軟件對FileIO和Mysql進行壓力測試。

????????本次我負責ARM芯片基礎(chǔ)架構(gòu)的測試，測試使用一臺天宮服務(wù)器，配置為2顆Kunpeng 920-4826（48核心2600 MHz頻率）內(nèi)存128G，存儲使用華為OceanstorDorado5600V6核心配置為1TBCache，11塊3.8T NVME SSD Palm硬盤，RAID-TP，1塊熱備。服務(wù)器通過兩塊8GHBA卡直連存儲。

1. 操作系統(tǒng)以及配置

?操作系統(tǒng)

由于ARM的環(huán)境在客戶現(xiàn)場，本次文檔記錄的環(huán)境我使用X86架構(gòu)的服務(wù)器進行重新演示與操作。核心軟件Sysbench與Mysql還有nmon使用源碼安裝，操作系統(tǒng)使用OpenEuler22.03SPH2。

????????

??????? Open Euler系統(tǒng)下載鏈接：

DownloadopenEuler is an open source, free Linux distribution platform. The platform provides an open community for global developers to build an open, diversified, and architecture-inclusive software ecosystem. openEuler is also an innovative platform that encourages everyone to propose new ideas, explore new approaches, and practice new solutions.https://www.openeuler.org/en/download/?version=openEuler%2022.03%20LTS%20SP2

操作系統(tǒng)安裝和IP地址配置此處省略，測試環(huán)境最好把selinux和firewalld關(guān)閉

系統(tǒng)參數(shù)優(yōu)化部分按照自身的環(huán)境具體分析，本次測試環(huán)境主要是修改了最大打開文件數(shù)這個參數(shù)，其他的設(shè)置基本沒動。

vi /etc/sysctl.conf

fs.file-max=65535

Yum源配置

配置YUM源這里也不說了，公網(wǎng)源我比較喜歡清華大學(xué)的yum源

#generic-repos is licensed under the Mulan PSL v2.

#You can use this software according to the terms and conditions of the Mulan PSL v2.

#You may obtain a copy of Mulan PSL v2 at:

#??? http://license.coscl.org.cn/MulanPSL2

#THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR

#IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR

#PURPOSE.

#See the Mulan PSL v2 for more details.

[OS]

name=OS

baseurl=https://mirrors.tuna.tsinghua.edu.cn/openeuler/openEuler-22.03-LTS/OS/$basearch/

enabled=1

gpgcheck=1

gpgkey=https://mirrors.tuna.tsinghua.edu.cn/openeuler/openEuler-22.03-LTS/OS/$basearch/RPM-GPG-KEY-openEuler

[everything]

name=everything

baseurl=https://mirrors.tuna.tsinghua.edu.cn/openeuler/openEuler-22.03-LTS/everything/$basearch/

enabled=1

gpgcheck=1

gpgkey=https://mirrors.tuna.tsinghua.edu.cn/openeuler/openEuler-22.03-LTS/everything/$basearch/RPM-GPG-KEY-openEuler

[EPOL]

name=EPOL

baseurl=https://mirrors.tuna.tsinghua.edu.cn/openeuler/openEuler-22.03-LTS/EPOL/main/$basearch/

enabled=1

gpgcheck=1

gpgkey=https://mirrors.tuna.tsinghua.edu.cn/openeuler/openEuler-22.03-LTS/OS/$basearch/RPM-GPG-KEY-openEuler

[debuginfo]

name=debuginfo

baseurl=https://mirrors.tuna.tsinghua.edu.cn/openeuler/openEuler-22.03-LTS/debuginfo/$basearch/

enabled=1

gpgcheck=1

gpgkey=https://mirrors.tuna.tsinghua.edu.cn/openeuler/openEuler-22.03-LTS/debuginfo/$basearch/RPM-GPG-KEY-openEuler

[source]

name=source

baseurl=https://mirrors.tuna.tsinghua.edu.cn/openeuler/openEuler-22.03-LTS/source/

enabled=1

gpgcheck=1

gpgkey=https://mirrors.tuna.tsinghua.edu.cn/openeuler/openEuler-22.03-LTS/source/RPM-GPG-KEY-openEuler

[update]

name=update

baseurl=https://mirrors.tuna.tsinghua.edu.cn/openeuler/openEuler-22.03-LTS/update/$basearch/

enabled=1

gpgcheck=1

gpgkey=https://mirrors.tuna.tsinghua.edu.cn/openeuler/openEuler-22.03-LTS/OS/$basearch/RPM-GPG-KEY-openEuler

?安裝多路徑軟件：

UltraPaht下載鏈接：

華為 UltraPath 軟件下載和補丁升級 - 華為UltraPath閃存存儲：包括軟件下載、補丁升級、HedEx文檔、配置手冊、案例、工具、產(chǎn)品公告、視頻、論壇等內(nèi)容。https://support.huawei.com/enterprise/zh/flash-storage/ultrapath-pid-8576127/software

由于本次操作系統(tǒng)使用OpenEuler 22.03在測試時（2023年12月）UltraPaht軟件還沒有對OpenEuler 22.03支持，所以測試使用的是Linux默認的Multipath?，安裝就不說了，這里貼一下華為存儲關(guān)于Multipath?的配置：

devices {

????? device {

?????????????? vendor????????????????? "HUAWEI"

?????????????? product???????????????? "XSG1"

?????????????? path_grouping_policy??? multibus

?????????????? path_checker??????????? tur

?????????????? prio??????????????????? const

?????????????? path_selector?????????? "round-robin 0"

?????????????? failback??????????????? immediate

?????????????? dev_loss_tmo??????????? 30

?????????????? fast_io_fail_tmo??????? 5

?????? ????????no_path_retry?????????? 15

}

}

華為Dorado 5600V6存儲關(guān)于linux Multipath?配置的相關(guān)鏈接：

https://support.huawei.com/enterprise/zh/doc/EDOC1100112792/814d66d3

?

NMON監(jiān)控工具

源碼安裝nmon監(jiān)控工具：

nmon監(jiān)控工具是IBM為AIX研發(fā)的系統(tǒng)監(jiān)控工具，可以通過源碼安裝的形式移植到Linux平臺，源碼下載地址：

nmon and njmon | Site / CompilingNmon https://nmon.sourceforge.io/pmwiki.php?n=Site.CompilingNmon

編輯makefile增加如下兩行，并修改c文件的名稱

nmon_ARM_euler2203: lmon16p.c

??????? cc -o nmon_ARM_euler2203 $(FILE) $(CFLAGS) $(LDFLAGS) -D ARM

安裝編譯軟件和相關(guān)軟件

yum install make gcc gcc-c++ ncurses-dev*

執(zhí)行make編譯

make

測試軟件是否安裝成功

./nmon_ARM_euler2203

新建軟連接給系統(tǒng)添加nmon命令

ln -s /root/nmon/nmon_ARM_euler2203? /usr/bin/nmon

?Mysql

源碼安裝mysql：

Mysql源碼下載地址：MySQL :: Download MySQL Community Server

下載boost軟件，編譯mysql時需要用到

https://boostorg.jfrog.io/artifactory/main/release/1.77.0/source/boost_1_77_0.tar.bz2

安裝依賴工具：

yum install? gcc make gcc-c++ libxml2 libxml2-devel openssl openssl-devel bzip2 bzip2-devel curl-devel postgresql-devel libpng libjpeg-devel libjpeg libpng-devel freetype freetype-devel libicu-devel? cmake ncurses-devel libzip ?libtirpc-devel ?rpcgen? --skip-broken

里邊有些沖突使用--skip-broken調(diào)過沖突軟件

創(chuàng)建mysql用戶和用戶組

groupadd -r mysql

useradd? -g mysql -s /sbin/nologin mysql

創(chuàng)建mysql安裝目錄和數(shù)據(jù)目錄：

mkdir /data/mysql -p

mkdir /data/data

chown -R mysql:mysql /data/

進入mysql解壓后的目錄執(zhí)行預(yù)編譯命令：

cmake . -DCMAKE_INSTALL_PREFIX=/data/mysql -DMYSQL_DATADIR=/data/data/ -DSYSCONFDIR=/etc -DMYSQL_USER=mysql -DMYSQL_TCP_PORT=3306 -DWITHOUT_TOKUDB=1 -DMYSQL_UNIX_ADDR=/tmp/mysql.sock -DDEFAULT_CHARSET=utf8 -DDEFAULT_COLLATION=utf8_general_ci?? -DFORCE_INSOURCE_BUILD=1?? -DDOWNLOAD_BOOST=1 -DWITH_BOOST=/root/mysql/boost_1_77_0.tar.bz2

libtirpc-devel?? rpcgen? 這兩個軟件本地yum源中估計沒有需要提前下載好過程中可能會報如下錯誤，到時候手動rpm或者yum安裝好即可

預(yù)編譯完成：

執(zhí)行mak命令

這里使用的是make -j ?如果不加j參數(shù)是單線程，加入j參數(shù)就是調(diào)用所有cpu進行編譯速度會快很多，如果不想使用全部CPU可以在j后加數(shù)字例如? make? -j? 4 ?這樣就是使用4顆CPU進行編譯

Make完成：

執(zhí)行make install 安裝

安裝完成：

配置mysql服務(wù)和開機自動啟動：

cp /data/mysql/support-files/mysql.server /etc/rc.d/init.d/mysql

systemctl daemon-reload?

systemctl status mysql?

systemctl enable mysql

修改my.cnf,對mysql進行配置：

[client]

[mysqld]

port = 3306

socket = /tmp/mysql.sock

log_error = /var/log/mysql-error.log

datadir = /data/data

user = mysql

max_connections = 8000

lower_case_table_names = 1

innodb_buffer_pool_size = 208G

innodb_page_size = 8192

innodb_buffer_pool_instances = 6

innodb_file_per_table = OFF

sort_buffer_size = 1M

#innodb_flush_method = O_DIRECT

innodb_io_capacity = 4000000

innodb_page_cleaners = 4

#innodb_fast_shutdown = 0

innodb_flush_neighbors = 0

#innodb_log_group_home_dir = /data/iblogs

innodb_log_files_in_group = 3

innodb_log_file_size = 1024M

innodb_log_buffer_size = 100M

innodb_flush_log_at_trx_commit = 0

innodb_write_io_threads = 48

innodb_read_io_threads = 48

innodb_thread_concurrency = 0

tmpdir = /tmp

max_prepared_stmt_count = 99999

sync_binlog = 1000

[mysqldump]

參數(shù)說明：

log_error = /var/log/mysql-error.log?????? #mysql錯誤日志，啟動不了可以看這個地方

datadir = /data/data????????????????????????????????????????????? #mysql數(shù)據(jù)存放位置

max_connections = 8000?????????????? #最大TCP鏈接輸

innodb_buffer_pool_size = 208G????????????????????? #關(guān)鍵參數(shù)innodb緩存大小設(shè)置成內(nèi)存的60-80%

innodb_page_size = 8192??????????????????????????????????? #緩存頁大小

innodb_buffer_pool_instances = 6??????? #緩存實例 設(shè)置成4-8

#innodb_flush_method = O_DIRECT????? #設(shè)置成O_DIRECT時數(shù)據(jù)落盤不經(jīng)過操作系統(tǒng)內(nèi)存

innodb_io_capacity = 4000000????????????????????????? #存儲的IOPS

innodb_page_cleaners = 4????????????????????????????????? #內(nèi)存臟頁刷新線程數(shù)

innodb_flush_neighbors = 0?????????????????????????????? #內(nèi)存臟頁刷新時帶不帶隔壁，對于傳統(tǒng)機械硬盤建議使用1開啟，而對于固態(tài)硬盤可以使用0關(guān)閉

#innodb_log_group_home_dir = /data/iblogs

innodb_log_files_in_group = 3????????????????????????? #比較關(guān)鍵參數(shù)事務(wù)日志個數(shù)

innodb_log_file_size = 1024M??????????????????????????? #比較關(guān)鍵參數(shù)事務(wù)日志大小，本配置中當事務(wù)日志達到3個1G時會觸發(fā)回收

innodb_log_buffer_size = 100M

innodb_flush_log_at_trx_commit = 0????????????? #關(guān)鍵參數(shù)值為0 : 提交事務(wù)的時候，不立即把 redo log buffer 里的數(shù)據(jù)刷入磁盤文件的，而是依靠 InnoDB 的主線程每秒執(zhí)行一次刷新到磁盤。此時可能你提交事務(wù)了，結(jié)果 mysql 宕機了，然后此時內(nèi)存里的數(shù)據(jù)全部丟失。值為1 : 提交事務(wù)的時候，就必須把 redo log 從內(nèi)存刷入到磁盤文件里去，只要事務(wù)提交成功，那么 redo log 就必然在磁盤里了。注意，因為操作系統(tǒng)的“延遲寫”特性，此時的刷入只是寫到了操作系統(tǒng)的緩沖區(qū)中，因此執(zhí)行同步操作才能保證一定持久化到了硬盤中。值為2 : 提交事務(wù)的時候，把 redo 日志寫入磁盤文件對應(yīng)的 os cache 緩存里去，而不是直接進入磁盤文件，可能 1 秒后才會把 os cache 里的數(shù)據(jù)寫入到磁盤文件里去。只有1才能真正地保證事務(wù)的持久性，但是由于刷新操作 fsync() 是阻塞的，直到完成后才返回，我們知道寫磁盤的速度是很慢的，因此 MySQL 的性能會明顯地下降。如果不在乎事務(wù)丟失，0和2能獲得更高的性能。

#sync_binlog = 1000????????????????????????????? #比較關(guān)鍵參數(shù)sync_binlog=0 的時候，表示每次提交事務(wù)都只 write，不 fsync；sync_binlog=1 的時候，表示每次提交事務(wù)都會執(zhí)行 fsync；sync_binlog=N(N>1) 的時候，表示每次提交事務(wù)都 write，但累積 N 個事務(wù)后才 fsync。因此，在出現(xiàn) IO 瓶頸的場景里，將 sync_binlog 設(shè)置成一個比較大的值，可以提升性能。在實際的業(yè)務(wù)場景中，考慮到丟失日志量的可控性，一般不建議將這個參數(shù)設(shè)成 0，比較常見的是將其設(shè)置為 100~1000 中的某個數(shù)值。但是，將 sync_binlog 設(shè)置為 N，對應(yīng)的風險是：如果主機發(fā)生異常重啟，會丟失最近 N 個事務(wù)的 binlog 日志。

innodb_write_io_threads = 48?????????????????????????? #比較關(guān)鍵參數(shù)數(shù)據(jù)庫寫時可以調(diào)用的線程建議設(shè)置成CPU核數(shù)的一半

innodb_read_io_threads = 48???????????????????????????????????? #比較關(guān)鍵參數(shù)數(shù)據(jù)庫讀時可以調(diào)用的線程建議設(shè)置成CPU核數(shù)的一半

innodb_thread_concurrency = 0?????????????????????? #比較關(guān)鍵參數(shù)數(shù)據(jù)庫可以使用的CPU核數(shù)，0是不限制

tmpdir = /tmp

#skip_ssl

max_prepared_stmt_count = 99999??????????????? #數(shù)據(jù)庫最大允許多少鏈接

執(zhí)行mysql初始化：

/data/mysql/bin/mysqld --initialize --basedir=/data/mysql --datadir=/data/data/ --lower-case-table-names=1 --user=mysql

查看root用戶的密碼：

tail -f /var/log/mysql-error.log

啟動mysql:

systemctl start mysql

添加mysql鏈接庫到系統(tǒng)

echo? /data/mysql/lib/? >>/etc/ld.so.conf

ldconfig -v

修改PATH，可以使用mysql命令：

echo 'export? PATH=$PATH:/data/mysql/bin/' >> /etc/profile

mysql初始化：

mysql_secure_installation

Mysql 至此安裝配置完畢。

?

Sysbench

Sysbench下載地址：

GitHub - akopytov/sysbench: Scriptable database and system performance benchmarkScriptable database and system performance benchmark - GitHub - akopytov/sysbench: Scriptable database and system performance benchmarkhttps://github.com/akopytov/sysbench/

安裝automake和相關(guān)軟件：

yum install automake libtool libsysfs

進入sysbench解壓路徑執(zhí)行automake腳本：

./autogen.sh

Automake完成：

執(zhí)行configure配置編譯：

./configure --with-pgsql --with-mysql-includes=/data/mysql/include/ --with-mysql-libs=/data/mysql/lib/

配置完成：

執(zhí)行make：

make -j

Make完成：

執(zhí)行make install安裝：

安裝完成：

?Sysbench測試

Sysbench對File IO進行測試：?

具體測試的步驟和方法我不介紹了，可以查看：

File IO測試主要分為兩種，一種是模擬正常的Linux文件讀寫操作，另一種是使用O_Direct模式下直接把讀寫請求下發(fā)到存儲，區(qū)別就是讀寫數(shù)據(jù)的時候經(jīng)不經(jīng)過系統(tǒng)緩存，關(guān)鍵參數(shù)有如下幾個，原理見圖：

--file-extra-flags=direct

--file-fsync-end=off

--file-fsync-freq=0???

理論上來說O_Direct模式下可以測試出當前環(huán)境下存儲的真實的性能，IOPS和帶寬。

本次測試以8K塊大小隨機讀寫性能為基準對存儲進行壓力測試。

測試環(huán)境Intel+Vmware+Huawei Oceanstor 5510V5環(huán)境測試結(jié)果：

sysbench fileio --file-num=20 --file-total-size=20G? --threads=16? prepare

sysbench? --report-interval=5 fileio --file-num=20 --file-total-size=20G --file-test-mode=rndrd --file-block-size=8K --file-extra-flags=direct --file-fsync-end=off --file-fsync-freq=0 --threads=128 --time=30? run

sysbench 1.1.0 (using bundled LuaJIT 2.1.0-beta3)

Running the test with following options:

Number of threads: 128

Report intermediate results every 5 second(s)

Initializing random number generator from current time

Extra file open flags: directio

20 files, 1GiB each

20GiB total file size

Block size 8KiB

Number of IO requests: 0

Read/Write ratio for combined random IO test: 1.50

Using synchronous I/O mode

Doing random read test

Initializing worker threads...

Threads started!

[ 5s ] reads: 24.02 MiB/s writes: 0.00 MiB/s fsyncs: 0.00/s latency (ms,95%): 137.350

[ 10s ] reads: 23.72 MiB/s writes: 0.00 MiB/s fsyncs: 0.00/s latency (ms,95%): 189.929

[ 15s ] reads: 28.08 MiB/s writes: 0.00 MiB/s fsyncs: 0.00/s latency (ms,95%): 167.437

[ 20s ] reads: 29.38 MiB/s writes: 0.00 MiB/s fsyncs: 0.00/s latency (ms,95%): 207.823

[ 25s ] reads: 29.94 MiB/s writes: 0.00 MiB/s fsyncs: 0.00/s latency (ms,95%): 196.894

[ 30s ] reads: 65.47 MiB/s writes: 0.00 MiB/s fsyncs: 0.00/s latency (ms,95%): 54.828

Throughput:

???????? read:? IOPS=4202.53 32.83 MiB/s (34.43 MB/s)

???????? write: IOPS=0.00 0.00 MiB/s (0.00 MB/s)

???????? fsync: IOPS=0.00

Latency (ms):

???????? min:????????????????????????????????? 0.29

???????? avg:???????????????????????????????? 30.07

???????? max:?????????????????????????????? 2297.32

???????? 95th percentile:??????????????????? 164.45

???????? sum:??????????????????????????? 3865105.81

sysbench? --report-interval=5 fileio --file-num=20 --file-total-size=20G --file-test-mode=rndwr --file-block-size=8K --file-extra-flags=direct --file-fsync-end=off --file-fsync-freq=0 --threads=128? run

sysbench 1.1.0 (using bundled LuaJIT 2.1.0-beta3)

Running the test with following options:

Number of threads: 128

Report intermediate results every 5 second(s)

Initializing random number generator from current time

Extra file open flags: directio

20 files, 1GiB each

20GiB total file size

Block size 8KiB

Number of IO requests: 0

Read/Write ratio for combined random IO test: 1.50

Using synchronous I/O mode

Doing random write test

Initializing worker threads...

Threads started!

[ 5s ] reads: 0.00 MiB/s writes: 125.69 MiB/s fsyncs: 0.00/s latency (ms,95%): 9.560

Throughput:

???????? read:? IOPS=0.00 0.00 MiB/s (0.00 MB/s)

???????? write: IOPS=16242.83 126.90 MiB/s (133.06 MB/s)

???????? fsync: IOPS=0.00

Latency (ms):

???????? min:????????????????????????????????? 3.09

???????? avg:????????????????????????????????? 7.88

???? ????max:??????????????????????????????? 444.62

???????? 95th percentile:????????????????????? 9.56

???????? sum:??????????????????????????? 1279924.48

結(jié)果分析：

讀的流量和IOPS數(shù)據(jù)為34.43 MB/s和4202.53

寫的流量和IOPS數(shù)據(jù)為133.06 MB/s 和16242.83

原因是測試環(huán)境中使用的存儲是Huawei Oceanstor5510V5混閃存儲，配置的是SSD+NL-sas硬盤，開啟了數(shù)據(jù)分層，由于我的測試環(huán)境讀寫流量不是連續(xù)觸發(fā)不了存儲的數(shù)據(jù)分層（存儲監(jiān)控15分鐘內(nèi)熱點數(shù)據(jù)放到SSD層），也就是這次測試的讀寫數(shù)據(jù)基本都落在了NL_SAS層，隨機讀由于隨機所以緩存命中率低，數(shù)據(jù)是真正從NL_SAS層一個一個找出來的，所以IOPS相對較小，隨機寫的時候數(shù)據(jù)首先會進入到存儲的緩存然后再進行異步的落盤操作，所以隨機寫的IOPS會高。如果我連續(xù)測試一段時間觸發(fā)了存儲的數(shù)據(jù)分層功能我認為IOPS數(shù)據(jù)會比當前測試出的數(shù)據(jù)高很多，尤其是隨機讀。

另外，在進行Sysbench時線程數(shù)也比較重要，判斷系統(tǒng)讀寫壓力是否已經(jīng)是最大的最簡單的辦法就是通過nmon工具查看CPU情況，當出現(xiàn)下圖這種情況基本就屬于壓力較大，當CPU還有空閑的情況時就需要考慮是否要增加線程

客戶ARM+Huawei Dorado5600V6環(huán)境測試結(jié)果：

隨機direct模式讀：（8K塊大小，1個文件20G）?? IOPS（187934）

sysbench? --report-interval=5 fileio --file-num=1 --file-total-size=20G --file-test-mode=rndrd --file-block-size=8K --file-extra-flags=direct --file-fsync-end=off --file-fsync-freq=0 --threads=128? run

sysbench 1.1.0 (using bundled LuaJIT 2.1.0-beta3)

Running the test with following options:

Number of threads: 128

Report intermediate results every 5 second(s)

Initializing random number generator from current time

Extra file open flags: directio

1 files, 20GiB each

20GiB total file size

Block size 8KiB

Number of IO requests: 0

Read/Write ratio for combined random IO test: 1.50

Using synchronous I/O mode

Doing random read test

Initializing worker threads...

Threads started!

[ 5s ] reads: 1468.34 MiB/s writes: 0.00 MiB/s fsyncs: 0.00/s latency (ms,95%): 1.142

[ 10s ] reads: 1468.20 MiB/s writes: 0.00 MiB/s fsyncs: 0.00/s latency (ms,95%): 1.163

Throughput:

???????? read:? IOPS=187934.67 1468.24 MiB/s (1539.56 MB/s)

???????? write: IOPS=0.00 0.00 MiB/s (0.00 MB/s)

???????? fsync: IOPS=0.00

Latency (ms):

???????? min:????????????????????????????????? 0.26

???????? avg:????????????????????????????????? 0.68

???????? max:????????????????????????????????? 9.40

???????? 95th percentile:????????????????????? 1.16

???????? sum:??????????????????????????? 1278010.57

隨機direct模式寫：（8K塊大小，1個文件20G）?? IOPS（187934）

sysbench? --report-interval=5 fileio --file-num=1 --file-total-size=20G --file-test-mode=rndwr --file-block-size=8K --file-extra-flags=direct --file-fsync-end=off --file-fsync-freq=0 --threads=128? run

sysbench 1.1.0 (using bundled LuaJIT 2.1.0-beta3)

Running the test with following options:

Number of threads: 128

Report intermediate results every 5 second(s)

Initializing random number generator from current time

Extra file open flags: directio

1 files, 20GiB each

20GiB total file size

Block size 8KiB

Number of IO requests: 0

Read/Write ratio for combined random IO test: 1.50

Using synchronous I/O mode

Doing random write test

Initializing worker threads...

Threads started!

[ 5s ] reads: 0.00 MiB/s writes: 1182.27 MiB/s fsyncs: 0.00/s latency (ms,95%): 1.205

[ 10s ] reads: 0.00 MiB/s writes: 1186.01 MiB/s fsyncs: 0.00/s latency (ms,95%): 1.163

Throughput:

???????? read:? IOPS=0.00 0.00 MiB/s (0.00 MB/s)

???????? write: IOPS=151560.62 1184.07 MiB/s (1241.58 MB/s)

???????? fsync: IOPS=0.00

Latency (ms):

???????? min:????????????????????????????????? 0.20

???????? avg:????????????????????????????????? 0.84

???????? max:???????????????????????????????? 14.72

???????? 95th percentile:????????????????????? 1.18

???????? sum:??????????????????????????? 1278377.38

結(jié)果分析：

讀的流量和IOPS分別為1539.56 MB/s（這個數(shù)接近兩個8GBHBA卡的理論值速度說明這個測試結(jié)果這里可能有瓶頸隨機讀的IOPS應(yīng)該比后邊的18萬7還要高，我在2K和4K的時候測出過20萬上下的IOPS）和187934.67。

寫的流量和IOPS分別為1241.58 MB/s和151560.62也就是隨機寫的IOPS在15萬上下。

PS: 20萬和15萬隨機讀寫的IOPS并不是這個存儲的極限值，聽同事說他們測試出的結(jié)果比這個20萬和15萬要大很多（40萬上下），可能是因為我的終端少的原因LUN少的原因，真正要測試到存儲性能極限應(yīng)該是多客戶的單文件系統(tǒng)多LUN場景。

Sysbench對Mysql壓力測試：

創(chuàng)建test1用戶：

create user 'test1'@'%' identified with mysql_native_password by 'admin@123';

GRANT ALL PRIVILEGES ON *.* TO 'test1'@'%';

Mysql新建測試庫：

create database test_db;

Sysbench數(shù)據(jù)庫測試10張1000萬的表，主要測試oltp_read_write.lua

新建10張1000萬的表：

sysbench --report-interval=3 /usr/local/share/sysbench/oltp_common.lua --mysql-host=127.0.0.1 --mysql-port=3306 --mysql-user=test1 --mysql-password=admin@123 --mysql-db=test_db --tables=10 --table-size=10000000? --threads=96? prepare

SELECT COUNT(*) FROM sbtest10;

64線程OLTP_rad_wirte測試? tps約2800?? ?qps約5.8萬

sysbench --report-interval=3? /usr/local/share/sysbench/oltp_read_write.lua --mysql-host=127.0.0.1 --mysql-port=3306 --mysql-user=test1 --mysql-password=admin@123 --mysql-db=test_db? --threads=64 --time=30 run

sysbench 1.1.0 (using bundled LuaJIT 2.1.0-beta3)

Running the test with following options:

Number of threads: 64

Report intermediate results every 3 second(s)

Initializing random number generator from current time

Initializing worker threads...

Threads started!

[ 3s ] thds: 64 tps: 2414.15 qps: 48545.07 (r/w/o: 34006.89/9689.23/4848.95) lat (ms,95%): 56.84 err/s: 0.00 reconn/s: 0.00

[ 6s ] thds: 64 tps: 2774.61 qps: 55472.89 (r/w/o: 38822.56/11101.45/5548.89) lat (ms,95%): 41.85 err/s: 0.00 reconn/s: 0.00

[ 9s ] thds: 64 tps: 2852.91 qps: 57166.46 (r/w/o: 40027.69/11432.29/5706.48) lat (ms,95%): 38.25 err/s: 0.00 reconn/s: 0.00

[ 12s ] thds: 64 tps: 2981.06 qps: 59553.84 (r/w/o: 41698.16/11894.24/5961.45) lat (ms,95%): 34.33 err/s: 0.00 reconn/s: 0.00

[ 15s ] thds: 64 tps: 3015.30 qps: 60204.28 (r/w/o: 42106.83/12068.19/6029.26) lat (ms,95%): 38.25 err/s: 0.00 reconn/s: 0.00

[ 18s ] thds: 64 tps: 3002.94 qps: 60183.77 (r/w/o: 42160.47/12015.09/6008.21) lat (ms,95%): 34.95 err/s: 0.00 reconn/s: 0.00

[ 21s ] thds: 64 tps: 2949.97 qps: 59042.72 (r/w/o: 41321.57/11821.21/5899.94) lat (ms,95%): 34.95 err/s: 0.00 reconn/s: 0.00

[ 24s ] thds: 64 tps: 3023.85 qps: 60381.25 (r/w/o: 42259.84/12074.05/6047.36) lat (ms,95%): 34.95 err/s: 0.00 reconn/s: 0.00

[ 27s ] thds: 64 tps: 2919.32 qps: 58432.00 (r/w/o: 40914.77/11678.27/5838.97) lat (ms,95%): 37.56 err/s: 0.00 reconn/s: 0.00

[ 30s ] thds: 64 tps: 2939.23 qps: 58713.62 (r/w/o: 41096.23/11740.59/5876.80) lat (ms,95%): 37.56 err/s: 0.00 reconn/s: 0.00

SQL statistics:

??? queries performed:

??????? read:?????????????????????? ?????1213660

??????? write:?????????????????????????? 346760

??????? other:?????????????????????????? 173380

??????? total:?????????????????????????? 1733800

??? transactions:??????????????????????? 86690? (2887.15 per sec.)

??? queries:?????????????????? ??????????1733800 (57742.96 per sec.)

??? ignored errors:????????????????????? 0????? (0.00 per sec.)

??? reconnects:????????????????????????? 0????? (0.00 per sec.)

Throughput:

??? events/s (eps):????????????????????? 2887.1478

??? time elapsed:??????????????????????? 30.0262s

??? total number of events:????????????? 86690

Latency (ms):

???????? min:??????????????????????????????????? 6.43

???????? avg:?????????????????????????????????? 22.15

???????? max:?????????????????????? ???????????261.41

???????? 95th percentile:?????????????????????? 38.94

???????? sum:????????????????????????????? 1920616.04

Threads fairness:

??? events (avg/stddev):?????????? 1354.5312/11.44

??? execution time (avg/stddev):?? 30.0096/0.00

128線程OLTP_rad_wirte測試? tps約3000?? ?qps約6.2萬

sysbench --report-interval=3? /usr/local/share/sysbench/oltp_read_write.lua --mysql-host=127.0.0.1 --mysql-port=3306 --mysql-user=test1 --mysql-password=admin@123 --mysql-db=test_db? --threads=128 --time=30 run

sysbench 1.1.0 (using bundled LuaJIT 2.1.0-beta3)

Running the test with following options:

Number of threads: 128

Report intermediate results every 3 second(s)

Initializing random number generator from current time

Initializing worker threads...

Threads started!

[ 3s ] thds: 128 tps: 2852.15 qps: 57714.30 (r/w/o: 40443.57/11525.81/5744.92) lat (ms,95%): 84.47 err/s: 0.00 reconn/s: 0.00

[ 6s ] thds: 128 tps: 3147.85 qps: 62967.95 (r/w/o: 44084.86/12585.39/6297.69) lat (ms,95%): 69.29 err/s: 0.00 reconn/s: 0.00

[ 9s ] thds: 128 tps: 3177.68 qps: 63611.67 (r/w/o: 44504.57/12752.73/6354.37) lat (ms,95%): 68.05 err/s: 0.00 reconn/s: 0.00

[ 12s ] thds: 128 tps: 3187.82 qps: 63741.04 (r/w/o: 44641.79/12722.95/6376.30) lat (ms,95%): 69.29 err/s: 0.00 reconn/s: 0.00

[ 15s ] thds: 128 tps: 3189.10 qps: 63685.27 (r/w/o: 44590.69/12717.05/6377.53) lat (ms,95%): 70.55 err/s: 0.00 reconn/s: 0.00

[ 18s ] thds: 128 tps: 3192.74 qps: 63926.05 (r/w/o: 44721.30/12819.28/6385.47) lat (ms,95%): 69.29 err/s: 0.00 reconn/s: 0.00

[ 21s ] thds: 128 tps: 3178.91 qps: 63541.63 (r/w/o: 44498.47/12684.66/6358.50) lat (ms,95%): 70.55 err/s: 0.00 reconn/s: 0.00

[ 24s ] thds: 128 tps: 3198.44 qps: 64075.53 (r/w/o: 44840.87/12837.44/6397.22) lat (ms,95%): 71.83 err/s: 0.00 reconn/s: 0.00

[ 27s ] thds: 128 tps: 3146.95 qps: 62864.71 (r/w/o: 44020.00/12551.14/6293.57) lat (ms,95%): 73.13 err/s: 0.00 reconn/s: 0.00

[ 30s ] thds: 128 tps: 3153.04 qps: 63103.44 (r/w/o: 44167.54/12629.82/6306.08) lat (ms,95%): 70.55 err/s: 0.00 reconn/s: 0.00

SQL statistics:

??? queries performed:

??????? read:??????????????????????????? 1321726

??????? write:?????????????????????????? 377636

??????? other:?????????????????????????? 188818

??????? total:?????????????????????????? 1888180

??? transactions:???????? ???????????????94409? (3141.64 per sec.)

??? queries:???????????????????????????? 1888180 (62832.82 per sec.)

??? ignored errors:????????????????????? 0????? (0.00 per sec.)

??? reconnects:????????????????????????? 0????? (0.00 per sec.)

Throughput:

??? events/s (eps):????????????????????? 3141.6409

??? time elapsed:??????????????????????? 30.0509s

??? total number of events:????????????? 94409

Latency (ms):

???????? min:??????????????????????????????????? 5.48

???????? avg:????????????????????????? ?????????40.69

???????? max:????????????????????????????????? 331.43

???????? 95th percentile:?????????????????????? 71.83

???????? sum:????????????????????????????? 3841561.01

Threads fairness:

??? events (avg/stddev):?????????? 737.5703/11.66

??? execution time (avg/stddev):?? 30.0122/0.01

256線程OLTP_rad_wirte測試? tps約3100?? ?qps約6.3萬

sysbench --report-interval=3? /usr/local/share/sysbench/oltp_read_write.lua --mysql-host=127.0.0.1 --mysql-port=3306 --mysql-user=test1 --mysql-password=admin@123 --mysql-db=test_db? --threads=256 --time=30 run

sysbench 1.1.0 (using bundled LuaJIT 2.1.0-beta3)

Running the test with following options:

Number of threads: 256

Report intermediate results every 3 second(s)

Initializing random number generator from current time

Initializing worker threads...

Threads started!

[ 3s ] thds: 256 tps: 2899.31 qps: 59378.73 (r/w/o: 41720.15/11775.72/5882.86) lat (ms,95%): 164.45 err/s: 0.00 reconn/s: 0.00

[ 6s ] thds: 256 tps: 3119.46 qps: 62484.27 (r/w/o: 43660.81/12584.53/6238.93) lat (ms,95%): 153.02 err/s: 0.00 reconn/s: 0.00

[ 9s ] thds: 256 tps: 3131.55 qps: 62724.31 (r/w/o: 43898.35/12562.20/6263.76) lat (ms,95%): 155.80 err/s: 0.00 reconn/s: 0.00

[ 12s ] thds: 256 tps: 3137.38 qps: 62512.35 (r/w/o: 43781.05/12459.54/6271.77) lat (ms,95%): 155.80 err/s: 0.00 reconn/s: 0.00

[ 15s ] thds: 256 tps: 3122.63 qps: 62485.62 (r/w/o: 43729.83/12509.19/6246.60) lat (ms,95%): 158.63 err/s: 0.00 reconn/s: 0.00

[ 18s ] thds: 256 tps: 3178.39 qps: 63726.76 (r/w/o: 44592.76/12775.22/6358.78) lat (ms,95%): 155.80 err/s: 0.00 reconn/s: 0.00

[ 21s ] thds: 256 tps: 3212.72 qps: 64082.44 (r/w/o: 44904.11/12754.55/6423.78) lat (ms,95%): 147.61 err/s: 0.00 reconn/s: 0.00

[ 24s ] thds: 256 tps: 3230.99 qps: 64863.03 (r/w/o: 45356.12/13043.27/6463.64) lat (ms,95%): 144.97 err/s: 0.00 reconn/s: 0.00

[ 27s ] thds: 256 tps: 3250.30 qps: 64906.97 (r/w/o: 45482.18/12924.19/6500.60) lat (ms,95%): 153.02 err/s: 0.00 reconn/s: 0.00

[ 30s ] thds: 256 tps: 3210.30 qps: 64200.43 (r/w/o: 44905.61/12875.22/6419.61) lat (ms,95%): 150.29 err/s: 0.00 reconn/s: 0.00

SQL statistics:

??? queries performed:

??????? read:??????????????????????????? 1326374

??????? write:?????????????????????????? 378964

??????? other:?????????????????????????? 189482

??????? total:??? ???????????????????????1894820

??? transactions:??????????????????????? 94741? (3148.91 per sec.)

??? queries:???????????????????????????? 1894820 (62978.13 per sec.)

??? ignored errors:????????????????????? 0????? (0.00 per sec.)

??? reconnects:????????????????????????? 0????? (0.00 per sec.)

Throughput:

??? events/s (eps):????????????????????? 3148.9064

??? time elapsed:??????????????????????? 30.0870s

??? total number of events:????????????? 94741

Latency (ms):

???????? min:?? ?????????????????????????????????6.56

???????? avg:?????????????????????????????????? 81.10

???????? max:????????????????????????????????? 416.75

???????? 95th percentile:????????????????????? 153.02

???????? sum:????????????????????????????? 7683421.86

Threads fairness:

??? events (avg/stddev):?????????? 370.0820/9.19

??? execution time (avg/stddev):?? 30.0134/0.01

系統(tǒng)壓力情況：

客戶ARM+Huawei Dorado5600V6環(huán)境測試結(jié)果：

128線程OLTP_rad_wirte測試? tps約1.5萬?? ?qps約30萬

sysbench --report-interval=3? /usr/local/share/sysbench/oltp_read_write.lua --mysql-host=127.0.0.1 --mysql-port=3306 --mysql-user=test1 --mysql-password=admin@123 --mysql-db=test_db? --threads=128 --time=30 run

sysbench 1.1.0 (using bundled LuaJIT 2.1.0-beta3)

Running the test with following options:

Number of threads: 128

Report intermediate results every 3 second(s)

Initializing random number generator from current time

Initializing worker threads...

Threads started!

[ 3s ] thds: 128 tps: 15166.66 qps: 304070.71 (r/w/o: 212913.86/60780.89/30375.95) lat (ms,95%): 13.46 err/s: 0.00 reconn/s: 0.00

[ 6s ] thds: 128 tps: 15603.67 qps: 311997.68 (r/w/o: 218398.34/62392.00/31207.33) lat (ms,95%): 11.45 err/s: 0.00 reconn/s: 0.00

[ 9s ] thds: 128 tps: 15337.83 qps: 306682.99 (r/w/o: 214663.33/61344.66/30675.00) lat (ms,95%): 11.45 err/s: 0.00 reconn/s: 0.00

[ 12s ] thds: 128 tps: 15525.01 qps: 310399.61 (r/w/o: 217271.86/62078.39/31049.36) lat (ms,95%): 11.45 err/s: 0.00 reconn/s: 0.00

[ 15s ] thds: 128 tps: 15302.99 qps: 305873.13 (r/w/o: 214109.52/61159.29/30604.31) lat (ms,95%): 12.08 err/s: 0.00 reconn/s: 0.00

[ 18s ] thds: 128 tps: 15213.24 qps: 304544.81 (r/w/o: 213194.03/60921.29/30429.48) lat (ms,95%): 12.30 err/s: 0.00 reconn/s: 0.00

[ 21s ] thds: 128 tps: 15258.80 qps: 305196.07 (r/w/o: 213659.59/61020.88/30515.61) lat (ms,95%): 12.08 err/s: 0.00 reconn/s: 0.00

[ 24s ] thds: 128 tps: 15058.19 qps: 301285.54 (r/w/o: 210859.38/60308.11/30118.05) lat (ms,95%): 12.30 err/s: 0.00 reconn/s: 0.00

[ 27s ] thds: 128 tps: 14756.63 qps: 295047.20 (r/w/o: 206570.44/58963.51/29513.25) lat (ms,95%): 13.22 err/s: 0.00 reconn/s: 0.00

[ 30s ] thds: 128 tps: 15142.77 qps: 302853.82 (r/w/o: 212006.84/60561.10/30285.88) lat (ms,95%): 12.30 err/s: 0.00 reconn/s: 0.00

SQL statistics:

??? queries performed:

??????? read:??????????????????????????? 6401584

??????? write:?????????????????????????? 1829024

??????? other:?????????????????????????? 914512

??????? total:?????????????????????????? 9145120

??? transactions:??????????????????????? 457256 (15225.84 per sec.)

??? queries:???????????????????????????? 9145120 (304516.81 per sec.)

??? ignored errors:????????????????????? 0????? (0.00 per sec.)

??? reconnects:????????????????????????? 0????? (0.00 per sec.)

Throughput:

??? events/s (eps):????????????????????? 15225.8405

??? time elapsed:??????????????????????? 30.0316s

??? total number of events:????????????? 457256

Latency (ms):

???????? min:??????????????????????????????????? 3.52

???????? avg: ???????????????????????????????????8.40

???????? max:?????????????????????????????????? 74.67

???????? 95th percentile:?????????????????????? 12.08

???????? sum:????????????????????????????? 3838723.94

Threads fairness:

??? events (avg/stddev):?????????? 3572.3125/28.17

??? execution time (avg/stddev):?? 29.9900/0.00

256線程OLTP_rad_wirte測試? tps約1.6萬?? ?qps約33萬

?sysbench --report-interval=3? /usr/local/share/sysbench/oltp_read_write.lua --mysql-host=127.0.0.1 --mysql?????????????????? -port=3306 --mysql-user=test1 --mysql-password=admin@123 --mysql-db=test_db? --threads=256 --time=30 run

sysbench 1.1.0 (using bundled LuaJIT 2.1.0-beta3)

Running the test with following options:

Number of threads: 256

Report intermediate results every 3 second(s)

Initializing random number generator from current time

Initializing worker threads...

Threads started!

[ 3s ] thds: 256 tps: 11849.98 qps: 238251.40 (r/w/o: 166883.92/47586.28/23781.20) lat (ms,95%): 78.60 err/s: 0.00 reconn/s: 0.00

[ 6s ] thds: 256 tps: 16479.07 qps: 329688.11 (r/w/o: 230777.68/65949.29/32961.14) lat (ms,95%): 24.83 err/s: 0.00 reconn/s: 0.00

[ 9s ] thds: 256 tps: 16407.63 qps: 328197.56 (r/w/o: 229743.79/65637.84/32815.92) lat (ms,95%): 23.95 err/s: 0.33 reconn/s: 0.00

[ 12s ] thds: 256 tps: 16534.86 qps: 330813.88 (r/w/o: 231555.71/66188.11/33070.05) lat (ms,95%): 23.52 err/s: 0.00 reconn/s: 0.0?????????????????? 0

[ 15s ] thds: 256 tps: 16298.66 qps: 325849.48 (r/w/o: 228110.54/65141.97/32596.98) lat (ms,95%): 25.28 err/s: 0.00 reconn/s: 0.0?????????????????? 0

[ 18s ] thds: 256 tps: 16594.61 qps: 331902.50 (r/w/o: 232315.18/66398.10/33189.22) lat (ms,95%): 23.95 err/s: 0.00 reconn/s: 0.00

[ 21s ] thds: 256 tps: 16883.17 qps: 337453.62 (r/w/o: 236238.97/67449.32/33765.33) lat (ms,95%): 23.10 err/s: 0.00 reconn/s: 0.00

[ 24s ] thds: 256 tps: 16509.38 qps: 330344.93 (r/w/o: 231198.31/66127.52/33019.09) lat (ms,95%): 23.95 err/s: 0.00 reconn/s: 0.00

[ 27s ] thds: 256 tps: 16554.83 qps: 331111.18 (r/w/o: 231812.23/66188.64/33110.32) lat (ms,95%): 23.95 err/s: 0.00 reconn/s: 0.00

[ 30s ] thds: 256 tps: 16695.55 qps: 332540.55 (r/w/o: 232671.28/66563.18/33306.09) lat (ms,95%): 23.52 err/s: 0.00 reconn/s: 0.00

SQL statistics:

??? queries performed:

??????? read:??????????????????????????? 6756316

??????? write:?????????????????????????? 1930373

??????? other:?????????????????????????? 965187

??????? total:?????????????????????????? 9651876

??? transactions:??????????????????????? 482593 (16058.05 per sec.)

??? queries:????????????????????????? ???9651876 (321161.44 per sec.)

??? ignored errors:????????????????????? 1????? (0.03 per sec.)

??? reconnects:????????????????????????? 0????? (0.00 per sec.)

Throughput:

??? events/s (eps):????????????????????? 16058.0454

??? time elapsed:??????????????????????? 30.0530s

??? total number of events:????????????? 482593

Latency (ms):

???????? min:??????????????????????????????????? 3.06

???????? avg:?????????????????????????????????? 15.91

???????? max:????????????????????? ????????????475.14

???????? 95th percentile:?????????????????????? 24.83

???????? sum:????????????????????????????? 7677627.58

Threads fairness:

??? events (avg/stddev):?????????? 1885.1289/26.20

??? execution time (avg/stddev):?? 29.9907/0.01文章來源地址http://www.zghlxwxcb.cn/news/detail-768886.html

到了這里，關(guān)于ARM CPU架構(gòu)下Mysql數(shù)據(jù)庫通過Sysbench壓力測試的文章就介紹完了。如果您還想了解更多內(nèi)容，請在右上角搜索TOY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！