作者：盧文雙 資深數(shù)據(jù)庫(kù)內(nèi)核研發(fā)

序言：

以前對(duì) MySQL 測(cè)試框架 MTR 的使用，主要集中于 SQL 正確性驗(yàn)證。近期由于工作需要，深入了解了 MTR 的方方面面，發(fā)現(xiàn) MTR 的能力不僅限于此，還支持單元測(cè)試、壓力測(cè)試、代碼覆蓋率測(cè)試、內(nèi)存錯(cuò)誤檢測(cè)、線程競(jìng)爭(zhēng)與死鎖等功能，因此，本著分享的精神，將其總結(jié)成一個(gè)系列。

主要內(nèi)容如下：

• 入門篇：工作機(jī)制、編譯安裝、參數(shù)、指令示例、推薦用法、添加 case、常見(jiàn)問(wèn)題、異常調(diào)試
• 進(jìn)階篇：高階用法，包括單元測(cè)試、壓力測(cè)試、代碼覆蓋率測(cè)試、內(nèi)存錯(cuò)誤檢測(cè)、線程競(jìng)爭(zhēng)與死鎖
• 源碼篇：分析 MTR 的源碼
• 語(yǔ)法篇：?jiǎn)卧獪y(cè)試、壓力測(cè)試、mysqltest 語(yǔ)法、異常調(diào)試

由于個(gè)人水平有限，所述難免有錯(cuò)誤之處，望雅正。

本文是第二篇進(jìn)階篇。

本文首發(fā)于 2023-05-01 21:03:44

MTR 系列基于 MySQL 8.0.29 版本（編譯情況也在 8.0.32 版本驗(yàn)證過(guò)），且主要在 Ubuntu 22.04 X86_64 驗(yàn)證（部分指令也在 Ubuntu 20.04 X86_64、Ubuntu 22.04 ARM64、MacOS M1 做了驗(yàn)證），如有例外，會(huì)特別說(shuō)明。

前言

上一篇文章《MySQL 測(cè)試框架 MTR 系列教程（一）：入門篇》介紹了 mtr 的原理、目錄結(jié)構(gòu)、參數(shù)及常見(jiàn)用法，側(cè)重于最常見(jiàn)的 SQL 正確性驗(yàn)證，但 mtr 能做更多的事情，比如 內(nèi)存錯(cuò)誤、線程競(jìng)爭(zhēng)、代碼覆蓋率、壓力測(cè)試等，本文主要介紹這些內(nèi)容，涉及的相關(guān)工具如下：

• valgrind ：用于內(nèi)存調(diào)試、內(nèi)存泄漏檢測(cè)以及性能分析。
• Sanitizier ：谷歌發(fā)起的開(kāi)源工具集。
  • ASAN/AddressSanitizier ：檢查內(nèi)存地址相關(guān)問(wèn)題，包括內(nèi)存泄漏、釋放后使用、重復(fù)釋放、堆溢出、棧溢出等等問(wèn)題。
  • LSAN/LeakSanitizer ：檢查內(nèi)存泄漏問(wèn)題。它是集成在 Address Sanitizer 中的一個(gè)相對(duì)獨(dú)立的工具，它工作在檢查過(guò)程的最后階段。
  • MSAN/MemorySanitizer ： 檢查使用未初始化內(nèi)存的問(wèn)題。
  • TSAN/ThreadSanitizer ： 檢查線程數(shù)據(jù)競(jìng)爭(zhēng)和死鎖問(wèn)題。
  • UBSAN/UndefinedBehaviorSanitizer ： 檢測(cè)未定義行為（使用空指針、有符號(hào)整數(shù)溢出等）。
• gcov ： 代碼覆蓋率測(cè)試。
• gprof ： 性能分析工具。
• 單元測(cè)試
• 壓力測(cè)試

本文將逐一介紹對(duì)各個(gè)工具的支持情況。

補(bǔ)充：

MariaDB 已經(jīng)很好的支持了以上工具集：

Compile and Using MariaDB with Sanitizers (ASAN, UBSAN, TSAN, MSAN) - MariaDB Knowledge Base

Compiling MariaDB for Debugging - MariaDB Knowledge Base （支持 valgrind）

MySQL 編譯選項(xiàng)

首先說(shuō)明一下與本文相關(guān)的 MySQL 編譯選項(xiàng)：

• -DCMAKE_BUILD_TYPE=type

  The type of build to produce:

  • RelWithDebInfo: default value。Enable optimizations and generate debugging information. This is the default MySQL build type.
  • Release: Enable optimizations but omit debugging information to reduce the build size. This build type was added in MySQL 8.0.13 (MySQL 5.7 is not supported).
  • Debug: Disable optimizations and generate debugging information. This build type is also used if the WITH_DEBUG option is enabled. That is, -DWITH_DEBUG=1 has the same effect as -DCMAKE_BUILD_TYPE=Debug.

• -DWITH_DEBUG=bool

  Whether to include debugging support. The default isOFF.

• -DWITH_ASAN=bool

  Whether to enable the AddressSanitizer, for compilers that support it. The default is off.

• -DWITH_ASAN_SCOPE=bool

  Whether to enable the AddressSanitizer -fsanitize-address-use-after-scope Clang flag for use-after-scope detection. The default is off. To use this option, -DWITH_ASAN must also be enabled.

• -DWITH_LSAN=bool

  Whether to run LeakSanitizer, without AddressSanitizer. The default isOFF.

  This option was added in MySQL 8.0.16.

• -DWITH_MSAN=bool

  Whether to enable MemorySanitizer, for compilers that support it. The default is off.

  For this option to have an effect if enabled, all libraries linked to MySQL must also have been compiled with the option enabled.

• -DWITH_TSAN=bool

  Whether to enable the ThreadSanitizer, for compilers that support it. The default is off.

• -DWITH_UBSAN=bool

  Whether to enable the Undefined Behavior Sanitizer, for compilers that support it. The default is off.

• -DWITH_UNIT_TESTS={ON|OFF}

  If enabled, compile MySQL with unit tests. The default is ON unless the server is not being compiled.

• -DWITH_VALGRIND=bool

  Whether to compile in the Valgrind header files, which exposes the Valgrind API to MySQL code. The default isOFF.

  To generate a Valgrind-aware debug build, -DWITH_VALGRIND=1 normally is combined with -DWITH_DEBUG=1. See Building Debug Configurations.

• -DENABLE_GCOV=bool

  Whether to include gcov support (Linux only).

• -DENABLE_GPROF=bool

  Whether to enable gprof (optimized Linux builds only). The default isOFF.

• -DWITH_TEST_TRACE_PLUGIN=bool

  Whether to build the test protocol trace client plugin (see Using the Test Protocol Trace Plugin). By default, this option is disabled. Enabling this option has no effect unless the WITH_CLIENT_PROTOCOL_TRACING option is enabled. If MySQL is configured with both options enabled, the libmysqlclient client library is built with the test protocol trace plugin built in, and all the standard MySQL clients load the plugin. However, even when the test plugin is enabled, it has no effect by default. Control over the plugin is afforded using environment variables; see Using the Test Protocol Trace Plugin.

  Note

  Do not enable the WITH_TEST_TRACE_PLUGIN option if you want to use your own protocol trace plugins because only one such plugin can be loaded at a time and an error occurs for attempts to load a second one. If you have already built MySQL with the test protocol trace plugin enabled to see how it works, you must rebuild MySQL without it before you can use your own plugins.

  For information about writing trace plugins, see Writing Protocol Trace Plugins.

• -DWITH_CLIENT_PROTOCOL_TRACING=bool

  Whether to build the client-side protocol tracing framework into the client library. By default, this option is enabled.

  For information about writing protocol trace client plugins, see Writing Protocol Trace Plugins.

  See also the WITH_TEST_TRACE_PLUGIN option.

• -DWITH_KEYRING_TEST=bool

  Whether to build the test program that accompanies the keyring_file plugin. The default isOFF. Test file source code is located in the plugin/keyring/keyring-test directory.

• -DWITH_NDB_TEST={ON|OFF}

  If enabled, include a set of NDB API test programs. The default is OFF.

詳見(jiàn)：MySQL :: MySQL 8.0 Reference Manual :: 2.8.7 MySQL Source-Configuration Options

以下是各組件或測(cè)試類型的詳細(xì)介紹。

valgrind

簡(jiǎn)介

valgrind 是一個(gè)工具集，主要集成了：

• memcheck ：內(nèi)存錯(cuò)誤檢測(cè)器。
• cachegrind ：緩存和分支預(yù)測(cè)分析器。
• callgrind ：可生成緩存分析器的調(diào)用圖。
• helgrind ：線程錯(cuò)誤檢測(cè)器。
• DRD ：也是線程錯(cuò)誤檢測(cè)器。
• massif ：堆分析器，它可以幫助程序使用更少的內(nèi)存。
• DHAT ：一種不同類型的堆分析器。使用它可以了解塊壽命，塊利用率和布局效率低下的問(wèn)題。

選項(xiàng)

mtr 提供的 valgrind 選項(xiàng)如下：

Options for valgrind

  callgrind             Instruct valgrind to use callgrind.
  helgrind              Instruct valgrind to use helgrind.
  valgrind              Run the "mysqltest" and "mysqld" executables using
                        valgrind with default options.
  valgrind-all          Synonym for --valgrind.
  valgrind-clients      Run clients started by .test files with valgrind.
  valgrind-mysqld       Run the "mysqld" executable with valgrind.
  valgrind-mysqltest    Run the "mysqltest" and "mysql_client_test" executable
                        with valgrind.
  valgrind-option=ARGS  Option to give valgrind, replaces default option(s), can
                        be specified more then once.
  valgrind-options=ARGS Deprecated, use --valgrind-option.
  valgrind-path=<EXE>   Path to the valgrind executable.

從代碼看：

1. 支持的工具不僅限于 callgrind、helgrind，還支持 memcheck、massif 。
2. 只有啟用--valgrind 或 --valgrind-mysqltest 選項(xiàng)，才會(huì)用到 mysql_client_test 。

使用方法

編譯選項(xiàng)： -DWITH_DEBUG=1 -DWITH_VALGRIND=1

使用建議：

1、推薦指令可參考 mysql-test/collections/ 目錄下的文件 default.daily-valgrind、default.push-valgrind、default.weekly-valgrind 。

2、據(jù)我實(shí)測(cè)，如需測(cè)試 valgrind 所有功能，需在原有指令基礎(chǔ)上添加--valgrind選項(xiàng)。比如：

# 官方 collections 中的示例指令：
perl mysql-test-run.pl --timer  --force --skip-rpl --comment=all_default_valgrind --vardir=var-all_default_valgrind --skip-ndb

# 添加 --valgrind
perl mysql-test-run.pl --timer  --force --skip-rpl --comment=all_default_valgrind --vardir=var-all_default_valgrind --skip-ndb --valgrind

3、在同時(shí)啟用 ASAN 和 valgrind 的情況下，并在運(yùn)行 mtr 時(shí)添加--valgrind選項(xiàng)，mtr 會(huì)因 valgrind memcheck 與 asan 沖突而 crash，因此，valgrind 與 asan 不建議同時(shí)啟用。

Sanitizier

Sanitizers 是谷歌發(fā)起的開(kāi)源工具集，已經(jīng)成為靜態(tài)和動(dòng)態(tài)代碼分析的利器，可以檢查內(nèi)存錯(cuò)誤、未初始化的讀取、線程安全和未定義的行為等相關(guān)的問(wèn)題。

優(yōu)點(diǎn)：與同類型分析工具相比，Sanitizers 帶來(lái)的性能損失通常要小得多，而且往往提供的信息更詳細(xì)。

缺點(diǎn)：代碼（可能還有工具鏈的一部分）需要使用附加的標(biāo)志重新編譯。

Sanitizers 包括如下組件：

• AddressSanitizer/ASAN

  檢查內(nèi)存地址相關(guān)問(wèn)題，包括內(nèi)存泄漏、釋放后使用、重復(fù)釋放、堆溢出、棧溢出等問(wèn)題。

  通過(guò)編譯插樁(CTI) ，能夠發(fā)現(xiàn)此堆/棧/全局變量讀寫(xiě)溢出，內(nèi)存泄露等問(wèn)題，并將信息直接打印到日志中。

  ASAN 是一個(gè)快速的內(nèi)存錯(cuò)誤檢測(cè)工具。它非?？?，只拖慢程序兩倍左右（比起 Valgrind 快多了）。

  它包括一個(gè)編譯器 instrumentation 模塊和一個(gè)提供malloc()/free() 替代項(xiàng)的運(yùn)行時(shí)庫(kù)。

• LeakSanitizer/LSAN

  檢查內(nèi)存泄漏問(wèn)題。它是集成在 Address Sanitizer 中的一個(gè)相對(duì)獨(dú)立的工具，它工作在檢查過(guò)程的最后階段。

• UndefinedBehaviorSanitizer/UBSAN

  檢測(cè)未定義行為（使用空指針、有符號(hào)整數(shù)溢出等）。

• ThreadSanitizer/TSAN

  檢查線程數(shù)據(jù)競(jìng)爭(zhēng)和死鎖問(wèn)題。

• MemorySanitizer/MSAN

  檢查使用未初始化內(nèi)存問(wèn)題。

• 內(nèi)核 Sanitizer包括KASAN和KMSAN

Sanitizers 項(xiàng)目本是 LLVM 項(xiàng)目的一部分，但 GNU 也將該系列工具加入到了自家的 GCC 編譯器中（clang 當(dāng)然也支持）。

• GCC 4.8 版本開(kāi)始支持 Address Sanitizer和 Thread Sanitizer。
• GCC 4.9 版本開(kāi)始支持 Leak Sanitizer 和 UndefinedBehaviorSanitizer。

ASAN

簡(jiǎn)介

ASAN/AddressSanitizer 能檢測(cè)很多種內(nèi)存錯(cuò)誤，主要包含如下類別：

• Out-of-bounds accesses to heap, stack and globals
• Use-after-free
• Use-after-return (clang flag -fsanitize-address-use-after-return=(never|runtime|always) default: runtime)
  • Enable with: ASAN_OPTIONS=detect_stack_use_after_return=1 (already enabled on Linux).
  • Disable with: ASAN_OPTIONS=detect_stack_use_after_return=0.
• Use-after-scope (clang flag -fsanitize-address-use-after-scope)
• Double-free, invalid free
• Memory leaks (experimental)

更詳細(xì)的示例 case：參考 https://learn.microsoft.com/zh-cn/cpp/sanitizers/asan-error-examples?view=msvc-170

• alloc-dealloc-mismatch
• allocation-size-too-big
• calloc-overflow
• container-overflow
• double-free
• dynamic-stack-buffer-overflow
• global-buffer-overflow
• heap-buffer-overflow
• heap-use-after-free
• invalid-allocation-alignment
• memcpy-param-overlap
• new-delete-type-mismatch
• stack-buffer-overflow
• stack-buffer-underflow
• stack-use-after-return
• stack-use-after-scope
• strncat-param-overlap
• use-after-poison

性能影響：使用 ASAN 后，性能會(huì)降低 2 倍。

使用方法

安裝：有一個(gè)單獨(dú)的動(dòng)態(tài)庫(kù)libasan6.so，會(huì)隨 gcc 安裝。

編譯選項(xiàng)：-DWITH_DEBUG=1 -DWITH_ASAN=1，可選擇啟用-DWITH_ASAN_SCOPE=1

驗(yàn)證版本：8.0.29

MTR 選項(xiàng)：--sanitize

使用建議：

1、ASAN 功能強(qiáng)大，相較于 valgrind，對(duì)性能影響小很多，建議作為主要的內(nèi)存檢測(cè)工具。

2、由于 mtr 需要用到/usr/bin/perl，因此，有可能出現(xiàn) perl 自身某些函數(shù)的內(nèi)存泄漏問(wèn)題被 Leak Sanitizer 檢測(cè)到，導(dǎo)致 mtr 測(cè)試失敗，此時(shí)，將問(wèn)題函數(shù)添加到lsan.supp文件即可解決。比如 Ubuntu 22.04 perl v5.34.0 會(huì)遇到內(nèi)存泄漏，同樣的，Ubuntu 20.04 perl v5.30.0 就無(wú)該問(wèn)題。

mysql-test/asan.supp 示例

# 語(yǔ)法
interceptor_via_fun:NameOfCFunctionToSuppress
interceptor_via_fun:-[ClassName objCMethodToSuppress:]
interceptor_via_lib:NameOfTheLibraryToSuppress

# 示例
interceptor_via_fun:Perl_safesyscalloc
interceptor_via_fun:Perl_safesysmalloc

參考：

• asan 文檔
• gcc/asan_suppressions.cpp at master · gcc-mirror/gcc · GitHub

mysql-test/lsan.supp 示例

# LSAN suppressions for gcc/clang
leak:Perl_safesyscalloc
leak:Perl_safesysmalloc
leak:Perl_safesysrealloc
leak:Perl_savesharedpv
leak:Perl_Slab_Alloc
leak:Perl_newUNOP_AUX
leak:Perl_newSTATEOP
leak:Perl_pmruntime
leak:/usr/bin/perl
leak:/lib64/libperl.so.*
leak:/bin/bash
leak:/usr/bin/zip
# OpenLDAP bug 9081
# Fixed in 2.4.49, we build with 2.4.48
leak:ldap_initialize

# sasl_client_init will load all available plugins with _sasl_load_plugins().
# It seems some of the SASL plugin have leaks.
# Both LSAN and Valgrind report leaks.
leak:sasl_client_add_plugin

該內(nèi)容來(lái)源于源碼文件，可見(jiàn)官方知曉 Perl 高版本的內(nèi)存泄漏問(wèn)題，以此方式來(lái)忽略。

指令示例

在Ubuntu 22.04 X86_64 運(yùn)行：

perl mysql-test-run.pl --timer --max-test-fail=0 --force --parallel=1 --vardir=var-rpl --suite=rpl --sanitize
perl mysql-test-run.pl --timer --max-test-fail=0 --force --parallel=1 --max-test-fail=0 --vardir=var-binlog --suite=binlog --sanitize

LSAN

簡(jiǎn)介

LSAN/LeakSanitizer 用于內(nèi)存泄漏檢測(cè)。

性能影響：使用 LSAN 后，除了執(zhí)行的最后階段會(huì)有一個(gè)內(nèi)存泄漏檢測(cè)之外，幾乎沒(méi)有性能開(kāi)銷。

使用方法

安裝：有一個(gè)單獨(dú)的動(dòng)態(tài)庫(kù)liblsan0.so，會(huì)隨 gcc 安裝。

編譯選項(xiàng)：-DWITH_DEBUG=1 -DWITH_LSAN=1

驗(yàn)證版本：8.0.29

MTR 選項(xiàng)：--sanitize

使用建議：

1. 由于 ASAN 集成了 LSAN，因此，只有不使用 ASAN、僅使用 LSAN 的情況下才需要設(shè)置該選項(xiàng)。
2. lsan.supp 格式見(jiàn) 「ASAN」小節(jié)。

指令示例

只要編譯時(shí)啟用 ASAN 或 LSAN，在運(yùn)行時(shí)添加--sanitize 選項(xiàng)即可。

perl mysql-test-run.pl --timer --max-test-fail=0 --force --parallel=1 --max-test-fail=0 --vardir=var-binlog --suite=binlog --sanitize

UBSAN

簡(jiǎn)介

UBSAN/UndefinedBehaviorSanitizer 是針對(duì)未定義行為的檢測(cè)器，速度很快。

UBSAN 需要在編譯時(shí)修改程序，以捕獲程序執(zhí)行期間的各種未定義行為。比如：

• 數(shù)組下標(biāo)越界：Array subscript out of bounds, where the bounds can be statically determined
• 位移位超過(guò)數(shù)據(jù)類型邊界：Bitwise shifts that are out of bounds for their data type
• 解除對(duì)未對(duì)齊指針或空指針的關(guān)聯(lián)：Dereferencing misaligned or null pointers
• 有符號(hào)整數(shù)溢出：Signed integer overflow
• 浮點(diǎn)數(shù)類型轉(zhuǎn)換導(dǎo)致的溢出：Conversion to, from, or between floating-point types which would overflow the destination

更多行為詳見(jiàn) ：

UndefinedBehaviorSanitizer — Clang 17.0.0git documentation (llvm.org)

性能影響：UBSAN 的運(yùn)行時(shí)成本很小，對(duì)地址空間布局或 ABI 沒(méi)有影響。

使用方法

安裝：有一個(gè)單獨(dú)的動(dòng)態(tài)庫(kù)libubsan1.so，會(huì)隨 gcc 安裝。

編譯選項(xiàng)：-DWITH_DEBUG=1 -DWITH_UBSAN=1

驗(yàn)證版本：8.0.29

MTR 選項(xiàng)：--sanitize

使用建議：

1. 與 ASAN、TSAN、gcov、gprof 都兼容，可一起啟用。
2. 若想要某些 case 跳過(guò) UBSAN 的檢查，可引用include/not_ubsan.inc。目前只有如下 case 會(huì)跳過(guò) UBSAN：

./t/innodb_tmp_table_heap_to_disk.test
./t/ssl-big.test
./t/count_distinct3.test
./t/multi_update2.test
./t/ds_mrr-big.test

./suite/gis/t/gis_not_ubsan.test
./suite/binlog_gtid/t/binlog_warning_same_server_id.test

指令示例

只要編譯時(shí)啟用 UBSAN，在運(yùn)行時(shí)添加--sanitize 選項(xiàng)即可。

perl mysql-test-run.pl --timer --max-test-fail=0 --force --parallel=1 --max-test-fail=0 --vardir=var-binlog --suite=binlog --sanitize

TSAN

簡(jiǎn)介

TSAN/ThreadSanitizer 是用于檢測(cè)數(shù)據(jù)競(jìng)爭(zhēng)和線程死鎖的工具。

性能影響：引入 TSAN 后，會(huì)降低 5-15 倍性能，同時(shí)，內(nèi)存占用率會(huì)提升 5-10 倍。

使用方法

安裝：有一個(gè)單獨(dú)的動(dòng)態(tài)庫(kù)libtsan0.so，會(huì)隨 gcc 安裝。

編譯選項(xiàng)：-DWITH_DEBUG=1 -DWITH_TSAN=1

驗(yàn)證版本：8.0.29、8.0.32

MTR 選項(xiàng)：--sanitize

使用建議：

1、TSAN 與 ASAN 不兼容（一起使用 cmake 會(huì)報(bào)錯(cuò)"No mysys timer support detected"），但TSAN 與 UBSAN、VALGRIND 兼容。

2、對(duì) TSAN 的支持是實(shí)驗(yàn)性的，尚不成熟，不建議使用。

-- Performing Test HAVE_SANITIZE_SCOPE
-- Performing Test HAVE_SANITIZE_SCOPE - Success
CMake Warning at CMakeLists.txt:1101 (MESSAGE):
  Thread sanitizer support is currently experimental.


-- Performing Test C_LD_LLD_RESULT
-- Performing Test C_LD_LLD_RESULT - Failed
-- Performing Test CXX_LD_LLD_RESULT
-- Performing Test CXX_LD_LLD_RESULT - Failed
-- Performing Test C_LD_GOLD_RESULT
-- Performing Test C_LD_GOLD_RESULT - Failed
-- Performing Test CXX_LD_GOLD_RESULT
-- Performing Test CXX_LD_GOLD_RESULT - Failed
-- Local boost dir /data/work/mysql/boost_1_77_0
-- Found /data/work/mysql/boost_1_77_0/boost/version.hpp
-- BOOST_VERSION_NUMBER is #define BOOST_VERSION 107700
-- BOOST_INCLUDE_DIR /data/work/mysql/boost_1_77_0
-- Looking for pthread.h
-- Looking for pthread.h - not found
-- Could NOT find Threads (missing: Threads_FOUND)
......
-- Looking for timer_create # 由于 timer_create/timer_settime 函數(shù)確實(shí)存在，嘗試調(diào)整過(guò) cmake，后續(xù)會(huì)報(bào)一系列錯(cuò)誤，該問(wèn)題不太好調(diào)。
-- Looking for timer_create - not found
-- Looking for timer_settime
-- Looking for timer_settime - not found
-- Looking for kqueue
-- Looking for kqueue - not found
-- Performing Test HAVE_SETNS
-- Performing Test HAVE_SETNS - Failed
-- Looking for EVFILT_TIMER
-- Looking for EVFILT_TIMER - not found
CMake Error at configure.cmake:334 (MESSAGE):
  No mysys timer support detected!
Call Stack (most recent call first):
  CMakeLists.txt:1487 (INCLUDE)

3、如果某些數(shù)據(jù)競(jìng)爭(zhēng)或死鎖情況是符合預(yù)期的，可以通過(guò) mysql-test/tsan.supp 跳過(guò)。

mysql-test/tsan.supp 示例

#
# Blacklist for Thread Sanitizer.
# Thread Sanitizer can be enabled with -DWITH_TSAN=1
#
# Suppression syntax is documented here:
# https://github.com/google/sanitizers/wiki/ThreadSanitizerSuppressions
#

race:innobase

race:client/dump/
deadlock:client/dump/

race:perfschema

race:plugin_vars_free_values
race:log_builtins_filter_run
race:MY_LOCALE_ERRMSGS::destroy
race:get_one_variable_ext
race:mysql_set_character_set_with_default_collation

race:ngs::Scheduler_dynamic::wait_if_idle_then_delete_worker
race:ngs::Socket_events::break_loop

deadlock:find_sys_var_ex
deadlock:Persisted_variables_cache::lock

signal:my_print_stacktrace

指令示例

只要編譯時(shí)啟用 TSAN，在運(yùn)行時(shí)添加--sanitize 選項(xiàng)即可。

perl mysql-test-run.pl --timer --max-test-fail=0 --force --parallel=1 --max-test-fail=0 --vardir=var-binlog --suite=binlog --sanitize

存在問(wèn)題

測(cè)試時(shí)，在 install database 階段，線程之間就會(huì)有大量 data race，報(bào)錯(cuò)示例如下：

// 報(bào)錯(cuò)信息位于 mysql-test/var-main-tsan/log/bootstrap.log
==================
WARNING: ThreadSanitizer: data race (pid=65314)
  Read of size 4 at 0x555ae20c13b0 by thread T4:
    #0 fil_validate_skip /data/work/mysql/mysql-server/storage/innobase/fil/fil0fil.cc:1953 (mysqld+0x541018e)
    #1 fil_aio_wait(unsigned long) /data/work/mysql/mysql-server/storage/innobase/fil/fil0fil.cc:8234 (mysqld+0x54109ca)
    #2 io_handler_thread /data/work/mysql/mysql-server/storage/innobase/srv/srv0start.cc:279 (mysqld+0x5143b04)
    ......
    #12 std::thread::_State_impl<std::thread::_Invoker<std::tuple<Detached_thread, void (*)(unsigned long), unsigned long> > >::_M_run() /usr/include/c++/11/bits/std_thread.h:211 (mysqld+0x5159b59)
    #13 <null> <null> (libstdc++.so.6+0xdc2b2)

  Previous write of size 4 at 0x555ae20c13b0 by thread T3:
    #0 fil_validate_skip /data/work/mysql/mysql-server/storage/innobase/fil/fil0fil.cc:1953 (mysqld+0x54101a7)
    #1 fil_aio_wait(unsigned long) /data/work/mysql/mysql-server/storage/innobase/fil/fil0fil.cc:8234 (mysqld+0x54109ca)
    #2 io_handler_thread /data/work/mysql/mysql-server/storage/innobase/srv/srv0start.cc:279 (mysqld+0x5143b04)
    ......
    #12 std::thread::_State_impl<std::thread::_Invoker<std::tuple<Detached_thread, void (*)(unsigned long), unsigned long> > >::_M_run() /usr/include/c++/11/bits/std_thread.h:211 (mysqld+0x5159b59)
    #13 <null> <null> (libstdc++.so.6+0xdc2b2)

  Location is global 'fil_validate_skip()::fil_validate_count' of size 4 at 0x555ae20c13b0 (mysqld+0x000007a5c3b0)

  Thread T4 (tid=65320, running) created by thread T1 at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (libstdc++.so.6+0xdc388)
    ......
    #8 handle_bootstrap /data/work/mysql/mysql-server/sql/bootstrap.cc:327 (mysqld+0x387778f)
    #9 pfs_spawn_thread /data/work/mysql/mysql-server/storage/perfschema/pfs.cc:2942 (mysqld+0x56751fb)

  Thread T3 (tid=65319, running) created by thread T1 at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (libstdc++.so.6+0xdc388)
    ......
    #8 handle_bootstrap /data/work/mysql/mysql-server/sql/bootstrap.cc:327 (mysqld+0x387778f)
    #9 pfs_spawn_thread /data/work/mysql/mysql-server/storage/perfschema/pfs.cc:2942 (mysqld+0x56751fb)

SUMMARY: ThreadSanitizer: data race /data/work/mysql/mysql-server/storage/innobase/fil/fil0fil.cc:1953 in fil_validate_skip
==================

......

install database 階段，類似的報(bào)錯(cuò)有 200 多個(gè)。雖然可以通過(guò)tsan.supp 文件跳過(guò)，但畢竟報(bào)錯(cuò)涉及較多函數(shù)，若全部跳過(guò)，可能會(huì)影響對(duì)正常情況下數(shù)據(jù)競(jìng)爭(zhēng)的判斷。因此，個(gè)人暫不建議使用。

MSAN

簡(jiǎn)介

MSAN/MemorySanitizer 用于檢測(cè)對(duì)未初始化內(nèi)存的讀?。╱ninitialized reads）問(wèn)題。

性能影響：引入 MSAN 后，性能會(huì)降低 3 倍。

使用方法

編譯選項(xiàng)：-DWITH_DEBUG=1 -DWITH_MSAN=1

驗(yàn)證版本：8.0.29

MTR 選項(xiàng)：--sanitize

使用建議：

1. 對(duì) MSAN 的支持是實(shí)驗(yàn)性的，尚不成熟，且與 ASAN 不兼容，考慮到 ASAN 的強(qiáng)大，因此，建議使用 ASAN，不建議使用 MSAN。

CMake Warning at CMakeLists.txt:1080 (MESSAGE):
  Memory sanitizer support is currently experimental.


CMake Error at CMakeLists.txt:1107 (MESSAGE):
  Cannot use AddressSanitizer and MemorySanitizer together

指令示例

只要編譯時(shí)啟用 MSAN，在運(yùn)行時(shí)添加--sanitize 選項(xiàng)即可。

perl mysql-test-run.pl --timer --max-test-fail=0 --force --parallel=1 --max-test-fail=0 --vardir=var-binlog --suite=binlog --sanitize

代碼覆蓋率測(cè)試

簡(jiǎn)介

gcov 用于分析代碼覆蓋率，gprof 用于分析 gcov 生成的統(tǒng)計(jì)數(shù)據(jù)，二者一般一起使用。

gprof 只支持 linux 操作系統(tǒng)，不支持 MacOS。

官方手冊(cè)：

Gcov (Using the GNU Compiler Collection (GCC))

gprof(1) - Linux manual page (man7.org)

超級(jí)方便的 Linux 自帶性能分析工具！gprof 介紹、安裝、使用及實(shí)踐 - 知乎 (zhihu.com)

使用方法

編譯選項(xiàng)：-DWITH_DEBUG=1 -DENABLE_GCOV=1 -DENABLE_GPROF=1

使用建議：

1. 與 ASAN、UBSAN 兼容，建議與 ASAN、UBSAN 同時(shí)啟用（未驗(yàn)證與 valgrind、MSAN、TSAN 的兼容性）。
2. 不能在make install的安裝目錄測(cè)試，必須在 執(zhí)行編譯的源碼目錄測(cè)試。
   1. 源碼根目錄/build-debug/mysql-test/mysql-test-run.pl （本人的編譯目錄是 build-debug）只是封裝了一層對(duì) 源碼根目錄/mysql-test/mysql-test-run.pl 的調(diào)用：

[apps@node6 mysql-test]$ pwd
/home/apps/mtr/mysql-oracle/mysql-8.0.26/build-debug/mysql-test

[apps@node6 mysql-test]$ cat mysql-test-run.pl
#!/usr/bin/perl
# Call mtr in out-of-source build
$ENV{MTR_BINDIR} = '/home/apps/mtr/mysql-oracle/mysql-8.0.26/build-debug';
chdir('/home/apps/mtr/mysql-oracle/mysql-8.0.26/mysql-test');
exit(system($^X, '/home/apps/mtr/mysql-oracle/mysql-8.0.26/mysql-test/mysql-test-run.pl', @ARGV) >> 8);

注意：

1. gcov、gprof 運(yùn)行需要較大內(nèi)存，尤其是開(kāi)啟-big-test的情況下。
2. 官方 collections/ 中沒(méi)有 gcov 的推薦用法。
3. mysql-test/README.gcov 文件的最后修改日期是 2006 年，已過(guò)時(shí)，沒(méi)有參考價(jià)值。

示例

直接在安裝目錄（make install）執(zhí)行測(cè)試，報(bào)錯(cuò)：

wslu@ubuntu:/data/work/mysql/mysql80-install.bak_asan_ubsan_gcov_gprof/mysql-test$ ./mtr --gcov
Logging: ./mtr  --gcov

MySQL Version 8.0.29
mysql-test-run: *** ERROR: Coverage test needs the source - please use source dist

在執(zhí)行編譯的目錄（比如 console-build-debug/mysql-test） 執(zhí)行：

wslu@ubuntu:/data/work/mysql/mysql-server/console-build-debug/mysql-test$ ./mtr --gcov -big-test
Logging: /data/work/mysql/mysql-server/mysql-test/mysql-test-run.pl  --gcov -big-test
MySQL Version 8.0.29
Checking supported features
 - Binaries are debug compiled
Purging gcov information from '/data/work/mysql/mysql-server'...
Using suite(s): auth_sec,binlog,binlog_gtid,binlog_nogtid,clone,collations,component_keyring_file,connection_control,encryption,federated,funcs_2,gcol,gis,information_schema,innodb,innodb_fts,innodb_gis,innodb_undo,innodb_zip,interactive_utilities,json,main,opt_trace,parts,perfschema,query_rewrite_plugins,rpl,rpl_gtid,rpl_nogtid,secondary_engine,service_status_var_registration,service_sys_var_registration,service_udf_registration,sys_vars,sysschema,test_service_sql_api,test_services,x
Collecting tests
 - Adding combinations for binlog
 - Adding combinations for binlog_gtid
 - Adding combinations for binlog_nogtid
 - Adding combinations for rpl
 - Adding combinations for rpl_gtid
 - Adding combinations for rpl_nogtid
Checking leftover processes
Removing old var directory
Creating var directory '/data/work/mysql/mysql-server/console-build-debug/mysql-test/var'
Installing system database
Using parallel: 1

==============================================================================
                  TEST NAME                       RESULT  TIME (ms) COMMENT
------------------------------------------------------------------------------
[  0%] binlog_gtid.binlog_xa_select_gtid_executed_explicitly_crash  [ disabled ]   Bug#28588717 Fails both on FreeBSD and other platforms
[  0%] binlog_nogtid.binlog_gtid_next_xa         [ disabled ]   BUG#33650776 Failure of XA COMMIT of prepared txn, can result in txn rollback
[  0%] sys_vars.innodb_log_writer_threads_basic  [ disabled ]   Bug#32129814 SYS_VARS.INNODB_LOG_WRITER_THREADS_BASIC TIMES OUT SPORADICALLY ON PB2
[  0%] sysschema.v_wait_classes_global_by_avg_latency  [ disabled ]   BUG#21550054 Test fails too often.
[  0%] binlog_gtid.binlog_gtid_mix_ok_packet_all_gtids 'mix'  [ pass ]    770
[  0%] binlog_gtid.binlog_gtid_mix_response_packet 'mix'  [ pass ]   6474
[  0%] binlog_gtid.binlog_xa_trx_gtid_response_packet 'mix'  [ pass ]    683
......
[  0%] binlog_gtid.binlog_gtid_errors 'mix'      [ pass ]   1583
......

如果想執(zhí)行測(cè)試后分析 gmon.out ，則可添加 -gprof 參數(shù)（僅支持 linux）：

wslu@ubuntu:/data/work/mysql/mysql-server/console-build-debug/mysql-test$ ./mtr --gcov -gprof -big-test

那么，在 gcov 執(zhí)行完成后，mtr 就會(huì)自動(dòng)調(diào)用 gprof 解析gmon.out文件。

存在問(wèn)題

但在 CentOS 7.6（云服務(wù)器 4C8G SSD）實(shí)測(cè)時(shí)遇到問(wèn)題——gprof 解析 gmon.out 時(shí)特別耗時(shí)，雖然該進(jìn)程 CPU 占用率 100%，看起來(lái)還在運(yùn)行，但并無(wú)任何輸出。

比如，在編譯的源碼目錄中執(zhí)行：

?  mysql-test git:(heads/mysql-8.0.26) ? ps -axf | grep mysql
30604 ?        SN     0:00  \_ perl mysql-test-run.pl --parallel=4 --timer --debug-server --force --testcase-timeout=180 --suite-timeout=1800 --comment=all-default-debug --vardir=var-all-default --skip-combinations --unit-tests-report --no-skip --exclude-platform=windows --skip-ndb --max-test-fail=0 --suite=rpl -gcov -gprof
30611 ?        SN     0:30      \_ /usr/bin/perl /home/wslu/work/mysql/mysql-server/mysql-test/mysql-test-run.pl --parallel=4 --timer --debug-server --force --testcase-timeout=180 --suite-timeout=1800 --comment=all-default-debug --vardir=var-all-default --skip-combinations --unit-tests-report --no-skip --exclude-platform=windows --skip-ndb --max-test-fail=0 --suite=rpl -gcov -gprof
32759 ?        SN     0:44          \_ /usr/bin/perl /home/wslu/work/mysql/mysql-server/mysql-test/mysql-test-run.pl --parallel=4 --timer --debug-server --force --testcase-timeout=180 --suite-timeout=1800 --comment=all-default-debug --vardir=var-all-default --skip-combinations --unit-tests-report --no-skip --exclude-platform=windows --skip-ndb --max-test-fail=0 --suite=rpl -gcov -gprof
 2829 ?        SN     0:00          |   \_ sh -c gprof /home/wslu/work/mysql/mysql-server/build-debug/runtime_output_directory/mysqld /home/wslu/work/mysql/mysql-server/mysql-test/var-all-default/3/mysqld.6/data/gmon.out 2 > /home/wslu/work/mysql/mysql-server/mysql-test/var-all-default/3/mysqld.6/data/gprof.err > /home/wslu/work/mysql/mysql-server/mysql-test/var-all-default/3/mysqld.6/data/gprof.msg
 2830 ?        RN     7:07          |       \_ gprof /home/wslu/work/mysql/mysql-server/build-debug/runtime_output_directory/mysqld /home/wslu/work/mysql/mysql-server/mysql-test/var-all-default/3/mysqld.6/data/gmon.out 2

gmon.out 文件只有 61MB，但 gprof 在解析 gmon.out 時(shí)，長(zhǎng)達(dá) 23 小時(shí)無(wú)任何輸出。

?  mysql-test git:(heads/mysql-8.0.26) ? ls /home/wslu/work/mysql/mysql-server/mysql-test/var-all-default/3/mysqld.6/data/gmon.out -lh
-rw-r--r-- 1 wslu wslu 61M Mar 27 20:21 /home/wslu/work/mysql/mysql-server/mysql-test/var-all-default/3/mysqld.6/data/gmon.out
?  mysql-test git:(heads/mysql-8.0.26) ? ll /home/wslu/work/mysql/mysql-server/mysql-test/var-all-default/3/mysqld.6/data/gprof.err
-rw-r--r-- 1 wslu wslu 0 Mar 28 09:23 /home/wslu/work/mysql/mysql-server/mysql-test/var-all-default/3/mysqld.6/data/gprof.err
?  mysql-test git:(heads/mysql-8.0.26) ? ll  /home/wslu/work/mysql/mysql-server/mysql-test/var-all-default/3/mysqld.6/data/gprof.msg
-rw-r--r-- 1 wslu wslu 0 Mar 28 09:23 /home/wslu/work/mysql/mysql-server/mysql-test/var-all-default/3/mysqld.6/data/gprof.msg

單元測(cè)試

簡(jiǎn)介

MySQL 使用 TAP（Test Anything Protocol） 和 Google Test Framework 來(lái)實(shí)現(xiàn)單元測(cè)試。

MyTAP

TAP 是 Perl 與測(cè)試模塊之間所使用的簡(jiǎn)單的基于文本的接口，主要用于開(kāi)發(fā) Perl 和 PHP 模塊。示例如下：

TAP version 13
ok 1 - testNewArrayIsEmpty(ArrayTest)
ok 2 - testArrayContainsAnElement(ArrayTest)
not ok 3 - Failure: testFailure(FailureErrorTest)
  ---
  message: 'Failed asserting that <integer:2> matches expected value <integer:1>.'
  severity: fail
  data:
    got: 2
    expected: 1
  ...
not ok 4 - Error: testError(FailureErrorTest)
1..4

為了實(shí)現(xiàn) C/C++ 的單元測(cè)試，MySQL 開(kāi)發(fā)了一個(gè)用于生成 TAP 文本的庫(kù)libmytap.a，源碼路徑位于unittest/mytap/。

Google Test Framework

Google Test Framework，與 MyTAP 類似，也是一個(gè)單元測(cè)試框架，但提供了更豐富的功能：

• A rich set of predicates
• User-defined predicates and assertions
• Automatic test registration
• Nice error reporting when a predicate fails (with line number, expected and actual values, and additional comments)
• Test fixtures, and setup/teardown logic
• Death tests
• Disabled tests
• Test filtering and shuffling

參考：

• MySQL: Creating and Executing Unit Tests
  • Home - Test Anything Protocol
  • GoogleTest User’s Guide | GoogleTest

使用方法

編譯選項(xiàng)：-DWITH_DEBUG=1 -DWITH_UNIT_TESTS={ON|OFF}，默認(rèn)是 ON 。

執(zhí)行路徑：必須在編譯的源碼目錄中執(zhí)行。

使用方法：

1. 編譯后，在執(zhí)行編譯（cmake）的目錄執(zhí)行 make test或make test-unit 指令，雖然按手冊(cè)描述兩個(gè)指令都能實(shí)現(xiàn)單元測(cè)試效果，但實(shí)測(cè)make test-unit會(huì)輸出更詳細(xì)的信息，因此，建議使用make test-unit 。
2. 編譯后，在編譯目錄/mysql-test 中執(zhí)行 mtr 指令時(shí)，添加--unit-tests-report 選項(xiàng)。

注意事項(xiàng)：

若啟用了 ASAN：

1. 直接在編譯目錄執(zhí)行make test-unit，可能會(huì)因 ASAN 檢測(cè)到單元測(cè)試代碼有內(nèi)存錯(cuò)誤（RUN_ALL_TESTS()的子函數(shù)）而導(dǎo)致 case 失敗。
2. 通過(guò) mtr 指令來(lái)運(yùn)行單元測(cè)試時(shí)，也可能會(huì)遇到 ASAN 檢測(cè)到內(nèi)存錯(cuò)誤或內(nèi)存泄漏，即使按如下方式修改 .supp 文件，也無(wú)法跳過(guò)：
   1. 若是 AddressSanitizer 范疇中的錯(cuò)誤，比如下表中的 heap-buffer-overflow，在asan.supp 文件添加 interceptor_via_fun:RUN_ALL_TESTS ，無(wú)法跳過(guò)該錯(cuò)誤。
   2. 同理，如果是 ASAN 中的LeakSanitizer檢測(cè)到內(nèi)存泄漏，在lsan.supp 文件添加 leak:RUN_ALL_TESTS，無(wú)法跳過(guò)該錯(cuò)誤。

==228225==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x620000098e90 at pc 0x5570c34c26fb bp 0x7ffe1d0d0590 sp 0x7ffe1d0d0580
READ of size 2 at 0x620000098e90 thread T0
    #0 0x5570c34c26fa in modify_all_zh_pages /data/work/mysql/mysql-server/strings/ctype-uca.cc:4178
    #1 0x5570c34c4d89 in init_weight_level /data/work/mysql/mysql-server/strings/ctype-uca.cc:4287
    ......
    #16 0x5570c36724b8 in testing::UnitTest::Run() /data/work/mysql/mysql-server/extra/googletest/googletest-release-1.11.0/googletest[表情]c/gtest.cc:5438
    #17 0x5570c3285445 in RUN_ALL_TESTS() /data/work/mysql/mysql-server/extra/googletest/googletest-release-1.11.0/googletest/include/gtest/gtest.h:2490
    #18 0x5570c3284f94 in main /data/work/mysql/mysql-server/unittest/gunit/gunit_test_main.cc:150
    #19 0x7f680a423d8f in __libc_start_call_main ../sysdeps/nptl[表情]bc_start_call_main.h:58
    #20 0x7f680a423e3f in __libc_start_main_impl ..[表情]u[表情]bc-start.c:392
    #21 0x5570c2682f34 in _start (/data/work/mysql/mysql-server/console-build-debug/runtime_output_directory/merge_small_tests-t+0x26a9f34)

綜上，運(yùn)行單元測(cè)試時(shí)，不建議同時(shí)啟用 ASAN。

指令示例

make test 示例

?  console-build-debug git:(my_learn_8.0.29) ? make test
Running tests...
Test project /Users/wslu/work/mysql/mysql-server-8.0/console-build-debug
        Start   1: hp_test1
  1/223 Test   #1: hp_test1 ...................................................   Passed    0.49 sec
        Start   2: hp_test2
  2/223 Test   #2: hp_test2 ...................................................   Passed    0.74 sec
        Start   3: pfs_instr_class
  3/223 Test   #3: pfs_instr_class ............................................   Passed    0.54 sec
 ......
        Start 206: routertest_component_rest_metadata_cache
 206/223 Test #206: routertest_component_rest_metadata_cache ...................***Failed   13.37 sec
        Start 207: routertest_component_rest_mock_server
 207/223 Test #207: routertest_component_rest_mock_server ......................   Passed   23.60 sec
        Start 208: routertest_component_rest_router
 208/223 Test #208: routertest_component_rest_router ...........................   Passed    3.87 sec
        Start 209: routertest_component_rest_routing
 209/223 Test #209: routertest_component_rest_routing ..........................***Failed  145.64 sec
        Start 210: routertest_component_rest_connection_pool
 210/223 Test #210: routertest_component_rest_connection_pool ..................   Passed    5.71 sec
        Start 211: routertest_component_router_configuration_errors
 211/223 Test #211: routertest_component_router_configuration_errors ...........   Passed    5.15 sec
        Start 212: routertest_component_routing
 212/223 Test #212: routertest_component_routing ...............................***Failed   39.04 sec
        Start 213: routertest_component_routing_connection
 213/223 Test #213: routertest_component_routing_connection ....................***Failed  117.32 sec
        Start 214: routertest_component_routing_strategy
 214/223 Test #214: routertest_component_routing_strategy ......................   Passed   83.88 sec
        Start 215: routertest_component_sd_notify
 215/223 Test #215: routertest_component_sd_notify .............................   Passed   22.63 sec
        Start 216: routertest_component_shutdown
 216/223 Test #216: routertest_component_shutdown ..............................   Passed    4.86 sec
        Start 217: routertest_component_state_file
 217/223 Test #217: routertest_component_state_file ............................   Passed   22.92 sec
        Start 218: routertest_component_user_option
 218/223 Test #218: routertest_component_user_option ...........................   Passed    0.74 sec
        Start 219: routertest_component_metadata_http_auth_backend
 219/223 Test #219: routertest_component_metadata_http_auth_backend ............***Failed   76.95 sec
        Start 220: routertest_component_socket_close
 220/223 Test #220: routertest_component_socket_close ..........................   Passed  162.52 sec
        Start 221: routertest_component_routing_splicer
 221/223 Test #221: routertest_component_routing_splicer .......................   Passed  287.30 sec
        Start 222: routertest_component_mock_server
 222/223 Test #222: routertest_component_mock_server ...........................   Passed   20.44 sec
        Start 223: routertest_integration_routing_reuse
 223/223 Test #223: routertest_integration_routing_reuse .......................   Passed  440.36 sec

97% tests passed, 6 tests failed out of 223

Total Test time (real) = 2940.80 sec

The following tests FAILED:
   14 - merge_small_tests (Failed)
  206 - routertest_component_rest_metadata_cache (Failed)
  209 - routertest_component_rest_routing (Failed)
  212 - routertest_component_routing (Failed)
  213 - routertest_component_routing_connection (Failed)
  219 - routertest_component_metadata_http_auth_backend (Failed)
Errors while running CTest
Output from these tests are in: /Users/wslu/work/mysql/mysql-server-8.0/console-build-debug/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
make: *** [test] Error 8

make test-unit 示例

wslu@ubuntu:/data/work/mysql/mysql-server/console-build-debug$ head -n 100 /tmp/maketestunit.out
Test project /data/work/mysql/mysql-server/console-build-debug
        Start  14: merge_small_tests
        Start  15: merge_large_tests
  1/223 Test  #14: merge_small_tests ..........................................***Failed   70.09 sec
[==========] Running 2386 tests from 132 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from BoundsCheckedArrayDeathTest
[ RUN      ] BoundsCheckedArrayDeathTest.BoundsCheckRead
[       OK ] BoundsCheckedArrayDeathTest.BoundsCheckRead (616 ms)
[ RUN      ] BoundsCheckedArrayDeathTest.BoundsCheckAssign
[       OK ] BoundsCheckedArrayDeathTest.BoundsCheckAssign (329 ms)
[ RUN      ] BoundsCheckedArrayDeathTest.BoundsCheckPopFront
[       OK ] BoundsCheckedArrayDeathTest.BoundsCheckPopFront (276 ms)
[ RUN      ] BoundsCheckedArrayDeathTest.BoundsCheckResize
[       OK ] BoundsCheckedArrayDeathTest.BoundsCheckResize (277 ms)
[ RUN      ] BoundsCheckedArrayDeathTest.BoundsCheckResizeAssign
[       OK ] BoundsCheckedArrayDeathTest.BoundsCheckResizeAssign (290 ms)
[----------] 5 tests from BoundsCheckedArrayDeathTest (1794 ms total)

[----------] 1 test from DebugDeathTest
[ RUN      ] DebugDeathTest.Suicide
[       OK ] DebugDeathTest.Suicide (178 ms)
[----------] 1 test from DebugDeathTest (178 ms total)

......

216/223 Test   #1: hp_test1 ...................................................   Passed    0.96 sec
        Start  12: pfs_misc
217/223 Test  #10: pfs_noop ...................................................   Passed    0.76 sec
        Start  83: basic
218/223 Test  #83: basic ......................................................   Passed    0.12 sec
        Start  79: skip
219/223 Test  #79: skip .......................................................   Passed    0.07 sec
        Start  80: todo
220/223 Test  #12: pfs_misc ...................................................   Passed    0.91 sec
        Start  81: skip_all
221/223 Test  #80: todo .......................................................   Passed    0.06 sec
        Start  82: no_plan
222/223 Test  #81: skip_all ...................................................   Passed    0.07 sec
223/223 Test  #82: no_plan ....................................................   Passed    0.03 sec

94% tests passed, 14 tests failed out of 223

Total Test time (real) = 4334.59 sec

The following tests FAILED:
   14 - merge_small_tests (Failed)
   15 - merge_large_tests (Failed)
   56 - gcs_xcom_xcom_cache (Subprocess killed)
   57 - gcs_xcom_control_interface (Failed)
   65 - merge_temptable_tests-t (Failed)
  187 - routertest_component_bootstrap (Subprocess aborted)
  189 - routertest_component_bootstrap_clusterset (Subprocess aborted)
  191 - routertest_component_bootstrap_tls_endpoint (Subprocess aborted)
  192 - routertest_component_clusterset (Subprocess aborted)
  197 - routertest_component_gr_notifications (Failed)
  205 - routertest_component_rest_api_enable (Subprocess aborted)
  213 - routertest_component_routing_connection (Subprocess aborted)
  221 - routertest_component_routing_splicer (Timeout)
  223 - routertest_integration_routing_reuse (Failed)
Errors while running CTest

通過(guò) mtr 執(zhí)行單元測(cè)試

在編譯目錄/mysql-test 執(zhí)行如下指令：

perl mysql-test-run.pl --timer --max-test-fail=0 --force --parallel=1 --max-test-fail=0 --vardir=var-binlog --suite=binlog --unit-tests-report

mtr 會(huì)首先運(yùn)行 binlog suite 的所有 case，之后才會(huì)運(yùn)行單元測(cè)試。

壓力測(cè)試

簡(jiǎn)介

涉及壓力測(cè)試的有兩部分：

壓力測(cè)試 suites

只有兩個(gè)：

• stress
• innodb_stress

如需要添加新 case，參考對(duì)應(yīng) suite 已有 case 照貓畫(huà)虎即可，后續(xù)文章會(huì)詳解介紹語(yǔ)法。

mysql-stress-test.pl

被 mysql-test-run.pl 調(diào)用，參數(shù)是--stress。

使用說(shuō)明位于mysql-test/README.stress文件：

The stress script is designed to perform testing of the MySQL server in
a multi-threaded environment.

All functionality regarding stress testing is implemented in the
mysql-stress-test.pl script.

The stress script allows:

 - To stress test the mysqltest binary test engine.
 - To stress test the regular test suite and any additional test suites
   (such as mysql-test-extra-5.0).
 - To specify files with lists of tests both for initialization of
   stress db and for further testing itself.
 - To define the number of threads to be concurrently used in testing.
 - To define limitations for the test run. such as the number of tests or
   loops for execution or duration of testing, delay between test
   executions, and so forth.
 - To get a readable log file that can be used for identification of
   errors that occur during testing.

There are two ways to run the mysql-stress-test.pl script:

 - For most cases, it is enough to use the options below for starting
   the stress test from the mysql-test-run wrapper. In this case, the
   server is run automatically, all preparation steps are performed,
   and after that the stress test is started.

 - In advanced case, you can run the mysql-stress-test.pl script directly.
   But this requires that you perform some preparation steps and to specify
   a bunch of options as well, so this invocation method may be a bit
   complicated.

可見(jiàn)，有兩種用法：

• 大部分情況下，通過(guò) mysql-test-run.pl --stress=[option1,option2,...] 運(yùn)行即可，該腳本實(shí)現(xiàn)了準(zhǔn)備階段、壓力測(cè)試階段所需的工作。

• 更高級(jí)的用法是直接執(zhí)行mysql-stress-test.pl 腳本，這就需要自行實(shí)現(xiàn)準(zhǔn)備階段、測(cè)試階段所需的工作。主要包括：

  • --stress-init-file[=path]

    file_name is the location of the file that contains the list of tests to be run once to initialize the database for the testing. If missing, the default file is stress_init.txt in the test suite directory.

  • --stress-tests-file[=file_name]

    Use this option to run the stress tests. file_name is the location of the file that contains the list of tests. If file_name is omitted, the default file is stress-test.txt in the stress suite directory. (See --stress-suite-basedir).

其他參數(shù)見(jiàn)手冊(cè) MySQL: mysql-stress-test.pl — Server Stress Test Program

指令示例

單獨(dú)運(yùn)行壓力測(cè)試 suites

沒(méi)找到手冊(cè)說(shuō)明，據(jù)我理解，只要未主動(dòng)關(guān)閉單元測(cè)試標(biāo)記（-DWITH_UNIT_TESTS={ON|OFF}選項(xiàng)，默認(rèn)是開(kāi)啟的），就肯定會(huì)編譯生成 stress suite。

在 Ubuntu 22.04 X86_64 執(zhí)行測(cè)試，成功。

wslu@ubuntu:/data/work/mysql/mysql80-install.bak_asan_ubsan/mysql-test$ perl mysql-test-run.pl --force --timer    --comment=stress --vardir=var-stress  --suite=stress --no-skip --max-test-fail=30
Logging: mysql-test-run.pl  --force --timer --comment=stress --vardir=var-stress --suite=stress --no-skip --max-test-fail=30


MySQL Version 8.0.29

##############################################################################
# stress
##############################################################################
Checking supported features
 - Binaries are debug compiled
Using suite(s): stress
Collecting tests
Removing old var directory
Creating var directory '/data/work/mysql/mysql80-install.bak_asan_ubsan/mysql-test/var-stress'
Installing system database
Using parallel: 1

==============================================================================
                  TEST NAME                       RESULT  TIME (ms) COMMENT
------------------------------------------------------------------------------


[ 16%] stress.ddl_myisam                         [ pass ]  88171
[ 33%] stress.ddl_archive                        [ pass ]  11868
[ 50%] stress.ddl_csv                            [ pass ]   8007
[ 66%] stress.ddl_innodb                         [ pass ]  163638
[ 83%] stress.ddl_memory                         [ pass ]  84721
[100%] shutdown_report                           [ pass ]
------------------------------------------------------------------------------
The servers were restarted 1 times
The servers were reinitialized 0 times
Spent 356.405 of 423 seconds executing testcases

Completed: All 6 tests were successful.

mysql-stress-test.pl 使用示例

指令示例：

perl mysql-stress-test.pl
--stress-suite-basedir=/opt/qa/mysql-test-extra-5.0/mysql-test
--stress-basedir=/opt/qa/test
--server-logs-dir=/opt/qa/logs
--test-count=20
--stress-tests-file=innodb-tests.txt
--stress-init-file=innodb-init.txt
--threads=5
--suite=funcs_1
--mysqltest=/opt/mysql/mysql-5.0/client/mysqltest
--server-user=root
--server-database=test
--cleanup

官方推薦的壓力測(cè)試用法

#### 提交代碼時(shí)執(zhí)行
perl mysql-test-run.pl --force --timer    --comment=stress --vardir=var-stress  --suite=stress --no-skip --max-test-fail=30

#### 每天執(zhí)行
perl mysql-test-run.pl --force --timer --big-test   --comment=stress --vardir=var-stress  --suite=stress --no-skip

##### 每周執(zhí)行 basic
# 相較于提交代碼時(shí)執(zhí)行的測(cè)試指令，多了 --debug-server 選項(xiàng)
perl mysql-test-run.pl --debug-server --force --timer --comment=stress --vardir=var-stress  --suite=stress --no-skip
# 相較于上一條多了 --big-test
perl mysql-test-run.pl --debug-server --force --timer --big-test    --comment=stress --vardir=var-stress  --suite=stress --no-skip

#### 每天執(zhí)行 valgrind
perl mysql-test-run.pl --force --timer    --comment=stress --vardir=var-stress  --suite=stress

#### 每周執(zhí)行 valgrind
# 指定了 --big-test
perl mysql-test-run.pl --force --timer --big-test --testcase-timeout=60 --debug-server  --comment=stress-debug-big --vardir=var-stress-debug-big  --suite=stress

# 其他
perl mysql-test-run.pl --force --timer    --comment=stress --vardir=var-stress  --suite=stress --no-skip --max-test-fail=30
perl mysql-test-run.pl --force --timer    --comment=innodb-stress --vardir=var-innodb-stress  --suite=innodb_stress --no-skip --max-test-fail=30

注意：mysql-test/README.stress 文件的最后修改日期是 2006 年，已過(guò)時(shí)，沒(méi)有參考價(jià)值。

結(jié)論

mtr 執(zhí)行路徑

• 代碼覆蓋率、單元測(cè)試只能在編譯的源碼目錄/mysql-test執(zhí)行。
• 其他測(cè)試在編譯的源碼目錄/mysql-test和 安裝目錄/mysql-test都可以執(zhí)行。
• 如無(wú)特殊需求，更建議在安裝目錄執(zhí)行 mtr 測(cè)試（目錄結(jié)構(gòu)更清晰）。

測(cè)試結(jié)果及兼容性

注意事項(xiàng)：

1. 由于 mtr case 特別多，運(yùn)行全部 case 時(shí)間過(guò)長(zhǎng)，因此，該表中所說(shuō)「正常執(zhí)行」是執(zhí)行一部分 suites（而不是全部）未報(bào)錯(cuò)。
2. 從執(zhí)行時(shí)長(zhǎng)來(lái)看，如需在 hyper 運(yùn)行官方推薦 collections ，至少要開(kāi)啟 32 并發(fā)。
3. 對(duì)于單元測(cè)試、代碼覆蓋率測(cè)試、內(nèi)存錯(cuò)誤檢測(cè)，建議在 X86_64 平臺(tái)運(yùn)行，某些選項(xiàng)對(duì) ARM 平臺(tái)不兼容（編譯失?。?/strong>。
4. 部分 perl 版本有內(nèi)存泄漏，會(huì)被 ASAN 檢測(cè)出來(lái)，導(dǎo)致 mtr 測(cè)試終止。目前驗(yàn)證 Ubuntu 22.04 所用的 perl 5.34.0 存在內(nèi)存泄漏，而 Ubuntu 20.04 所用的 perl 5.30.0 不存在內(nèi)存泄漏。
5. 由于我主要是在虛擬機(jī)進(jìn)行驗(yàn)證的，而 mtr 運(yùn)行太過(guò)耗時(shí)，因此，本文章節(jié)涉及的 mtr 指令，大部分并未完整運(yùn)行（進(jìn)度小于 10%）。