算法
參考：Greedy search與beam search
在下面會用到解碼的方法選擇
- greedy_search：貪心搜索，貪心搜索是一種來自計算機科學的算法，生成第一個詞的分布以后，它將會根據(jù)你的條件語言模型挑選出最有可能的第一個詞進入你的機器翻譯模型中，在挑選出第一個詞之后它將會繼續(xù)挑選出最有可能的第二個詞，然后繼續(xù)挑選第三個最有可能的詞，這種算法就叫做貪心搜索。
- beam_search：集束搜索，對于語音識別，給定一個輸入的語音片段，你不會想要一個隨機的文本翻譯結(jié)果，你想要最好的，最接近原意的翻譯結(jié)果，集束搜索就是解決這個最常用的算法。
一、環(huán)境
1.1 硬件環(huán)境–RV1126開發(fā)板
- 硬件：RV-1126
[root@RV1126_RV1109:/mnt/nfs/sherpa]# uname -r
4.19.111
[root@RV1126_RV1109:/mnt/nfs/sherpa]# uname -m
armv7l
[root@RV1126_RV1109:/mnt/nfs/sherpa]# cat /proc/cpuinfo
processor : 0
model name : ARMv7 Processor rev 5 (v7l)
BogoMIPS : 70.52
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xc07
CPU revision : 5
processor : 1
model name : ARMv7 Processor rev 5 (v7l)
BogoMIPS : 70.52
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xc07
CPU revision : 5
processor : 2
model name : ARMv7 Processor rev 5 (v7l)
BogoMIPS : 70.52
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xc07
CPU revision : 5
processor : 3
model name : ARMv7 Processor rev 5 (v7l)
BogoMIPS : 70.52
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xc07
CPU revision : 5
Hardware : Generic DT based system
Revision : 0000
Serial : 3ea3a71e7adec418
1.2 交叉編譯器
編輯/etc/profile文件將SDK的交叉編譯器（gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf）添加到環(huán)境。
export PATH=/home/liefyuan/rv1126/rp_rv1126_sdk/prebuilts/gcc/linux-x86/arm/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/bin:$PATH
1.3 需要Cmake版本大于3.1以上
$ cmake -version
cmake version 3.26.0
CMake suite maintained and supported by Kitware (kitware.com/cmake).
二、交叉編譯sherpa
2.1 下載sherpa
$ git clone https://github.com/k2-fsa/sherpa-ncnn
2.2 編譯sherpa
$ cd sherpa-ncnn/
$ ./build-arm-linux-gnueabihf.sh
編譯成功后會生成兩個文件：
~/rv1126/kaldi/sherpa-ncnn$ ls -lh build-arm-linux-gnueabihf/install/bin/
total 4.1M
-rwxr-xr-x 1 liefyuan liefyuan 2.1M Apr 25 23:26 sherpa-ncnn
-rwxr-xr-x 1 liefyuan liefyuan 2.1M Apr 25 23:26 sherpa-ncnn-alsa
將這兩個可執(zhí)行文件拷貝到開發(fā)板上去。
2.3 運行測試
運行sherpa-ncnn
[root@RV1126_RV1109:/mnt/nfs/sherpa]# ./sherpa-ncnn
./sherpa-ncnn: error while loading shared libraries: libgomp.so.1: cannot open shared object file: No such file or directory
運行提示缺少libgomp.so.1
缺少的文件在SDK里面找一下：
liefyuan@ubuntu:~/rv1126/rp_rv1126_sdk$ find ./ -name libgomp*
./prebuilts/gcc/linux-x86/arm/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/share/info/libgomp.info
./prebuilts/gcc/linux-x86/arm/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/arm-linux-gnueabihf/libc/usr/lib/libgomp.so.1
./prebuilts/gcc/linux-x86/arm/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/arm-linux-gnueabihf/libc/usr/lib/libgomp.so
./prebuilts/gcc/linux-x86/arm/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/arm-linux-gnueabihf/libc/usr/lib/libgomp.so.1.0.0
./prebuilts/gcc/linux-x86/arm/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/arm-linux-gnueabihf/lib/libgomp.so.1
./prebuilts/gcc/linux-x86/arm/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/arm-linux-gnueabihf/lib/libgomp.a
./prebuilts/gcc/linux-x86/arm/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/arm-linux-gnueabihf/lib/libgomp.so
./prebuilts/gcc/linux-x86/arm/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/arm-linux-gnueabihf/lib/libgomp.spec
./prebuilts/gcc/linux-x86/arm/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/arm-linux-gnueabihf/lib/libgomp.so.1.0.0
./prebuilts/gcc/linux-x86/arm/gcc-linaro-6.3.1-2017.05-x86_64_arm-linux-gnueabihf/share/doc/libgomp.html
./prebuilts/gcc/linux-x86/arm/gcc-linaro-6.3.1-2017.05-x86_64_arm-linux-gnueabihf/share/info/libgomp.info
./prebuilts/gcc/linux-x86/arm/gcc-linaro-6.3.1-2017.05-x86_64_arm-linux-gnueabihf/arm-linux-gnueabihf/libc/lib/libgomp.so.1
./prebuilts/gcc/linux-x86/arm/gcc-linaro-6.3.1-2017.05-x86_64_arm-linux-gnueabihf/arm-linux-gnueabihf/libc/lib/libgomp.a
./prebuilts/gcc/linux-x86/arm/gcc-linaro-6.3.1-2017.05-x86_64_arm-linux-gnueabihf/arm-linux-gnueabihf/libc/lib/libgomp.so
./prebuilts/gcc/linux-x86/arm/gcc-linaro-6.3.1-2017.05-x86_64_arm-linux-gnueabihf/arm-linux-gnueabihf/libc/lib/libgomp.spec
./prebuilts/gcc/linux-x86/arm/gcc-linaro-6.3.1-2017.05-x86_64_arm-linux-gnueabihf/arm-linux-gnueabihf/libc/lib/libgomp.so.1.0.0
./prebuilts/gcc/linux-x86/arm/gcc-linaro-6.3.1-2017.05-x86_64_arm-linux-gnueabihf/arm-linux-gnueabihf/lib/libgomp.so.1
./prebuilts/gcc/linux-x86/arm/gcc-linaro-6.3.1-2017.05-x86_64_arm-linux-gnueabihf/arm-linux-gnueabihf/lib/libgomp.a
./prebuilts/gcc/linux-x86/arm/gcc-linaro-6.3.1-2017.05-x86_64_arm-linux-gnueabihf/arm-linux-gnueabihf/lib/libgomp.so
./prebuilts/gcc/linux-x86/arm/gcc-linaro-6.3.1-2017.05-x86_64_arm-linux-gnueabihf/arm-linux-gnueabihf/lib/libgomp.spec
./prebuilts/gcc/linux-x86/arm/gcc-linaro-6.3.1-2017.05-x86_64_arm-linux-gnueabihf/arm-linux-gnueabihf/lib/libgomp.so.1.0.0
./buildroot/output/rockchip_rv1126_rv1109_facial_gate/host/arm-buildroot-linux-gnueabihf/sysroot/usr/lib/libgomp.so.1
./buildroot/output/rockchip_rv1126_rv1109_facial_gate/host/arm-buildroot-linux-gnueabihf/sysroot/usr/lib/libgomp.so
./buildroot/output/rockchip_rv1126_rv1109_facial_gate/host/arm-buildroot-linux-gnueabihf/sysroot/usr/lib/libgomp.so.1.0.0
./buildroot/output/rockchip_rv1126_rv1109_facial_gate/host/arm-buildroot-linux-gnueabihf/sysroot/lib/libgomp.so.1
./buildroot/output/rockchip_rv1126_rv1109_facial_gate/host/arm-buildroot-linux-gnueabihf/sysroot/lib/libgomp.a
./buildroot/output/rockchip_rv1126_rv1109_facial_gate/host/arm-buildroot-linux-gnueabihf/sysroot/lib/libgomp.so
./buildroot/output/rockchip_rv1126_rv1109_facial_gate/host/arm-buildroot-linux-gnueabihf/sysroot/lib/libgomp.spec
./buildroot/output/rockchip_rv1126_rv1109_facial_gate/host/arm-buildroot-linux-gnueabihf/sysroot/lib/libgomp.so.1.0.0
./buildroot/output/rockchip_rv1126_rv1109_recovery/host/arm-buildroot-linux-gnueabihf/sysroot/usr/lib/libgomp.so.1
./buildroot/output/rockchip_rv1126_rv1109_recovery/host/arm-buildroot-linux-gnueabihf/sysroot/usr/lib/libgomp.so
./buildroot/output/rockchip_rv1126_rv1109_recovery/host/arm-buildroot-linux-gnueabihf/sysroot/usr/lib/libgomp.so.1.0.0
./buildroot/output/rockchip_rv1126_rv1109_recovery/host/arm-buildroot-linux-gnueabihf/sysroot/lib/libgomp.so.1
./buildroot/output/rockchip_rv1126_rv1109_recovery/host/arm-buildroot-linux-gnueabihf/sysroot/lib/libgomp.a
./buildroot/output/rockchip_rv1126_rv1109_recovery/host/arm-buildroot-linux-gnueabihf/sysroot/lib/libgomp.so
./buildroot/output/rockchip_rv1126_rv1109_recovery/host/arm-buildroot-linux-gnueabihf/sysroot/lib/libgomp.spec
./buildroot/output/rockchip_rv1126_rv1109_recovery/host/arm-buildroot-linux-gnueabihf/sysroot/lib/libgomp.so.1.0.0
從這里
./prebuilts/gcc/linux-x86/arm/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/arm-linux-gnueabihf/libc/usr/lib/libgomp.so.1
./prebuilts/gcc/linux-x86/arm/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/arm-linux-gnueabihf/libc/usr/lib/libgomp.so
拷貝libgomp.so.1和libgomp.so文件到開發(fā)板的/usr/lib里面去。
然后運行，出現(xiàn)下面信息代表正常！
[root@RV1126_RV1109:/mnt/nfs/sherpa]# ./sherpa-ncnn
Usage:
./bin/sherpa-ncnn \
/path/to/tokens.txt \
/path/to/encoder.ncnn.param \
/path/to/encoder.ncnn.bin \
/path/to/decoder.ncnn.param \
/path/to/decoder.ncnn.bin \
/path/to/joiner.ncnn.param \
/path/to/joiner.ncnn.bin \
/path/to/foo.wav [num_threads] [decode_method, can be greedy_search/modified_beam_search]
Please refer to
https://k2-fsa.github.io/sherpa/ncnn/pretrained_models/index.html
for a list of pre-trained models to download.
運行OK！
三、下載模型
使用的模型是小模型，下載地址：Small models
對應(yīng)的文檔：https://k2-fsa.github.io/sherpa/ncnn/pretrained_models/zipformer-transucer-models.html#sherpa-ncnn-streaming-zipformer-small-bilingual-zh-en-2023-02-16
cd /home/rv1126/kaldi/
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/csukuangfj/sherpa-ncnn-streaming-zipformer-small-bilingual-zh-en-2023-02-16
cd sherpa-ncnn-streaming-zipformer-small-bilingual-zh-en-2023-02-16
git lfs pull --include "*.bin"
運行最后一步報錯
liefyuan@ubuntu:/home/rv1126/kaldi/sherpa-ncnn-streaming-zipformer-small-bilingual-zh-en-2023-02-16$ git lfs pull --include "*.bin"
git: 'lfs' is not a git command. See 'git --help'.
The most similar command is
log
處理方法：
$ curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | sudo bash
$ sudo apt-get install git-lfs
將文件夾里面的.bin，.param，.txt文件拷貝到開發(fā)板上去，與可執(zhí)行文件放一起。
我使用的是nfs，開發(fā)板和虛擬機共享一個文件夾我的sherpa庫和sherpa可執(zhí)行文件也都在這個文件夾里面：
四、語音測試
4.1 單個語音文件解碼測試
./sherpa-ncnn \
./tokens.txt \
./encoder_jit_trace-pnnx.ncnn.param \
./encoder_jit_trace-pnnx.ncnn.bin \
./decoder_jit_trace-pnnx.ncnn.param \
./decoder_jit_trace-pnnx.ncnn.bin \
./joiner_jit_trace-pnnx.ncnn.param \
./joiner_jit_trace-pnnx.ncnn.bin \
./1.wav \
3 \
greedy_search
3：是指3個線程
greedy_search：貪心搜索算法
modified_beam_search：改進集束搜索算法
4.2 開發(fā)板上使用alsa架構(gòu)從MIC說話測試
./sherpa-ncnn-alsa \
./tokens.txt \
./encoder_jit_trace-pnnx.ncnn.param \
./encoder_jit_trace-pnnx.ncnn.bin \
./decoder_jit_trace-pnnx.ncnn.param \
./decoder_jit_trace-pnnx.ncnn.bin \
./joiner_jit_trace-pnnx.ncnn.param \
./joiner_jit_trace-pnnx.ncnn.bin \
"default" \
4 \
greedy_search
4：是指4個線程
greedy_search：貪心搜索算法
modified_beam_search：改進集束搜索算法
運行起來后對著板子的MIC說話，背誦一個古詩看看。
測試log
[root@RV1126_RV1109:/mnt/nfs/sherpa]# ./sherpa-ncnn-alsa ./tokens.txt ./encoder_
jit_trace-pnnx.ncnn.param ./encoder_jit_trace-pnnx.ncnn.bin ./decoder_jit_trace-
pnnx.ncnn.param ./decoder_jit_trace-pnnx.ncnn.bin ./joiner_jit_trace-pnnx.ncnn.p
aram ./joiner_jit_trace-pnnx.ncnn.bin "default" 4 greedy_search
RecognizerConfig(feat_config=FeatureExtractorConfig(sampling_rate=16000, feature_dim=80), model_config=ModelConfig(encoder_param="./encoder_jit_trace-pnnx.ncnn.param", encoder_bin="./encoder_jit_trace-pnnx.ncnn.bin", decoder_param="./decoder_jit_trace-pnnx.ncnn.param", decoder_bin="./decoder_jit_trace-pnnx.ncnn.bin", joiner_param="./joiner_jit_trace-pnnx.ncnn.param", joiner_bin="./joiner_jit_trace-pnnx.ncnn.bin", tokens="./tokens.txt", encoder num_threads=4, decoder num_threads=4, joiner num_threads=4), decoder_config=DecoderConfig(method="greedy_search", num_active_paths=4), endpoint_config=EndpointConfig(rule1=EndpointRule(must_contain_nonsilence=False, min_trailing_silence=2.4, min_utterance_length=0), rule2=EndpointRule(must_contain_nonsilence=True, min_trailing_silence=1.2, min_utterance_length=0), rule3=EndpointRule(must_contain_nonsilence=False, min_trailing_silence=0, min_utterance_length=300)), enable_endpoint=True)
Disable fp16 for Zipformer encoder
Don't Use GPU. has_gpu: 0, config.use_vulkan_compute: 1
Failed to set number of channels to 1. Invalid argument
Channel count is set to 2. Will use only 1 channel of it.
Current sample rate: 16000
Recording started!
Use recording device: default
0:床前明月光
1:疑似地上霜
2:舉頭望明月
3:低頭思故鄉(xiāng)
4:好啦這是一段測試^C
Caught Ctrl + C. Exiting...
很好！中英文都可以識別。文章來源：http://www.zghlxwxcb.cn/news/detail-444595.html
文檔寫的真不錯：
https://k2-fsa.github.io/sherpa/ncnn/pretrained_models/zipformer-transucer-models.html#sherpa-ncnn-streaming-zipformer-small-bilingual-zh-en-2023-02-16文章來源地址http://www.zghlxwxcb.cn/news/detail-444595.html
到了這里，關(guān)于【RV1126】移植kaldi實時語音識別的文章就介紹完了。如果您還想了解更多內(nèi)容，請在右上角搜索TOY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！
