前言
本文的原文連接是:
https://blog.csdn.net/freewebsys/article/details/108971807文章來源：http://www.zghlxwxcb.cn/news/detail-484193.html
未經(jīng)博主允許不得轉(zhuǎn)載。
博主CSDN地址是：https://blog.csdn.net/freewebsys
博主掘金地址是：https://juejin.cn/user/585379920479288
博主知乎地址是：https://www.zhihu.com/people/freewebsystem
1，關(guān)于gpt2的幾個(gè)例子學(xué)習(xí)
快速使用docker 鏡像進(jìn)行環(huán)境搭建。
相關(guān)的chatGpt項(xiàng)目有：
gpt2官方模型：
https://github.com/openai/gpt-2
6.1K 星星：
https://github.com/Morizeyao/GPT2-Chinese
2.4K 星星：
https://github.com/yangjianxin1/GPT2-chitchat
1.6K 星星：
https://github.com/imcaspar/gpt2-ml
先找個(gè)簡單的進(jìn)行研究：
3.2K 星星：
https://github.com/minimaxir/gpt-2-simple
2，使用docker配置環(huán)境
先弄官方的例子，使用tensorflow的2.12 的鏡像，因顯卡驅(qū)動的問題，只能用cpu進(jìn)行運(yùn)算：
git clone https://github.com/minimaxir/gpt-2-simple
cd gpt-2-simple
docker run --name gpt2simple -itd -v `pwd`:/data -p 8888:8888 tensorflow/tensorflow:latest
版本說明，這邊用的就是最小的版本：能跑就行。
latest: minimal image with TensorFlow Serving binary installed and ready to serve!
:latest-gpu: minimal image with TensorFlow Serving binary installed and ready to serve on GPUs!
:latest-devel - include all source/dependencies/toolchain to develop, along with a compiled binary that works on CPUs
:latest-devel-gpu - include all source dependencies/toolchain (cuda9/cudnn7) to develop, along with a compiled binary that works on NVIDIA GPUs.
然后進(jìn)入docker 鏡像中執(zhí)行命令：
當(dāng)然也可以使用Dockerfile 但是網(wǎng)速慢，且容易出錯：
docker exec -it gpt2simple bash
############### 以下是登陸后執(zhí)行：
sed -i 's/archive.ubuntu.com/mirrors.aliyun.com/g' /etc/apt/sources.list
sed -i 's/security.ubuntu.com/mirrors.aliyun.com/g' /etc/apt/sources.list
mkdir /root/.pip/
# 增加 pip 的源
echo "[global]" > ~/.pip/pip.conf
echo "index-url = https://mirrors.aliyun.com/pypi/simple/" >> ~/.pip/pip.conf
echo "[install]" >> ~/.pip/pip.conf
echo "trusted-host=mirrors.aliyun.com" >> ~/.pip/pip.conf
cd /data
#注釋掉 tensorflow 依賴
sed -i 's/tensorflow/#tensorflow/g' requirements.txt
pip3 install -r requirements.txt
3，使用uget工具下載模型，文件大容易卡死
sudo apt install uget
然后就是網(wǎng)絡(luò)特別的慢了。根本下載不了，就卡在進(jìn)度中。幾個(gè)特別大的模型，最大的6G。
一個(gè)比一個(gè)大，不知道壓縮沒有：
498M：
https://openaipublic.blob.core.windows.net/gpt-2/models/124M/model.ckpt.data-00000-of-00001
1.42G
https://openaipublic.blob.core.windows.net/gpt-2/models/355M/model.ckpt.data-00000-of-00001
3.10G
https://openaipublic.blob.core.windows.net/gpt-2/models/774M/model.ckpt.data-00000-of-00001
6.23G
https://openaipublic.blob.core.windows.net/gpt-2/models/1558M/model.ckpt.data-00000-of-00001
使用工具下載模型，命令行執(zhí)行的時(shí)候容易卡死：
這個(gè)云地址不支持多線程下載，就下載了一個(gè)最小的124M的模型。
先嘗個(gè)新鮮就行。
剩下的文件可以單獨(dú)下載：
gpt2 里面的代碼，去掉模型文件其他用腳本下載，哎網(wǎng)絡(luò)是個(gè)大問題。
也沒有國內(nèi)的鏡像。
download_model.py 124M
修改了代碼，去掉了最大的model.ckpt.data 這個(gè)單獨(dú)下載，下載了拷貝進(jìn)去。
import os
import sys
import requests
from tqdm import tqdm
if len(sys.argv) != 2:
print('You must enter the model name as a parameter, e.g.: download_model.py 124M')
sys.exit(1)
model = sys.argv[1]
subdir = os.path.join('models', model)
if not os.path.exists(subdir):
os.makedirs(subdir)
subdir = subdir.replace('\\','/') # needed for Windows
for filename in ['checkpoint','encoder.json','hparams.json', 'model.ckpt.index', 'model.ckpt.meta', 'vocab.bpe']:
r = requests.get("https://openaipublic.blob.core.windows.net/gpt-2/" + subdir + "/" + filename, stream=True)
with open(os.path.join(subdir, filename), 'wb') as f:
file_size = int(r.headers["content-length"])
chunk_size = 1000
with tqdm(ncols=100, desc="Fetching " + filename, total=file_size, unit_scale=True) as pbar:
# 1k for chunk_size, since Ethernet packet size is around 1500 bytes
for chunk in r.iter_content(chunk_size=chunk_size):
f.write(chunk)
pbar.update(chunk_size)
4，研究使用gpt2-simple執(zhí)行demo，訓(xùn)練200次
然后運(yùn)行demo.py 代碼
項(xiàng)目代碼：
提前把模型和文件準(zhǔn)備好：
https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt
另存為，在工程目錄
shakespeare.txt
gpt-2-simple/models$ tree
.
└── 124M
├── checkpoint
├── encoder.json
├── hparams.json
├── model.ckpt.data-00000-of-00001
├── model.ckpt.index
├── model.ckpt.meta
└── vocab.bpe
1 directory, 7 files
https://github.com/minimaxir/gpt-2-simple
import gpt_2_simple as gpt2
import os
import requests
model_name = "124M"
file_name = "shakespeare.txt"
sess = gpt2.start_tf_sess()
print("########### init start ###########")
gpt2.finetune(sess,
file_name,
model_name=model_name,
steps=200) # steps is max number of training steps
gpt2.generate(sess)
print("########### finish ###########")
執(zhí)行：
time python demo.py
real 80m14.186s
user 513m37.158s
sys 37m45.501s
開始訓(xùn)練，做測試，模型訓(xùn)練200次。耗時(shí)是 1小時(shí) 20分鐘。
用的是 Intel? Core? i7-9700 CPU @ 3.00GHz，8核8線程的。
使用CPU訓(xùn)練，沒有顯卡。
cpu都是80%，load 是 7 ，風(fēng)扇已經(jīng)呼呼轉(zhuǎn)了。
然后生成對話：
demo-run.py
import gpt_2_simple as gpt2
sess = gpt2.start_tf_sess()
gpt2.load_gpt2(sess)
gpt2.generate(sess)
執(zhí)行結(jié)果，沒有cpu/gpu 優(yōu)化：
python demo_generate.py
2023-03-03 13:11:53.801232: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-03-03 13:11:55.191519: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-03-03 13:11:57.054783: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:357] MLIR V1 optimization pass is not enabled
Loading checkpoint checkpoint/run1/model-200
Ministers' policy: policy
I am the king, and
I shall have none of you;
But, in the desire of your majesty,
I shall take your honour's honour,
And give you no better honour than
To be a king and a king's son,
And my honour shall have no more than that
Which you have given to me.
GLOUCESTER:
MONTAGUE:
Mistress:
Go, go, go, go, go, go, go, go, go!
GLOUCESTER:
You have done well, my lord;
I was but a piece of a body;
And, if thou meet me, I'll take thy pleasure;
And, if thou be not satisfied
I'll give thee another way, or let
My tongue hope that thou wilt find a friend:
I'll be your business, my lord.
MONTAGUE:
Go, go, go, go!
GLOUCESTER:
Go, go, go!
MONTAGUE:
Go, go, go!
GLOUCESTER:
You have been so well met, my lord,
I'll look you to the point:
If thou wilt find a friend, I'll be satisfied;
Thou hast no other choice but to be a king.
MONTAGUE:
Go, go, go!
GLOUCESTER:
Go, go, go!
MONTAGUE:
Go, go, go!
GLOUCESTER:
Go, go, go!
MONTAGUE:
Go, go, go!
GLOUCESTER:
Go, go, go!
KING RICHARD II:
A villain, if you have any, is a villain without a villain.
WARWICK:
I have seen the villain, not a villain,
But--
KING RICHARD II:
Here is the villain.
WARWICK:
A villain.
KING RICHARD II:
But a villain, let him not speak with you.
WARWICK:
Why, then, is there in this house no man of valour?
KING RICHARD II:
The Lord Northumberland, the Earl of Wiltshire,
The noble Earl of Wiltshire, and the Duke of Norfolk
All villainous.
WARWICK:
And here comes the villain?
KING RICHARD II:
He is a villain, if you be a villain.
每次生成的對話都不一樣呢?？梢远噙\(yùn)行幾次，生成的內(nèi)容都是不一樣的。
5，總結(jié)
ai果然是高技術(shù)含量的東西，代碼啥的不多，就是沒有太看懂。
然后消耗CPU和GPU資源，也是非常消耗硬件的。
這個(gè)很小的模型訓(xùn)練200次，都這么費(fèi)時(shí)間，更何況是大數(shù)據(jù)量多參數(shù)的模型呢??！
同時(shí)這個(gè)基礎(chǔ)設(shè)施也要搭建起來呢，有個(gè)項(xiàng)目要研究下了，就是
https://www.kubeflow.org/
得去研究服務(wù)器集群了，因?yàn)镹vidia的限制，服務(wù)器上跑的都是又貴又性能低的顯卡。
但是可以本地跑集群做訓(xùn)練呢?。?！
本文的原文連接是:
https://blog.csdn.net/freewebsys/article/details/108971807
文章來源地址http://www.zghlxwxcb.cn/news/detail-484193.html
到了這里，關(guān)于研究開源gpt-2-simple項(xiàng)目，跑一個(gè)簡單的模型，然后生成一段對話。用的是 Intel(R) Core(TM) i7-9700，8核8線程，訓(xùn)練最小的模型200次跑1個(gè)小時(shí)20分鐘的文章就介紹完了。如果您還想了解更多內(nèi)容，請?jiān)谟疑辖撬阉鱐OY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！
