一、說(shuō)明
因內(nèi)網(wǎng)主機(jī)需要拉取鏡像進(jìn)行Docker應(yīng)用，因此需要一臺(tái)帶外主機(jī)作為內(nèi)網(wǎng)私有倉(cāng)庫(kù)來(lái)提供內(nèi)外其他docker業(yè)務(wù)主機(jī)使用。參考架構(gòu)如下：
相關(guān)資源：加密、Distribution registry、Create and Configure Docker Registry、Registry部署、Distribution Registry v2 authentication、docker-registry
二、部署配置
注：不要在已有業(yè)務(wù)的生產(chǎn)環(huán)境主機(jī)搭建，因?yàn)檫^(guò)程需要重啟docker，這對(duì)生產(chǎn)來(lái)說(shuō)是不可被接受的；Docker 配置文件的默認(rèn)位置如下：
Linux：/etc/docker/daemon.json
macOS：/Library/Preferences/com.docker.docker/daemon.json
Windows：C:\ProgramData\Docker\config\daemon.json
環(huán)境變量：export DOCKER_CONFIG=/home/my/docker/config
2.1、Docker基礎(chǔ)環(huán)境部署
#因在帶外主機(jī)上部署，直接yum即可
yum install yum-utils -y #yum-config-manager依賴(lài)
--------------------------------------------------------------------------------------------------------------------------
Total 1.4 MB/s | 1.3 MB 00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : python-kitchen-1.1.1-5.el7.noarch 1/5
Updating : libxml2-2.9.1-6.el7_9.6.x86_64 2/5
Installing : libxml2-python-2.9.1-6.el7_9.6.x86_64 3/5
Installing : yum-utils-1.1.31-54.el7_8.noarch 4/5
Cleanup : libxml2-2.9.1-6.el7_2.3.x86_64 5/5
Verifying : libxml2-2.9.1-6.el7_9.6.x86_64 1/5
Verifying : python-kitchen-1.1.1-5.el7.noarch 2/5
Verifying : yum-utils-1.1.31-54.el7_8.noarch 3/5
Verifying : libxml2-python-2.9.1-6.el7_9.6.x86_64 4/5
Verifying : libxml2-2.9.1-6.el7_2.3.x86_64 5/5
Installed:
yum-utils.noarch 0:1.1.31-54.el7_8
Dependency Installed:
libxml2-python.x86_64 0:2.9.1-6.el7_9.6 python-kitchen.noarch 0:1.1.1-5.el7
Dependency Updated:
libxml2.x86_64 0:2.9.1-6.el7_9.6
Complete!
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
Loaded plugins: fastestmirror, versionlock
adding repo from: http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
grabbing file http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo to /etc/yum.repos.d/docker-ce.repo
repo saved to /etc/yum.repos.d/docker-ce.repo
#基礎(chǔ)包（可忽略）
yum install -y wget net-tools nfs-utils lrzsz gcc gcc-c++ make cmake libxml2-devel openssl-devel curl curl-devel unzip sudo ntp libaio-devel wget vim ncurses-devel autoconf automake zlib-devel python-devel epel-release openssh-server socat ipvsadm conntrack ntpdate telnet
#安裝依賴(lài)
yum install -y device-mapper-persistent-data lvm2
……
Updated:
device-mapper-persistent-data.x86_64 0:0.8.5-3.el7_9.2 lvm2.x86_64 7:2.02.187-6.el7_9.5
Dependency Updated:
device-mapper.x86_64 7:1.02.170-6.el7_9.5 device-mapper-event.x86_64 7:1.02.170-6.el7_9.5
device-mapper-event-libs.x86_64 7:1.02.170-6.el7_9.5 device-mapper-libs.x86_64 7:1.02.170-6.el7_9.5
lvm2-libs.x86_64 7:2.02.187-6.el7_9.5
Complete!
#安裝docker環(huán)境，
yum install docker-ce docker-ce-cli containerd.io -y
#或
yum install -y docker-ce-23.0.1-1.el7 docker-ce-cli-23.0.1-1.el7 containerd.io docker-buildx-plugin docker-compose-plugin #前三個(gè)必安裝，推薦安裝1.11.2以上版本的容器引擎客戶端
#創(chuàng)建用戶
usermod -aG docker $(USER)
#啟動(dòng)
systemctl start docker
systemctl status docker #輸出如下
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Active: active (running) since Tue 2024-01-16 17:48:54 CST; 5s ago
Docs: https://docs.docker.com
Main PID: 4549 (dockerd)
Memory: 25.0M
CGroup: /system.slice/docker.service
└─4549 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
Jan 16 17:48:52 Desktopvideo systemd[1]: Starting Docker Application Container Engine...
Jan 16 17:48:52 Desktopvideo dockerd[4549]: time="2024-01-16T17:48:52.895562767+08:00" level=info msg="Starting up"
Jan 16 17:48:53 Desktopvideo dockerd[4549]: time="2024-01-16T17:48:53.034501644+08:00" level=info msg="Loading...art."
Jan 16 17:48:53 Desktopvideo dockerd[4549]: time="2024-01-16T17:48:53.894486165+08:00" level=info msg="Loading...one."
Jan 16 17:48:53 Desktopvideo dockerd[4549]: time="2024-01-16T17:48:53.994229637+08:00" level=info msg="Docker ...4.0.7
Jan 16 17:48:53 Desktopvideo dockerd[4549]: time="2024-01-16T17:48:53.994411964+08:00" level=info msg="Daemon ...tion"
Jan 16 17:48:54 Desktopvideo dockerd[4549]: time="2024-01-16T17:48:54.145085504+08:00" level=info msg="API lis...sock"
Jan 16 17:48:54 Desktopvideo systemd[1]: Started Docker Application Container Engine.
Hint: Some lines were ellipsized, use -l to show in full.
#配置開(kāi)機(jī)自啟
systemctl enable docker #輸出如下
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service
#驗(yàn)證
docker version #輸出如下
Client: Docker Engine - Community
Version: 24.0.7
API version: 1.43
Go version: go1.20.10
Git commit: afdd53b
Built: Thu Oct 26 09:11:35 2023
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 24.0.7
API version: 1.43 (minimum version 1.12)
Go version: go1.20.10
Git commit: 311b9ff
Built: Thu Oct 26 09:10:36 2023
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.27
GitCommit: a1496014c916f9e62104b33d1bb5bd03b0858e59
runc:
Version: 1.1.11
GitCommit: v1.1.11-0-g4bccb38
docker-init:
Version: 0.19.0
GitCommit: de40ad0
===============================
docker run hello-world #驗(yàn)證，第一次會(huì)下載，然后再輸出Hello from Docker!
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
c1ec31eb5944: Pull complete
Digest: sha256:4bd78111b6914a99dbc560e6a20eab57ff6655aea4a80c50b0c5491968cbc2e6
Status: Downloaded newer image for hello-world:latest
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
(amd64)
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/
For more examples and ideas, visit:
https://docs.docker.com/get-started/
docker history docker.io/hello-world #輸出如下
IMAGE CREATED CREATED BY SIZE COMMENT
d2c94e258dcb 8 months ago CMD ["/hello"] 0B buildkit.dockerfile.v0
<missing> 8 months ago COPY hello / # buildkit 13.3kB buildkit.dockerfile.v0
docker info #輸出如下
Client: Docker Engine - Community
Version: 24.0.7
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.11.2
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.21.0
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 2
Running: 0
Paused: 0
Stopped: 2
Images: 1
Server Version: 24.0.7
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc #這里
Default Runtime: runc
Init Binary: docker-init
containerd version: a1496014c916f9e62104b33d1bb5bd03b0858e59
runc version: v1.1.11-0-g4bccb38
init version: de40ad0
Security Options:
seccomp
Profile: builtin
Kernel Version: 3.10.0-693.21.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.51GiB
Name: Desktopvideo
ID: 07c30337-2af0-4f9f-bb63-fc5e0af01a76
Docker Root Dir: /var/lib/docker #默認(rèn)位置
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
2.2、配置
注：安裝完docker后會(huì)生成一個(gè)docker0的網(wǎng)卡和br-x開(kāi)頭的網(wǎng)橋，從外部訪問(wèn)容器內(nèi)部時(shí)只需要訪問(wèn)宿主機(jī)的地址和對(duì)應(yīng)的容器映射的地址，訪問(wèn)的數(shù)據(jù)包到宿主機(jī)上后經(jīng)過(guò)ip包解析后通過(guò)目的port和iptables的規(guī)則會(huì)將數(shù)據(jù)包由eth0網(wǎng)卡轉(zhuǎn)發(fā)至網(wǎng)橋上的docker0上進(jìn)行下一步路由。所以如果容器的宿主機(jī)上的ip_forward未打開(kāi)，那么該宿主機(jī)上的容器則不能被其他宿主機(jī)訪問(wèn)；net.bridge.bridge-nf-call-iptables用于轉(zhuǎn)發(fā)是因默認(rèn)情況下，從容器發(fā)送到默認(rèn)網(wǎng)橋的流量，不會(huì)被轉(zhuǎn)發(fā)到外部。
#開(kāi)啟包轉(zhuǎn)發(fā)功能和修改內(nèi)核參數(shù)
modprobe br_netfilter #用于linux iptables/netfilter通過(guò)和linux bridge功能聯(lián)動(dòng)，以實(shí)現(xiàn)透明防火墻功能，它也稱(chēng)橋接模式防火墻（Bridge Firewall），也就是在網(wǎng)橋設(shè)備上加入防火墻功能，否則下面內(nèi)核參數(shù)啟用時(shí)會(huì)報(bào)錯(cuò)
echo "modprobe br_netfilter" >> /etc/profile
cat > /etc/sysctl.d/docker.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
#應(yīng)用生效
sysctl -p /etc/sysctl.d/docker.conf
#拉取倉(cāng)庫(kù)鏡像
docker pull distribution/registry #輸出如下
Using default tag: latest
latest: Pulling from distribution/registry
90f4dba627d6: Pull complete
6ca70669e0c7: Pull complete
8afccc587e11: Pull complete
5d5d4c33684a: Pull complete
6865c91bf9e0: Pull complete
24ec77c4f34d: Pull complete
d998fd3ae604: Pull complete
9ae2aa709472: Pull complete
dfbbb717df56: Pull complete
9f9a21385c8d: Pull complete
30cb17477211: Pull complete
Digest: sha256:910528144c2dcbfbf1ee85b9b3c8b8312ef4222e61e7d6282b0b17d6e363611c
Status: Downloaded newer image for distribution/registry:latest
docker.io/distribution/registry:latest
docker images |grep registry #驗(yàn)證
distribution/registry latest 71b9fde93b3f 5 years ago 299MB
#使用registry鏡像搭建一個(gè)私有倉(cāng)庫(kù),名稱(chēng)建議小寫(xiě)；默認(rèn)情況下，Registry存放鏡像的目錄是/var/lib/registry目錄下，這樣如果容器被刪除，則存放于容器中的鏡像也會(huì)丟失，所以我們一般情況下會(huì)使用-v參數(shù)指定本地持久的路徑，映射或掛載到容器的/var/lib/registry下，其中registry目錄會(huì)自動(dòng)創(chuàng)建
docker run -d -v /data/docker/registry:/var/lib/registry -p 5000:5000 --restart=always --name myhub distribution/registry
docker ps -a|grep regis #驗(yàn)證
b7715545dda9 distribution/registry "registry serve /etc…" 14 seconds ago Up 11 seconds 0.0.0.0:5000->5000/tcp, :::5000->5000/tcp myhub
curl http://localhost:5000/v2/_catalog
{"repositories":[]}
2.3、構(gòu)建私有倉(cāng)庫(kù)
1）私有倉(cāng)庫(kù)因外聯(lián)公網(wǎng)，可以直接從hub下載鏡像，然后執(zhí)行docker load -i xx.tar進(jìn)行上傳到本地倉(cāng)庫(kù)，也可直接pull
docker images #驗(yàn)證本地倉(cāng)庫(kù)已有鏡像，這里就用hello-world測(cè)試，你可自行pull其他鏡像，比如busybox，它是一個(gè)集成了一百多個(gè)最常用Linux命令和工具的工具箱，更多參看：www.busybox.net
REPOSITORY TAG IMAGE ID CREATED SIZE
hello-world latest d2c94e258dcb 8 months ago 13.3kB
distribution/registry latest 71b9fde93b3f 5 years ago 299MB
#修改鏡像標(biāo)簽為你的私有倉(cāng)庫(kù)鏡像
#語(yǔ)法： docker tag 原鏡像名:標(biāo)簽 私有倉(cāng)庫(kù)地址/新鏡像名:標(biāo)簽，如果不寫(xiě)鏡像標(biāo)簽，默認(rèn)是：latest
#docker tag <public-image-name> <your-registry-address>/<image-name>
docker tag hello-world 172.1.18.9:5000/hello-world:v1
docker images #驗(yàn)證
REPOSITORY TAG IMAGE ID CREATED SIZE
172.1.18.9:5000/hello-world v1 d2c94e258dcb 8 months ago 13.3kB
hello-world latest d2c94e258dcb 8 months ago 13.3kB
distribution/registry latest 71b9fde93b3f 5 years ago 299MB
#修改私有倉(cāng)庫(kù)本地配置文件
vim /etc/docker/deamon.json #添加如下
{
"insecure-registries":["172.1.18.9:5000"]
}
#完成后，重啟docker
systemctl restart docker
docker info #驗(yàn)證，輸出如下即可
……
Insecure Registries:
172.1.18.9:5000
127.0.0.0/8
#將172.1.18.9:5000/hello-world:v1 push到私有倉(cāng)庫(kù)里
docker push 172.1.18.9:5000/hello-world:v1 #輸出如下
The push refers to repository [172.1.18.9:5000/hello-world]
ac28800ec8bb: Pushed
v1: digest: sha256:d37ada95d47ad12224c205a938129de52345828b4fa27b03a98825d1e2e7 size: 524
#驗(yàn)證
curl -S http://172.16.18.9:5000/v2/_catalog #如下所示，已經(jīng)可見(jiàn)剛才上傳的鏡像了，當(dāng)然客戶端也可驗(yàn)證
{"repositories":["hello-world"]}
2）啟動(dòng)registry-web
由于官方的registry只是提供了api，沒(méi)有一個(gè)圖形化界面供顯示和操作，所以我們可以利用第三方提供的鏡像啟動(dòng)一個(gè)圖形化頁(yè)面。進(jìn)入容器的工作目錄（/var/lib/docker/containers//）， 創(chuàng)建config.yml文件（容器的配置文件），配置：
docker inspect <container-id> | grep "ConfigPath" #查看指定容器去配置文件
cd /var/lib/docker/b7715545dda975dd41b4d571d329e8edb34c07f236e03f144b1c68e552fc8ec4/
ls #輸出如下
b7715545dda975dd41b4d571d329e8edb34c07f236e03f144b1c68e552fc8ec4-json.log
checkpoints
config.v2.json
hostconfig.json
hostname
hosts
mounts
resolv.conf
resolv.conf.hash
#創(chuàng)建config.yml文件，配置：
registry:
# Docker registry url
url: http://172.1.18.9:5000/v2
# Docker registry fqdn
name: localhost:5000
# To allow image delete, should be false
readonly: false
auth:
# Disable authentication
enabled: false
#拉取web容器，找一個(gè)stars多的
docker pull hyper/docker-registry-web
#啟動(dòng)容器
mkdir -p /etc/docker/container/conf/
docker run -d -p 18080:18080 --name my-registry-web --restart=always -v $(pwd)/config.yml:/etc/docker/container/conf/config.yml:ro hyper/docker-registry-web
#驗(yàn)證
docker inspect --format '{{ .NetworkSettings.IPAddress }}' registry_id
curl -XGET http://localhost:5000/v2/_catalog
curl -s -XGET localhost:5000/v2/_catalog
curl -XGET http://localhost:5000/v2/hello-world/tags/list #輸出如下
{"name":"hello-world","tags":["v1"]}
#刪除鏡像：DELETE /v2/<name>/manifests/<reference>（鏡像對(duì)應(yīng)sha256值）
curl -I -X DELETE http://localhost:5000/v2/hello-world/manifests/sha256:48c7f9c92844bbbb5d0a101392f7c2a7949e40f8ea90c8b3bc396879d95e897e
#報(bào)錯(cuò)
024-01-17 04:21:29,399 [http-bio-8080-exec-1] ERROR web.RepositoryController - Can't access registry: _catalog?n=100
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://localhost:5000/v2/_catalog?n=100":Connection refused; nested exception is java.net.ConnectException: Connection refused
at grails.plugins.rest.client.RestBuilder.invokeRestTemplate(RestBuilder.groovy:312)
at docker.registry.web.CustomRestBuilder.request(CustomRestBuilder.groovy:22)
at docker.registry.web.RestService.requestInternal(RestService.groovy:70)
at docker.registry.web.RestService.request(RestService.groovy:63)
at docker.registry.web.RestService.get(RestService.groovy:42)
at docker.registry.web.RepositoryController.index(RepositoryController.groovy:29)
at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198)
at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
at grails.plugin.springsecurity.web.filter.GrailsAnonymousAuthenticationFilter.doFilter(GrailsAnonymousAuthenticationFilter.java:53)
at grails.plugin.springsecurity.web.authentication.logout.MutableLogoutFilter.doFilter(MutableLogoutFilter.java:62)
at grails.plugin.springsecurity.web.SecurityRequestHolderFilter.doFilter(SecurityRequestHolderFilter.java:59)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
... 14 more
現(xiàn)場(chǎng)遇到了報(bào)錯(cuò)：
#源命令
docker run -d -p 8080:8080 --name my-registry-web --restart=always -v $(pwd)/config.yml:/etc/docker/container/conf/config.yml:ro hyper/docker-registry-web
#修改
docker run -d -p 8080:8080 --restart=always --name my-registry-web --link myhub -e REGISTRY_URL=http://172.1.18.9:5000/v2 -e REGISTRY_NAME=localhost:5000 hyper/docker-registry-web
3）設(shè)置http訪問(wèn)的私有鏡像倉(cāng)庫(kù)容器
docker從1.3.X之后，與docker registry交互默認(rèn)使用的是https，修改docker啟動(dòng)配置文件，在啟動(dòng)docker server時(shí)增加啟動(dòng)參數(shù)為默認(rèn)使用http訪問(wèn)?？赏ㄟ^(guò)修改啟動(dòng)服務(wù)來(lái)，修改docker啟動(dòng)時(shí)的倉(cāng)庫(kù)訪問(wèn)
cp /usr/lib/systemd/system/docker.service /usr/lib/systemd/system/docker.service.bak
修改docker.service
vim /usr/lib/systemd/system/docker.service #找到 ExecStart的行，修改成下面這樣
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --insecure-registry localhost:5000
#或
ExecStart=/usr/bin/dockerd $OPTIONS \
$DOCKER_STORAGE_OPTIONS \
$DOCKER_NETWORK_OPTIONS \
$INSECURE_REGISTRY \
--insecure-registry localhost:5000
注意上面的不要與配置文件沖突了，二選一，否則報(bào)錯(cuò)：
6 dockerd[1011290]: unable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives are specified both as a flag and in the configuration file: insecure-registries: (from flag: [172.1.18.9:5000], from file: [172.1.18.9:5000])
4）設(shè)置https訪問(wèn)的私有鏡像倉(cāng)庫(kù)容器
yum -y install httpd-tools #安裝加密工具
cd /etc/docker/
mkdir certs
#生產(chǎn)自簽證書(shū)，跟ssh配置一樣
openssl req -newkey rsa:4096 -nodes -sha256 -keyout certs/registry.key -x509 -days 365 -out certs/registry.crt #類(lèi)似如下
Generating a 4096 bit RSA private key
......................................++
...........++
writing new private key to 'certs/registry.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:beijing
Locality Name (eg, city) [Default City]:
Organization Name (eg, company) [Default Company Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (eg, your name or your server's hostname) []:localhost
Email Address []:
#生成鑒權(quán)密碼文件：username和password替換成對(duì)應(yīng)的用戶名和密碼,B強(qiáng)制密碼加密；b在命令中輸入密碼，不提示輸入密碼；n不更新密鑰文件
docker run --entrypoint htpasswd registry -Bbn username password > auth/htpasswd
#啟動(dòng)registry容器
docker run -d -p 5000:5000 --restart=always --name localhost \
-v `pwd`/auth:/auth \
-e "REGISTRY_AUTH=htpasswd" \
-e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" \
-e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
-v `pwd`/certs:/certs \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/registry.crt \
-e REGISTRY_HTTP_TLS_KEY=/certs/registry.key \
-v /opt/data/registry:/var/lib/registry \
-v /usr/local/docker/config.yml:/etc/docker/registry/config.yml registry
#如果不啟用密碼
docker run -d -p 5000:5000 --restart=always --name localhost \
-v `pwd`/etc/docker/certs:/certs \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/etc/docker/certs/registry.crt \
-e REGISTRY_HTTP_TLS_KEY=/etc/docker/certs/registry.key \
-v /data/registry:/var/lib/registry \
-v /var/lib/docker/containers/config.yml:/etc/docker/registry/config.yml registry
#客戶端側(cè)
mkdir -p /etc/docker/certs.d/
#從私有倉(cāng)庫(kù)服務(wù)器將證書(shū)拷貝到client中， client不安裝證書(shū)的話，進(jìn)行pull/push操作，會(huì)出現(xiàn)x509: certificate signed by unknown authority的報(bào)錯(cuò)。
scp ……
#client重啟Docker Daemon
2.4、Docker內(nèi)網(wǎng)客戶端側(cè)配置
1）修改docker配置文件（沒(méi)有的新建即可）：/etc/docker/deamon.json，指定docker鏡像加速結(jié)點(diǎn)為：上文建立的私有倉(cāng)庫(kù)的地址
{
"insecure-registries":["172.1.18.9:5000"],
"registry-mirrors": [
"http://hub-mirror.c.163.com",
"https://docker.mirrors.ustc.edu.cn",
"https://cr.console.aliyun.com",
"https://hub.daocloud.io",
"https://5ufvppm7.mirror.aliyuncs.com",
"https://6iuzoxz4.mirror.aliyuncs.com",
"http://192.168.1.12:8080" #可只寫(xiě)這一項(xiàng)
]
}
完成后，重啟docker：systemctl restart docker
docker info #驗(yàn)證，輸出如下
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 18.09.0
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Hugetlb Pagesize: 2MB, 1GB, 2MB, 1GB (default is 2MB)
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 5dd35b83aeab1780082734b68e92a3991d3f8eee
runc version: 1f4c5ce0be8a1bfc8486bbcb7e84e92848366c93
init version: N/A (expected: )
Security Options:
seccomp
Profile: default
Kernel Version: 5.10.0-60.70.0.94.oe2203.bclinux.x86_64
Operating System: BigCloud Enterprise Linux For Euler 22.10 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 14.88GiB
Name: ygcg-manger206
ID: WTZ5:5RCE:ZOOP:7FPP:ZWCS:PY2F:P54Q:P5HP:W6MN:DRC3:2AM4:ATPX
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
172.1.18.9:5000
127.0.0.0/8
Registry Mirrors:
http://hub-mirror.c.163.com/
https://docker.mirrors.ustc.edu.cn/
https://cr.console.aliyun.com/
https://hub.daocloud.io/
https://5ufvppm7.mirror.aliyuncs.com/
Live Restore Enabled: true
2）驗(yàn)證使用
目前無(wú)法再客戶端使用 docker search host:5000/registry進(jìn)行查詢(xún)
# 目前無(wú)法用docker search
#下面是一個(gè)網(wǎng)絡(luò)示例，可列出私有倉(cāng)庫(kù)的鏡像腳本
#!/usr/bin/env python
#-*- coding:utf-8 -*-
import requests
import json
import traceback
import sys
repo_ip = sys.argv[1]
repo_port = sys.argv[2]
def getImagesNames(repo_ip,repo_port):
docker_images = []
try:
url = "http://" + repo_ip + ":" + repo_port + "/v2/_catalog"
res =requests.get(url).content.strip()
res_dic = json.loads(res)
images_type = res_dic['repositories']
for i in images_type:
url2 = "http://" + repo_ip + ":" + repo_port +"/v2/" + str(i) + "/tags/list"
res2 =requests.get(url2).content.strip()
res_dic2 = json.loads(res2)
name = res_dic2['name']
tags = res_dic2['tags']
if tags != None:
for tag in tags:
docker_name = str(repo_ip) + ":" + repo_port + "/" + name + ":" + tag
docker_images.append(docker_name)
print(docker_name)
except:
traceback.print_exc()
return docker_images
getImagesNames(repo_ip, repo_port)
#shell腳本
#!/bin/bash
file=$(mktemp)
curl -s $1:5000/v2/_catalog | jq | egrep -v '\{|\}|\[|]' | awk -F\" '{print $2}' > $file
while read aa ; do
tag=($(curl -s $1:5000/v2/$aa/tags/list | jq | egrep -v '\{|\}|\[|]|name' | awk -F\" '{print $2}'))
for i in ${tag[*]} ; do
echo $1:5000/${aa}:$i
done
done < $file
rm -rf $file
#查看是，執(zhí)行
python list_private_images.py ip:5000
#容器鏡像刪除后垃圾回收
docker exec -ti myhub sh
registry garbage-collect /etc/docker/registry/config.yml
#刪除鏡像倉(cāng)庫(kù)的腳本
#!/usr/bin/env python
"""
Usage:
Shut down your registry service to avoid race conditions and possible data loss
and then run the command with an image repo like this:
delete_docker_registry_image.py --image awesomeimage --dry-run
"""
import argparse
import json
import logging
import os
import sys
import shutil
import glob
logger = logging.getLogger(__name__)
def del_empty_dirs(s_dir, top_level):
"""recursively delete empty directories"""
b_empty = True
for s_target in os.listdir(s_dir):
s_path = os.path.join(s_dir, s_target)
if os.path.isdir(s_path):
if not del_empty_dirs(s_path, False):
b_empty = False
else:
b_empty = False
if b_empty:
logger.debug("Deleting empty directory '%s'", s_dir)
if not top_level:
os.rmdir(s_dir)
return b_empty
def get_layers_from_blob(path):
"""parse json blob and get set of layer digests"""
try:
with open(path, "r") as blob:
data_raw = blob.read()
data = json.loads(data_raw)
if data["schemaVersion"] == 1:
result = set([entry["blobSum"].split(":")[1] for entry in data["fsLayers"]])
else:
result = set([entry["digest"].split(":")[1] for entry in data["layers"]])
if "config" in data:
result.add(data["config"]["digest"].split(":")[1])
return result
except Exception as error:
logger.critical("Failed to read layers from blob:%s", error)
return set()
def get_digest_from_blob(path):
"""parse file and get digest"""
try:
with open(path, "r") as blob:
return blob.read().split(":")[1]
except Exception as error:
logger.critical("Failed to read digest from blob:%s", error)
return ""
def get_links(path, _filter=None):
"""recursively walk `path` and parse every link inside"""
result = []
for root, _, files in os.walk(path):
for each in files:
if each == "link":
filepath = os.path.join(root, each)
if not _filter or _filter in filepath:
result.append(get_digest_from_blob(filepath))
return result
class RegistryCleanerError(Exception):
pass
class RegistryCleaner(object):
"""Clean registry"""
def __init__(self, registry_data_dir, dry_run=False):
self.registry_data_dir = registry_data_dir
if not os.path.isdir(self.registry_data_dir):
raise RegistryCleanerError("No repositories directory found inside " \
"REGISTRY_DATA_DIR '{0}'.".
format(self.registry_data_dir))
self.dry_run = dry_run
def _delete_layer(self, repo, digest):
"""remove blob directory from filesystem"""
path = os.path.join(self.registry_data_dir, "repositories", repo, "_layers/sha256", digest)
self._delete_dir(path)
def _delete_blob(self, digest):
"""remove blob directory from filesystem"""
path = os.path.join(self.registry_data_dir, "blobs/sha256", digest[0:2], digest)
self._delete_dir(path)
def _blob_path_for_revision(self, digest):
"""where we can find the blob that contains the json describing this digest"""
return os.path.join(self.registry_data_dir, "blobs/sha256",
digest[0:2], digest, "data")
def _blob_path_for_revision_is_missing(self, digest):
"""for each revision, there should be a blob describing it"""
return not os.path.isfile(self._blob_path_for_revision(digest))
def _get_layers_from_blob(self, digest):
"""get layers from blob by digest"""
return get_layers_from_blob(self._blob_path_for_revision(digest))
def _delete_dir(self, path):
"""remove directory from filesystem"""
if self.dry_run:
logger.info("DRY_RUN: would have deleted %s", path)
else:
logger.info("Deleting %s", path)
try:
shutil.rmtree(path)
except Exception as error:
logger.critical("Failed to delete directory:%s", error)
def _delete_from_tag_index_for_revision(self, repo, digest):
"""delete revision from tag indexes"""
paths = glob.glob(
os.path.join(self.registry_data_dir, "repositories", repo,
"_manifests/tags/*/index/sha256", digest)
)
for path in paths:
self._delete_dir(path)
def _delete_revisions(self, repo, revisions, blobs_to_keep=None):
"""delete revisions from list of directories"""
if blobs_to_keep is None:
blobs_to_keep = []
for revision_dir in revisions:
digests = get_links(revision_dir)
for digest in digests:
self._delete_from_tag_index_for_revision(repo, digest)
if digest not in blobs_to_keep:
self._delete_blob(digest)
self._delete_dir(revision_dir)
def _get_tags(self, repo):
"""get all tags for given repository"""
path = os.path.join(self.registry_data_dir, "repositories", repo, "_manifests/tags")
if not os.path.isdir(path):
logger.critical("No repository '%s' found in repositories directory %s",
repo, self.registry_data_dir)
return None
result = []
for each in os.listdir(path):
filepath = os.path.join(path, each)
if os.path.isdir(filepath):
result.append(each)
return result
def _get_repositories(self):
"""get all repository repos"""
result = []
root = os.path.join(self.registry_data_dir, "repositories")
for each in os.listdir(root):
filepath = os.path.join(root, each)
if os.path.isdir(filepath):
inside = os.listdir(filepath)
if "_layers" in inside:
result.append(each)
else:
for inner in inside:
result.append(os.path.join(each, inner))
return result
def _get_all_links(self, except_repo=""):
"""get links for every repository"""
result = []
repositories = self._get_repositories()
for repo in [r for r in repositories if r != except_repo]:
path = os.path.join(self.registry_data_dir, "repositories", repo)
for link in get_links(path):
result.append(link)
return result
def prune(self):
"""delete all empty directories in registry_data_dir"""
del_empty_dirs(self.registry_data_dir, True)
def _layer_in_same_repo(self, repo, tag, layer):
"""check if layer is found in other tags of same repository"""
for other_tag in [t for t in self._get_tags(repo) if t != tag]:
path = os.path.join(self.registry_data_dir, "repositories", repo,
"_manifests/tags", other_tag, "current/link")
manifest = get_digest_from_blob(path)
try:
layers = self._get_layers_from_blob(manifest)
if layer in layers:
return True
except IOError:
if self._blob_path_for_revision_is_missing(manifest):
logger.warn("Blob for digest %s does not exist. Deleting tag manifest: %s", manifest, other_tag)
tag_dir = os.path.join(self.registry_data_dir, "repositories", repo,
"_manifests/tags", other_tag)
self._delete_dir(tag_dir)
else:
raise
return False
def _manifest_in_same_repo(self, repo, tag, manifest):
"""check if manifest is found in other tags of same repository"""
for other_tag in [t for t in self._get_tags(repo) if t != tag]:
path = os.path.join(self.registry_data_dir, "repositories", repo,
"_manifests/tags", other_tag, "current/link")
other_manifest = get_digest_from_blob(path)
if other_manifest == manifest:
return True
return False
def delete_entire_repository(self, repo):
"""delete all blobs for given repository repo"""
logger.debug("Deleting entire repository '%s'", repo)
repo_dir = os.path.join(self.registry_data_dir, "repositories", repo)
if not os.path.isdir(repo_dir):
raise RegistryCleanerError("No repository '{0}' found in repositories "
"directory {1}/repositories".
format(repo, self.registry_data_dir))
links = set(get_links(repo_dir))
all_links_but_current = set(self._get_all_links(except_repo=repo))
for layer in links:
if layer in all_links_but_current:
logger.debug("Blob found in another repository. Not deleting: %s", layer)
else:
self._delete_blob(layer)
self._delete_dir(repo_dir)
def delete_repository_tag(self, repo, tag):
"""delete all blobs only for given tag of repository"""
logger.debug("Deleting repository '%s' with tag '%s'", repo, tag)
tag_dir = os.path.join(self.registry_data_dir, "repositories", repo, "_manifests/tags", tag)
if not os.path.isdir(tag_dir):
raise RegistryCleanerError("No repository '{0}' tag '{1}' found in repositories "
"directory {2}/repositories".
format(repo, tag, self.registry_data_dir))
manifests_for_tag = set(get_links(tag_dir))
revisions_to_delete = []
blobs_to_keep = []
layers = []
all_links_not_in_current_repo = set(self._get_all_links(except_repo=repo))
for manifest in manifests_for_tag:
logger.debug("Looking up filesystem layers for manifest digest %s", manifest)
if self._manifest_in_same_repo(repo, tag, manifest):
logger.debug("Not deleting since we found another tag using manifest: %s", manifest)
continue
else:
revisions_to_delete.append(
os.path.join(self.registry_data_dir, "repositories", repo,
"_manifests/revisions/sha256", manifest)
)
if manifest in all_links_not_in_current_repo:
logger.debug("Not deleting the blob data since we found another repo using manifest: %s", manifest)
blobs_to_keep.append(manifest)
layers.extend(self._get_layers_from_blob(manifest))
layers_uniq = set(layers)
for layer in layers_uniq:
if self._layer_in_same_repo(repo, tag, layer):
logger.debug("Not deleting since we found another tag using digest: %s", layer)
continue
self._delete_layer(repo, layer)
if layer in all_links_not_in_current_repo:
logger.debug("Blob found in another repository. Not deleting: %s", layer)
else:
self._delete_blob(layer)
self._delete_revisions(repo, revisions_to_delete, blobs_to_keep)
self._delete_dir(tag_dir)
def delete_untagged(self, repo):
"""delete all untagged data from repo"""
logger.debug("Deleting utagged data from repository '%s'", repo)
repositories_dir = os.path.join(self.registry_data_dir, "repositories")
repo_dir = os.path.join(repositories_dir, repo)
if not os.path.isdir(repo_dir):
raise RegistryCleanerError("No repository '{0}' found in repositories "
"directory {1}/repositories".
format(repo, self.registry_data_dir))
tagged_links = set(get_links(repositories_dir, _filter="current"))
layers_to_protect = []
for link in tagged_links:
layers_to_protect.extend(self._get_layers_from_blob(link))
unique_layers_to_protect = set(layers_to_protect)
for layer in unique_layers_to_protect:
logger.debug("layer_to_protect: %s", layer)
tagged_revisions = set(get_links(repo_dir, _filter="current"))
revisions_to_delete = []
layers_to_delete = []
dir_for_revisions = os.path.join(repo_dir, "_manifests/revisions/sha256")
for rev in os.listdir(dir_for_revisions):
if rev not in tagged_revisions:
revisions_to_delete.append(os.path.join(dir_for_revisions, rev))
for layer in self._get_layers_from_blob(rev):
if layer not in unique_layers_to_protect:
layers_to_delete.append(layer)
unique_layers_to_delete = set(layers_to_delete)
self._delete_revisions(repo, revisions_to_delete)
for layer in unique_layers_to_delete:
self._delete_blob(layer)
self._delete_layer(repo, layer)
def get_tag_count(self, repo):
logger.debug("Get tag count of repository '%s'", repo)
repo_dir = os.path.join(self.registry_data_dir, "repositories", repo)
tags_dir = os.path.join(repo_dir, "_manifests/tags")
if os.path.isdir(tags_dir):
tags = os.listdir(tags_dir)
return len(tags)
else:
logger.info("Tags directory does not exist: '%s'", tags_dir)
return -1
def main():
"""cli entrypoint"""
parser = argparse.ArgumentParser(description="Cleanup docker registry")
parser.add_argument("-i", "--image",
dest="image",
required=True,
help="Docker image to cleanup")
parser.add_argument("-v", "--verbose",
dest="verbose",
action="store_true",
help="verbose")
parser.add_argument("-n", "--dry-run",
dest="dry_run",
action="store_true",
help="Dry run")
parser.add_argument("-f", "--force",
dest="force",
action="store_true",
help="Force delete (deprecated)")
parser.add_argument("-p", "--prune",
dest="prune",
action="store_true",
help="Prune")
parser.add_argument("-u", "--untagged",
dest="untagged",
action="store_true",
help="Delete all untagged blobs for image")
args = parser.parse_args()
handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter(u'%(levelname)-8s [%(asctime)s] %(message)s'))
logger.addHandler(handler)
if args.verbose:
logger.setLevel(logging.DEBUG)
else:
logger.setLevel(logging.INFO)
# make sure not to log before logging is setup. that'll hose your logging config.
if args.force:
logger.info(
"You supplied the force switch, which is deprecated. It has no effect now, and the script defaults to doing what used to be only happen when force was true")
splitted = args.image.split(":")
if len(splitted) == 2:
image = splitted[0]
tag = splitted[1]
else:
image = args.image
tag = None
if 'REGISTRY_DATA_DIR' in os.environ:
registry_data_dir = os.environ['REGISTRY_DATA_DIR']
else:
registry_data_dir = "/opt/registry_data/docker/registry/v2"
try:
cleaner = RegistryCleaner(registry_data_dir, dry_run=args.dry_run)
if args.untagged:
cleaner.delete_untagged(image)
else:
if tag:
tag_count = cleaner.get_tag_count(image)
if tag_count == 1:
cleaner.delete_entire_repository(image)
else:
cleaner.delete_repository_tag(image, tag)
else:
cleaner.delete_entire_repository(image)
if args.prune:
cleaner.prune()
except RegistryCleanerError as error:
logger.fatal(error)
sys.exit(1)
if __name__ == "__main__":
main()
#刪除執(zhí)行
./delete_docker_registry_image -i localhost:5000/registry:latest
3）也可再內(nèi)網(wǎng)倉(cāng)庫(kù)上導(dǎo)出鏡像為tar包然后復(fù)制到內(nèi)網(wǎng)docker主機(jī)上使用
docker save -o hello-world.tar hello-world #導(dǎo)出
#導(dǎo)入
docker load -i hello-world.tar
三、附錄
1）容器重啟策略回顧
–restart=always重啟策略：
no：默認(rèn)策略，容器退出不重啟
on-failure：容器非正常退出重啟容器
on-failure：3：容器非正常退出最多重啟4次容器
always：退出總是重啟容器
unless-stopped：在容器退出時(shí)總是重啟容器，但是不考慮在Docker守護(hù)進(jìn)程啟動(dòng)時(shí)就已經(jīng)停止了的容器
2）使用Harbor 構(gòu)建私有倉(cāng)庫(kù)
Harbor 是 VMware 公司開(kāi)源的企業(yè)級(jí) Docker Registry 項(xiàng)目，其目標(biāo)是幫助用戶迅速搭建一個(gè)企業(yè)級(jí)的 Docker Registry 服務(wù)。Harbor以 Docker 公司開(kāi)源的 Registry 為基礎(chǔ)，提供了圖形管理 UI 、基于角色的訪問(wèn)控制(Role Based AccessControl,RBAC) 、AD/LDAP 集成、以及審計(jì)日志(Auditlogging) 、管理界面、自我注冊(cè)、鏡像復(fù)制等企業(yè)用戶需求的功能，同時(shí)還原生支持中文。更多參見(jiàn)：官網(wǎng)
Harbor 的每個(gè)組件都是以 Docker 容器的形式構(gòu)建的，使用 docker-compose 來(lái)對(duì)它進(jìn)行部署。用于部署 Harbor 的 docker-compose 模板位于 harbor/docker-compose.yml。
Harbor 在架構(gòu)上主要有 Proxy、Registry、Core services、Database（Harbor-db）、Log collector（Harbor-log）、Job services 六個(gè)組件。
①Proxy: 是一個(gè) nginx 的前端代理，Harbor 的 Registry、UI、Token 服務(wù)等組件，都處在 nginx 反向代理后邊。 該代理將來(lái)自瀏覽器、docker clients 的請(qǐng)求轉(zhuǎn)發(fā)到后端不同的服務(wù)上。
②Registry: 負(fù)責(zé)儲(chǔ)存 Docker 鏡像，并處理 Docker push/pull 命令。由于要對(duì)用戶進(jìn)行訪問(wèn)控制，即不同用戶對(duì) Docker 鏡像 有不同的讀寫(xiě)權(quán)限，Registry 會(huì)指向一個(gè) Token 服務(wù)，強(qiáng)制用戶的每次 Docker pull/push 請(qǐng)求都要攜帶一個(gè)合法的 Token， Registry 會(huì)通過(guò)公鑰對(duì) Token 進(jìn)行解密驗(yàn)證。
③Core services: Harbor的核心功能，主要提供以下3個(gè)服務(wù): 1）UI（harbor-ui）: 提供圖形化界面，幫助用戶管理 Registry 上的鏡像（image）, 并對(duì)用戶進(jìn)行授權(quán)。 2）WebHook：為了及時(shí)獲取Registry上image 狀態(tài)變化的情況，在Registry 上配置 Webhook，把狀態(tài)變化傳遞給 UI 模塊。 3）Token 服務(wù)：負(fù)責(zé)根據(jù)用戶權(quán)限給每個(gè) Docker push/pull 命令簽發(fā) Token。Docker 客戶端向 Registry 服務(wù)發(fā)起的請(qǐng)求， 如果不包含 Token，會(huì)被重定向到 Token 服務(wù)，獲得 Token 后再重新向 Registry 進(jìn)行請(qǐng)求。
④Database（harbor-db）：為core services提供數(shù)據(jù)庫(kù)服務(wù)，負(fù)責(zé)儲(chǔ)存用戶權(quán)限、審計(jì)日志、Docker 鏡像分組信息等數(shù)據(jù)。
⑤Job services: 主要用于鏡像復(fù)制，本地鏡像可以被同步到遠(yuǎn)程 Harbor 實(shí)例上。
⑥Log collector（harbor-log）: 負(fù)責(zé)收集其他組件的日志到一個(gè)地方。
Harbor的特性如下：文章來(lái)源：http://www.zghlxwxcb.cn/news/detail-800734.html
①、基于角色控制：用戶和倉(cāng)庫(kù)都是基于項(xiàng)目進(jìn)行組織的，而用戶在項(xiàng)目中可以擁有不同的權(quán)限。
②、基于鏡像的復(fù)制策略：鏡像可以在多個(gè)Harbor實(shí)例之間進(jìn)行復(fù)制（同步）。
③、支持 LDAP/AD：Harbor 可以集成企業(yè)內(nèi)部已有的 AD/LDAP（類(lèi)似數(shù)據(jù)庫(kù)的一張表），用于對(duì)已經(jīng)存在的用戶認(rèn)證和管理。
④、鏡像刪除和垃圾回收：鏡像可以被刪除，也可以回收鏡像占用的空間。
⑤、圖形化用戶界面：用戶可以通過(guò)瀏覽器來(lái)瀏覽，搜索鏡像倉(cāng)庫(kù)以及對(duì)項(xiàng)目進(jìn)行管理。
⑥、審計(jì)管理：所有針對(duì)鏡像倉(cāng)庫(kù)的操作都可以被記錄追溯，用于審計(jì)管理。
⑦、支持 RESTful API：RESTful API 提供給管理員對(duì)于 Harbor 更多的操控, 使得與其它管理軟件集成變得更容易。
⑧、Harbor和docker registry的關(guān)系：Harbor實(shí)質(zhì)上是對(duì)docker registry做了封裝，擴(kuò)展了自己的業(yè)務(wù)模板。文章來(lái)源地址http://www.zghlxwxcb.cn/news/detail-800734.html
到了這里，關(guān)于Docker本地私有倉(cāng)庫(kù)搭建配置指導(dǎo)的文章就介紹完了。如果您還想了解更多內(nèi)容，請(qǐng)?jiān)谟疑辖撬阉鱐OY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！
