Python網(wǎng)絡(luò)爬蟲(chóng)urllib庫(kù)使用指南
前言??
前篇簡(jiǎn)單介紹了什么是網(wǎng)絡(luò)爬蟲(chóng)及相關(guān)概念，這篇開(kāi)始講解爬蟲(chóng)中的第一個(gè)庫(kù)——urllib。
urllib??
urllib是Python標(biāo)準(zhǔn)庫(kù)中的一個(gè)模塊，提供了一些用于處理URL的功能。
使用urllib可以進(jìn)行URL的解析、發(fā)送HTTP請(qǐng)求、文件下載等操作。
1、urllib的基本使用??
使用urIlib來(lái)獲取百度首頁(yè)的源碼??
# 使用urI1ib來(lái)獲取百度首頁(yè)的源碼 import urllib.request # (1)定義一個(gè)urI 贏是你要訪問(wèn)的地址 url = 'http://www.baidu.com/' # (2)模擬瀏微器向服務(wù)器發(fā)達(dá)求 response阿應(yīng) response = urllib.request.urlopen(url) # (3)獲取響應(yīng)中的頁(yè)面的源碼 (content 內(nèi)容意思) # read方法 返回的是字節(jié)形式的二進(jìn)制數(shù)據(jù) content = response.read() # (4)打印數(shù)據(jù) print(content)
打印結(jié)果：
b'<!DOCTYPE html><!--STATUS OK--><html><head><meta http-equiv="Content-Type" content="text/html;charset=utf-8"><meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"><meta content="always" name="referrer"><meta name="theme-color" content="#ffffff"><meta name="description" content="\xe5\x85\xa8\xe7\x90\x83\xe9\xa2\x86\xe5\x85\x88\xe7\x9a\x84\xe4\xb8\xad\xe6\x96\x87\xe6\x90\x9c\xe7\xb4\xa2\xe5\xbc\x95\xe6\x93\x8e\xe3\x80\x81\xe8\x87\xb4\xe5\x8a\x9b\xe4\xba\x8e\xe8\xae\xa9\xe7\xbd\x91\xe6\xb0\x91\xe6\x9b\xb4\xe4\xbe\xbf\xe6\x8d\xb7\xe5\x9c\xb0\xe8\x8e\xb7\xe5\x8f\x96\xe4\xbf\xa1\xe6\x81\xaf\xef\xbc\x8c\xe6\x89\xbe\xe5\x88\xb0\xe6\x89\x80\xe6\xb1\x82\xe3\x80\x82\xe7\x99\xbe\xe5\xba\xa6\xe8\xb6\x85\xe8\xbf\x87\xe5\x8d\x83\xe4\xba\xbf\xe7\x9a\x84\xe4\xb8\xad\xe6\x96\x87\xe7\xbd\x91\xe9\xa1\xb5\xe6\x95\xb0\xe6\x8d\xae\xe5\xba\x93\xef\xbc\x8c\xe5\x8f\xaf\xe4\xbb\xa5\xe7\x9e\xac\xe9\x97\xb4\xe6\x89\xbe\xe5\x88\xb0\xe7\x9b\xb8\xe5\x85\xb3\xe7\x9a\x84\xe6\x90\x9c\xe7\xb4\xa2\xe7\xbb\x93\xe6\x9e\x9c\xe3\x80\x82"><link rel="shortcut icon" href="https://www.baidu.com/favicon.ico" type="image/x-icon" /><link rel="search" type="application/opensearchdescription+xml" href="/content-search.xml" title="\xe7\x99\xbe\xe5\xba\xa6\xe6\x90\x9c\xe7\xb4\xa2" /><link rel="icon" sizes="any" mask href="https://www.baidu.com/favicon.ico"><link rel="dns-prefetch" href="//dss0.bdstatic.com"/><link rel="dns-prefetch" href="//dss1.bdstatic.com"/><link rel="dns-prefetch" href="//ss1.bdstatic.com"/><link rel="dns-prefetch" href="//sp0.baidu.com"/><link rel="dns-prefetch" href="//sp1.baidu.com"/><link rel="dns-prefetch" href="//sp2.baidu.com"/><link rel="dns-prefetch" href="//pss.bdstatic.com"/><link rel="apple-touch-icon-precomposed" href="https://psstatic.cdn.bcebos.com/video/wiseindex/aa6eef91f8b5b1a33b454c401_1660835115000.png"><title>\xe7\x99\xbe\xe5\xba\xa6\xe4\xb8\x80\xe4\xb8\x8b\xef\xbc\x8c\xe4\xbd\xa0\xe5\xb0\xb1\xe7\x9f\xa5\xe9 ......
可以看到開(kāi)頭字母為b，這代表這返回的是字節(jié)形式的二進(jìn)制數(shù)據(jù)。我們想要看懂它需要將它轉(zhuǎn)換成字符串，也就是解碼（二進(jìn)制->字符串）。
源碼：源碼
# 使用urI1ib來(lái)獲取百度首頁(yè)的源碼
import urllib.request
# (1)定義一個(gè)urI 贏是你要訪問(wèn)的地址
url = 'http://www.baidu.com'
# (2)模擬瀏微器向服務(wù)器發(fā)達(dá)求 response阿應(yīng)
response = urllib.request.urlopen(url)
# (3)獲取響應(yīng)中的頁(yè)面的源碼 (content 內(nèi)容意思)
# read方法 返回的是字節(jié)形式的二進(jìn)制數(shù)據(jù) 我們使用decode進(jìn)行主動(dòng)解碼，將其轉(zhuǎn)換成字符串
content = response.read().decode("utf-8")
# (4)打印數(shù)據(jù)
print(content)打印結(jié)果：
2、一個(gè)類型六個(gè)方法??
一個(gè)類型??
為什么要講一個(gè)類型呢？因?yàn)楹竺嬉v的一個(gè)庫(kù)叫request庫(kù)，request庫(kù)也有相對(duì)應(yīng)的響應(yīng)，但是它的類型不一樣，為了進(jìn)行一個(gè)對(duì)比，才進(jìn)行這樣的一個(gè)講解。
import urllib.request url = "http://www.baidu.com" # 模擬瀏覽器向服務(wù)器發(fā)送請(qǐng)求 response = urllib.request.urlopen(url) # 一個(gè)類型六個(gè)方法 print(type(response))
類型打?。?nbsp;
<class 'http.client.HTTPResponse'>
從上面我們可以知道response的數(shù)據(jù)類型是HttpResponse，我們需要去記住。
六個(gè)方法??
Ⅰ、read方法
read方法是按照一個(gè)字節(jié)一個(gè)字節(jié)的去讀
import urllib.request url = "http://www.baidu.com" # 模擬瀏覽器向服務(wù)器發(fā)送請(qǐng)求 response = urllib.request.urlopen(url) # 按照一個(gè)字節(jié)一個(gè)字節(jié)的去讀 content = response.read() print(content)
read方法按字節(jié)讀
import urllib.request url = "http://www.baidu.com" # 模擬瀏覽器向服務(wù)器發(fā)送請(qǐng)求 response = urllib.request.urlopen(url) # read方法按字節(jié)讀 content = response.read(5) print(content)
打印結(jié)果：
b'<!DOC'
Ⅱ、readline方法按行讀
import urllib.request url = "http://www.baidu.com" # 模擬瀏覽器向服務(wù)器發(fā)送請(qǐng)求 response = urllib.request.urlopen(url) # readline方法按行讀 content = response.readline() print(content)
打印結(jié)果：
b'<!DOCTYPE html><!--STATUS OK--><html><head><meta http-equiv="Content-Type" content="text/html;charset=utf-8"><meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"><meta content="always" name="referrer"><meta name="theme-color" content="#ffffff"><meta name="description" content="\xe5\x85\xa8\xe7\x90\x83\xe9\xa2\x86\xe5\x85\x88\xe7\x9a\x84\xe4\xb8\xad\xe6\x96\x87\xe6\x90\x9c\xe7\xb4\xa2\xe5\xbc\x95\xe6\x93\x8e\xe3\x80\x81\xe8\x87\xb4\xe5\x8a\x9b\xe4\xba\x8e\xe8\xae\xa9\xe7\xbd\x91\xe6\xb0\x91\xe6\x9b\xb4\xe4\xbe\xbf\xe6\x8d\xb7\xe5\x9c\xb0\xe8\x8e\xb7\xe5\x8f\x96\xe4\xbf\xa1\xe6\x81\xaf\xef\xbc\x8c\xe6\x89\xbe\xe5\x88\xb0\xe6\x89\x80\xe6\xb1\x82\xe3\x80\x82\xe7\x99\xbe\xe5\xba\xa6\xe8\xb6\x85\xe8\xbf\x87\xe5\x8d\x83\xe4\xba\xbf\xe7\x9a\x84\xe4\xb8\xad\xe6\x96\x87\xe7\xbd\x91\xe9\xa1\xb5\xe6\x95\xb0\xe6\x8d\xae\xe5\xba\x93\xef\xbc\x8c\xe5\x8f\xaf\xe4\xbb\xa5\xe7\x9e\xac\xe9\x97\xb4\xe6\x89\xbe\xe5\x88\xb0\xe7\x9b\xb8\xe5\x85\xb3\xe7\x9a\x84\xe6\x90\x9c\xe7\xb4\xa2\xe7\xbb\x93\xe6\x9e\x9c\xe3\x80\x82"><link rel="shortcut icon" href="https://www.baidu.com/favicon.ico" type="image/x-icon" /><link rel="search" type="application/opensearchdescription+xml" href="/content-search.xml" title="\xe7\x99\xbe\xe5\xba\xa6\xe6\x90\x9c\xe7\xb4\xa2" /><link rel="icon" sizes="any" mask href="https://www.baidu.com/favicon.ico"><link rel="dns-prefetch" href="//dss0.bdstatic.com"/><link rel="dns-prefetch" href="//dss1.bdstatic.com"/><link rel="dns-prefetch" href="//ss1.bdstatic.com"/><link rel="dns-prefetch" href="//sp0.baidu.com"/><link rel="dns-prefetch" href="//sp1.baidu.com"/><link rel="dns-prefetch" href="//sp2.baidu.com"/><link rel="dns-prefetch" href="//pss.bdstatic.com"/><link rel="apple-touch-icon-precomposed" href="https://psstatic.cdn.bcebos.com/video/wiseindex/aa6eef91f8b5b1a33b454c401_1660835115000.png"><title>\xe7\x99\xbe\xe5\xba\xa6\xe4\xb8\x80\xe4\xb8\x8b\xef\xbc\x8c\xe4\xbd\xa0\xe5\xb0\xb1\xe7\x9f\xa5\xe9\x81\x93</title><style index="newi" type="text/css">#form .bdsug{top:39px}.bdsug{display:none;position:absolute;width:535px;background:#fff;border:1px solid #ccc!important;_overflow:hidden;box-shadow:1px 1px 3px #ededed;-webkit-box-shadow:1px 1px 3px #ededed;-moz-box-shadow:1px 1px 3px #ededed;-o-box-shadow:1px 1px 3px #ededed}.bdsug li{width:519px;color:#000;font:14px arial;line-height:25px;padding:0 8px;position:relative;cursor:default}.bdsug li.bdsug-s{background:#f0f0f0}.bdsug-store span,.bdsug-store b{color:#7A77C8}.bdsug-store-del{font-size:12px;color:#666;text-decoration:underline;position:absolute;right:8px;top:0;cursor:pointer;display:none}.bdsug-s .bdsug-store-del{display:inline-block}.bdsug-ala{display:inline-block;border-bottom:1px solid #e6e6e6}.bdsug-ala h3{line-height:14px;background:url(//www.baidu.com/img/sug_bd.png?v=09816787.png) no-repeat left center;margin:6px 0 4px;font-size:12px;font-weight:400;color:#7B7B7B;padding-left:20px}.bdsug-ala p{font-size:14px;font-weight:700;padding-left:20px}#m .bdsug .bdsug-direct p{color:#00c;font-weight:700;line-height:34px;padding:0 8px;margin-top:0;cursor:pointer;white-space:nowrap;overflow:hidden}#m .bdsug .bdsug-direct p img{width:16px;height:16px;margin:7px 6px 9px 0;vertical-align:middle}#m .bdsug .bdsug-direct p span{margin-left:8px}#form .bdsug .bdsug-direct{width:auto;padding:0;border-bottom:1px solid #f1f1f1}#form .bdsug .bdsug-direct p i{font-size:12px;line-height:100%;font-style:normal;font-weight:400;color:#fff;background-color:#2b99ff;display:inline;text-align:center;padding:1px 5px;*padding:2px 5px 0;margin-left:8px;overflow:hidden}.bdsug .bdsug-pcDirect{color:#000;font-size:14px;line-height:30px;height:30px;background-color:#f8f8f8}.bdsug .bdsug-pc-direct-tip{position:absolute;right:15px;top:8px;width:55px;height:15px;display:block;background:url(http://pss.bdstatic.com/r/www/cache/static/global/img/pc_direct_42d6311.png) no-repeat 0 0}.bdsug li.bdsug-pcDirect-s{background-color:#f0f0f0}.bdsug .bdsug-pcDirect-is{color:#000;font-size:14px;line-height:22px;background-color:#f5f5f5}.bdsug .bdsug-pc-direct-tip-is{position:absolute;right:15px;top:3px;width:55px;height:15px;display:block;background:url(http://pss.bdstatic.com/r/www/cache/static/global/img/pc_direct_42d6311.png) no-repeat 0 0}.bdsug li.bdsug-pcDirect-is-s{background-color:#f0f0f0}.bdsug .bdsug-pcDirect-s .bdsug-pc-direct-tip,.bdsug .bdsug-pcDirect-is-s .bdsug-pc-direct-tip-is{background-position:0 -15px}.bdsug .bdsug-newicon{color:#929292;opacity:.7;font-size:12px;display:inline-block;line-height:22px;letter-spacing:2px}.bdsug .bdsug-s .bdsug-newicon{opacity:1}.bdsug .bdsug-newicon i{letter-spacing:0;font-style:normal}.bdsug .bdsug-feedback-wrap{display:none}.toggle-underline{text-decoration:none}.toggle-underline:hover{text-decoration:underline}.bdpfmenu,.usermenu{border:1px solid #d1d1d1;position:absolute;width:105px;top:36px;z-index:302;box-shadow:1px 1px 5px #d1d1d1;-webkit-box-shadow:1px 1px 5px #d1d1d1;-moz-box-shadow:1px 1px 5px #d1d1d1;-o-box-shadow:1px 1px 5px #d1d1d1}.bdpfmenu{font-size:12px;background-color:#fff}.bdpfmenu a,.usermenu a{display:block;text-align:left;margin:0!important;padding:0 9px;line-height:26px;text-decoration:none}.briiconsbg{background-repeat:no-repeat;background-size:300px 18px;background-image:url(http://pss.bdstatic.com/r/www/cache/static/home/img/icons_0c37e9b.png);background-image:url(http://pss.bdstatic.com/r/www/cache/static/home/img/icons_809ae65.gif)\\9}.bdpfmenu a:link,.bdpfmenu a:visited,#u .usermenu a:link,#u .usermenu a:visited{background:#fff;color:#333}.bdpfmenu a:hover,.bdpfmenu a:active,#u .usermenu a:hover,#u .usermenu a:active{background:#38f;text-decoration:none;color:#fff}.bdpfmenu{width:70px}#wrapper .bdnuarrow{width:0;height:0;font-size:0;line-height:0;display:block;position:absolute;top:-10px;left:50%;margin-left:-5px}#wrapper .bdnuarrow em,#wrapper .bdnuarrow i{width:0;height:0;font-size:0;line-height:0;display:block;position:absolute;border:5px solid transparent;border-style:dashed dashed solid}#wrapper .bdnuarrow em{border-bottom-color:#d8d8d8;top:-1px}#wrapper .bdnuarrow i{border-bottom-color:#fff;top:0}#gxszHead .prefpanelclose{cursor:pointer;width:16px;height:16px;float:right;margin-top:7px;background-position:-248px 0}#gxszHead .prefpanelclose:hover{background-position:-264px 0}.s_ipt::-webkit-input-placeholder{padding-left:3px;color:#aaa;font-size:13px}.s_ipt::-moz-placeholder{padding-left:3px;color:#aaa;font-size:13px}.s_ipt:-ms-input-placeholder{padding-left:3px;color:#aaa;font-size:13px}.s_ipt::placeholder{padding-left:3px;color:#aaa;font-size:13px}.kw-placeholder{position:absolute;top:0;left:0;color:#aaa;font-size:13px;height:40px;line-height:40px;padding-left:10px;max-width:360px;z-index:99;pointer-events:none}.kw-placeholder.kw-placehlder-high{height:40px;line-height:40px}.kw-placeholder.placeholders-hidden{visibility:hidden}#head_wrapper #form .bdsug-new{width:544px;top:35px;border-radius:0 0 10px 10px;border:2px solid #4E6EF2!important;border-top:0!important;box-shadow:none;font-family:Arial,sans-serif;z-index:1}#head_wrapper.sam_head_wrapper2 #form .bdsug-new{width:545px;z-index:1;border:1px solid #4E6EF2!important;border-top:0!important}#head_wrapper #form .bdsug-new ul{margin:7px 14px 0;padding:8px 0 7px;background:0 0;border-top:2px solid #f5f5f6}#head_wrapper #form .bdsug-new ul li{width:auto;padding-left:14px;margin-left:-14px;margin-right:-14px;color:#626675;line-height:28px;background:0 0;font-family:Arial,sans-serif}#head_wrapper #form .bdsug-new ul li .sug-search-icon,#head_wrapper #form .bdsug-new ul li .sug-history-icon{margin-right:4px;color:#222}#head_wrapper #form .bdsug-new ul li span{color:#626675}#head_wrapper #form .bdsug-new ul li b{font-weight:400;color:#222}#head_wrapper #form .bdsug-new .bdsug-store-del{font-size:13px;text-decoration:none;color:#9195A3;right:16px}#head_wrapper #form .bdsug-new .bdsug-store-del:hover{color:#315EFB;cursor:pointer}#head_wrapper #form .bdsug-new ul li:hover,#head_wrapper #form .bdsug-new ul li:hover span,#head_wrapper #form .bdsug-new ul li:hover b{cursor:pointer}.wrapper_new #form .bdsug-new .bdsug-s{background-color:#F5F5F6!important}.wrapper_new #form .sam_search .bdsug-new .bdsug-s{background-color:#F1F3FD!important}#head_wrapper #form .sam_search .bdsug-new .bdsug-s{background-color:#F1F3FD!important}#head .s-down #form .bdsug-new{top:32px}.s-skin-hasbg #head_wrapper #form .bdsug-new{border-color:#4569ff!important;border-top:0!important}.s-skin-hasbg #head_wrapper.s-down #form .bdsug-new{border-color:#4e6ef2!important;border-top:0!important}.s-skin-hasbg #head_wrapper.s-down #form.sam_search .bdsug-new{border-color:rgba(0,0,0,.05)!important;border-top:1px solid rgba(0,0,0,.05)!important;top:54px!important}#head_wrapper #form .bdsug-new .bdsug-s,#head_wrapper #form .bdsug-new .bdsug-s span,#head_wrapper #form .bdsug-new .bdsug-s b{color:#315EFB}#head_wrapper #form .bdsug-new>div span:hover,#head_wrapper #form .bdsug-new>div a:hover{color:#315EFB!important}#head_wrapper #form #kw.new-ipt-focus{border-color:#4e6ef2}#head_wrapper #form .bdsug-new ul li{}#head_wrapper #form .bdsug-new ul li .sug-hot-orange,#head_wrapper #form .bdsug-new ul li .sug-hot-grey,#head_wrapper #form .bdsug-new ul li .sug-hot-blue{display:inline-block;width:12px;height:12px;font-size:12px;line-height:12px;padding:2px;text-align:center;font-weight:500;margin-left:6px;vertical-align:text-bottom}#head_wrapper #form .bdsug-new ul li .sug-hot-orange{display:inline-block;color:#fff;background:#F60;border-radius:4px}#head_wrapper #form .bdsug-new ul li .sug-new-tag{text-align:center;margin-left:6px;box-sizing:border-box;font-size:12px;line-height:14px;padding:1px 4px;font-weight:500}#head_wrapper #form .bdsug-new ul li .sug-new-tag-grey{color:#858585;border:1px solid rgba(133,133,133,.5);border-radius:4px}#head_wrapper #form .bdsug-new ul li .sug-new-tag-blue{color:#36F;border:1px solid rgba(51,102,255,.4);border-radius:4px}#head_wrapper #form .bdsug-new ul li .sug-new-tag-orange{color:#F33;border:1px solid rgba(255,51,51,.4);border-radius:4px}#head_wrapper #form .bdsug-new ul li .sug-tag-text{display:inline-block;box-sizing:border-box;margin-left:6px;height:18px;padding:2px 4px;font-size:12px;line-height:12px;font-weight:500;text-align:center;border-radius:4px;vertical-align:middle;border-width:1px;border-style:solid}#head_wrapper #form .bdsug-new ul li .sug-tag-img{height:18px;max-width:80px;vertical-align:middle;margin-left:6px}#head_wrapper #form .bdsug-new ul li .direct-sug-wrap{display:block;height:40px;padding:7px 0;color:#222}#head_wrapper #form .bdsug-new ul li .direct-sug-wrap:visited{color:#222}#head_wrapper #form .bdsug-new ul li .direct-sug-wrap .left-img-wrap{position:relative}#head_wrapper #form .bdsug-new ul li .direct-sug-wrap .left-img-wrap::before{content:"";width:38px;height:38px;position:absolute;top:0;right:0;left:0;bottom:0;border:1px solid rgba(0,0,0,.06);border-radius:9px}#head_wrapper #form .bdsug-new ul li .direct-sug-wrap .left-img{box-sizing:border-box;float:left;width:40px;height:40px;border:1px solid rgba(0,0,0,.06);border-radius:9px;margin-right:6px}#head_wrapper #form .bdsug-new ul li .direct-sug-wrap .content{float:left}#head_wrapper #form .bdsug-new ul li .direct-sug-wrap .header{margin-bottom:6px;line-height:18px;height:18px;vertical-align:middle}#head_wrapper #form .bdsug-new ul li .direct-sug-wrap .name{float:left;font-size:18px;color:#222;font-weight:500}#head_wrapper #form .bdsug-new ul li .direct-sug-wrap .tag-img{width:18px;margin-left:4px}#head_wrapper #form .bdsug-new ul li .direct-sug-wrap .more{line-height:14px;width:240px}#head_wrapper #form .bdsug-new ul li .direct-sug-wrap .brief,#head_wrapper #form .bdsug-new ul li .direct-sug-wrap .info{font-size:14px;color:#222;font-weight:400}#head_wrapper #form .bdsug-new ul li .direct-sug-wrap .brief{margin-right:6px}#head_wrapper #form .bdsug-new ul li .direct-sug-wrap .right-btn{float:right;margin-top:5px;margin-right:6px;width:60px;height:24px;text-align:center;font-size:14px;color:#36F;line-height:24px;background-image:linear-gradient(111deg,#e8f7ff 0,#edf0ff 100%);border-radius:12px}#head_wrapper #form .bdsug-new ul li .direct-sug-wrap .right-btn:hover{background:#315efb;color:#fff}#head_wrapper #form .bdsug-new ul li:hover .direct-sug-wrap .name{color:#315EFB}#head_wrapper #form .bdsug-new ul li:hover .direct-sug-wrap .brief{color:#222}#head_wrapper.s-down #form .sam-bdsug.bdsug-new{top:52px}#head_wrapper #form .sam-bdsug.bdsug-new{width:100%;box-shadow:0 4px 4px 0 rgba(0,0,0,.1);border:1px solid rgba(0,0,0,.05)!important;border-radius:12px;top:56px}#head_wrapper #form .sam-bdsug.bdsug-new ul{border:0;padding:0 0 7px}#head_wrapper #form .sam-bdsug.bdsug-new ul li{line-height:32px}#head_wrapper #form .sam-bdsug.bdsug-new ul .bdsug-s{background-color:#F1F3FD!important}#head_wrapper #form .sam-bdsug.bdsug-new .bdsug-store-del{right:15px}.sam_search .sam_search_rec,.sam_search .sam_search_soutu{z-index:1;display:none;position:absolute;top:50%;margin-top:-12px;font-size:24px;color:#4E6EF2;height:24px;line-height:24px;width:24px;cursor:pointer;-webkit-transform:translate3d(0,0,0);transform:translate3d(0,0,0);transition:transform .3s ease}.sam_search .sam_search_rec{right:54px}.sam_search .sam_search_soutu{right:14px}.sam_search .sam_search_rec:hover,.sam_search .sam_search_soutu:hover{color:#1D4FFF!important;transform:scale(1.08,1.08)}.sam_search .sam_search_rec_hover,.sam_search .sam_search_soutu_hover{background:#626675;border-radius:8px;height:32px;width:76px;text-align:center;line-height:32px;font-size:13px;color:#FFF;position:absolute;z-index:2;top:50px}.sam_search .sam_search_rec_hover:before,.sam_search .sam_search_soutu_hover:before{content:\'\';border:4px solid transparent;border-bottom:4px solid #626675;position:absolute;left:50%;top:-8px;margin-left:-4px}.sam_search .sam_search_rec_hover{right:29px}.sam_search .sam_search_soutu_hover{display:none;right:-12px}</style><style type="text/css" index="superbase">blockquote,body,button,dd,dl,dt,fieldset,form,h1,h2,h3,h4,h5,h6,hr,input,legend,li,ol,p,pre,td,textarea,th,ul{margin:0;padding:0}\n'Ⅲ、readlines按行讀直到讀完
import urllib.request url = "http://www.baidu.com" # 模擬瀏覽器向服務(wù)器發(fā)送請(qǐng)求 response = urllib.request.urlopen(url) # readlines按行讀直到讀完 content = response.readlines() print(content)
Ⅳ、getcode獲取狀態(tài)碼
import urllib.request url = "http://www.baidu.com" # 模擬瀏覽器向服務(wù)器發(fā)送請(qǐng)求 response = urllib.request.urlopen(url) # getcode獲取狀態(tài)碼 print(response.getcode())
打印結(jié)果:
200
常見(jiàn)狀態(tài)碼
在HTTP協(xié)議中，常見(jiàn)的狀態(tài)碼用于表示服務(wù)器對(duì)請(qǐng)求的處理結(jié)果。以下是一些常見(jiàn)的狀態(tài)碼及其含義：
200 OK：請(qǐng)求成功，服務(wù)器成功處理了請(qǐng)求并返回了所需的數(shù)據(jù)。
301 Moved Permanently：永久重定向，請(qǐng)求的資源已永久移動(dòng)到新的URL。
302 Found：臨時(shí)重定向，請(qǐng)求的資源暫時(shí)移動(dòng)到了新的URL。
400 Bad Request：請(qǐng)求無(wú)效，服務(wù)器無(wú)法理解或處理請(qǐng)求。
401 Unauthorized：未授權(quán)，請(qǐng)求需要身份驗(yàn)證。
403 Forbidden：禁止訪問(wèn)，服務(wù)器拒絕請(qǐng)求訪問(wèn)資源。
404 Not Found：未找到，服務(wù)器無(wú)法找到請(qǐng)求的資源。
500 Internal Server Error：服務(wù)器內(nèi)部錯(cuò)誤，服務(wù)器在處理請(qǐng)求過(guò)程中發(fā)生了錯(cuò)誤。
502 Bad Gateway：錯(cuò)誤的網(wǎng)關(guān)，作為代理或網(wǎng)關(guān)的服務(wù)器從上游服務(wù)器接收到無(wú)效的響應(yīng)。
503 Service Unavailable：服務(wù)不可用，服務(wù)器暫時(shí)無(wú)法處理請(qǐng)求，通常是由于過(guò)載或維護(hù)。
這只是一些常見(jiàn)的狀態(tài)碼，HTTP協(xié)議中還有更多的狀態(tài)碼，每個(gè)狀態(tài)碼都有特定的含義和用途。在開(kāi)發(fā)或使用HTTP請(qǐng)求時(shí)，了解常見(jiàn)的狀態(tài)碼可以幫助你更好地理解服務(wù)器對(duì)請(qǐng)求的響應(yīng)。
Ⅴ、geturl獲取url地址
import urllib.request url = "http://www.baidu.com" # 模擬瀏覽器向服務(wù)器發(fā)送請(qǐng)求 response = urllib.request.urlopen(url) # getcode獲取狀態(tài)碼 print(response.geturl())
打印結(jié)果：
http://www.baidu.com
Ⅵ、getheaders方法獲取headers（一個(gè)狀態(tài)信息）
import urllib.request url = "http://www.baidu.com" # 模擬瀏覽器向服務(wù)器發(fā)送請(qǐng)求 response = urllib.request.urlopen(url) # getcode獲取狀態(tài)碼 print(response.getheaders())
打印信息：
[('Content-Length', '410774'), ('Content-Security-Policy', "frame-ancestors 'self' https://chat.baidu.com http://mirror-chat.baidu.com https://fj-chat.baidu.com https://hba-chat.baidu.com https://hbe-chat.baidu.com https://njjs-chat.baidu.com https://nj-chat.baidu.com https://hna-chat.baidu.com https://hnb-chat.baidu.com http://debug.baidu-int.com;"), ('Content-Type', 'text/html; charset=utf-8'), ('Date', 'Sat, 23 Sep 2023 10:36:09 GMT'), ('P3p', 'CP=" OTI DSP COR IVA OUR IND COM "'), ('P3p', 'CP=" OTI DSP COR IVA OUR IND COM "'), ('Server', 'BWS/1.1'), ('Set-Cookie', 'BAIDUID=50E2111E01C6C8C836180B5DFDFF669E:FG=1; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com'), ('Set-Cookie', 'BIDUPSID=50E2111E01C6C8C836180B5DFDFF669E; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com'), ('Set-Cookie', 'PSTM=1695465369; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com'), ('Set-Cookie', 'BAIDUID=50E2111E01C6C8C85CBF086F53BBCFB9:FG=1; max-age=31536000; expires=Sun, 22-Sep-24 10:36:09 GMT; domain=.baidu.com; path=/; version=1; comment=bd'), ('Traceid', '1695465369023779380213415997353182430819'), ('Vary', 'Accept-Encoding'), ('X-Ua-Compatible', 'IE=Edge,chrome=1'), ('Connection', 'close')]3、下載??
我們可以通過(guò)爬蟲(chóng)（urllib.request.urlretrieve()）將網(wǎng)頁(yè)、圖片、視頻下載下來(lái)。
下載網(wǎng)頁(yè)??
下載百度的網(wǎng)頁(yè)
import urllib.request # 下載網(wǎng)頁(yè) url_page = "http://www,.baidu, com" # urI代表的是下載的路徑 filename文件的名字# 在python中 可以變量的名字 也可以直接寫(xiě)值 urllib.request.urlretrieve(url_page, 'baidu.html')
下載好了之后瀏覽器打開(kāi)：
下載圖片??
import urllib.request # 下載圖片 url_img="https://profile-avatar.yssmx.cn/82351d3cb9754fda97cb85258d7e74a0_m0_63951142.jpg!1" urllib.request.urlretrieve(url_img, filename="tx.jpg")
把我的頭像鏈接輸入，下載下來(lái)：
下載視頻??
再找到里面的src
這就是我們的視頻地址
import urllib.request # 下載網(wǎng)頁(yè) # url_page = "http://www.baidu.com" # # urI代表的是下載的路徑 filename文件的名字# 在python中 可以變量的名字 也可以直接寫(xiě)值 # urllib.request.urlretrieve(url_page, 'baidu.html') url_video="https://v16m-default.akamaized.net/5ec5b30ab203de382ab98284de179428/650f2c82/video/tos/alisg/tos-alisg-v-0051c001-sg/oAoAVUpAqIizX8BESwEmjdNsBWAfpFhywEKEVq/?a=0&ch=0&cr=0&dr=0&er=0&lr=default&cd=0%7C0%7C0%7C0&br=2016&bt=1008&bti=cmd5ZGgxZGZiZmIrY2E6&cs=0&ds=3&ft=dl9~j-Inz7TlNPJZiyq8Z&mime_type=video_mp4&qs=13&rc=M3A8ZDQ6ZmhrbDMzODYzNEBpM3A8ZDQ6ZmhrbDMzODYzNEAyXmBycjRnYGhgLS1kMC1zYSMyXmBycjRnYGhgLS1kMC1zcw%3D%3D&l=202309231156594B50CCEF2A58CC50DD84&btag=e00068000https://v16m-default.akamaized.net/5ec5b30ab203de382ab98284de179428/650f2c82/video/tos/alisg/tos-alisg-v-0051c001-sg/oAoAVUpAqIizX8BESwEmjdNsBWAfpFhywEKEVq/?a=0&ch=0&cr=0&dr=0&er=0&lr=default&cd=0%7C0%7C0%7C0&br=2016&bt=1008&bti=cmd5ZGgxZGZiZmIrY2E6&cs=0&ds=3&ft=dl9~j-Inz7TlNPJZiyq8Z&mime_type=video_mp4&qs=13&rc=M3A8ZDQ6ZmhrbDMzODYzNEBpM3A8ZDQ6ZmhrbDMzODYzNEAyXmBycjRnYGhgLS1kMC1zYSMyXmBycjRnYGhgLS1kMC1zcw%3D%3D&l=202309231156594B50CCEF2A58CC50DD84&btag=e00068000" urllib.request.urlretrieve(url_video, "dm.mp4")
運(yùn)行代碼
下載好了之后我們?cè)趐ycharm是打不開(kāi)的，得去本地文件中找：
點(diǎn)擊就可以看了： 文章來(lái)源：http://www.zghlxwxcb.cn/news/detail-715606.html
文章來(lái)源地址http://www.zghlxwxcb.cn/news/detail-715606.html
到了這里，關(guān)于Python網(wǎng)絡(luò)爬蟲(chóng)庫(kù)urllib詳解的文章就介紹完了。如果您還想了解更多內(nèi)容，請(qǐng)?jiān)谟疑辖撬阉鱐OY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！
![[Python]爬蟲(chóng)基礎(chǔ)——urllib庫(kù)](https://imgs.yssmx.com/Uploads/2024/02/435181-1.png)
