爬虫：UnicodeDecodeError: 'gbk' codec can't decode byte 0xa6 in position

由于爬取的网页编码格式是“gb2312”格式的，所以第一反应就是也用这个格式编码和解码

import re
from lxml import etree
import html
with open('test.html','r',encoding='gbk') as f:c = f.read()
s = re.sub(r'\n',' ',c)
tree = etree.HTML(c)
rows = tree.xpath("//ul[@class='bang_list clearfix bang_list_mode']/li")
for row in rows:boards = {}s1 = etree.tostring(row).decode('gbk')s1 = html.unescape(s1)print(s1)break

由于 “gbk” 包括 “gb2312”所以使用了 “gbk”，其实结果都一样
翻看了好多博客发现：

爬取的所有网页无论何种编码格式，都转化为 utf-8 格式进行存储

具体什么原因现在我也没清楚，留着后续补充吧

但是关于 gbk 或者 gb2312 格式的网页牵扯到存储时，转换成 utf-8 格式是没错的

import re
from lxml import etree
import html
with open('test.html','r',encoding='utf-8') as f:c = f.read()
s = re.sub(r'\n',' ',c)
tree = etree.HTML(c)
rows = tree.xpath("//ul[@class='bang_list clearfix bang_list_mode']/li")for row in rows:boards = {}s1 = etree.tostring(row).decode('utf-8')s1 = html.unescape(s1)print(s1)break

正常显示

爬虫：UnicodeDecodeError: 'gbk' codec can't decode byte 0xa6 in position相关推荐

踩坑记-- UnicodeDecodeError: ‘gbk‘ codec can‘t decode byte 0xa6 in position 17: illegal multibyte seque
在使用exejs运行js代码的时候发生如下报错,但是在命令行去运行js文件正常: Exception in thread Thread-1: Traceback (most recent call l ...
UnicodeDecodeError: 'gbk' codec can't decode byte 0xa6 in position 34: illegal multibyte sequence
在做写诗机器人的诗词预处理时,出现错误.原代码如下: poems = [] file = open('poetryTang.txt','r' ) for line in file.readlines( ...
UnicodeDecodeError: ‘gbk‘ codec can‘t decode byte 0xa6 in position 34: illegal multibyte sequence
python读取TXT文件时出现错误 with open(r'高等数学.txt') as file_object:contents=file_object.read()print(contents) ...
关于UnicodeDecodeError: ‘gbk‘ codec can‘t decode byte 0xa6 in position 9737: 的解决
异常原因: 判断这个版本django由于出错信息的代码有编码读取问题造成无法正常显示. 根据提示找到 python安装路径\Lib\site-packages\django\views\debug. ...
anaconda -spyder报错解决-UnicodeDecodeError: 'gbk' codec can't decode byte 0x93 in position 611: illegal
此文首发于我的个人博客:anaconda -spyder报错解决-UnicodeDecodeError 'gbk' codec can't decode byte 0x93 in position 6 ...
解决Python报错UnicodeDecodeError: 'gbk' codec can't decode byte 0x80 in position 658: illegal multibyte
解决Python报错–UnicodeDecodeError: 'gbk' codec can't decode byte 0x80 in position 658: illegal multibyte ...
UnicodeDecodeError: ‘gbk’ codec can’t decode byte 0x80 in position 658: illegal multibyte sequence
解决Python报错–UnicodeDecodeError: 'gbk' codec can't decode byte 0x80 in position 658: illegal multibyte ...
编码调试：UnicodeDecodeError: ‘gbk‘ codec can‘t decode byte 0xaf in position 12: illegal multibyte sequen
在程序段: stopkey = [w.strip() for w in codecs.open('data/stopWord.txt', 'r').readlines()] 出现错误: Unicode ...
成功解决UnicodeDecodeError: ‘gbk‘ codec can‘t decode byte 0xba in position 2: illegal multibyte sequence
成功解决UnicodeDecodeError: 'gbk' codec can't decode byte 0xba in position 2: illegal multibyte sequence ...

爬虫：UnicodeDecodeError: 'gbk' codec can't decode byte 0xa6 in position

爬虫：UnicodeDecodeError: 'gbk' codec can't decode byte 0xa6 in position相关推荐

最新文章

热门文章