1.通过对知网期刊的链接去解析包含内容的详情页链接,但是知网不存在页面状态码依旧是 200, 所以转而想用网页存在某些内容来决定取舍,但是判断条件好像对知网内容不起作用,起不到筛查的效果.

2.完整代码:

import requests

from bs4 import BeautifulSoup

import time

def get_url(years,month,num):#month用两位数表示，num用三位数表示。

times = 'GJXW'+str(years)+str(month).zfill(2)+str(num).zfill(3)

url='http://kns.cnki.net/kcms/detail/detail.aspx?dbcode=CJFD&filename={}'.format(times)

headers={

'User-Agent':'Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.104 Safari/537.36 Core/1.53.4033.400 QQBrowser/9.6.12624.400',

'Cookie':'Ecp_notFirstLogin=ezBGYf; SID=120162; ASP.NET_SessionId=pj040hrhxe43fsrsa2mk24ep; Ecp_ClientId=3171112155604138976; c_m_LinID=LinID=WEEvREcwSlJHSldRa1Fhb09jMjVzQmVYUXBzNTNlbzJreWFVTU5NeVRmST0=$9A4hF_YAuvQ5obgVAqNKPCYcEjKensW4IQMovwHtwkF4VYPoHbKxJw!!&ot=11/12/2017 16:16:52; c_m_expire=2017-11-12 16:16:52; Ecp_session=1; LID=WEEvREcwSlJHSldRa1Fhb09jMjVzQmVYUXBzNTNlbzJreWFVTU5NeVRmST0=$9A4hF_YAuvQ5obgVAqNKPCYcEjKensW4IQMovwHtwkF4VYPoHbKxJw!!; Ecp_LoginStuts=%7B%22IsAutoLogin%22%3Afalse%2C%22UserName%22%3A%22gz0289%22%2C%22ShowName%22%3A%22%25E5%2590%2589%25E6%259E%2597%25E5%25A4%25A7%25E5%25AD%25A6%25E7%258F%25A0%25E6%25B5%25B7%25E5%25AD%25A6%25E9%2599%25A2%22%2C%22UserType%22%3A%22bk%22%2C%22r%22%3A%22ezBGYf%22%7D'

}

html=requests.get(url,headers=headers)

soup = BeautifulSoup(html.text,'lxml')

if soup.find('body > div:nth-child(1) > div.sorry > p'): #body > div:nth-child(1) > div.sorry > p

pass

else:

print(url)

def main():

years='2017'

for month in range(1,13):

for num in range(1,15):

get_url(years, month, num)

time.sleep(2)

if __name__ == '__main__':

main()

3.判断条件是根据这部分选择

预想结果是会根据 main 函数里的内容进行循环判断, 但是 201701013 和 14 后缀的连接仍旧会跳转到上面的部分, 但是会被作为结果返回.!!!

4.我进行的处理是反其道而行之, 设想通过判断存在某部分内容来选择, 但依旧没有任何效果.