很多数据，需要用搜索引擎帮忙搜下网页，再进行爬。找了半天就发现google和bing的可用--免费，但有一定限制。

google得接vpn什么的，你懂地。。。

使用bing搜索引擎服务，首先你需要一个ms帐号，然后去申请一个接入APP ID，免费版的有使用次数和频率限制。

bing搜索中文页： http://cn.bing.com/partners/developers#BingSearchApi

bing搜索APP ID页： https://www.microsoft.com/cognitive-services/en-us/subscriptions

bing搜索API: https://dev.cognitive.microsoft.com/docs/services/56b43eeccf5ff8098cef3807/operations/56b4447dcf5ff8098cef380d

bing搜索测试页：https://dev.cognitive.microsoft.com/docs/services/56b43eeccf5ff8098cef3807/operations/56b4447dcf5ff8098cef380d/console

如果返回http_code为200表示调用成功，结果是一个json格式的字符串，包含网页url、描述、日期等。

def t5():import jsonimport requestssearch_key = '黄金'search_count = 10search_offset = 10search_language = 'zh-cn' #mktms_app_id = 'a499*****************' #APP IDurl = 'https://api.cognitive.microsoft.com/bing/v5.0/search?q=%s&count=%d&offset=%d&mkt=%s&safesearch=Off' % (search_key, search_count, search_offset, search_language)headers = {'Ocp-Apim-Subscription-Key': ms_app_id} #必需，用于验证ret = requests.get(url=url, headers=headers)ret_code = ret.status_codeif not 200 == ret_code:print ret_coderet.close()returnret_content = ret.contentret.close()ret_content_dict = json.loads(ret_content)ret_items = ret_content_dict.get('webPages').get('value')for i in ret_items:print i

响应数据：

{"_type": "SearchResponse","webPages": {"webSearchUrl": "https://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=hLmod1xUQ8sDTA4JbO1WZgkMJxe7ha9J5lEvKntxvxw&v=1&r=https%3a%2f%2fwww.bing.com%2fsearch%3fq%3d%25e9%25bb%2584%25e9%2587%2591&p=DevEx,5437.1","totalEstimatedMatches": 14100000,"value": [{"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.0","name": "国际黄金价格走势图-黄金频道-和讯网","url": "http://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=QoccehSy0QAhzg1f_JmllllfGvDb6dednFmprKry0Sw&v=1&r=http%3a%2f%2fgold.hexun.com%2fhjxh%2f&p=DevEx,5103.1","thumbnailUrl": "https://www.bing.com/th?id=OIP.M7e0fabd47cfc3d75f0581207207d9fb9o0&pid=Api","displayUrl": "gold.hexun.com/hjxh","snippet": "品种 中间价 买入价 卖出价 最高价 最低价 今开盘 昨收盘 涨跌量 更新时间 美元账户黄金 1275.05 1273.75 1276.35 1283.92 1267.72 1267.72 1275.10-1.35 2016-10-29 04:00:00","deepLinks": [{"name": "纸黄金","url": "http://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=-bKusIVEhz02UqGzDdFOs1ibm2SDxq66RC1ahvdaivg&v=1&r=http%3a%2f%2fgold.hexun.com%2faccount%2f&p=DevEx,5096.1","snippet": "指黄金的纸上交易，投资者的买卖交易记录只在个人预先开立的“黄金存折账户”上体现，而不涉及实物金的提取。盈利模式即通过低买高卖，获取差价利润。"},{"name": "实物金","url": "http://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=csDxPOP2bxG85JED-8VY5ExkhGhP0CiLoPy7X416T3E&v=1&r=http%3a%2f%2fdata.gold.hexun.com%2fGoldHistoryPage.aspx&p=DevEx,5099.1","snippet": "时间 品牌 产品 价格 手工费 涨跌 2016-11-07 中国黄金 足金饰品 330.00----2016-11-07 中国黄金 高端工艺金条 355.00----2016-11-07 中国黄金 工艺金条 308.90----2016-11-07"},{"name": "金市动态","url": "http://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=Rad4DWLTda_sa8wjqBcKSB4CQjwcefoRVKDdEHDJEcA&v=1&r=http%3a%2f%2fgold.hexun.com%2fmarket%2f&p=DevEx,5094.1","snippet": "剑心解盘：9.18-9.19原油沥青一周行情回顾及下周布局操作建议 剑心解盘：8.26-27周评原油沥青行情分析及操作建议，原油宁贵沥青下周行情预测及操作建议"},{"name": "实物黄金","url": "http://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=QkRsF_ZBY43VUim_elwqfij-1iCbFxtepkKJA536Q6w&v=1&r=http%3a%2f%2fgold.hexun.com%2fbullion%2findex.html&p=DevEx,5095.1","snippet": "菜百简介 菜百主要经营足金、千足金饰品、金条、白玉、珠宝、金银币章、奥运特许商品及18K金、银饰品等大类商品。[详细]"},{"name": "沙龙首页","url": "http://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=pAoPOZbZHmTruR3EsGZl0dhlZ_AL6FoIiLfHEgCEVtY&v=1&r=http%3a%2f%2fsalon.hexun.com%2fgold%2findex.aspx&p=DevEx,5098.1","snippet": "2012年12月4日短期黄金下方支撑1705-1708 2012年12月3日今年年前白银高点在37一线 2012年11月29日白银12月35是较强阴力位 2012年11月28日白银近期回到33.5以下概率不 ..."},{"name": "现货黄金","url": "http://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=4LiFFvOy2k4jktXUHhSiYShyLQ3OcJx-0oGwEvGDPk0&v=1&r=http%3a%2f%2fgold.hexun.com%2fspot%2findex.html&p=DevEx,5100.1","snippet": "黄金现货市场上实物黄金的主要形式是金条（Gold Bullion）和金块，也有金币、金质奖章和首饰等。金条有低纯度的砂金和高纯度的条金，条金一般重400盎司。"}],"dateLastCrawled": "2016-10-30T10:18:00"},{"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.1","name": "黄金（贵金属）_百度百科","url": "http://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=tFhHEizzpk-OEzaqAoHLPhW8Jb_6-PxoF5MZjJ3QsJQ&v=1&r=http%3a%2f%2fbaike.baidu.com%2fview%2f5937.htm&p=DevEx,5117.1","displayUrl": "baike.baidu.com/view/5937","snippet": "黄金（Gold）是化学元素金（化学元素符号Au）的单质形式，是一种软的，金黄色的，抗腐蚀的贵金属。金是最稀有、最珍贵和最被人看重的金属之一。国际上一般黄金都是 ...","dateLastCrawled": "2016-10-31T16:35:00"},{"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.2","name": "黄金行情_东方财富网","url": "http://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=UKVszzZuN1jaW-6lWooqOGjIvbr0FD8m_eswRJDDYrk&v=1&r=http%3a%2f%2fquote.eastmoney.com%2fcenter%2fgold.html&p=DevEx,5130.1","displayUrl": "quote.eastmoney.com/center/gold.html","snippet": "东方财富网黄金行情提供权威、全面、专业、及时的行情数据，将黄金行情、国际贵金属期货、国际贵金属现货、上海黄金期货、上海黄金现货、货币贵金属、纸 黄金 ...","dateLastCrawled": "2016-11-05T19:31:00"},{"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.3","name": "黄金网_国内最具影响力的黄金投资服务平台","url": "http://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=3rfiJbhnHx7qNre3kYIhMLvafJb_TCZiTNVuDaXdThg&v=1&r=http%3a%2f%2fgold.cnfol.com%2f&p=DevEx,5144.1","thumbnailUrl": "https://www.bing.com/th?id=OIP.M886fd1231b6be43eaec53890052a18d8o2&pid=Api","displayUrl": "gold.cnfol.com","snippet": "黄金网为您提供最及时、最全面全球24小时黄金、白银市场最新动态、国际经济数据和财经大事，满足贵金属投资市场参与者对基本面消息及时、全面的需求；提供业界最 ...","dateLastCrawled": "2016-11-06T01:19:00"},{"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.4","name": "黄金价格|白银价格|黄金|白银_新浪财经-贵金属_新浪网","url": "http://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=_pEXiOXNdR73LGeqyiUQPVhGMQcXt9LU5OtzS8HqrZQ&v=1&r=http%3a%2f%2ffinance.sina.com.cn%2fnmetal%2f&p=DevEx,5158.1","thumbnailUrl": "https://www.bing.com/th?id=OIP.M83d7877098e4a17b6497e3957777cb55o0&pid=Api","displayUrl": "finance.sina.com.cn/nmetal","snippet": "新浪贵金属提供全面专业的贵金属资讯及报价服务,覆盖黄金、白银等多个品种的现货和期货市场的投资理财服务.例如提供黄金价格,白银价格,纸黄金,黄金期货等贵金属信息。","dateLastCrawled": "2016-11-05T18:06:00"},{"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.5","name": "中国黄金投资网(中国黄金网),黄金,金价,黄金价格,今日金价,纸 ...","url": "http://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=CKVXV-pMW-evHTkLcQSW8LDY5601qWGGRKMsCnjgbZE&v=1&r=http%3a%2f%2fwww.cngold.org%2f&p=DevEx,5172.1","displayUrl": "www.cngold.org","snippet": "中国黄金网提供国际国内黄金价格,纸黄金价格,今日金价,铂金(白金)价格,黄金T+D开户,白银价格,纸黄金价格走势图,伦敦金,黄金投资资讯,金价实时行情,金价走势图 ...","dateLastCrawled": "2016-11-05T10:55:00"},{"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.6","name": "COMEX黄金(GC)期货行情,新闻,报价_新浪财经_新浪网","url": "http://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=YgrXMCJAerRufageBbWIQLPtALhl7HQn9DWik5yqdjY&v=1&r=http%3a%2f%2ffinance.sina.com.cn%2ffutures%2fquotes%2fGC.shtml&p=DevEx,5186.1","displayUrl": "finance.sina.com.cn/futures/quotes/GC.shtml","snippet": "新浪财经-期货频道为您提供COMEX黄金(GC)期货行情,期货资料,期货新闻,报价,机构报告,评论,现货价格,持仓分析等与COMEX黄金(COMEX黄金)期货相关的信息与 ...","dateLastCrawled": "2016-11-05T13:16:00"},{"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.7","name": "黄金_理财频道_腾讯网","url": "http://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=-lcwVd1Mt0qmAasOXzJTwzdbSSsmxryMumr6_1hPRdE&v=1&r=http%3a%2f%2ffinance.qq.com%2fgold%2f&p=DevEx,5200.1","displayUrl": "finance.qq.com/gold","snippet": "腾讯黄金频道为您提供专业丰富的黄金资讯，包括黄金，白银，实时行情，分析评论等内容。 ... 股市再遭杀戮 避险推升黄金大涨 金价目前交易于所有均线上方，5 ...","dateLastCrawled": "2016-10-22T03:26:00"},{"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.8","name": "行业价值资讯领导者-贵金属频道-金融界","url": "http://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=0o8GT89KHtGV5Vk3KBG1gwmEVJ5CtdPxc43XEVaI85k&v=1&r=http%3a%2f%2fgold.jrj.com.cn%2f&p=DevEx,5213.1","thumbnailUrl": "https://www.bing.com/th?id=OIP.Md28f6f64d420827d0c08eca680c94805o1&pid=Api","displayUrl": "gold.jrj.com.cn","snippet": "金融界黄金频道，国内最早的黄金频道，最权威的黄金投资指南，行情24小时实时更新，内容覆盖现货黄金、纸黄金、黄金T+D、黄金期货、现货白银、纸白银及铂金钯金等 ...","dateLastCrawled": "2016-11-06T03:09:00"},{"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.9","name": "和讯黄金-国内黄金门户专业权威网站","url": "http://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=oCHwBwAb3-CWzxFjXqLPH6PnrL8y1Cm9GwwwwNcCZhA&v=1&r=http%3a%2f%2fgold.hexun.com%2f&p=DevEx,5225.1","displayUrl": "gold.hexun.com","snippet": "和讯黄金为中国最大的黄金专业网站，国际财经资讯、贵金属市场最新动态、专家金评发表量第一，最专业最全的黄金白银市场数据，拥有中国最火的外汇论坛和博客互动 ...","dateLastCrawled": "2016-11-05T20:29:00"}]},"relatedSearches": {"id": "https://api.cognitive.microsoft.com/api/v5/#RelatedSearches","value": [{"text": "黄金外汇交易平台","displayText": "黄金外汇交易平台","webSearchUrl": "https://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=nxYqrIsLkTTqL72Bkk6Svzcd75ZPVQvHuKAbldl70DE&v=1&r=https%3a%2f%2fwww.bing.com%2fsearch%3fq%3d%25e9%25bb%2584%25e9%2587%2591%25e5%25a4%2596%25e6%25b1%2587%25e4%25ba%25a4%25e6%2598%2593%25e5%25b9%25b3%25e5%258f%25b0&p=DevEx,5438.1"},{"text": "黄金白银价格","displayText": "黄金白银价格","webSearchUrl": "https://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=yl9EcI_HpemXrC1gRk87sd8_IeDEn4tkLmk8499KaO8&v=1&r=https%3a%2f%2fwww.bing.com%2fsearch%3fq%3d%25e9%25bb%2584%25e9%2587%2591%25e7%2599%25bd%25e9%2593%25b6%25e4%25bb%25b7%25e6%25a0%25bc&p=DevEx,5439.1"},{"text": "黄金投资公司","displayText": "黄金投资公司","webSearchUrl": "https://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=u6JrcBTpg0HR3RPlP4HlyBjkkvqJjJVlsxpZQa0ft6E&v=1&r=https%3a%2f%2fwww.bing.com%2fsearch%3fq%3d%25e9%25bb%2584%25e9%2587%2591%25e6%258a%2595%25e8%25b5%2584%25e5%2585%25ac%25e5%258f%25b8&p=DevEx,5440.1"},{"text": "国际黄金价格网","displayText": "国际黄金价格网","webSearchUrl": "https://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=EZ4s0Zz5yG6SP9CkjCm3k2Pno4QxB0BeS3hb7qjVv5U&v=1&r=https%3a%2f%2fwww.bing.com%2fsearch%3fq%3d%25e5%259b%25bd%25e9%2599%2585%25e9%25bb%2584%25e9%2587%2591%25e4%25bb%25b7%25e6%25a0%25bc%25e7%25bd%2591&p=DevEx,5441.1"},{"text": "黄金收购","displayText": "黄金收购","webSearchUrl": "https://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=fs94jtaF3YmvI7f4NoMatRX7RYnr5Hkliqr3JjAinFw&v=1&r=https%3a%2f%2fwww.bing.com%2fsearch%3fq%3d%25e9%25bb%2584%25e9%2587%2591%25e6%2594%25b6%25e8%25b4%25ad&p=DevEx,5442.1"},{"text": "国际黄金期货","displayText": "国际黄金期货","webSearchUrl": "https://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=x8FskOKYJT5k2w20DN2xbRrU3UNSEauMv9INljPhmog&v=1&r=https%3a%2f%2fwww.bing.com%2fsearch%3fq%3d%25e5%259b%25bd%25e9%2599%2585%25e9%25bb%2584%25e9%2587%2591%25e6%259c%259f%25e8%25b4%25a7&p=DevEx,5443.1"},{"text": "搜狐黄金","displayText": "搜狐黄金","webSearchUrl": "https://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=sNftjaCimcu4OKF6255d2yhu7HJf_CdNa3xQsDF3rQI&v=1&r=https%3a%2f%2fwww.bing.com%2fsearch%3fq%3d%25e6%2590%259c%25e7%258b%2590%25e9%25bb%2584%25e9%2587%2591&p=DevEx,5444.1"},{"text": "期货 黄金 股票","displayText": "期货 黄金 股票","webSearchUrl": "https://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=-4pROTsvM90_hEltKIbVgxHiCDBlzsR77Fy90P15DHc&v=1&r=https%3a%2f%2fwww.bing.com%2fsearch%3fq%3d%25e6%259c%259f%25e8%25b4%25a7%2b%25e9%25bb%2584%25e9%2587%2591%2b%25e8%2582%25a1%25e7%25a5%25a8&p=DevEx,5445.1"},{"text": "中国黄金期货","displayText": "中国黄金期货","webSearchUrl": "https://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=dvcMe80xmGUA8p_NAc4hHfyL_NXX90Ql3G10ZRqXzHA&v=1&r=https%3a%2f%2fwww.bing.com%2fsearch%3fq%3d%25e4%25b8%25ad%25e5%259b%25bd%25e9%25bb%2584%25e9%2587%2591%25e6%259c%259f%25e8%25b4%25a7&p=DevEx,5446.1"},{"text": "在线黄金","displayText": "在线黄金","webSearchUrl": "https://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=JKjSWXuNjinKVqTYxd-Yx-6TnyC3TZt9BzMWWMAWKkg&v=1&r=https%3a%2f%2fwww.bing.com%2fsearch%3fq%3d%25e5%259c%25a8%25e7%25ba%25bf%25e9%25bb%2584%25e9%2587%2591&p=DevEx,5447.1"}]},"rankingResponse": {"mainline": {"items": [{"answerType": "WebPages","resultIndex": 0,"value": {"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.0"}},{"answerType": "WebPages","resultIndex": 1,"value": {"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.1"}},{"answerType": "WebPages","resultIndex": 2,"value": {"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.2"}},{"answerType": "WebPages","resultIndex": 3,"value": {"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.3"}},{"answerType": "WebPages","resultIndex": 4,"value": {"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.4"}},{"answerType": "WebPages","resultIndex": 5,"value": {"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.5"}},{"answerType": "WebPages","resultIndex": 6,"value": {"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.6"}},{"answerType": "WebPages","resultIndex": 7,"value": {"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.7"}},{"answerType": "WebPages","resultIndex": 8,"value": {"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.8"}},{"answerType": "WebPages","resultIndex": 9,"value": {"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.9"}}]},"sidebar": {"items": [{"answerType": "RelatedSearches","value": {"id": "https://api.cognitive.microsoft.com/api/v5/#RelatedSearches"}}]}}
}