文章目录

  • 示例介绍
  • 示例代码
  • 输出

转载请标明出处：
http://blog.csdn.net/qq_27818541/article/details/112198134
本文出自:【BigManing的博客】

示例介绍

  1. 首先访问 URL
  2. 找到所有含有.product-grid-item 的条目
  3. 解析 ，组装数据
  4. 输出到文件

示例代码

package mainimport ("encoding/csv""log""os""github.com/gocolly/colly/v2"
)func main() {fName := "xkcd_store_items.csv"file, err := os.Create(fName)if err != nil {log.Fatalf("Cannot create file %q: %s\n", fName, err)return}defer file.Close()writer := csv.NewWriter(file)defer writer.Flush()// Write CSV headerwriter.Write([]string{"Name", "Price", "URL", "Image URL"})// Instantiate default collectorc := colly.NewCollector(// Allow requests only to store.xkcd.comcolly.AllowedDomains("store.xkcd.com"),)// Extract product detailsc.OnHTML(".product-grid-item", func(e *colly.HTMLElement) {writer.Write([]string{e.ChildAttr("a", "title"),e.ChildText("span"),e.Request.AbsoluteURL(e.ChildAttr("a", "href")),"https:" + e.ChildAttr("img", "src"),})})// Find and visit next page linksc.OnHTML(`.next a[href]`, func(e *colly.HTMLElement) {e.Request.Visit(e.Attr("href"))})c.Visit("https://store.xkcd.com/collections/everything")log.Printf("Scraping finished, check file %q for results\n", fName)// Display collector's statisticslog.Println(c)
}

筛选的标签：

输出

最后输出到文件： xkcd_store_items.csv


附原始文件数据：

Name,Price,URL,Image URL
Actual Size Stickers,$5.00,https://store.xkcd.com/collections/everything/products/actual-size-stickers,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/actual_size_1_medium.png?v=1479786233
Switch and Button Stickers,$5.00,https://store.xkcd.com/collections/everything/products/switch-and-button-stickers,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/stickerpack_1_medium.png?v=1479786235
Opinions Bumper Sticker,$5.00,https://store.xkcd.com/collections/everything/products/opinions-bumper-sticker,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/opinions2_medium.jpg?v=1479786237
Citation Needed Sticker Pack,$5.00,https://store.xkcd.com/collections/everything/products/citation-needed-sticker-pack,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/citation_stickers_300x300_f72b56cc-8bbe-426c-a52a-10fb939056e9_medium.png?v=1479786241
Formal Logic Bumper Sticker,$5.00,https://store.xkcd.com/collections/everything/products/formal-logic-bumper-sticker,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/formal_logic_store_1_medium.png?v=1479786243
Doughnut Mug,$7.50,https://store.xkcd.com/collections/everything/products/doughnut-mug,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/doughnut_mug_300_medium.png?v=1479786171
Move Fast and Break Things,$7.50,https://store.xkcd.com/collections/everything/products/move-fast-and-break-things,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/movefast-mug_back_grande_aa546fff-ef3d-4834-a624-4e8bcb68a41e_medium.jpg?v=1479786173
7 Button Pack,$10.00,https://store.xkcd.com/collections/everything/products/7-button-pack,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/xkcd_buttons_1_medium.jpg?v=1479786169
xkcd store gift card,from $13.00,https://store.xkcd.com/collections/everything/products/xkcd-store-gift-card,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/giftcard_medium.png?v=1479827615
Online Communities (2007) Poster,$15.00,https://store.xkcd.com/collections/everything/products/online-communities-2007-poster,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/online_communities_1_medium.png?v=1479786198
Height Poster,$15.00,https://store.xkcd.com/collections/everything/products/height-poster,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/height_sample_1_medium.png?v=1479786200
Gravity Wells Poster,$15.00,https://store.xkcd.com/collections/everything/products/gravity-wells-poster,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/gravity_wells_store_1_c8544af5-2308-4ef9-abad-b62056276011_medium.png?v=1479786202
Movie Narrative Charts Poster,$15.00,https://store.xkcd.com/collections/everything/products/movie-narrative-charts-poster,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/movie_narrative_store_1_medium.png?v=1479786204
Online Communities Poster,from $15.00,https://store.xkcd.com/collections/everything/products/online-communities-poster,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/online_communities2_1_medium.png?v=1479786207
Money Poster,from $15.00,https://store.xkcd.com/collections/everything/products/money-poster,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/money_store_1_fa972ce9-c535-42b2-9e04-76eade314652_medium.png?v=1479786209
Lakes & Oceans Poster,$15.00,https://store.xkcd.com/collections/everything/products/lakes-oceans-poster,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/lakes_and_oceans_store_1_medium.png?v=1479786211
United Shapes Poster,$15.00,https://store.xkcd.com/collections/everything/products/united-shapes-poster,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/united_shapes_1_medium.png?v=1479786214
Congress Poster,$15.00,https://store.xkcd.com/collections/everything/products/congress-poster,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/congress_1_10599aa5-cbc9-4309-b6a0-bac41287b18c_medium.png?v=1479786218
Up Goer Five Poster,$15.00,https://store.xkcd.com/collections/everything/products/up-goer-five-poster,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/upgoerfive-1_6b55bf97-5d10-47d5-b1ee-cda3f58f2bf8_medium.png?v=1479786220
Subways,$15.00,https://store.xkcd.com/collections/everything/products/subways,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/subways_store_1_medium.png?v=1479786223
Exoplanet Neighborhood Poster,$15.00,https://store.xkcd.com/collections/everything/products/exoplanet-neighborhood-poster,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/neighborhood_store_1_medium.png?v=1479786226
Signed Prints,$15.00,https://store.xkcd.com/collections/everything/products/signed-prints,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/Randall_Munroe_Sig_35c507f6-9bac-4480-8fef-3150d418c2e8_medium.png?v=1479786231
xkcd: volume 0,$18.00,https://store.xkcd.com/collections/everything/products/xkcd-volume-0,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/xkcd_book_1_90ff6434-e1c5-4a7e-a169-865ccd1bf1d5_medium.png?v=1479786168
Useless,from $19.00,https://store.xkcd.com/collections/everything/products/useless,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/f_useless_shirt_1_medium.jpg?v=1479786257
No Raptors,from $19.00,https://store.xkcd.com/collections/everything/products/no-raptors,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/raptors_shirt_1_medium.jpg?v=1479786279
Sudo,from $19.00,https://store.xkcd.com/collections/everything/products/sudo,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/sudo_shirt_1_72b5ef3b-9b22-46bb-a8ae-9a0ba54e5d5a_medium.jpg?v=1479786286
Witty,from $19.00,https://store.xkcd.com/collections/everything/products/witty,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/witty_shirt_1_medium.jpg?v=1479786292
Just Shy,from $19.00,https://store.xkcd.com/collections/everything/products/just-shy,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/just-shy_shirt_1_medium.jpg?v=1479786299
I Know Regular Expressions,from $19.00,https://store.xkcd.com/collections/everything/products/i-know-regular-expressions,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/regex_shirt_1_medium.jpg?v=1479786306
Try Science,from $19.00,https://store.xkcd.com/collections/everything/products/try-science,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/science_shirt_1_medium.jpg?v=1479786316
XKCD College-Style Shirt,from $19.00,https://store.xkcd.com/collections/everything/products/xkcd-college-style-shirt,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/college_shirt_1_medium.jpg?v=1479786326
Compiling,from $19.00,https://store.xkcd.com/collections/everything/products/compiling,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/compiling_shirt_1_medium.jpg?v=1479786335
Linux Cheat Shirt,$19.00,https://store.xkcd.com/collections/everything/products/linux-cheat-shirt,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/linux-cheat_shirt_1_medium.jpg?v=1479786346
Woodpecker,from $19.00,https://store.xkcd.com/collections/everything/products/woodpecker,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/woodpecker_shirt_1_medium.jpg?v=1479786351
Correlation,from $19.00,https://store.xkcd.com/collections/everything/products/correlation,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/correlation_shirt_1_medium.jpg?v=1479786358
Shark,from $19.00,https://store.xkcd.com/collections/everything/products/shark,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/shark_shirt_1_medium.jpg?v=1479786362
Tech Support,from $19.00,https://store.xkcd.com/collections/everything/products/tech-support,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/tech_support_shirt_1_1730f41c-cbb2-4cb3-b916-2e6e5295580c_medium.jpg?v=1479786369
QR Code,from $19.00,https://store.xkcd.com/collections/everything/products/qr-code,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/qr_shirt_1_ec39110e-ad86-4ec0-864e-d4b5c1421941_medium.jpg?v=1479786378
Sysadmin,from $19.00,https://store.xkcd.com/collections/everything/products/sysadmin,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/sysadmin_shirt_1_medium.jpg?v=1479786391
Centrifuge,from $19.00,https://store.xkcd.com/collections/everything/products/centrifuge,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/centrifuge_shirt_1_medium.jpg?v=1479786400
Collatz Conjecture,from $19.00,https://store.xkcd.com/collections/everything/products/collatz-conjecture,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/collatz_shirt_1_medium.jpg?v=1479786413
Self Reference,from $19.00,https://store.xkcd.com/collections/everything/products/self-reference,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/self_reference_shirt_1_medium.jpg?v=1479786424
Greek Letter Shirt,from $19.00,https://store.xkcd.com/collections/everything/products/greek-letter-shirt,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/greek_medium.png?v=1479786433
Issues Shirt,from $19.00,https://store.xkcd.com/collections/everything/products/issues-shirt,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/notability-black_medium.png?v=1479786439
Flying Ferret Tank Top,$20.00,https://store.xkcd.com/collections/everything/products/flying-ferret-tank-top,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/tank_top_1_medium.jpg?v=1479786246
Overthinking Shirt,$21.00,https://store.xkcd.com/collections/everything/products/overthinking-shirt,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/overthinking1_medium.png?v=1479786447
XKCD Figures Shirt,$21.00,https://store.xkcd.com/collections/everything/products/xkcd-figures-shirt,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/xkcd1_medium.png?v=1479786455
Rover Shirt,$21.00,https://store.xkcd.com/collections/everything/products/rover-shirt,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/rover1_medium.png?v=1479786463
Exoplanets Shirt,$21.00,https://store.xkcd.com/collections/everything/products/exoplanets-shirt,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/exoplanets1_medium.png?v=1479786470
Hoodie,$40.00,https://store.xkcd.com/collections/everything/products/hoodie,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/hoodie_1_7f9223f9-6933-47c6-9af5-d06b8227774a_medium.png?v=1479786341
Hat Guy Polo,$42.00,https://store.xkcd.com/collections/everything/products/hat-guy-polo,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/hat_polo_1_medium.jpg?v=1479786180
Raptor Polo,$42.00,https://store.xkcd.com/collections/everything/products/raptor-polo,https://cdn.shopifycdn.net/s/files/1/0149/3544/products/raptor_polo_1_medium.jpg?v=1479786192

Go爬虫colly官方示例六【xkcd_store】- 抓取xkcd商店商品列表相关推荐

  1. php 获取京东商品详情,用正则表达式，抓取京东的商品列表

    用正则表达式,抓取京东的商品列表 保存到服务器 抓取文件 实例 require 'function.php'; $url = 'https://list.jd.com/list.html?cat=73 ...

  2. Go爬虫colly官方示例二十一【shopify_sitemap】- 使用XPath请求抓取shopify网站地图里url

    文章目录 示例介绍 示例代码 名词解释 Sitemap 简介 XPath 输出 转载请标明出处: http://blog.csdn.net/qq_27818541/article/details/11 ...

  3. Go爬虫colly官方示例十五【proxy_switcher】- 设置代理池

    文章目录 示例介绍 示例代码 输出 转载请标明出处: http://blog.csdn.net/qq_27818541/article/details/112489172 本文出自:[BigManin ...

  4. 采集淘宝API数据，抓取淘宝商品资料无需申请appkey

    为了进行淘宝的API开发,首先我们需要做下面几件事情. 1)开发者注册一个账号 2)然后为每个淘宝应用注册一个应用程序键(App Key) . 3)下载淘宝API的SDK并掌握基本的API基础知识和调 ...

  5. Python爬虫实战六之抓取爱问知识人问题并保存至数据库

    大家好,本次为大家带来的是抓取爱问知识人的问题并将问题和答案保存到数据库的方法,涉及的内容包括: Urllib的用法及异常处理 Beautiful Soup的简单应用 MySQLdb的基础用法 正则表 ...

  6. [Python从零到壹] 四.网络爬虫之入门基础及正则表达式抓取博客案例

    欢迎大家来到"Python从零到壹",在这里我将分享约200篇Python系列文章,带大家一起去学习和玩耍,看看Python这个有趣的世界.所有文章都将结合案例.代码和作者的经验讲 ...

  7. c#使用正则表达式获取TR中的多个TD_[Python从零到壹] 四.网络爬虫之入门基础及正则表达式抓取博客案例...

    首先祝大家中秋节和国庆节快乐,欢迎大家来到"Python从零到壹",在这里我将分享约200篇Python系列文章,带大家一起去学习和玩耍,看看Python这个有趣的世界.所有文章都 ...

  8. Python爬虫淘宝基于selenium抓取淘宝商品数据2021年测试过滑动验证

    配置一下"可能需要修改的参数",就可以食用底部代码了,ps:可能已失效 本文章代码功能 准备工作 Python用到的库和准备工作 可能需要修改的参数 在CMD中打开一个Chrome ...

  9. 爬虫练习（二）—股票信息抓取

    爬虫练习(二)-股票信息抓取 1 思路 1.首先需要找到一个将股票信息存放在html上的股票网站 2.因为每一个股票都是一个独立的页面,所以在抓取时,首先要了解每个股票网址的规律(网址最后结尾都是股票 ...

最新文章

  1. sh脚本每天创建一个文件夹_我每天创建一个月的视频。 这就是发生的事
  2. 搭建Cacti监控系统（一）-- 搭建LNMP 环境
  3. 百度大脑开放日：24 种新 AI 能力 20 个经典案例 助力 AI 落地
  4. 马云不会因支付宝事件走下神坛
  5. mysql使用 CONCAT(字段,字段) 函数拼接
  6. Java笔记-通过4个小程序对Java内存初步探究
  7. 应用在tomcat下的四种部署方式(原创)
  8. 良好的用户界面设计技巧
  9. linux下载镜像的命令wget,Linux wget命令整站下载做网站镜像
  10. element-ui可编辑行增加行或删除行
  11. 5分钟搞懂如何在Spring Boot中Schedule Tasks
  12. PHP仿网易云原创音乐分享平台网站源码
  13. 【STM32】基于STM32F407实现串口通信
  14. 《梦的解析》读后感（来给自己解解梦吧）
  15. Web3j使用教程(1)
  16. 技术人的修炼之道：从业余到专业
  17. 重庆理工大学计算机考研资料汇总
  18. 投影幕布选购避坑指南！
  19. 转：高效人士执行的四个原则
  20. 笔记本内存条的选择指南

热门文章

  1. 『往事』之---我的童年少年时代
  2. 最有福气的家庭：不翻旧账、不争对错、不慕虚荣
  3. python：pyautocad
  4. android分析手机唤醒源
  5. python 保障系统（一）
  6. HPC Game小结
  7. Apache Tomcat 下载网站惊现校内（xiaonei）镜像
  8. HTML5之placeholder属性
  9. 不会使用以下命令，别说你会Linux
  10. H5全新红包直通车网站源码 包含多款游戏已对接支付