Go爬虫colly官方示例二十一【shopify_sitemap】- 使用XPath请求抓取shopify网站地图里url

文章目录

示例介绍
示例代码
名词解释
- Sitemap 简介
- XPath
输出

转载请标明出处：
http://blog.csdn.net/qq_27818541/article/details/112587216
本文出自:【BigManing的博客】

示例介绍

访问shopify网站地图url https://www.shopify.com/sitemap.xml
c.OnXML 爬取 //urlset/url/loc 对应的值
组装到集合中
打印

示例代码

package mainimport ("fmt""github.com/gocolly/colly/v2"
)func main() {// Array containing all the known URLs in a sitemapknownUrls := []string{}// Create a Collector specifically for Shopify//设置domainc := colly.NewCollector(colly.AllowedDomains("www.shopify.com"))// Create a callback on the XPath query searching for the URLs    //由于数据结构是xml，所以使用XPath查询 爬取每个urlc.OnXML("//urlset/url/loc", func(e *colly.XMLElement) {knownUrls = append(knownUrls, e.Text)})// Start the collector //访问urlc.Visit("https://www.shopify.com/sitemap.xml")fmt.Println("All known URLs:")for _, url := range knownUrls {fmt.Println("\t", url)} // 打印数据fmt.Println("Collected", len(knownUrls), "URLs")
}

名词解释

Sitemap 简介

可方便网站管理员通知搜索引擎他们网站上有哪些可供抓取的网页。最简单的 Sitemap 形式，就是XML 文件，在其中列出网站中的网址以及关于每个网址的其他元数据（上次更新的时间、更改的频率以及相对于网站上其他网址的重要程度为何等），以便搜索引擎可以更加智能地抓取网站。

XPath

XPath，全称 XML Path Language，即 XML 路径语言，它是一门在 XML 文档中查找信息的语言。最初是用来搜寻 XML 文档的，但同样适用于 HTML 文档的搜索。

输出

All known URLs:https://www.shopify.com/https://www.shopify.com/3dhttps://www.shopify.com/abouthttps://www.shopify.com/about/economic-impacthttps://www.shopify.com/about/economic-impact/canadahttps://www.shopify.com/about/economic-impact/globalhttps://www.shopify.com/about/economic-impact/individualhttps://www.shopify.com/about/economic-impact/regionalhttps://www.shopify.com/about/environmenthttps://www.shopify.com/about/environment/sustainability-fundhttps://www.shopify.com/about/environment/sustainability-fund/biomasshttps://www.shopify.com/about/environment/sustainability-fund/collaborationhttps://www.shopify.com/success-stories/novo-watchhttps://www.shopify.com/success-stories/modern-markethttps://www.shopify.com/success-stories/datsusarahttps://www.shopify.com/success-stories/catherine-zoraidahttps://www.shopify.com/success-stories/orange-and-parkhttps://www.shopify.com/success-stories/nj-swingsetshttps://www.shopify.com/success-stories/bija-bharhttps://www.shopify.com/success-stories/groomstandhttps://www.shopify.com/success-stories/regal-gentlemenhttps://www.shopify.com/success-stories/lightbox-photography-cardshttps://www.shopify.com/success-stories/suffolk-latch-companyhttps://www.shopify.com/success-stories/one-tribe-apparelhttps://www.shopify.com/success-stories/tikker-watcheshttps://www.shopify.com/success-stories/mstrmnd-collectivehttps://www.shopify.com/success-stories/old-world-kitchen-by-polder-s-old-world-markethttps://www.shopify.com/success-stories/growing-footprintshttps://www.shopify.com/success-stories/love-that-baghttps://www.shopify.com/success-stories/sweet-spot-skirtshttps://www.shopify.com/success-stories/christmas-shirt-companyhttps://www.shopify.com/success-stories/unsung-hero-apparelhttps://www.shopify.com/success-stories/enjoy-meditationhttps://www.shopify.com/success-stories/fragrance-buyhttps://www.shopify.com/success-stories/laguna-salt-companyhttps://www.shopify.com/success-stories/subehttps://www.shopify.com/success-stories/tiny-devotionshttps://www.shopify.com/success-stories/modish-storehttps://www.shopify.com/success-stories/bogarthttps://www.shopify.com/success-stories/the-future-kepthttps://www.shopify.com/success-stories/longsands-clothing-cohttps://www.shopify.com/success-stories/maitri-learninghttps://www.shopify.com/success-stories/koogalhttps://www.shopify.com/success-stories/mezel-modshttps://www.shopify.com/success-stories/shoreline-musichttps://www.shopify.com/success-stories/the-last-hunthttps://www.shopify.com/success-stories/krama-wheelhttps://www.shopify.com/success-stories/girly-go-garterhttps://www.shopify.com/success-stories/dewerstonehttps://www.shopify.com/success-stories/lotus-belle-usahttps://www.shopify.com/success-stories/all-mountain-stylehttps://www.shopify.com/success-stories/mountain-standardhttps://www.shopify.com/success-stories/jones-of-boerum-hillhttps://www.shopify.com/success-stories/kristin-schmucker-shophttps://www.shopify.com/success-stories/brooklyn-bambinihttps://www.shopify.com/success-stories/the-comfortable-boxers-cohttps://www.shopify.com/success-stories/latched-mamahttps://www.shopify.com/success-stories/floor-tape-storehttps://www.shopify.com/success-stories/mukhayogahttps://www.shopify.com/success-stories/tilit-chef-goodshttps://www.shopify.com/success-stories/gem-chocolateshttps://www.shopify.com/success-stories/plant-with-purposehttps://www.shopify.com/success-stories/brownfields-shophttps://www.shopify.com/success-stories/notabag-totehttps://www.shopify.com/success-stories/project-repathttps://www.shopify.com/success-stories/cape-whoopieshttps://www.shopify.com/success-stories/be-green-packaging-storehttps://www.shopify.com/success-stories/think-labshttps://www.shopify.com/success-stories/vlu-accessorieshttps://www.shopify.com/success-stories/skin-owl-oilhttps://www.shopify.com/success-stories/desires-by-mikolayhttps://www.shopify.com/success-stories/verttyhttps://www.shopify.com/success-stories/jerky-spothttps://www.shopify.com/success-stories/the-yes-barhttps://www.shopify.com/success-stories/holy-craphttps://www.shopify.com/success-stories/mud-gearhttps://www.shopify.com/success-stories/econica-lingeriehttps://www.shopify.com/success-stories/not-another-bunch-of-flowershttps://www.shopify.com/success-stories/the-uncommon-greenhttps://www.shopify.com/success-stories/qubits-toyhttps://www.shopify.com/success-stories/shinesty-clothinghttps://www.shopify.com/success-stories/betsy-and-iyahttps://www.shopify.com/success-stories/quest-award-shophttps://www.shopify.com/success-stories/retro-city-sunglasseshttps://www.shopify.com/success-stories/lets-hug-it-outhttps://www.shopify.com/success-stories/leaves-of-treeshttps://www.shopify.com/success-stories/biko-jewelryhttps://www.shopify.com/success-stories/timber-plateshttps://www.shopify.com/success-stories/baguerahttps://www.shopify.com/success-stories/chappy-happyhttps://www.shopify.com/success-stories/digital-tintypeshttps://www.shopify.com/success-stories/the-net-returnhttps://www.shopify.com/success-stories/me-camhttps://www.shopify.com/success-stories/tigerlily-sandalshttps://www.shopify.com/success-stories/warren-london-dog-productshttps://www.shopify.com/success-stories/famous-games-cohttps://www.shopify.com/success-stories/outline-the-skyhttps://www.shopify.com/success-stories/buicedhttps://www.shopify.com/success-stories/encircledhttps://www.shopify.com/success-stories/soaring-hearthttps://www.shopify.com/success-stories/desert-farms-camel-milkhttps://www.shopify.com/success-stories/chemo-cozyhttps://www.shopify.com/success-stories/humble-hilohttps://www.shopify.com/success-stories/oxygen-plushttps://www.shopify.com/success-stories/beauty-empirehttps://www.shopify.com/success-stories/sockhopnyhttps://www.shopify.com/success-stories/matter-goodshttps://www.shopify.com/success-stories/encarded-custom-cardshttps://www.shopify.com/success-stories/adorable-essentialshttps://www.shopify.com/success-stories/equsshttps://www.shopify.com/success-stories/lazy-fruitshttps://www.shopify.com/success-stories/mixed-madehttps://www.shopify.com/success-stories/wish-boxhttps://www.shopify.com/success-stories/esqido-lasheshttps://www.shopify.com/success-stories/brooklyn-delhihttps://www.shopify.com/success-stories/same-skyhttps://www.shopify.com/success-stories/stayhold-standhttps://www.shopify.com/success-stories/top-anchor-quiltinghttps://www.shopify.com/success-stories/kigurumihttps://www.shopify.com/success-stories/fantasy-jockshttps://www.shopify.com/success-stories/my-swim-stuffhttps://www.shopify.com/success-stories/united-by-bluehttps://www.shopify.com/success-stories/shop-lovelyhttps://www.shopify.com/success-stories/lilitabhttps://www.shopify.com/success-stories/beautiful-photo-propshttps://www.shopify.com/success-stories/top-and-derbyhttps://www.shopify.com/success-stories/3d-universehttps://www.shopify.com/success-stories/proof-eyewearhttps://www.shopify.com/success-stories/kitchen-safehttps://www.shopify.com/success-stories/suitablehttps://www.shopify.com/success-stories/64-facetshttps://www.shopify.com/success-stories/panda-sunglasseshttps://www.shopify.com/success-stories/slide-beltshttps://www.shopify.com/success-stories/strobepro-studio-lightinghttps://www.shopify.com/success-stories/caljava-onlinehttps://www.shopify.com/success-stories/hammock-universehttps://www.shopify.com/success-stories/get-back-supply-cohttps://www.shopify.com/success-stories/packed-partyhttps://www.shopify.com/success-stories/tradlands....Collected 1251 URLs

Go爬虫colly官方示例二十一【shopify_sitemap】- 使用XPath请求抓取shopify网站地图里url相关推荐

[Python爬虫] 之二十二：Selenium +phantomjs 利用 pyquery抓取界面网站数据
一.介绍本例子用Selenium +phantomjs爬取界面(https://a.jiemian.com/index.php?m=search&a=index&type=news& ...
Go爬虫colly官方示例六【xkcd_store】- 抓取xkcd商店商品列表
文章目录示例介绍示例代码输出转载请标明出处: http://blog.csdn.net/qq_27818541/article/details/112198134 本文出自:[BigManin ...
Go爬虫colly官方示例十五【proxy_switcher】- 设置代理池
文章目录示例介绍示例代码输出转载请标明出处: http://blog.csdn.net/qq_27818541/article/details/112489172 本文出自:[BigManin ...
NeuralProphet官方示例二：日照辐射强度预测(Forecasting hourly solar irradiance)
时间序列论文: NeuralProphet: Explainable Forecasting at Scale NeuralProphet之一:安装与使用 NeuralProphet之二:季节性(Se ...
[转]网络爬虫（一）：抓取网页的含义和URL基本构成
一.网络爬虫的定义网络爬虫,即Web Spider,是一个很形象的名字. 把互联网比喻成一个蜘蛛网,那么Spider就是在网上爬来爬去的蜘蛛. 网络蜘蛛是通过网页的链接地址来寻找网页的. 从网站某一 ...
python抓取网站重要url_[Python]网络爬虫（一）：抓取网页的含义和URL基本构成
一.网络爬虫的定义网络爬虫,即Web Spider,是一个很形象的名字. 把互联网比喻成一个蜘蛛网,那么Spider就是在网上爬来爬去的蜘蛛. 网络蜘蛛是通过网页的链接地址来寻找网页的. 从网站某一 ...
Java爬虫实战（一）：抓取一个网站上的全部链接
前言:写这篇文章之前,主要是我看了几篇类似的爬虫写法,有的是用的队列来写,感觉不是很直观,还有的只有一个请求然后进行页面解析,根本就没有自动爬起来这也叫爬虫?因此我结合自己的思路写了一下简单的爬虫,测 ...
爬虫实践---抓取小说网站
电子书网站:http://www.7kanshu.net/ 对网页进行分析可知,玄幻奇幻排行武侠仙侠排行历史军事排行完本小说排行观察发现,历史军事和完本小说的时一致的,其余类别是一样的,所以进行分开处 ...
Python爬虫学习第三章-4.3-使用xpath解析爬取全国城市名称
Python爬虫学习第三章-4.3-使用xpath解析爬取全国城市名称这一节主要是使用xpath解析爬取全国城市名称这里使用的网址是:空气质量历史数据查询这一个案例体现的点主要是xpat ...