本文是在 mac 系统操作

Chrome headless 模式介绍

Chrome-headless 模式， Google 针对 Chrome 浏览器 59版新增加的一种模式，可以让你不打开UI界面的情况下使用 Chrome 浏览器，所以运行效果与 Chrome 保持完美一致。
官方：
https://developers.google.cn/web/updates/2017/04/headless-chrome
其他：
https://linux.cn/article-8850-1.html
https://www.cnblogs.com/fnng/p/7797839.html

安装必备条件

1. golang 环境及环境变量配置

https://blog.csdn.net/fenglailea/article/details/70216964

2. 必要的包(主要是被墙的包)

mkdir -p $GOPATH/src/golang.org/x
cd $GOPATH/src/golang.org/x
git clone https://github.com/golang/net.git #这个就是net包
git clone https://github.com/golang/crypto.git #这个就是crypto包
git clone https://github.com/golang/sys.git
git clone https://github.com/golang/mobile.git
git clone https://github.com/golang/text.git
git clone https://github.com/golang/tools.git
git clone https://github.com/golang/image.git
... 等等 需要的包，请自行替换

3.安装chrome 并设置环境变量

安装 chrome

https://www.google.cn/intl/zh-CN/chrome/
如果你被墙，请自行更换其他下载地址

设置环境变量

来自
https://developers.google.cn/web/updates/2017/04/headless-chrome

vim ~/.bash_profile

最后一行加入

alias chrome="/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome"
alias chrome-canary="/Applications/Google\ Chrome\ Canary.app/Contents/MacOS/Google\ Chrome\ Canary"
alias chromium="/Applications/Chromium.app/Contents/MacOS/Chromium"

使生效

source ~/.bash_profile

4.XCODE

mac xcode 这个工具必须要安装，否则报错
在 App Store 搜索 xcode，进行安装(注意：因为此软件非常大，下载速度比较慢)

安装 chromedp

go get -u github.com/chromedp/chromedp

官方案例

https://github.com/chromedp/chromedp

go get -u -d github.com/chromedp/examples

执行案例格式

go run $GOPATH/src/github.com/chromedp/examples/xxxxxx/main.go

xxxxxx : 表示目录名称

具体目录请看官方 https://github.com/chromedp/examples#available-examples

执行案例

go run $GOPATH/src/github.com/chromedp/examples/eval/main.go

输出:

2018/10/05 11:29:40 <- {"id":1,"method":"Log.enable","params":{}}
2018/10/05 11:29:40 -> {"id":1,"result":{}}
2018/10/05 11:29:40 <- {"id":2,"method":"Runtime.enable","params":{}}
2018/10/05 11:29:40 -> {"method":"Runtime.executionContextCreated","params":{"context":{"id":1,"origin":"://","name":"","auxData":{"isDefault":true,"frameId":"E1EA547C788F3E5457AB5D1B4FB25D83"}}}}
2018/10/05 11:29:40 -> {"id":2,"result":{}}
2018/10/05 11:29:40 <- {"id":3,"method":"Inspector.enable","params":{}}
2018/10/05 11:29:40 -> {"id":3,"result":{}}
2018/10/05 11:29:40 <- {"id":4,"method":"Page.enable","params":{}}
2018/10/05 11:29:40 -> {"id":4,"result":{}}
2018/10/05 11:29:40 <- {"id":5,"method":"DOM.enable","params":{}}
2018/10/05 11:29:40 -> {"id":5,"result":{}}
2018/10/05 11:29:40 <- {"id":6,"method":"CSS.enable","params":{}}
2018/10/05 11:29:40 -> {"id":6,"result":{}}
2018/10/05 11:29:40 <- {"id":7,"method":"Page.getResourceTree","params":{}}
2018/10/05 11:29:40 -> {"id":7,"result":{"frameTree":{"frame":{"id":"E1EA547C788F3E5457AB5D1B4FB25D83","loaderId":"37D3BC533258DA6D9BE27DDC937D8FAF","url":"about:blank","securityOrigin":"://","mimeType":"text/html"},"resources":[]}}}
2018/10/05 11:29:40 <- {"id":8,"method":"DOM.getDocument","params":{"pierce":true}}
2018/10/05 11:29:40 -> {"id":8,"result":{"root":{"nodeId":1,"backendNodeId":1,"nodeType":9,"nodeName":"#document","localName":"","nodeValue":"","childNodeCount":1,"children":[{"nodeId":2,"parentId":1,"backendNodeId":2,"nodeType":1,"nodeName":"HTML","localName":"html","nodeValue":"","childNodeCount":2,"children":[{"nodeId":3,"parentId":2,"backendNodeId":3,"nodeType":1,"nodeName":"HEAD","localName":"head","nodeValue":"","childNodeCount":0,"attributes":[]},{"nodeId":4,"parentId":2,"backendNodeId":4,"nodeType":1,"nodeName":"BODY","localName":"body","nodeValue":"","childNodeCount":0,"attributes":[]}],"attributes":[],"frameId":"E1EA547C788F3E5457AB5D1B4FB25D83"}],"documentURL":"about:blank","baseURL":"about:blank","xmlVersion":""}}}
2018/10/05 11:29:40 <- {"id":9,"method":"Page.navigate","params":{"url":"https://www.baidu.com"}}
2018/10/05 11:29:41 -> {"id":9,"result":{"frameId":"E1EA547C788F3E5457AB5D1B4FB25D83","loaderId":"68AE558838E29AB64CFD2C5506D46BBE"}}
2018/10/05 11:29:41 <- {"id":10,"method":"Runtime.evaluate","params":{"expression":"Object.keys(window);","returnByValue":true}}
2018/10/05 11:29:41 -> {"method":"Page.frameStartedLoading","params":{"frameId":"E1EA547C788F3E5457AB5D1B4FB25D83"}}
2018/10/05 11:29:41 -> {"method":"Runtime.executionContextDestroyed","params":{"executionContextId":1}}
2018/10/05 11:29:41 -> {"method":"Runtime.executionContextsCleared","params":{}}
2018/10/05 11:29:41 -> {"method":"Page.frameNavigated","params":{"frame":{"id":"E1EA547C788F3E5457AB5D1B4FB25D83","loaderId":"68AE558838E29AB64CFD2C5506D46BBE","url":"https://www.baidu.com/","securityOrigin":"https://www.baidu.com","mimeType":"text/html"}}}
2018/10/05 11:29:41 -> {"method":"Runtime.executionContextCreated","params":{"context":{"id":2,"origin":"https://www.baidu.com","name":"","auxData":{"isDefault":true,"frameId":"E1EA547C788F3E5457AB5D1B4FB25D83"}}}}
2018/10/05 11:29:41 -> {"method":"DOM.documentUpdated","params":{}}
2018/10/05 11:29:41 <- {"id":11,"method":"DOM.getDocument","params":{"pierce":true}}
2018/10/05 11:29:41 -> {"id":10,"result":{"result":{"type":"object","value":["postMessage","blur","focus","close","frames","self","window","parent","opener","top","length","closed","location","document","origin","name","history","locationbar","menubar","personalbar","scrollbars","statusbar","toolbar","status","frameElement","navigator","customElements","external","screen","innerWidth","innerHeight","scrollX","pageXOffset","scrollY","pageYOffset","screenX","screenY","outerWidth","outerHeight","devicePixelRatio","clientInformation","screenLeft","screenTop","defaultStatus","defaultstatus","styleMedia","onanimationend","onanimationiteration","onanimationstart","onsearch","ontransitionend","onwebkitanimationend","onwebkitanimationiteration","onwebkitanimationstart","onwebkittransitionend","isSecureContext","onabort","onblur","oncancel","oncanplay","oncanplaythrough","onchange","onclick","onclose","oncontextmenu","oncuechange","ondblclick","ondrag","ondragend","ondragenter","ondragleave","ondragover","ondragstart","ondrop","ondurationchange","onemptied","onended","onerror","onfocus","oninput","oninvalid","onkeydown","onkeypress","onkeyup","onload","onloadeddata","onloadedmetadata","onloadstart","onmousedown","onmouseenter","onmouseleave","onmousemove","onmouseout","onmouseover","onmouseup","onmousewheel","onpause","onplay","onplaying","onprogress","onratechange","onreset","onresize","onscroll","onseeked","onseeking","onselect","onstalled","onsubmit","onsuspend","ontimeupdate","ontoggle","onvolumechange","onwaiting","onwheel","onauxclick","ongotpointercapture","onlostpointercapture","onpointerdown","onpointermove","onpointerup","onpointercancel","onpointerover","onpointerout","onpointerenter","onpointerleave","onafterprint","onbeforeprint","onbeforeunload","onhashchange","onlanguagechange","onmessage","onmessageerror","onoffline","ononline","onpagehide","onpageshow","onpopstate","onrejectionhandled","onstorage","onunhandledrejection","onunload","performance","stop","open","alert","confirm","prompt","print","requestAnimationFrame","cancelAnimationFrame","requestIdleCallback","cancelIdleCallback","captureEvents","releaseEvents","getComputedStyle","matchMedia","moveTo","moveBy","resizeTo","resizeBy","getSelection","find","webkitRequestAnimationFrame","webkitCancelAnimationFrame","fetch","btoa","atob","setTimeout","clearTimeout","setInterval","clearInterval","createImageBitmap","scroll","scrollTo","scrollBy","onappinstalled","onbeforeinstallprompt","crypto","ondevicemotion","ondeviceorientation","ondeviceorientationabsolute","indexedDB","webkitStorageInfo","sessionStorage","localStorage","chrome","visualViewport","speechSynthesis","webkitRequestFileSystem","webkitResolveLocalFileSystemURL","openDatabase","applicationCache","caches","h","_ASYNC_START","_chrome_37_fix","__async_strategy","bds","navigate","al_arr","selfOpen","isIE","E","bdUser","bdQuery","bdUseFavo","bdFavoOn","bdCid","bdSid","bdServerTime","bdQid","bdstoken","login_success"]}}}
2018/10/05 11:29:41 -> {"method":"Inspector.detached","params":{"reason":"Render process gone."}}
2018/10/05 11:29:41 -> { "method": "Inspector.detached", "params": { "reason": "target_closed"} }
2018/10/05 11:29:41 window object keys: [postMessage blur focus close frames self window parent opener top length closed location document origin name history locationbar menubar personalbar scrollbars statusbar toolbar status frameElement navigator customElements external screen innerWidth innerHeight scrollX pageXOffset scrollY pageYOffset screenX screenY outerWidth outerHeight devicePixelRatio clientInformation screenLeft screenTop defaultStatus defaultstatus styleMedia onanimationend onanimationiteration onanimationstart onsearch ontransitionend onwebkitanimationend onwebkitanimationiteration onwebkitanimationstart onwebkittransitionend isSecureContext onabort onblur oncancel oncanplay oncanplaythrough onchange onclick onclose oncontextmenu oncuechange ondblclick ondrag ondragend ondragenter ondragleave ondragover ondragstart ondrop ondurationchange onemptied onended onerror onfocus oninput oninvalid onkeydown onkeypress onkeyup onload onloadeddata onloadedmetadata onloadstart onmousedown onmouseenter onmouseleave onmousemove onmouseout onmouseover onmouseup onmousewheel onpause onplay onplaying onprogress onratechange onreset onresize onscroll onseeked onseeking onselect onstalled onsubmit onsuspend ontimeupdate ontoggle onvolumechange onwaiting onwheel onauxclick ongotpointercapture onlostpointercapture onpointerdown onpointermove onpointerup onpointercancel onpointerover onpointerout onpointerenter onpointerleave onafterprint onbeforeprint onbeforeunload onhashchange onlanguagechange onmessage onmessageerror onoffline ononline onpagehide onpageshow onpopstate onrejectionhandled onstorage onunhandledrejection onunload performance stop open alert confirm prompt print requestAnimationFrame cancelAnimationFrame requestIdleCallback cancelIdleCallback captureEvents releaseEvents getComputedStyle matchMedia moveTo moveBy resizeTo resizeBy getSelection find webkitRequestAnimationFrame webkitCancelAnimationFrame fetch btoa atob setTimeout clearTimeout setInterval clearInterval createImageBitmap scroll scrollTo scrollBy onappinstalled onbeforeinstallprompt crypto ondevicemotion ondeviceorientation ondeviceorientationabsolute indexedDB webkitStorageInfo sessionStorage localStorage chrome visualViewport speechSynthesis webkitRequestFileSystem webkitResolveLocalFileSystemURL openDatabase applicationCache caches h _ASYNC_START _chrome_37_fix __async_strategy bds navigate al_arr selfOpen isIE E bdUser bdQuery bdUseFavo bdFavoOn bdCid bdSid bdServerTime bdQid bdstoken login_success]

FAQ

invalid active developer path (/Library/Developer/CommandLineTools)

解决方法：
在 App Store 搜索 xcode，进行安装(注意：因为此软件非常大，下载速度比较慢)
其他解决方法：https://blog.csdn.net/zhufuing/article/details/53068185

出现 exit 1 等等情况时(我这里无法重现此情况了)

只要重启系统即可解决
原因：xcode 还没有安装，你就执行 go 运行命令了，并且命令启动chrome的进程无法自动结束，所以一直报错

来源

golang使用chrome headless获取网页内容
https://www.cnblogs.com/apocelipes/p/9264673.html

https://github.com/chromedp/chromedp

https://blog.csdn.net/qq_28796345/article/details/79698035

http://www.stats.gov.cn/tjsj/tjbz/tjyqhdmhcxhfdm/

go lang chrome 爬虫（MAC 系统）相关推荐

python拿什么做可视化界面好-用python打造可视化爬虫监控系统，酷炫的图形化界面...
原标题:用python打造可视化爬虫监控系统,酷炫的图形化界面本文并不是讲解爬虫的相关技术实现的,而是从实用性的角度,将抓取并存入 MongoDB 的数据用 InfluxDB 进行处理,而后又通过 ...
Mac系统好用快捷键简记
文章目录前言一.常见MAC按键图标说明二.几种类型好用的快捷键 1.输入法(mac自带) 2.截图(mac自带) 3.切换 4.文件管理 5.在chrome中的使用(没试过其它) 总结前言对 ...
mac php怎么做网页,Mac_mac系统中safari怎么添加书签? 把常用网页添加到收藏夹的效果，苹果电脑mac系统自带safari浏览 - phpStudy...
mac系统中safari怎么添加书签? 把常用网页添加到收藏夹的效果苹果电脑mac系统自带safari浏览器与普通的win系统是不一样的,收藏常用的网页也是不一样的,下面是将你常用的网页加入到书签里 ...
Mac系统使用终端检测局域网内开放端口的设备ip,扫描本地网络上的监控摄像头
如果支持http管理的摄像头,会开放80端口,如果确认设备连接到网络通常会自动获取IP地址(路由器已经开启了DHCP的情况下) 但是摄像头获取到的摄像头地址时什么呢?可以通过网络扫描来实现.如果是用m ...
MYJ2C一款java2c混淆工具 java代码转C代码交叉编译windows，linux，mac系统动态链接库
MYJ2C MYJ2C将编译的Java的Class字节码转换为C语言代码.交叉编译(您不用自己配置编译环境,MYJ2C自动完成)可以生成Windows,Linux,Mac系统X86,ARM平台的动态链 ...
黑苹果更改时间命令_黑苹果Mac系统快捷键修改
今天我们介绍一下黑苹果Mac系统快捷键修改的问题: 由于苹果机的键盘和普通PC机的键盘不同,因此苹果机的快捷键也会与普通PC不同,这对于我们这些经常使用键盘的人来说非常不便,下面附上两者的不同: 普通 ...
mac网页java无法加载,chrome浏览器mac版无法加载怎么办_chrome浏览器mac版打不开网页解决方法-win7之家...
对于使用苹果电脑的用户来说,想要安装chrome浏览器时,就需要选择相应版本的mac版chrome浏览器,然而最近有些用户在使用chrome浏览器mac版时却出现了无法加载,打不开网页的问题,那么ch ...
『Python学习笔记』Mac系统汇总：终端文件显示不同颜色Pycharm代码模板JAVA环境变量
Mac系统:终端文件显示不同颜色&Pycharm代码模板&JAVA环境变量! 文章目录一. 显示不同颜色二. mac电脑怎么设置触控板拖动三. mac系统中termius终端中文 ...
黑马程序员——ios面试学习一：Mac系统使用教程——黑马 ios 技术博客
------Java培训.Android培训.iOS培训..Net培训.期待与您交流! ------- 学习Mac从这里开始!!! 谢谢大家热烈支持, 感动! 你们的支持是我更新的动力!大家留言说 ...
mac系统清理软件MacClean支持文件类型分享
MacClean是一款设计巧妙且创造性的清洁应用程序,它为数百万Mac用户带来了清理,优化和保护Mac的好消息.只需点击几下,您就可以清除系统和浏览器生成的所有垃圾.借助多功能工具,您可以管理文件,扩 ...

go lang chrome 爬虫（MAC 系统）

Chrome headless 模式介绍

安装必备条件

1. golang 环境及环境变量配置

2. 必要的包(主要是被墙的包)

3.安装chrome 并设置环境变量

安装 chrome

设置环境变量

4.XCODE

安装 chromedp

官方案例

FAQ

invalid active developer path (/Library/Developer/CommandLineTools)

出现 exit 1 等等情况时(我这里无法重现此情况了)

go lang chrome 爬虫（MAC 系统）相关推荐

最新文章

热门文章

go lang chrome 爬虫 （MAC 系统）

Chrome headless 模式 介绍

安装必备条件

1. golang 环境及环境变量配置

2. 必要的包(主要是被墙的包)

3.安装chrome 并设置环境变量

安装 chrome

设置环境变量

4.XCODE

安装 chromedp

官方案例

FAQ

invalid active developer path (/Library/Developer/CommandLineTools)

出现 exit 1 等等情况时(我这里无法重现此情况了)

go lang chrome 爬虫 （MAC 系统）相关推荐

最新文章

热门文章

go lang chrome 爬虫（MAC 系统）

Chrome headless 模式介绍

go lang chrome 爬虫（MAC 系统）相关推荐