前不久。看到一篇文章我用爬虫一天时间“偷了”知乎一百万用户。仅仅为证明PHP是世界上最好的语言，该文章中使用的登录方式是直接复制cookie到代码中，这里呢，我不以爬信息为目的。仅仅是简单的介绍使用java来进行模拟登录的基本过程。之前写过的文章android 项目实战——打造超级课程表一键提取课表功能事实上就是模拟登录的范畴。再加上近期在知乎上看到非常多人问关于超级课程表的实现，事实上本质就是模拟登录，掌握了这篇文章的内容，你不再操心抓不到信息了。然后，这篇文章会使用到之前的一篇Cookie保持的文章Android OkHttp的Cookie自己主动化管理，还有Jsoup的使用Jsoup库使用全然解析，为了简单处理，直接使用javaSE来，而不再使用Android进行。假设要移植到Android，唯一的处理可能就是把网络请求工作扔到子线程中去 。

首先使用Chrome打开知乎首页 , 点击登录，你会看到以下这个界面

在Chorme中按F12，调出开发人员工具，切到Network选项卡，勾选Preserve Log。记得一定要勾选，不然你会看不到信息。

一切就绪后，在输入框中输出账号密码点击登录。登录成功后你会看到这么一条记录

点击图中的email，在最下方你会看到本次请求提交了4个參数，以及在上方，你会看到本次请求的地址是http://www.zhihu.com/login/email

你会吃惊的发现知乎的密码是明文传输的，提交的參数的意思也非常easy，email就是账号，password就是密码。remember_me就是是否记住，这里传true就能够了，另一个_xsrf參数，这个毛估估应该是防爬虫的。

因此在提交前我们要从源代码中将这个值抓取下来。该值在表单的隐藏域中

一切准备就绪后，你就兴高採烈的用代码去模拟登录，然后你会发现会返回一个验证码错误的信息。事实上，我们还须要提交一个验证码，其參数名为captcha，验证码的地址为，

http://www.zhihu.com/captcha.gif?r=时间戳

于是我们得出了这种一个数据。

请求地址

http://www.zhihu.com/login/email

请求參数

_xsrf 表单中提取的隐藏域的值

captcha 验证码

email 邮箱

password 密码

remember_me 记住我

另一个问题。验证码的值怎么得到呢。答案是人工输入。将验证码保存到本地进行觉得识别，输入后进行登陆就可以。

这里的网络请求使用OkHttp。以及解析使用Jsoup，然后我们会使用到Gson，将他们增加maven依赖

com.squareup.okhttp

okhttp

2.4.0

org.jsoup

jsoup

1.8.3

com.google.code.gson

gson

2.3.1

在编码之前。我们得想想怎么维持登陆状态。没错，就是Cookie怎样保持，我们仅仅进行登陆一次，兴许都直接採集数据就能够了，因此须要将cookie持久化。对之前的文章中的一个Android类进行改造。使其变成java平台可用的类。能够看到我们将它从之前保存到SharePrefrences中改成了保存到文件里，并以json形式存储，这就是为什么会用到Gson的原因了

package cn.edu.zafu.zhihu;

import com.google.gson.Gson;

import com.google.gson.GsonBuilder;

import com.google.gson.reflect.TypeToken;

import java.io.*;

import java.net.CookieStore;

import java.net.HttpCookie;

import java.net.URI;

import java.net.URISyntaxException;

import java.util.*;

import java.util.concurrent.ConcurrentHashMap;

/**

* User:lizhangqu(513163535@qq.com)

* Date:2015-07-18

* Time: 16:54

*/

public class PersistentCookieStore implements CookieStore {

private static final Gson gson= new GsonBuilder().setPrettyPrinting().create();

private static final String LOG_TAG = "PersistentCookieStore";

private static final String COOKIE_PREFS = "CookiePrefsFile";

private static final String COOKIE_NAME_PREFIX = "cookie_";

private final HashMap> cookies;

private Map cookiePrefs=new HashMap();

/**

* Construct a persistent cookie store.

*

*/

public PersistentCookieStore() {

String cookieJson = readFile("cookie.json");

Map fromJson = gson.fromJson(cookieJson,new TypeToken>() {}.getType());

if(fromJson!=null){

System.out.println(fromJson);

cookiePrefs=fromJson;

}

cookies = new HashMap>();

// Load any previously stored cookies into the store

for(Map.Entry entry : cookiePrefs.entrySet()) {

if (((String)entry.getValue()) != null && !((String)entry.getValue()).startsWith(COOKIE_NAME_PREFIX)) {

String[] cookieNames = split((String) entry.getValue(), ",");

for (String name : cookieNames) {

String encodedCookie = cookiePrefs.get(COOKIE_NAME_PREFIX + name);

if (encodedCookie != null) {

HttpCookie decodedCookie = decodeCookie(encodedCookie);

if (decodedCookie != null) {

if(!cookies.containsKey(entry.getKey()))

cookies.put(entry.getKey(), new ConcurrentHashMap());

cookies.get(entry.getKey()).put(name, decodedCookie);

}

}

}

}

}

}

public void add(URI uri, HttpCookie cookie) {

String name = getCookieToken(uri, cookie);

// Save cookie into local store, or remove if expired

if (!cookie.hasExpired()) {

if(!cookies.containsKey(uri.getHost()))

cookies.put(uri.getHost(), new ConcurrentHashMap());

cookies.get(uri.getHost()).put(name, cookie);

} else {

if(cookies.containsKey(uri.toString()))

cookies.get(uri.getHost()).remove(name);

}

cookiePrefs.put(uri.getHost(), join(",", cookies.get(uri.getHost()).keySet()));

cookiePrefs.put(COOKIE_NAME_PREFIX + name, encodeCookie(new SerializableHttpCookie(cookie)));

String json=gson.toJson(cookiePrefs);

saveFile(json.getBytes(), "cookie.json");

}

protected String getCookieToken(URI uri, HttpCookie cookie) {

return cookie.getName() + cookie.getDomain();

}

public List get(URI uri) {

ArrayList ret = new ArrayList();

if(cookies.containsKey(uri.getHost()))

ret.addAll(cookies.get(uri.getHost()).values());

return ret;

}

public boolean removeAll() {

cookiePrefs.clear();

cookies.clear();

return true;

}

public boolean remove(URI uri, HttpCookie cookie) {

String name = getCookieToken(uri, cookie);

if(cookies.containsKey(uri.getHost()) && cookies.get(uri.getHost()).containsKey(name)) {

cookies.get(uri.getHost()).remove(name);

if(cookiePrefs.containsKey(COOKIE_NAME_PREFIX + name)) {

cookiePrefs.remove(COOKIE_NAME_PREFIX + name);

}

cookiePrefs.put(uri.getHost(), join(",", cookies.get(uri.getHost()).keySet()));

return true;

} else {

return false;

}

}

public List getCookies() {

ArrayList ret = new ArrayList();

for (String key : cookies.keySet())

ret.addAll(cookies.get(key).values());

return ret;

}

public List getURIs() {

ArrayList ret = new ArrayList();

for (String key : cookies.keySet())

try {

ret.add(new URI(key));

} catch (URISyntaxException e) {

e.printStackTrace();

}

return ret;

}

/**

* Serializes Cookie object into String

*

*@param cookie cookie to be encoded, can be null

*@return cookie encoded as String

*/

protected String encodeCookie(SerializableHttpCookie cookie) {

if (cookie == null)

return null;

ByteArrayOutputStream os = new ByteArrayOutputStream();

try {

ObjectOutputStream outputStream = new ObjectOutputStream(os);

outputStream.writeObject(cookie);

} catch (IOException e) {

System.out.println("IOException in encodeCookie"+ e);

return null;

}

return byteArrayToHexString(os.toByteArray());

}

/**

* Returns cookie decoded from cookie string

*

*@param cookieString string of cookie as returned from http request

*@return decoded cookie or null if exception occured

*/

protected HttpCookie decodeCookie(String cookieString) {

byte[] bytes = hexStringToByteArray(cookieString);

ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(bytes);

HttpCookie cookie = null;

try {

ObjectInputStream objectInputStream = new ObjectInputStream(byteArrayInputStream);

cookie = ((SerializableHttpCookie) objectInputStream.readObject()).getCookie();

} catch (IOException e) {

System.out.println("IOException in decodeCookie"+e);

} catch (ClassNotFoundException e) {

System.out.println("ClassNotFoundException in decodeCookie"+e);

}

return cookie;

}

/**

* Using some super basic byte array <-> hex conversions so we don't have to rely on any

* large Base64 libraries. Can be overridden if you like!

*

*@param bytes byte array to be converted

*@return string containing hex values

*/

protected String byteArrayToHexString(byte[] bytes) {

StringBuilder sb = new StringBuilder(bytes.length * 2);

for (byte element : bytes) {

int v = element & 0xff;

if (v < 16) {

sb.append('0');

}

sb.append(Integer.toHexString(v));

}

return sb.toString().toUpperCase(Locale.US);

}

/**

* Converts hex values from strings to byte arra

*

*@param hexString string of hex-encoded values

*@return decoded byte array

*/

protected byte[] hexStringToByteArray(String hexString) {

int len = hexString.length();

byte[] data = new byte[len / 2];

for (int i = 0; i < len; i += 2) {

data[i / 2] = (byte) ((Character.digit(hexString.charAt(i), 16) << 4) + Character.digit(hexString.charAt(i + 1), 16));

}

return data;

}

public static String join(CharSequence delimiter, Iterable tokens) {

StringBuilder sb = new StringBuilder();

boolean firstTime = true;

for (Object token: tokens) {

if (firstTime) {

firstTime = false;

} else {

sb.append(delimiter);

}

sb.append(token);

}

return sb.toString();

}

public static String[] split(String text, String expression) {

if (text.length() == 0) {

return new String[]{};

} else {

return text.split(expression, -1);

}

}

public static void saveFile(byte[] bfile, String fileName) {

BufferedOutputStream bos = null;

FileOutputStream fos = null;

File file = null;

try {

file = new File(fileName);

fos = new FileOutputStream(file);

bos = new BufferedOutputStream(fos);

bos.write(bfile);

} catch (Exception e) {

e.printStackTrace();

} finally {

if (bos != null) {

try {

bos.close();

} catch (IOException e1) {

e1.printStackTrace();

}

}

if (fos != null) {

try {

fos.close();

} catch (IOException e1) {

e1.printStackTrace();

}

}

}

}

public static String readFile(String fileName) {

BufferedInputStream bis = null;

FileInputStream fis = null;

File file = null;

try {

file = new File(fileName);

fis = new FileInputStream(file);

bis = new BufferedInputStream(fis);

int available = bis.available();

byte[] bytes=new byte[available];

bis.read(bytes);

String str=new String(bytes);

return str;

} catch (Exception e) {

e.printStackTrace();

} finally {

if (bis != null) {

try {

bis.close();

} catch (IOException e1) {

e1.printStackTrace();

}

}

if (fis != null) {

try {

fis.close();

} catch (IOException e1) {

e1.printStackTrace();

}

}

}

return "";

}

}

然后新建一个OkHttp请求类，并设置其Cookie处理类为我们编写的类。

private static OkHttpClient client = new OkHttpClient();

client.setCookieHandler(new CookieManager(new PersistentCookieStore(), CookiePolicy.ACCEPT_ALL));

好了。能够開始获取_xsrf以及验证码了。验证码保存在项目根文件夹下名为code.png的文件

private static String xsrf;

public static void getCode() throws IOException{

Request request = new Request.Builder()

.url("http://www.zhihu.com/")

.addHeader("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36")

.build();

Response response = client.newCall(request).execute();

String result = response.body().string();

Document parse = Jsoup.parse(result);

System.out.println(parse + "");

result = parse.select("input[type=hidden]").get(0).attr("value")

.trim();

xsrf=result;

System.out.println("_xsrf:" + result);

String codeUrl = "http://www.zhihu.com/captcha.gif?r=";

codeUrl += System.currentTimeMillis();

System.out.println("codeUrl:" + codeUrl);

Request getcode = new Request.Builder()

.url(codeUrl)

.addHeader("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36")

.build();

Response code = client.newCall(getcode).execute();

byte[] bytes = code.body().bytes();

saveCode(bytes, "code.png");

}

public static void saveCode(byte[] bfile, String fileName) {

BufferedOutputStream bos = null;

FileOutputStream fos = null;

File file = null;

try {

file = new File(fileName);

fos = new FileOutputStream(file);

bos = new BufferedOutputStream(fos);

bos.write(bfile);

} catch (Exception e) {

e.printStackTrace();

} finally {

if (bos != null) {

try {

bos.close();

} catch (IOException e1) {

e1.printStackTrace();

}

}

if (fos != null) {

try {

fos.close();

} catch (IOException e1) {

e1.printStackTrace();

}

}

}

}

然后将获取来的參数连同账号密码进行提交登录

public static void login(String randCode,String email,String password) throws IOException{

RequestBody formBody = new FormEncodingBuilder()

.add("_xsrf", xsrf)

.add("captcha", randCode)

.add("email", email)

.add("password", password)

.add("remember_me", "true")

.build();

Request login = new Request.Builder()

.url("http://www.zhihu.com/login/email")

.post(formBody)

.addHeader("User-Agent","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36")

.build();

Response execute = client.newCall(login).execute();

System.out.println(decode(execute.body().string()));

}

public static String decode(String unicodeStr) {

if (unicodeStr == null) {

return null;

}

StringBuffer retBuf = new StringBuffer();

int maxLoop = unicodeStr.length();

for (int i = 0; i < maxLoop; i++) {

if (unicodeStr.charAt(i) == '\\') {

if ((i < maxLoop - 5)

&& ((unicodeStr.charAt(i + 1) == 'u') || (unicodeStr

.charAt(i + 1) == 'U')))

try {

retBuf.append((char) Integer.parseInt(

unicodeStr.substring(i + 2, i + 6), 16));

i += 5;

} catch (NumberFormatException localNumberFormatException) {

retBuf.append(unicodeStr.charAt(i));

}

else

retBuf.append(unicodeStr.charAt(i));

} else {

retBuf.append(unicodeStr.charAt(i));

}

}

return retBuf.toString();

}

当看到以下的信息就代码登录成功了

之后你就能够获取你想要的信息了。这里简单获取一些信息，比方我要获取轮子哥的followers的昵称。分页自己处理下就ok了。

public static void getFollowers() throws IOException{

Request request = new Request.Builder()

.url("http://www.zhihu.com/people/zord-vczh/followees")

.addHeader("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36")

.build();

Response response = client.newCall(request).execute();

String result=response.body().string();

Document parse = Jsoup.parse(result);

Elements select = parse.select("div.zm-profile-card");

StringBuilder builder=new StringBuilder();

for (int i=0;i

Element element = select.get(i);

String name=element.select("h2").text();

System.out.println(name+"");

builder.append(name);

builder.append("\n");

}

}

下图就是获取到的信息。当然。仅仅要你登录了。什么信息你都能够获取到。