Python利用hadoop Streaming编写的Map-Reduce程序命令运行和本地调试运行

一、Hadoop的 Streaming介绍

Hadoop本身是用Java开发的，程序也需要用Java编写，但是通过Hadoop Streaming，可以使用任意语言（python、ruby、C++等）来编写程序，让Hadoop运行。

Hadoop提供了MapReduce的API,并允许使用非java的其他语言来写自己的map和reduce函数。而Streaming使用Unix标准流作为Hadoop和应用程序之间的接口，所以允许我们可以使用任何编程语言通过标准输入/输出来写MapReduce程序。Streaming的相关源码可以在hadoop的源码中查看。

Streaming天生适合用于文本处理，在文本模式下使用时，它有一个数据的行视图。map的输入数据通过标准输入流传递给map函数，并且是一行一行地传输，最后结果写入标准输出。map输出的键值对以一个制表符分割的行，它以这样的形式写到标准输出。reduce函数的输入格式相同——通过制表符来分割的键值对——并通过标准输入流进行传输。reduce函数从标准输入流中读取输入行，该输入已由hadoop框架根据键排序，最后将结果写入标准输出。

二、评价Hadoop的 Streaming

优点
- 可以使用自己喜欢的语言来编写MapReduce程序
- 不需要像写Java的MR程序那样import一大堆库，在代码里做一大堆配置，很多东西都抽象到了stdio上，代码量显著减少
- 因为没有库的依赖，调试方便，并且可以脱离Hadoop先在本地用管道pipes模拟调试
缺点
- 只能通过命令行参数来控制MapReduce框架，不像Java的程序那样可以在代码里使用API，控制力比较弱，有些东西鞭长莫及
- 因为中间隔着一层处理，效率会比较慢

所以Hadoop Streaming比较适合做一些简单的任务，比如用python写只有一两百行的脚本。如果项目比较复杂，或者需要进行比较细致的优化，使用Streaming就容易出现一些束手束脚的地方。

三、Python编写的MapReduce程序

1）输入的数据为books.json：把文本数据放入fs建立好的input文件夹中

["milton-paradise.txt", "[ Paradise Lost by John Milton 1667 ] Book I Of Man ' s first disobedience , and the fruit Of that forbidden tree whose mortal taste Brought death into the World , and all our woe , With loss of Eden , till one greater Man Restore us , and regain the blissful seat , Sing , Heavenly Muse , that , on the secret top Of Oreb , or of Sinai , didst inspire That shepherd who first taught the chosen seed In the beginning how the heavens and earth Rose out of Chaos : or , if Sion hill Delight thee more , and Siloa ' s brook that flowed Fast by the oracle of God , I thence Invoke thy aid to my adventurous song , That with no middle flight intends to soar Above th ' Aonian mount , while it pursues Things unattempted yet in prose or rhyme ."]
["edgeworth-parents.txt", "[ The Parent ' s Assistant , by Maria Edgeworth ] THE ORPHANS . Near the ruins of the castle of Rossmore , in Ireland , is a small cabin , in which there once lived a widow and her four children . As long as she was able to work , she was very industrious , and was accounted the best spinner in the parish ; but she overworked herself at last , and fell ill , so that she could not sit to her wheel as she used to do , and was obliged to give it up to her eldest daughter , Mary ."]
["austen-emma.txt", "[ Emma by Jane Austen 1816 ] VOLUME I CHAPTER I Emma Woodhouse , handsome , clever , and rich , with a comfortable home and happy disposition , seemed to unite some of the best blessings of existence ; and had lived nearly twenty - one years in the world with very little to distress or vex her . She was the youngest of the two daughters of a most affectionate , indulgent father ; and had , in consequence of her sister ' s marriage , been mistress of his house from a very early period . Her mother had died too long ago for her to have more than an indistinct remembrance of her caresses ; and her place had been supplied by an excellent woman as governess , who had fallen little short of a mother in affection ."]
["chesterton-ball.txt", "[ The Ball and The Cross by G . K . Chesterton 1909 ] I . A DISCUSSION SOMEWHAT IN THE AIR The flying ship of Professor Lucifer sang through the skies like a silver arrow ; the bleak white steel of it , gleaming in the bleak blue emptiness of the evening . That it was far above the earth was no expression for it ; to the two men in it , it seemed to be far above the stars . The professor had himself invented the flying machine , and had also invented nearly everything in it ."]
["bible-kjv.txt", "[ The King James Bible ] The Old Testament of the King James Bible The First Book of Moses : Called Genesis 1 : 1 In the beginning God created the heaven and the earth . 1 : 2 And the earth was without form , and void ; and darkness was upon the face of the deep . And the Spirit of God moved upon the face of the waters . 1 : 3 And God said , Let there be light : and there was light . 1 : 4 And God saw the light , that it was good : and God divided the light from the darkness . 1 : 5 And God called the light Day , and the darkness he called Night . And the evening and the morning were the first day ."]
["chesterton-thursday.txt", "[ The Man Who Was Thursday by G . K . Chesterton 1908 ] To Edmund Clerihew Bentley A cloud was on the mind of men , and wailing went the weather , Yea , a sick cloud upon the soul when we were boys together . Science announced nonentity and art admired decay ; The world was old and ended : but you and I were gay ; Round us in antic order their crippled vices came -- Lust that had lost its laughter , fear that had lost its shame . Like the white lock of Whistler , that lit our aimless gloom , Men showed their own white feather as proudly as a plume . Life was a fly that faded , and death a drone that stung ; The world was very old indeed when you and I were young ."]
["blake-poems.txt", "[ Poems by William Blake 1789 ] SONGS OF INNOCENCE AND OF EXPERIENCE and THE BOOK of THEL SONGS OF INNOCENCE INTRODUCTION Piping down the valleys wild , Piping songs of pleasant glee , On a cloud I saw a child , And he laughing said to me : \" Pipe a song about a Lamb !\" So I piped with merry cheer . \" Piper , pipe that song again ;\" So I piped : he wept to hear . \" Drop thy pipe , thy happy pipe ; Sing thy songs of happy cheer :!\" So I sang the same again , While he wept with joy to hear . \" Piper , sit thee down and write In a book , that all may read .\" So he vanish ' d from my sight ; And I pluck ' d a hollow reed , And I made a rural pen , And I stain ' d the water clear , And I wrote my happy songs Every child may joy to hear ."]
["shakespeare-caesar.txt", "[ The Tragedie of Julius Caesar by William Shakespeare 1599 ] Actus Primus . Scoena Prima . Enter Flauius , Murellus , and certaine Commoners ouer the Stage . Flauius . Hence : home you idle Creatures , get you home : Is this a Holiday ? What , know you not ( Being Mechanicall ) you ought not walke Vpon a labouring day , without the signe Of your Profession ? Speake , what Trade art thou ? Car . Why Sir , a Carpenter Mur . Where is thy Leather Apron , and thy Rule ? What dost thou with thy best Apparrell on ? You sir , what Trade are you ? Cobl . Truely Sir , in respect of a fine Workman , I am but as you would say , a Cobler Mur . But what Trade art thou ? Answer me directly Cob . A Trade Sir , that I hope I may vse , with a safe Conscience , which is indeed Sir , a Mender of bad soules Fla ."]
["whitman-leaves.txt", "[ Leaves of Grass by Walt Whitman 1855 ] Come , said my soul , Such verses for my Body let us write , ( for we are one ,) That should I after return , Or , long , long hence , in other spheres , There to some group of mates the chants resuming , ( Tallying Earth ' s soil , trees , winds , tumultuous waves ,) Ever with pleas ' d smile I may keep on , Ever and ever yet the verses owning -- as , first , I here and now Signing for Soul and Body , set to them my name , Walt Whitman [ BOOK I . INSCRIPTIONS ] } One ' s - Self I Sing One ' s - self I sing , a simple separate person , Yet utter the word Democratic , the word En - Masse ."]
["melville-moby_dick.txt", "[ Moby Dick by Herman Melville 1851 ] ETYMOLOGY . ( Supplied by a Late Consumptive Usher to a Grammar School ) The pale Usher -- threadbare in coat , heart , body , and brain ; I see him now . He was ever dusting his old lexicons and grammars , with a queer handkerchief , mockingly embellished with all the gay flags of all the known nations of the world . He loved to dust his old grammars ; it somehow mildly reminded him of his mortality ."]

2）把文本数据books.json放入fs建立好的1024_input文件夹中

localhost:experiment_data a6$ hadoop fs -mkdir /1024_input
localhost:experiment_data a6$ hadoop dfs -ls /
localhost:experiment_data a6$ hadoop fs -copyFromLocal book.json /1024_input
上传本地到hdfs上的命令格式如下：
$ hadoop fs -copyFromLocal <PATH TO LOCAL FILE(S)> input/

3）编写代码之一：统计每个单词在哪些文档里出现

map函数 invertedMapper.py 文件，内容如下：

#!/usr/bin/env python
'''
Test by yyz
Date: 2017/10/24
'''
import sys
import json
for line in sys.stdin:line = line.strip()record = json.loads(line)fileName = record[0];value = record[1];words = value.split()for word in words:print '%s\t%s' % (word, fileName)

4）编写代码之二：形成单词的简单倒排索引

reduce函数 invertedReducer.py 文件，内容如下:

# !/usr/bin/env python
'''
Test by yyz
Date: 2017/10/24
'''
import sys
# maps words to their counts
media = {}
wordInMedia = {}
# input comes from STDIN
for line in sys.stdin:(word, fileName) = line.strip().split('\t', 1)media.setdefault(word, [])media[word].append(fileName)
for word in media:wordInMedia.setdefault(word, list(set(media[word])))
# write the results to STDOUT (standard output)
for word in wordInMedia:print '%s\t%s' % (word, wordInMedia[word])

四、运行Python编写的MapReduce程序

在hadoop-2.6.5中测试本代码

1）确保输出文件夹1024_output不存在

localhost:experiment_data a6$ hadoop fs -rm -r /1024_output

2）在本地进行测试，书写模式如下：

cat <input path> | python <path to mapper script> | sort -t $'\t' -k1,1 | python <path to reducer script> > <output path>

我的本地测试命令如下：

cat map_reduce_program/book.json | python map_reduce_program/invertedMapper.py | sort | python map_reduce_program/invertedReducer.py

2）hadoop命令运行MR程序，书写模式如下：

$hadoop jar /Users/a6/Applications/hadoop-2.6.5/share/hadoop/tools/lib/hadoop-streaming-2.6.5.jar \
-mapper 'python mapper.py'
-file mapper.py\
-reducer 'python reducer.py'
-file reducer.py \
-input input/*
-output output

我的运行命令如下：

localhost:~ a6$ hadoop jar /Users/a6/Applications/hadoop-2.6.5/share/hadoop/tools/lib/hadoop-streaming-2.6.5.jar  -mapper "python invertedMapper.py" -reducer "python invertedReducer.py" -input "/1024_input/*" -output "/1024_output" -file "/Users/a6/Downloads/PycharmProjects/map_reduce_program/invertedMapper.py" -file "/Users/a6/Downloads/PycharmProjects/map_reduce_program/invertedReducer.py"

书写格式如下：

localhost:~ a6$ hadoop jar /Users/a6/Applications/hadoop-2.6.5/share/hadoop/tools/lib/hadoop-streaming-2.6.5.jar  \
-mapper "python invertedMapper.py" \
-reducer "python invertedReducer.py" \
-input "/1024_input/*"  \
-output "/1024_output" \
-file "/Users/a6/Downloads/PycharmProjects/map_reduce_program/invertedMapper.py"  \
-file "/Users/a6/Downloads/PycharmProjects/map_reduce_program/invertedReducer.py"

五、注意事项

1）Hadoop默认按照tab（\t）来分割key和value，以第一个分割出的部分为key，按key进行排序，因此这里使用
sort -t $'\t' -k1,1 来模拟。
2）如果你有其他需求，在交给Hadoop Streaming执行时可以通过命令行参数调，本地调试也可以进行相应的调整，主要是调整sort的参数。因此为了能够熟练进行本地调试，建议先掌握sort命令的用法。
3）如果你在python脚本里加上了shabang标记（#! /usr/bin/env python），并且为它们添加了执行权限，也可以用类似于
./mapper.py 代替 python mapper.py

六、生成结果如下

all  ['milton-paradise.txt', 'blake-poems.txt', 'melville-moby_dick.txt']
Rossmore    ['edgeworth-parents.txt']
Consumptive ['melville-moby_dick.txt']
Self    ['whitman-leaves.txt']
Night   ['bible-kjv.txt']
eldest  ['edgeworth-parents.txt']
four    ['edgeworth-parents.txt']
Caesar  ['shakespeare-caesar.txt']
winds   ['whitman-leaves.txt']
Moses   ['bible-kjv.txt']
children    ['edgeworth-parents.txt']
seemed  ['chesterton-ball.txt', 'austen-emma.txt']
whose   ['milton-paradise.txt']
Parent  ['edgeworth-parents.txt']
father  ['austen-emma.txt']
young   ['chesterton-thursday.txt']
vex ['austen-emma.txt']
to  ['milton-paradise.txt', 'austen-emma.txt', 'chesterton-ball.txt', 'blake-poems.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt']
th  ['milton-paradise.txt']
smile   ['whitman-leaves.txt']
Day ['bible-kjv.txt']
hill    ['milton-paradise.txt']
me  ['blake-poems.txt', 'shakespeare-caesar.txt']
woman   ['austen-emma.txt']
Whitman ['whitman-leaves.txt']
song    ['milton-paradise.txt', 'blake-poems.txt']
far ['chesterton-ball.txt']
void    ['bible-kjv.txt']
rhyme   ['milton-paradise.txt']
Oreb    ['milton-paradise.txt']
Pipe    ['blake-poems.txt']
Where   ['shakespeare-caesar.txt']
heaven  ['bible-kjv.txt']
Bentley ['chesterton-thursday.txt']
--  ['whitman-leaves.txt', 'melville-moby_dick.txt', 'chesterton-thursday.txt']
sang    ['chesterton-ball.txt', 'blake-poems.txt']
1667    ['milton-paradise.txt']
pleas   ['whitman-leaves.txt']
Paradise    ['milton-paradise.txt']
Fast    ['milton-paradise.txt']
VOLUME  ['austen-emma.txt']
Shakespeare ['shakespeare-caesar.txt']
earth   ['chesterton-ball.txt', 'bible-kjv.txt', 'milton-paradise.txt']
Jane    ['austen-emma.txt']
emptiness   ['chesterton-ball.txt']
blue    ['chesterton-ball.txt']
what    ['shakespeare-caesar.txt']
darkness    ['bible-kjv.txt']
Cobler  ['shakespeare-caesar.txt']
supplied    ['austen-emma.txt']
Delight ['milton-paradise.txt']
above   ['chesterton-ball.txt']
;   ['austen-emma.txt', 'chesterton-ball.txt', 'bible-kjv.txt', 'chesterton-thursday.txt', 'blake-poems.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt']
ever    ['whitman-leaves.txt', 'melville-moby_dick.txt']
piped   ['blake-poems.txt']
Apparrell   ['shakespeare-caesar.txt']
body    ['melville-moby_dick.txt']
Emma    ['austen-emma.txt']
Professor   ['chesterton-ball.txt']
men ['chesterton-ball.txt', 'chesterton-thursday.txt']
here    ['whitman-leaves.txt']
water   ['blake-poems.txt']
let ['whitman-leaves.txt']
crippled    ['chesterton-thursday.txt']
sing    ['whitman-leaves.txt']
[   ['milton-paradise.txt', 'austen-emma.txt', 'chesterton-ball.txt', 'bible-kjv.txt', 'chesterton-thursday.txt', 'blake-poems.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt']
invented    ['chesterton-ball.txt']
daughter    ['edgeworth-parents.txt']
my  ['milton-paradise.txt', 'blake-poems.txt', 'whitman-leaves.txt']
spheres ['whitman-leaves.txt']
DISCUSSION  ['chesterton-ball.txt']
Blake   ['blake-poems.txt']
heart   ['melville-moby_dick.txt']
Things  ['milton-paradise.txt']
Actus   ['shakespeare-caesar.txt']
till    ['milton-paradise.txt']
Book    ['milton-paradise.txt', 'bible-kjv.txt']
1908    ['chesterton-thursday.txt']
1909    ['chesterton-ball.txt']
Answer  ['shakespeare-caesar.txt']
cloud   ['blake-poems.txt', 'chesterton-thursday.txt']
from    ['bible-kjv.txt', 'austen-emma.txt', 'blake-poems.txt']
would   ['shakespeare-caesar.txt']
give    ['edgeworth-parents.txt']
two ['chesterton-ball.txt', 'austen-emma.txt']
more    ['milton-paradise.txt', 'austen-emma.txt']
Piper   ['blake-poems.txt']
Leather ['shakespeare-caesar.txt']
about   ['blake-poems.txt']
Rule    ['shakespeare-caesar.txt']
So  ['blake-poems.txt']
excellent   ['austen-emma.txt']
known   ['melville-moby_dick.txt']
fly ['chesterton-thursday.txt']
Signing ['whitman-leaves.txt']
word    ['whitman-leaves.txt']
middle  ['milton-paradise.txt']
this    ['shakespeare-caesar.txt']
work    ['edgeworth-parents.txt']
soul    ['whitman-leaves.txt', 'chesterton-thursday.txt']
hope    ['shakespeare-caesar.txt']
INNOCENCE   ['blake-poems.txt']
laughter    ['chesterton-thursday.txt']
admired ['chesterton-thursday.txt']
shepherd    ['milton-paradise.txt']
lock    ['chesterton-thursday.txt']
Edmund  ['chesterton-thursday.txt']
Aonian  ['milton-paradise.txt']
sir ['shakespeare-caesar.txt']
sit ['blake-poems.txt', 'edgeworth-parents.txt']
divided ['bible-kjv.txt']
Who ['chesterton-thursday.txt']
Carpenter   ['shakespeare-caesar.txt']
,)  ['whitman-leaves.txt']
machine ['chesterton-ball.txt']
how ['milton-paradise.txt']
Truely  ['shakespeare-caesar.txt']
Why ['shakespeare-caesar.txt']
embellished ['melville-moby_dick.txt']
A   ['chesterton-ball.txt', 'shakespeare-caesar.txt', 'chesterton-thursday.txt']
may ['blake-poems.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt']
ORPHANS ['edgeworth-parents.txt']
after   ['whitman-leaves.txt']
:!"    ['blake-poems.txt']
consequence ['austen-emma.txt']
waves   ['whitman-leaves.txt']
Hence   ['shakespeare-caesar.txt']
bleak   ['chesterton-ball.txt']
a   ['austen-emma.txt', 'chesterton-ball.txt', 'chesterton-thursday.txt', 'blake-poems.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt']
short   ['austen-emma.txt']
coat    ['melville-moby_dick.txt']
One ['whitman-leaves.txt']
so  ['edgeworth-parents.txt']
order   ['chesterton-thursday.txt']
signe   ['shakespeare-caesar.txt']
What    ['shakespeare-caesar.txt']
indeed  ['shakespeare-caesar.txt', 'chesterton-thursday.txt']
Supplied    ['melville-moby_dick.txt']
years   ['austen-emma.txt']
ended   ['chesterton-thursday.txt']
!" ['blake-poems.txt']
through ['chesterton-ball.txt']
same    ['blake-poems.txt']
existence   ['austen-emma.txt']
its ['chesterton-thursday.txt']
Body    ['whitman-leaves.txt']
1   ['bible-kjv.txt']
industrious ['edgeworth-parents.txt']
group   ['whitman-leaves.txt']
decay   ['chesterton-thursday.txt']
chosen  ['milton-paradise.txt']
glee    ['blake-poems.txt']
brook   ['milton-paradise.txt']
,   ['milton-paradise.txt', 'austen-emma.txt', 'chesterton-ball.txt', 'bible-kjv.txt', 'chesterton-thursday.txt', 'blake-poems.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt']
Sir ['shakespeare-caesar.txt']
Sing    ['milton-paradise.txt', 'blake-poems.txt', 'whitman-leaves.txt']
Tallying    ['whitman-leaves.txt']
them    ['whitman-leaves.txt']
evening ['chesterton-ball.txt', 'bible-kjv.txt']
Grammar ['melville-moby_dick.txt']
greater ['milton-paradise.txt']
died    ['austen-emma.txt']
Car ['shakespeare-caesar.txt']
thee    ['milton-paradise.txt', 'blake-poems.txt']
safe    ['shakespeare-caesar.txt']
Murellus    ['shakespeare-caesar.txt']
She ['austen-emma.txt']
not ['shakespeare-caesar.txt', 'edgeworth-parents.txt']
now ['whitman-leaves.txt', 'melville-moby_dick.txt']
silver  ['chesterton-ball.txt']
gloom   ['chesterton-thursday.txt']
name    ['whitman-leaves.txt']
arrow   ['chesterton-ball.txt']
went    ['chesterton-thursday.txt']
CHAPTER ['austen-emma.txt']
Mur ['shakespeare-caesar.txt']
Rose    ['milton-paradise.txt']
taught  ['milton-paradise.txt']
our ['milton-paradise.txt', 'chesterton-thursday.txt']
out ['milton-paradise.txt']
INSCRIPTIONS    ['whitman-leaves.txt']
'  ['milton-paradise.txt', 'blake-poems.txt', 'whitman-leaves.txt', 'edgeworth-parents.txt', 'austen-emma.txt']
day ['bible-kjv.txt', 'shakespeare-caesar.txt']
Muse    ['milton-paradise.txt']
Genesis ['bible-kjv.txt']
William ['blake-poems.txt', 'shakespeare-caesar.txt']
flowed  ['milton-paradise.txt']
waters  ['bible-kjv.txt']
announced   ['chesterton-thursday.txt']
Stage   ['shakespeare-caesar.txt']
disobedience    ['milton-paradise.txt']
G   ['chesterton-ball.txt', 'chesterton-thursday.txt']
pleasant    ['blake-poems.txt']
small   ['edgeworth-parents.txt']
ETYMOLOGY   ['melville-moby_dick.txt']
disposition ['austen-emma.txt']
Mechanicall ['shakespeare-caesar.txt']
mistress    ['austen-emma.txt']
beginning   ['milton-paradise.txt', 'bible-kjv.txt']
En  ['whitman-leaves.txt']
created ['bible-kjv.txt']
Being   ['shakespeare-caesar.txt']
could   ['edgeworth-parents.txt']
keep    ['whitman-leaves.txt']
fruit   ['milton-paradise.txt']
place   ['austen-emma.txt']
mortal  ['milton-paradise.txt']
hence   ['whitman-leaves.txt']
castle  ['edgeworth-parents.txt']
fallen  ['austen-emma.txt']
first   ['milton-paradise.txt', 'bible-kjv.txt', 'whitman-leaves.txt']
Lamb    ['blake-poems.txt']
flying  ['chesterton-ball.txt']
There   ['whitman-leaves.txt']
Speake  ['shakespeare-caesar.txt']
one ['milton-paradise.txt', 'whitman-leaves.txt', 'austen-emma.txt']
Scoena  ['shakespeare-caesar.txt']
directly    ['shakespeare-caesar.txt']
hear    ['blake-poems.txt']
Ever    ['whitman-leaves.txt']
Sinai   ['milton-paradise.txt']
owning  ['whitman-leaves.txt']
Chesterton  ['chesterton-ball.txt', 'chesterton-thursday.txt']
little  ['austen-emma.txt']
"  ['blake-poems.txt']
indulgent   ['austen-emma.txt']
twenty  ['austen-emma.txt']
top ['milton-paradise.txt']
their   ['chesterton-thursday.txt']
2   ['bible-kjv.txt']
too ['austen-emma.txt']
Such    ['whitman-leaves.txt']
white   ['chesterton-ball.txt', 'chesterton-thursday.txt']
cabin   ['edgeworth-parents.txt']
that    ['milton-paradise.txt', 'bible-kjv.txt', 'chesterton-thursday.txt', 'blake-poems.txt', 'shakespeare-caesar.txt', 'edgeworth-parents.txt']
1599    ['shakespeare-caesar.txt']
aimless ['chesterton-thursday.txt']
SOMEWHAT    ['chesterton-ball.txt']
Democratic  ['whitman-leaves.txt']
herself ['edgeworth-parents.txt']
than    ['austen-emma.txt']
steel   ['chesterton-ball.txt']
loved   ['melville-moby_dick.txt']
showed  ['chesterton-thursday.txt']
Come    ['whitman-leaves.txt']
tree    ['milton-paradise.txt']
nations ['melville-moby_dick.txt']
idle    ['shakespeare-caesar.txt']
Usher   ['melville-moby_dick.txt']
light   ['bible-kjv.txt']
threadbare  ['melville-moby_dick.txt']
indistinct  ['austen-emma.txt']
and ['milton-paradise.txt', 'austen-emma.txt', 'chesterton-ball.txt', 'bible-kjv.txt', 'chesterton-thursday.txt', 'blake-poems.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt']
handkerchief    ['melville-moby_dick.txt']
God ['milton-paradise.txt', 'bible-kjv.txt']
labouring   ['shakespeare-caesar.txt']
mind    ['chesterton-thursday.txt']
comfortable ['austen-emma.txt']
Late    ['melville-moby_dick.txt']
say ['shakespeare-caesar.txt']
mates   ['whitman-leaves.txt']
have    ['austen-emma.txt']
blissful    ['milton-paradise.txt']
saw ['bible-kjv.txt', 'blake-poems.txt']
Invoke  ['milton-paradise.txt']
self    ['whitman-leaves.txt']
able    ['edgeworth-parents.txt']
soules  ['shakespeare-caesar.txt']
lit ['chesterton-thursday.txt']
also    ['chesterton-ball.txt']
forbidden   ['milton-paradise.txt']
drone   ['chesterton-thursday.txt']
which   ['shakespeare-caesar.txt', 'edgeworth-parents.txt']
flags   ['melville-moby_dick.txt']
With    ['milton-paradise.txt']
Spirit  ['bible-kjv.txt']
pursues ['milton-paradise.txt']
soar    ['milton-paradise.txt']
Drop    ['blake-poems.txt']
most    ['austen-emma.txt']
Piping  ['blake-poems.txt']
Enter   ['shakespeare-caesar.txt']
The ['chesterton-ball.txt', 'bible-kjv.txt', 'chesterton-thursday.txt', 'shakespeare-caesar.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt']
]   ['milton-paradise.txt', 'austen-emma.txt', 'chesterton-ball.txt', 'bible-kjv.txt', 'chesterton-thursday.txt', 'blake-poems.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt']
Called  ['bible-kjv.txt']
SONGS   ['blake-poems.txt']
Restore ['milton-paradise.txt']
clear   ['blake-poems.txt']
face    ['bible-kjv.txt']
pipe    ['blake-poems.txt']
accounted   ['edgeworth-parents.txt']
Earth   ['whitman-leaves.txt']
}   ['whitman-leaves.txt']
came    ['chesterton-thursday.txt']
inspire ['milton-paradise.txt']
wailing ['chesterton-thursday.txt']
queer   ['melville-moby_dick.txt']
utter   ['whitman-leaves.txt']
fear    ['chesterton-thursday.txt']
heavens ['milton-paradise.txt']
fine    ['shakespeare-caesar.txt']
(   ['shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt']
plume   ['chesterton-thursday.txt']
should  ['whitman-leaves.txt']
rich    ['austen-emma.txt']
stain   ['blake-poems.txt']
handsome    ['austen-emma.txt']
mortality   ['melville-moby_dick.txt']
do  ['edgeworth-parents.txt']
his ['melville-moby_dick.txt', 'austen-emma.txt']
Prima   ['shakespeare-caesar.txt']
get ['shakespeare-caesar.txt']
Leaves  ['whitman-leaves.txt']
trees   ['whitman-leaves.txt']
While   ['blake-poems.txt']
nearly  ['chesterton-ball.txt', 'austen-emma.txt']
THE ['chesterton-ball.txt', 'blake-poems.txt', 'edgeworth-parents.txt']
him ['melville-moby_dick.txt']
child   ['blake-poems.txt']
Maria   ['edgeworth-parents.txt']
brain   ['melville-moby_dick.txt']
EXPERIENCE  ['blake-poems.txt']
morning ['bible-kjv.txt']
overworked  ['edgeworth-parents.txt']
bad ['shakespeare-caesar.txt']
antic   ['chesterton-thursday.txt']
she ['edgeworth-parents.txt']
Austen  ['austen-emma.txt']
ruins   ['edgeworth-parents.txt']
wrote   ['blake-poems.txt']
widow   ['edgeworth-parents.txt']
set ['whitman-leaves.txt']
art ['shakespeare-caesar.txt', 'chesterton-thursday.txt']
Chaos   ['milton-paradise.txt']
seat    ['milton-paradise.txt']
ouer    ['shakespeare-caesar.txt']
see ['melville-moby_dick.txt']
Lucifer ['chesterton-ball.txt']
are ['shakespeare-caesar.txt', 'whitman-leaves.txt']
John    ['milton-paradise.txt']
best    ['shakespeare-caesar.txt', 'edgeworth-parents.txt', 'austen-emma.txt']
said    ['bible-kjv.txt', 'whitman-leaves.txt', 'blake-poems.txt']
That    ['chesterton-ball.txt', 'milton-paradise.txt', 'whitman-leaves.txt']
INTRODUCTION    ['blake-poems.txt']
3   ['bible-kjv.txt']
woe ['milton-paradise.txt']
marriage    ['austen-emma.txt']
K   ['chesterton-ball.txt', 'chesterton-thursday.txt']
King    ['bible-kjv.txt']
we  ['whitman-leaves.txt', 'chesterton-thursday.txt']
Bible   ['bible-kjv.txt']
Commoners   ['shakespeare-caesar.txt']
joy ['blake-poems.txt']
were    ['bible-kjv.txt', 'chesterton-thursday.txt']
pluck   ['blake-poems.txt']
merry   ['blake-poems.txt']
AND ['blake-poems.txt']
youngest    ['austen-emma.txt']
last    ['edgeworth-parents.txt']
Yea ['chesterton-thursday.txt']
ill ['edgeworth-parents.txt']
thence  ['milton-paradise.txt']
called  ['bible-kjv.txt']
s   ['milton-paradise.txt', 'whitman-leaves.txt', 'edgeworth-parents.txt', 'austen-emma.txt']
mildly  ['melville-moby_dick.txt']
Sion    ['milton-paradise.txt']
expression  ['chesterton-ball.txt']
Yet ['whitman-leaves.txt']
Conscience  ['shakespeare-caesar.txt']
THEL    ['blake-poems.txt']
distress    ['austen-emma.txt']
Her ['austen-emma.txt']
Clerihew    ['chesterton-thursday.txt']
simple  ['whitman-leaves.txt']
period  ['austen-emma.txt']
hollow  ['blake-poems.txt']
gleaming    ['chesterton-ball.txt']
Ireland ['edgeworth-parents.txt']
Cobl    ['shakespeare-caesar.txt']
respect ['shakespeare-caesar.txt']
Holiday ['shakespeare-caesar.txt']
Was ['chesterton-thursday.txt']
Flauius ['shakespeare-caesar.txt']
Apron   ['shakespeare-caesar.txt']
Mender  ['shakespeare-caesar.txt']
dusting ['melville-moby_dick.txt']
been    ['austen-emma.txt']
.   ['milton-paradise.txt', 'austen-emma.txt', 'chesterton-ball.txt', 'bible-kjv.txt', 'chesterton-thursday.txt', 'blake-poems.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt']
secret  ['milton-paradise.txt']
feather ['chesterton-thursday.txt']
affectionate    ['austen-emma.txt']
lived   ['edgeworth-parents.txt', 'austen-emma.txt']
flight  ['milton-paradise.txt']
gay ['melville-moby_dick.txt', 'chesterton-thursday.txt']
am  ['shakespeare-caesar.txt']
deep    ['bible-kjv.txt']
World   ['milton-paradise.txt']
Whistler    ['chesterton-thursday.txt']
Round   ['chesterton-thursday.txt']
And ['bible-kjv.txt', 'blake-poems.txt']
Old ['bible-kjv.txt']
mount   ['milton-paradise.txt']
thou    ['shakespeare-caesar.txt']
while   ['milton-paradise.txt']
seed    ['milton-paradise.txt']
ought   ['shakespeare-caesar.txt']
lexicons    ['melville-moby_dick.txt']
wild    ['blake-poems.txt']
aid ['milton-paradise.txt']
Creatures   ['shakespeare-caesar.txt']
soil    ['whitman-leaves.txt']
is  ['shakespeare-caesar.txt', 'edgeworth-parents.txt']
Dick    ['melville-moby_dick.txt']
it  ['chesterton-ball.txt', 'bible-kjv.txt', 'melville-moby_dick.txt', 'milton-paradise.txt', 'edgeworth-parents.txt']
AIR ['chesterton-ball.txt']
shame   ['chesterton-thursday.txt']
grammars    ['melville-moby_dick.txt']
good    ['bible-kjv.txt']
Siloa   ['milton-paradise.txt']
in  ['milton-paradise.txt', 'austen-emma.txt', 'chesterton-ball.txt', 'chesterton-thursday.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt']
prose   ['milton-paradise.txt']
return  ['whitman-leaves.txt']
You ['shakespeare-caesar.txt']
vanish  ['blake-poems.txt']
if  ['milton-paradise.txt']
1816    ['austen-emma.txt']
)   ['shakespeare-caesar.txt', 'melville-moby_dick.txt']
parish  ['edgeworth-parents.txt']
reminded    ['melville-moby_dick.txt']
unite   ['austen-emma.txt']
chants  ['whitman-leaves.txt']
Of  ['milton-paradise.txt', 'shakespeare-caesar.txt']
Workman ['shakespeare-caesar.txt']
daughters   ['austen-emma.txt']
wheel   ['edgeworth-parents.txt']
used    ['edgeworth-parents.txt']
I   ['milton-paradise.txt', 'austen-emma.txt', 'chesterton-ball.txt', 'chesterton-thursday.txt', 'blake-poems.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt']
upon    ['bible-kjv.txt', 'chesterton-thursday.txt']
Soul    ['whitman-leaves.txt']
Cob ['shakespeare-caesar.txt']
IN  ['chesterton-ball.txt']
dust    ['melville-moby_dick.txt']
thy ['milton-paradise.txt', 'blake-poems.txt', 'shakespeare-caesar.txt']
Is  ['shakespeare-caesar.txt']
didst   ['milton-paradise.txt']
nonentity   ['chesterton-thursday.txt']
person  ['whitman-leaves.txt']
cheer   ['blake-poems.txt']
without ['bible-kjv.txt', 'shakespeare-caesar.txt']
resuming    ['whitman-leaves.txt']
In  ['milton-paradise.txt', 'bible-kjv.txt', 'blake-poems.txt']
very    ['edgeworth-parents.txt', 'chesterton-thursday.txt', 'austen-emma.txt']
the ['milton-paradise.txt', 'austen-emma.txt', 'chesterton-ball.txt', 'bible-kjv.txt', 'chesterton-thursday.txt', 'blake-poems.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt']
1789    ['blake-poems.txt']
-   ['whitman-leaves.txt', 'austen-emma.txt']
verses  ['whitman-leaves.txt']
Tragedie    ['shakespeare-caesar.txt']
Heavenly    ['milton-paradise.txt']
yet ['milton-paradise.txt', 'whitman-leaves.txt']
dost    ['shakespeare-caesar.txt']
vices   ['chesterton-thursday.txt']
death   ['milton-paradise.txt', 'chesterton-thursday.txt']
Lust    ['chesterton-thursday.txt']
mother  ['austen-emma.txt']
had ['chesterton-ball.txt', 'chesterton-thursday.txt', 'austen-emma.txt']
intends ['milton-paradise.txt']
adventurous ['milton-paradise.txt']
spinner ['edgeworth-parents.txt']
book    ['blake-poems.txt']
4   ['bible-kjv.txt']
skies   ['chesterton-ball.txt']
On  ['blake-poems.txt']
certaine    ['shakespeare-caesar.txt']
read    ['blake-poems.txt']
James   ['bible-kjv.txt']
stung   ['chesterton-thursday.txt']
early   ['austen-emma.txt']
know    ['shakespeare-caesar.txt']
world   ['melville-moby_dick.txt', 'chesterton-thursday.txt', 'austen-emma.txt']
Grass   ['whitman-leaves.txt']
Or  ['whitman-leaves.txt']
d   ['blake-poems.txt', 'whitman-leaves.txt']
loss    ['milton-paradise.txt']
like    ['chesterton-ball.txt']
lost    ['chesterton-thursday.txt']
OF  ['blake-poems.txt']
Profession  ['shakespeare-caesar.txt']
laughing    ['blake-poems.txt']
governess   ['austen-emma.txt']
Milton  ['milton-paradise.txt']
Vpon    ['shakespeare-caesar.txt']
valleys ['blake-poems.txt']
old ['melville-moby_dick.txt', 'chesterton-thursday.txt']
obliged ['edgeworth-parents.txt']
proudly ['chesterton-thursday.txt']
somehow ['melville-moby_dick.txt']
Woodhouse   ['austen-emma.txt']
Primus  ['shakespeare-caesar.txt']
vse ['shakespeare-caesar.txt']
sight   ['blake-poems.txt']
home    ['shakespeare-caesar.txt', 'austen-emma.txt']
pale    ['melville-moby_dick.txt']
Mary    ['edgeworth-parents.txt']
Life    ['chesterton-thursday.txt']
for ['chesterton-ball.txt', 'whitman-leaves.txt', 'austen-emma.txt']
1851    ['melville-moby_dick.txt']
Men ['chesterton-thursday.txt']
1855    ['whitman-leaves.txt']
everything  ['chesterton-ball.txt']
pen ['blake-poems.txt']
Let ['bible-kjv.txt']
?   ['shakespeare-caesar.txt']
He  ['melville-moby_dick.txt']
regain  ['milton-paradise.txt']
be  ['chesterton-ball.txt', 'bible-kjv.txt']
School  ['melville-moby_dick.txt']
who ['milton-paradise.txt', 'austen-emma.txt']
Eden    ['milton-paradise.txt']
;" ['blake-poems.txt']
Near    ['edgeworth-parents.txt']
by  ['milton-paradise.txt', 'austen-emma.txt', 'chesterton-ball.txt', 'chesterton-thursday.txt', 'blake-poems.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt']
First   ['bible-kjv.txt']
wept    ['blake-poems.txt']
on  ['milton-paradise.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'chesterton-thursday.txt']
sister  ['austen-emma.txt']
of  ['milton-paradise.txt', 'austen-emma.txt', 'chesterton-ball.txt', 'bible-kjv.txt', 'chesterton-thursday.txt', 'blake-poems.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt']
Trade   ['shakespeare-caesar.txt']
blessings   ['austen-emma.txt']
or  ['milton-paradise.txt', 'austen-emma.txt']
own ['chesterton-thursday.txt']
into    ['milton-paradise.txt']
down    ['blake-poems.txt']
weather ['chesterton-thursday.txt']
But ['shakespeare-caesar.txt']
remembrance ['austen-emma.txt']
Edgeworth   ['edgeworth-parents.txt']
your    ['shakespeare-caesar.txt']
Ball    ['chesterton-ball.txt']
her ['edgeworth-parents.txt', 'austen-emma.txt']
Assistant   ['edgeworth-parents.txt']
." ['blake-poems.txt']
there   ['bible-kjv.txt', 'edgeworth-parents.txt']
long    ['whitman-leaves.txt', 'edgeworth-parents.txt', 'austen-emma.txt']
Every   ['blake-poems.txt']
caresses    ['austen-emma.txt']
stars   ['chesterton-ball.txt']
house   ['austen-emma.txt']
:   ['milton-paradise.txt', 'bible-kjv.txt', 'shakespeare-caesar.txt', 'chesterton-thursday.txt', 'blake-poems.txt']
was ['austen-emma.txt', 'chesterton-ball.txt', 'bible-kjv.txt', 'chesterton-thursday.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt']
Herman  ['melville-moby_dick.txt']
happy   ['blake-poems.txt', 'austen-emma.txt']
himself ['chesterton-ball.txt']
form    ['bible-kjv.txt']
Science ['chesterton-thursday.txt']
but ['shakespeare-caesar.txt', 'chesterton-thursday.txt', 'edgeworth-parents.txt']
boys    ['chesterton-thursday.txt']
Moby    ['melville-moby_dick.txt']
reed    ['blake-poems.txt']
Fla ['shakespeare-caesar.txt']
with    ['milton-paradise.txt', 'austen-emma.txt', 'blake-poems.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt']
mockingly   ['melville-moby_dick.txt']
he  ['bible-kjv.txt', 'blake-poems.txt']
unattempted ['milton-paradise.txt']
made    ['blake-poems.txt']
up  ['edgeworth-parents.txt']
us  ['milton-paradise.txt', 'whitman-leaves.txt', 'chesterton-thursday.txt']
Walt    ['whitman-leaves.txt']
BOOK    ['blake-poems.txt', 'whitman-leaves.txt']
faded   ['chesterton-thursday.txt']
professor   ['chesterton-ball.txt']
affection   ['austen-emma.txt']
walke   ['shakespeare-caesar.txt']
taste   ['milton-paradise.txt']
moved   ['bible-kjv.txt']
Cross   ['chesterton-ball.txt']
Thursday    ['chesterton-thursday.txt']
an  ['austen-emma.txt']
Masse   ['whitman-leaves.txt']
To  ['chesterton-thursday.txt']
as  ['edgeworth-parents.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'chesterton-thursday.txt', 'austen-emma.txt']
clever  ['austen-emma.txt']
at  ['edgeworth-parents.txt']
Above   ['milton-paradise.txt']
tumultuous  ['whitman-leaves.txt']
some    ['whitman-leaves.txt', 'austen-emma.txt']
ship    ['chesterton-ball.txt']
again   ['blake-poems.txt']
no  ['chesterton-ball.txt', 'milton-paradise.txt']
when    ['chesterton-thursday.txt']
other   ['whitman-leaves.txt']
5   ['bible-kjv.txt']
sick    ['chesterton-thursday.txt']
Poems   ['blake-poems.txt']
you ['shakespeare-caesar.txt', 'chesterton-thursday.txt']
Man ['milton-paradise.txt', 'chesterton-thursday.txt']
Like    ['chesterton-thursday.txt']
Lost    ['milton-paradise.txt']
separate    ['whitman-leaves.txt']
Julius  ['shakespeare-caesar.txt']
fell    ['edgeworth-parents.txt']
Testament   ['bible-kjv.txt']
Melville    ['melville-moby_dick.txt']
ago ['austen-emma.txt']
together    ['chesterton-thursday.txt']
Brought ['milton-paradise.txt']
As  ['edgeworth-parents.txt']
write   ['blake-poems.txt', 'whitman-leaves.txt']
oracle  ['milton-paradise.txt']
rural   ['blake-poems.txt']
once    ['edgeworth-parents.txt']
songs   ['blake-poems.txt']