hbase数据库数据存入_七个星期中的七个数据库

hbase数据库数据存入

这篇文章是《 七个星期的七个数据库》一书中Hbase第二天的回顾。
大部分命令和脚本可以在GitHub上找到： https : //github.com/eyalgo/seven-dbs-in-seven-weeks/tree/master/hbase/day_2

流脚本

第2天的第一件事是下载大量数据（大数据）并将其流式传输到Hbase。有一个JRuby脚本，我必须对其进行更改才能使其工作： https : //github.com/eyalgo/seven-dbs-in-seven-weeks/blob/master/hbase/day_2/import_from_wikipedia.rb

正如书中所建议的那样，在对其进行了更改之后，我不得不对列族进行一些压缩。之后，我可以运行脚本：

curl http://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2 | bzcat | /opt/hbase/hbase-0.94.18/bin/hbase shell /home/eyalgo/seven-dbs-in-seven-weeks/hbase/day_2/import_from_wikipedia.rbcurl http://dumps.wikimedia.org/enwiktionary/latest/enwiktionary-latest-pages-articles.xml.bz2 | bzcat | /opt/hbase/hbase-0.94.18/bin/hbase shell import_from_wikipedia.rb

这是脚本运行时的输出

1 10.0G    1  128M    0     0   456k      0  6:23:37  0:04:48  6:18:49  817k19000 records inserted (Serotonin)1 10.0G    1  131M    0     0   461k      0  6:19:03  0:04:51  6:14:12  921k19500 records inserted (Serotonin specific reuptake inhibitors)1 10.0G    1  135M    0     0   469k      0  6:12:45  0:04:54  6:07:51 1109k20000 records inserted (Tennis court)1 10.0G    1  138M    0     0   477k      0  6:06:12  0:04:57  6:01:15 1269k20500 records inserted (Tape drive)

本章的下一部分将讨论区域和其他一些管道内容。

构建链接表

在这一部分中，源是大的Wiki表，输出是“链接”表。每个链接都有“发件人：”和“收件人：”。这是更改后的工作脚本的链接： https : //github.com/eyalgo/seven-dbs-in-seven-weeks/blob/master/hbase/day_2/generate_wiki_links.rb

本章的其余部分显示了如何查看数据，对数据进行计数等等。

家庭作业

作业的主要部分是创建一个新表：“ foods”，该表从XML中获取数据，可以从美国的健康与营养站点下载该数据。该数据显示每种食物的营养成分。

我决定创建一个非常简单的表。列族没有任何特殊选项。我创建了一个专栏文章：事实。 XML文件中的每一行数据都是事实的一部分。我还决定该行的键为Display_Name 。毕竟，按键而不是按ID查找要容易得多。

create 'foods' , 'facts'

为了了解如何创建脚本，我查看了两个资源：

为Wiki表导入数据的脚本
XML中的一种元素（食物）

这是一个元素：

<Food_Display_Row><Food_Code>12350000</Food_Code><Display_Name>Sour cream dip</Display_Name><Portion_Default>1.00000</Portion_Default><Portion_Amount>.25000</Portion_Amount><Portion_Display_Name>cup </Portion_Display_Name><Factor>.25000</Factor><Increment>.25000</Increment><Multiplier>1.00000</Multiplier><Grains>.04799</Grains><Whole_Grains>.00000</Whole_Grains><Vegetables>.04070</Vegetables><Orange_Vegetables>.00000</Orange_Vegetables><Drkgreen_Vegetables>.00000</Drkgreen_Vegetables><Starchy_vegetables>.00000</Starchy_vegetables><Other_Vegetables>.04070</Other_Vegetables><Fruits>.00000</Fruits><Milk>.00000</Milk><Meats>.00000</Meats><Soy>.00000</Soy><Drybeans_Peas>.00000</Drybeans_Peas><Oils>.00000</Oils><Solid_Fats>105.64850</Solid_Fats><Added_Sugars>1.57001</Added_Sugars><Alcohol>.00000</Alcohol><Calories>133.65000</Calories><Saturated_Fats>7.36898</Saturated_Fats>
</Food_Display_Row>

我通过检查Wiki脚本和一个元素创建了脚本。打开文档是在看到一个打开的XML元素标签： Food_Display_Row时 。当将Food_Display_Row视为关闭标签时，脚本将创建文档。

include Java
import 'org.apache.hadoop.hbase.client.HTable'
import 'org.apache.hadoop.hbase.client.Put'
import 'org.apache.hadoop.hbase.HBaseConfiguration'
import 'javax.xml.stream.XMLStreamConstants'def jbytes( *args )args.map { |arg| arg.to_s.to_java_bytes }
endfactory = javax.xml.stream.XMLInputFactory.newInstance
reader = factory.createXMLStreamReader(java.lang.System.in)document = nil
buffer = nil
count = 0puts( @hbase )
conf = HBaseConfiguration.new
table = HTable.new( conf, "foods" )
table.setAutoFlush( false )while reader.has_nexttype = reader.nextif type == XMLStreamConstants::START_ELEMENT # (3)case reader.local_namewhen 'Food_Display_Row' then document = {}when /Display_Name|Portion_Default|Portion_Amount|Portion_Display_Name|Factor/ then buffer = []when /Increment|Multiplier|Grains|Whole_Grains|Vegetables|Orange_Vegetables/ then buffer = []when /Drkgreen_Vegetables|Starchy_vegetables|Other_Vegetables|Fruits|Milk|Meats/ then buffer = []when /Drybeans_Peas|Soy|Oils|Solid_Fats|Added_Sugars|Alcohol|Calories|Saturated_Fats/ then buffer = []endelsif type == XMLStreamConstants::CHARACTERSbuffer << reader.text unless buffer.nil?elsif type == XMLStreamConstants::END_ELEMENTcase reader.local_namewhen /Display_Name|Portion_Default|Portion_Amount|Portion_Display_Name|Factor/document[reader.local_name] = buffer.joinwhen /Increment|Multiplier|Grains|Whole_Grains|Vegetables|Orange_Vegetables/document[reader.local_name] = buffer.joinwhen /Drkgreen_Vegetables|Starchy_vegetables|Other_Vegetables|Fruits|Milk|Meats/document[reader.local_name] = buffer.joinwhen /Drybeans_Peas|Soy|Oils|Solid_Fats|Added_Sugars|Alcohol|Calories|Saturated_Fats/document[reader.local_name] = buffer.joinwhen 'Food_Display_Row'key = document['Display_Name'].to_java_bytesp = Put.new( key )p.add( *jbytes( "facts", "Display_Name", document['Display_Name'] ) )p.add( *jbytes( "facts", "Portion_Default", document['Portion_Default'] ) )p.add( *jbytes( "facts", "Portion_Amount", document['Portion_Amount'] ) )p.add( *jbytes( "facts", "Portion_Display_Name", document['Portion_Display_Name'] ) )p.add( *jbytes( "facts", "Factor", document['Factor'] ) )p.add( *jbytes( "facts", "Increment", document['Increment'] ) )p.add( *jbytes( "facts", "Multiplier", document['Multiplier'] ) )p.add( *jbytes( "facts", "Grains", document['Grains'] ) )p.add( *jbytes( "facts", "Whole_Grains", document['Whole_Grains'] ) )p.add( *jbytes( "facts", "Vegetables", document['Vegetables'] ) )p.add( *jbytes( "facts", "Orange_Vegetables", document['Orange_Vegetables'] ) )p.add( *jbytes( "facts", "Drkgreen_Vegetables", document['Drkgreen_Vegetables'] ) )p.add( *jbytes( "facts", "Starchy_vegetables", document['Starchy_vegetables'] ) )p.add( *jbytes( "facts", "Other_Vegetables", document['Other_Vegetables'] ) )p.add( *jbytes( "facts", "Fruits", document['Fruits'] ) )p.add( *jbytes( "facts", "Milk", document['Milk'] ) )p.add( *jbytes( "facts", "Meats", document['Meats'] ) )p.add( *jbytes( "facts", "Drybeans_Peas", document['Drybeans_Peas'] ) )p.add( *jbytes( "facts", "Soy", document['Soy'] ) )p.add( *jbytes( "facts", "Oils", document['Oils'] ) )p.add( *jbytes( "facts", "Solid_Fats", document['Solid_Fats'] ) )p.add( *jbytes( "facts", "Added_Sugars", document['Added_Sugars'] ) )p.add( *jbytes( "facts", "Alcohol", document['Alcohol'] ) )p.add( *jbytes( "facts", "Calories", document['Calories'] ) )p.add( *jbytes( "facts", "Saturated_Fats", document['Saturated_Fats'] ) )table.put( p )count += 1table.flushCommits() if count % 10 == 0if count % 500 == 0puts "#{count} records inserted (#{document['Display_Name']})"endendend
endtable.flushCommits()
exit

以下是获取XML文件并将其流式传输到Hbase的Shell命令。第一个命令针对具有单个元素的文件运行。验证正确性后，我将其针对完整文件。

curl file:///home/eyalgo/seven-dbs-in-seven-weeks/hbase/day_2/food-display-example.xml | cat | /opt/hbase/hbase-0.94.18/bin/hbase shell /home/eyalgo/seven-dbs-in-seven-weeks/hbase/day_2/import_food_display.rbcurl file:///home/eyalgo/seven-dbs-in-seven-weeks/hbase/day_2/MyFoodapediaData/Food_Display_Table.xml | cat | /opt/hbase/hbase-0.94.18/bin/hbase shell /home/eyalgo/seven-dbs-in-seven-weeks/hbase/day_2/import_food_display.rb

我们去吃点饭吧...

get 'foods' , 'fruit smoothie made with milk'

结果：

COLUMN CELL
facts:Added_Sugars timestamp=1399932481440, value=82.54236
facts:Alcohol timestamp=1399932481440, value=.00000
facts:Calories timestamp=1399932481440, value=197.96000
facts:Display_Name timestamp=1399932481440, value=fruit smoothie made with milk
facts:Drkgreen_Vegetables timestamp=1399932481440, value=.00000
facts:Drybeans_Peas timestamp=1399932481440, value=.00000
facts:Factor timestamp=1399932481440, value=1.00000
facts:Fruits timestamp=1399932481440, value=.56358
facts:Grains timestamp=1399932481440, value=.00000
facts:Increment timestamp=1399932481440, value=.25000
facts:Meats timestamp=1399932481440, value=.00000
facts:Milk timestamp=1399932481440, value=.22624
facts:Multiplier timestamp=1399932481440, value=.25000
facts:Oils timestamp=1399932481440, value=.00808
facts:Orange_Vegetables timestamp=1399932481440, value=.00000
facts:Other_Vegetables timestamp=1399932481440, value=.00000
facts:Portion_Amount timestamp=1399932481440, value=1.00000
facts:Portion_Default timestamp=1399932481440, value=2.00000
facts:Portion_Display_Name timestamp=1399932481440, value=cup
facts:Saturated_Fats timestamp=1399932481440, value=1.91092
facts:Solid_Fats timestamp=1399932481440, value=24.14304
facts:Soy timestamp=1399932481440, value=.00000
facts:Starchy_vegetables timestamp=1399932481440, value=.00000
facts:Vegetables timestamp=1399932481440, value=.00000
facts:Whole_Grains timestamp=1399932481440, value=.00000

翻译自: https://www.javacodegeeks.com/2014/05/seven-databases-in-seven-weeks-hbase-day-2.html

hbase数据库数据存入