flume学习（五）：使用hive来分析flume收集的日志数据

前面已经讲过如何将log4j的日志输出到指定的hdfs目录，我们前面的指定目录为/flume/events。

如果想用Hive来分析采集来的日志，我们可以将/flume/events下面的日志数据都load到hive中的表当中去。

如果了解hive的load data原理的话，还有一种更简便的方式，可以省去load data这一步，就是直接将sink1.hdfs.path指定为hive表的目录。

下面我将详细描述具体的操作步骤。

我们还是从需求驱动来讲解，前面我们采集的数据，都是接口的访问日志数据，数据格式是JSON格式如下：

{"requestTime":1405651379758,"requestParams":{"timestamp":1405651377211,"phone":"02038824941","cardName":"测试商家名称","provinceCode":"440000","cityCode":"440106"},"requestUrl":"/reporter-api/reporter/reporter12/init.do"}

现在有一个需求，我们要统计接口的总调用量。

我第一想法就是，hive中建一张表：test 然后将hdfs.path指定为tier1.sinks.sink1.hdfs.path=hdfs://master68:8020/user/hive/warehouse/besttone.db/test

然后select count(*) from test; 完事。

这个方案简单，粗暴，先这么干着。于是会遇到一个问题，我的日志数据时JSON格式的，需要hive来序列化和反序列化JSON格式的数据到test表的具体字段当中去。

这有点糟糕，因为hive本身没有提供JSON的SERDE,但是有提供函数来解析JSON字符串，

第一个是（UDF）:

get_json_object(string json_string,string path) 从给定路径上的JSON字符串中抽取出JSON对象，并返回这个对象的JSON字符串形式，如果输入的JSON字符串是非法的，则返回NULL。

第二个是表生成函数（UDTF）：json_tuple(string jsonstr,p1,p2,...,pn) 本函数可以接受多个标签名称，对输入的JSON字符串进行处理，这个和get_json_object这个UDF类似，不过更高效，其通过一次调用就可以获得多个键值，例：select b.* from test_json a lateral view json_tuple(a.id,'id','name') b as f1,f2;通过lateral view行转列。

最理想的方式就是能有一种JSON SERDE，只要我们LOAD完数据，就直接可以select * from test,而不是select get_json_object这种方式来获取，N个字段就要解析N次，效率太低了。

好在cloudrea wiki里提供了一个json serde类（这个类没有在发行的hive的jar包中），于是我把它搬来了，如下：

[java] view plaincopy

package com.besttone.hive.serde;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hive.serde.serdeConstants;
import org.apache.hadoop.hive.serde2.SerDe;
import org.apache.hadoop.hive.serde2.SerDeException;
import org.apache.hadoop.hive.serde2.SerDeStats;
import org.apache.hadoop.hive.serde2.objectinspector.ListObjectInspector;
import org.apache.hadoop.hive.serde2.objectinspector.MapObjectInspector;
import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
import org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector;
import org.apache.hadoop.hive.serde2.objectinspector.StructField;
import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
import org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo;
import org.apache.hadoop.hive.serde2.typeinfo.MapTypeInfo;
import org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo;
import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory;
import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.Writable;
import org.codehaus.jackson.map.ObjectMapper;
/**
* This SerDe can be used for processing JSON data in Hive. It supports
* arbitrary JSON data, and can handle all Hive types except for UNION. However,
* the JSON data is expected to be a series of discrete records, rather than a
* JSON array of objects.
*
* The Hive table is expected to contain columns with names corresponding to
* fields in the JSON data, but it is not necessary for every JSON field to have
* a corresponding Hive column. Those JSON fields will be ignored during
* queries.
*
* Example:
*
* { "a": 1, "b": [ "str1", "str2" ], "c": { "field1": "val1" } }
*
* Could correspond to a table:
*
* CREATE TABLE foo (a INT, b ARRAY<STRING>, c STRUCT<field1:STRING>);
*
* JSON objects can also interpreted as a Hive MAP type, so long as the keys and
* values in the JSON object are all of the appropriate types. For example, in
* the JSON above, another valid table declaraction would be:
*
* CREATE TABLE foo (a INT, b ARRAY<STRING>, c MAP<STRING,STRING>);
*
* Only STRING keys are supported for Hive MAPs.
*/
public class JSONSerDe implements SerDe {
private StructTypeInfo rowTypeInfo;
private ObjectInspector rowOI;
private List<String> colNames;
private List<Object> row = new ArrayList<Object>();
//遇到非JSON格式输入的时候的处理。
private boolean ignoreInvalidInput;
/**
* An initialization function used to gather information about the table.
* Typically, a SerDe implementation will be interested in the list of
* column names and their types. That information will be used to help
* perform actual serialization and deserialization of data.
*/
@Override
public void initialize(Configuration conf, Properties tbl)
throws SerDeException {
// 遇到无法转换成JSON对象的字符串时，是否忽略，默认不忽略，抛出异常，设置为true将跳过异常。
ignoreInvalidInput = Boolean.valueOf(tbl.getProperty(
"input.invalid.ignore", "false"));
// Get a list of the table's column names.
String colNamesStr = tbl.getProperty(serdeConstants.LIST_COLUMNS);
colNames = Arrays.asList(colNamesStr.split(","));
// Get a list of TypeInfos for the columns. This list lines up with
// the list of column names.
String colTypesStr = tbl.getProperty(serdeConstants.LIST_COLUMN_TYPES);
List<TypeInfo> colTypes = TypeInfoUtils
.getTypeInfosFromTypeString(colTypesStr);
rowTypeInfo = (StructTypeInfo) TypeInfoFactory.getStructTypeInfo(
colNames, colTypes);
rowOI = TypeInfoUtils
.getStandardJavaObjectInspectorFromTypeInfo(rowTypeInfo);
}
/**
* This method does the work of deserializing a record into Java objects
* that Hive can work with via the ObjectInspector interface. For this
* SerDe, the blob that is passed in is a JSON string, and the Jackson JSON
* parser is being used to translate the string into Java objects.
*
* The JSON deserialization works by taking the column names in the Hive
* table, and looking up those fields in the parsed JSON object. If the
* value of the field is not a primitive, the object is parsed further.
*/
@Override
public Object deserialize(Writable blob) throws SerDeException {
Map<?, ?> root = null;
row.clear();
try {
ObjectMapper mapper = new ObjectMapper();
// This is really a Map<String, Object>. For more information about
// how
// Jackson parses JSON in this example, see
// http://wiki.fasterxml.com/JacksonDataBinding
root = mapper.readValue(blob.toString(), Map.class);
} catch (Exception e) {
// 如果为true,不抛出异常，忽略该行数据
if (!ignoreInvalidInput)
throw new SerDeException(e);
else {
return null;
}
}
// Lowercase the keys as expected by hive
Map<String, Object> lowerRoot = new HashMap();
for (Map.Entry entry : root.entrySet()) {
lowerRoot.put(((String) entry.getKey()).toLowerCase(),
entry.getValue());
}
root = lowerRoot;
Object value = null;
for (String fieldName : rowTypeInfo.getAllStructFieldNames()) {
try {
TypeInfo fieldTypeInfo = rowTypeInfo
.getStructFieldTypeInfo(fieldName);
value = parseField(root.get(fieldName), fieldTypeInfo);
} catch (Exception e) {
value = null;
}
row.add(value);
}
return row;
}
/**
* Parses a JSON object according to the Hive column's type.
*
* @param field
* - The JSON object to parse
* @param fieldTypeInfo
* - Metadata about the Hive column
* @return - The parsed value of the field
*/
private Object parseField(Object field, TypeInfo fieldTypeInfo) {
switch (fieldTypeInfo.getCategory()) {
case PRIMITIVE:
// Jackson will return the right thing in this case, so just return
// the object
if (field instanceof String) {
field = field.toString().replaceAll("\n", "\\\\n");
}
return field;
case LIST:
return parseList(field, (ListTypeInfo) fieldTypeInfo);
case MAP:
return parseMap(field, (MapTypeInfo) fieldTypeInfo);
case STRUCT:
return parseStruct(field, (StructTypeInfo) fieldTypeInfo);
case UNION:
// Unsupported by JSON
default:
return null;
}
}
/**
* Parses a JSON object and its fields. The Hive metadata is used to
* determine how to parse the object fields.
*
* @param field
* - The JSON object to parse
* @param fieldTypeInfo
* - Metadata about the Hive column
* @return - A map representing the object and its fields
*/
private Object parseStruct(Object field, StructTypeInfo fieldTypeInfo) {
Map<Object, Object> map = (Map<Object, Object>) field;
ArrayList<TypeInfo> structTypes = fieldTypeInfo
.getAllStructFieldTypeInfos();
ArrayList<String> structNames = fieldTypeInfo.getAllStructFieldNames();
List<Object> structRow = new ArrayList<Object>(structTypes.size());
for (int i = 0; i < structNames.size(); i++) {
structRow.add(parseField(map.get(structNames.get(i)),
structTypes.get(i)));
}
return structRow;
}
/**
* Parse a JSON list and its elements. This uses the Hive metadata for the
* list elements to determine how to parse the elements.
*
* @param field
* - The JSON list to parse
* @param fieldTypeInfo
* - Metadata about the Hive column
* @return - A list of the parsed elements
*/
private Object parseList(Object field, ListTypeInfo fieldTypeInfo) {
ArrayList<Object> list = (ArrayList<Object>) field;
TypeInfo elemTypeInfo = fieldTypeInfo.getListElementTypeInfo();
for (int i = 0; i < list.size(); i++) {
list.set(i, parseField(list.get(i), elemTypeInfo));
}
return list.toArray();
}
/**
* Parse a JSON object as a map. This uses the Hive metadata for the map
* values to determine how to parse the values. The map is assumed to have a
* string for a key.
*
* @param field
* - The JSON list to parse
* @param fieldTypeInfo
* - Metadata about the Hive column
* @return
*/
private Object parseMap(Object field, MapTypeInfo fieldTypeInfo) {
Map<Object, Object> map = (Map<Object, Object>) field;
TypeInfo valueTypeInfo = fieldTypeInfo.getMapValueTypeInfo();
for (Map.Entry<Object, Object> entry : map.entrySet()) {
map.put(entry.getKey(), parseField(entry.getValue(), valueTypeInfo));
}
return map;
}
/**
* Return an ObjectInspector for the row of data
*/
@Override
public ObjectInspector getObjectInspector() throws SerDeException {
return rowOI;
}
/**
* Unimplemented
*/
@Override
public SerDeStats getSerDeStats() {
return null;
}
/**
* JSON is just a textual representation, so our serialized class is just
* Text.
*/
@Override
public Class<? extends Writable> getSerializedClass() {
return Text.class;
}
/**
* This method takes an object representing a row of data from Hive, and
* uses the ObjectInspector to get the data for each column and serialize
* it. This implementation deparses the row into an object that Jackson can
* easily serialize into a JSON blob.
*/
@Override
public Writable serialize(Object obj, ObjectInspector oi)
throws SerDeException {
Object deparsedObj = deparseRow(obj, oi);
ObjectMapper mapper = new ObjectMapper();
try {
// Let Jackson do the work of serializing the object
return new Text(mapper.writeValueAsString(deparsedObj));
} catch (Exception e) {
throw new SerDeException(e);
}
}
/**
* Deparse a Hive object into a Jackson-serializable object. This uses the
* ObjectInspector to extract the column data.
*
* @param obj
* - Hive object to deparse
* @param oi
* - ObjectInspector for the object
* @return - A deparsed object
*/
private Object deparseObject(Object obj, ObjectInspector oi) {
switch (oi.getCategory()) {
case LIST:
return deparseList(obj, (ListObjectInspector) oi);
case MAP:
return deparseMap(obj, (MapObjectInspector) oi);
case PRIMITIVE:
return deparsePrimitive(obj, (PrimitiveObjectInspector) oi);
case STRUCT:
return deparseStruct(obj, (StructObjectInspector) oi, false);
case UNION:
// Unsupported by JSON
default:
return null;
}
}
/**
* Deparses a row of data. We have to treat this one differently from other
* structs, because the field names for the root object do not match the
* column names for the Hive table.
*
* @param obj
* - Object representing the top-level row
* @param structOI
* - ObjectInspector for the row
* @return - A deparsed row of data
*/
private Object deparseRow(Object obj, ObjectInspector structOI) {
return deparseStruct(obj, (StructObjectInspector) structOI, true);
}
/**
* Deparses struct data into a serializable JSON object.
*
* @param obj
* - Hive struct data
* @param structOI
* - ObjectInspector for the struct
* @param isRow
* - Whether or not this struct represents a top-level row
* @return - A deparsed struct
*/
private Object deparseStruct(Object obj, StructObjectInspector structOI,
boolean isRow) {
Map<Object, Object> struct = new HashMap<Object, Object>();
List<? extends StructField> fields = structOI.getAllStructFieldRefs();
for (int i = 0; i < fields.size(); i++) {
StructField field = fields.get(i);
// The top-level row object is treated slightly differently from
// other
// structs, because the field names for the row do not correctly
// reflect
// the Hive column names. For lower-level structs, we can get the
// field
// name from the associated StructField object.
String fieldName = isRow ? colNames.get(i) : field.getFieldName();
ObjectInspector fieldOI = field.getFieldObjectInspector();
Object fieldObj = structOI.getStructFieldData(obj, field);
struct.put(fieldName, deparseObject(fieldObj, fieldOI));
}
return struct;
}
/**
* Deparses a primitive type.
*
* @param obj
* - Hive object to deparse
* @param oi
* - ObjectInspector for the object
* @return - A deparsed object
*/
private Object deparsePrimitive(Object obj, PrimitiveObjectInspector primOI) {
return primOI.getPrimitiveJavaObject(obj);
}
private Object deparseMap(Object obj, MapObjectInspector mapOI) {
Map<Object, Object> map = new HashMap<Object, Object>();
ObjectInspector mapValOI = mapOI.getMapValueObjectInspector();
Map<?, ?> fields = mapOI.getMap(obj);
for (Map.Entry<?, ?> field : fields.entrySet()) {
Object fieldName = field.getKey();
Object fieldObj = field.getValue();
map.put(fieldName, deparseObject(fieldObj, mapValOI));
}
return map;
}
/**
* Deparses a list and its elements.
*
* @param obj
* - Hive object to deparse
* @param oi
* - ObjectInspector for the object
* @return - A deparsed object
*/
private Object deparseList(Object obj, ListObjectInspector listOI) {
List<Object> list = new ArrayList<Object>();
List<?> field = listOI.getList(obj);
ObjectInspector elemOI = listOI.getListElementObjectInspector();
for (Object elem : field) {
list.add(deparseObject(elem, elemOI));
}
return list;
}
}

我稍微修改了一点东西，多加了一个参数input.invalid.ignore,对应的变量为：

//遇到非JSON格式输入的时候的处理。
private boolean ignoreInvalidInput;

在deserialize方法中原来是如果传入的是非JSON格式字符串的话，直接抛出了SerDeException，我加了一个参数来控制它是否抛出异常，在initialize方法中初始化这个变量（默认为false）：

// 遇到无法转换成JSON对象的字符串时，是否忽略，默认不忽略，抛出异常，设置为true将跳过异常。
ignoreInvalidInput = Boolean.valueOf(tbl.getProperty(
"input.invalid.ignore", "false"));

好的，现在将这个类打成JAR包： JSONSerDe.jar，放在hive_home的auxlib目录下（我的是/etc/hive/auxlib），然后修改hive-env.sh，添加HIVE_AUX_JARS_PATH=/etc/hive/auxlib/JSONSerDe.jar，这样每次运行hive客户端的时候都会将这个jar包添加到classpath，否则在设置SERDE的时候会报找不到类。

现在我们在HIVE中创建一张表用来存放日志数据：

[plain] view plaincopy

create table test(
requestTime BIGINT,
requestParams STRUCT<timestamp:BIGINT,phone:STRING,cardName:STRING,provinceCode:STRING,cityCode:STRING>,
requestUrl STRING)
row format serde "com.besttone.hive.serde.JSONSerDe"
WITH SERDEPROPERTIES(
"input.invalid.ignore"="true",
"requestTime"="$.requestTime",
"requestParams.timestamp"="$.requestParams.timestamp",
"requestParams.phone"="$.requestParams.phone",
"requestParams.cardName"="$.requestParams.cardName",
"requestParams.provinceCode"="$.requestParams.provinceCode",
"requestParams.cityCode"="$.requestParams.cityCode",
"requestUrl"="$.requestUrl");

这个表结构就是按照日志格式设计的，还记得前面说过的日志数据如下：

我使用了一个STRUCT类型来保存requestParams的值，row format我们用的是自定义的json serde:com.besttone.hive.serde.JSONSerDe，SERDEPROPERTIES中，除了设置JSON对象的映射关系外，我还设置了一个自定义的参数："input.invalid.ignore"="true"，忽略掉所有非JSON格式的输入行。这里不是真正意义的忽略，只是非法行的每个输出字段都为NULL了，要在结果集上忽略，必须这样写：select * from test where requestUrl is not null;

OK表建好了，现在就差数据了，我们启动flumedemo的WriteLog,往hive表test目录下面输出一些日志数据，然后在进入hive客户端，select * from test;所以字段都正确的解析，大功告成。

flume.conf如下：

[plain] view plaincopy

tier1.sources=source1
tier1.channels=channel1
tier1.sinks=sink1
tier1.sources.source1.type=avro
tier1.sources.source1.bind=0.0.0.0
tier1.sources.source1.port=44444
tier1.sources.source1.channels=channel1
tier1.sources.source1.interceptors=i1 i2
tier1.sources.source1.interceptors.i1.type=regex_filter
tier1.sources.source1.interceptors.i1.regex=\\{.*\\}
tier1.sources.source1.interceptors.i2.type=timestamp
tier1.channels.channel1.type=memory
tier1.channels.channel1.capacity=10000
tier1.channels.channel1.transactionCapacity=1000
tier1.channels.channel1.keep-alive=30
tier1.sinks.sink1.type=hdfs
tier1.sinks.sink1.channel=channel1
tier1.sinks.sink1.hdfs.path=hdfs://master68:8020/user/hive/warehouse/besttone.db/test
tier1.sinks.sink1.hdfs.fileType=DataStream
tier1.sinks.sink1.hdfs.writeFormat=Text
tier1.sinks.sink1.hdfs.rollInterval=0
tier1.sinks.sink1.hdfs.rollSize=10240
tier1.sinks.sink1.hdfs.rollCount=0
tier1.sinks.sink1.hdfs.idleTimeout=60

besttone.db是我在hive中创建的数据库，了解hive的应该理解没多大问题。

OK，到这篇文章为止，整个从LOG4J生产日志，到flume收集日志，再到用hive离线分析日志，一整套流水线都讲解完了。

flume学习（五）：使用hive来分析flume收集的日志数据相关推荐

蓝牙学习五（广播包分析wireshark）
1.简介软件工具:wireshark 硬件抓包工具:nrf52840 dongle 2.数据包分析使用wireshark抓到的数据包分为两个部分,一部分是软件自己添加的内容,另一部分才是广播出来的 ...
大数据相关书籍（包含Java, Scala, R, Linux, Spark, Hadoop, Hive, Hbase, Sqoop, Flume, Strom）
下面书单,有一部分英文版原版,当然价格也相对高一点,英文版部分需要在在亚马逊搜索 ,中文版大部分在京东有售! <Hadoop核心技术> 翟周伟著 <Storm分布式实时计算模式 & ...
学习笔记Hive（五） —— Hive应用（3）—— Hive查询
五.Hive查询 5.1.创建职工信息表任务实现: 1.创建表dept,emp和salgrade 2.导入数据将数据emp.txt导入到表emp #emp.txt 7369,SMITH,CLERK ...
flume学习-含安装
1.Flume是什么:Flume是Cloudera提供的一个高可用的,高可靠的,分布式的海量日志采集.聚合和传输的系统.Flume基于流式架构,灵活简单. Flume组成架构下面我们来详细介绍一下F ...
BT源代码学习心得(十五)：客户端源代码分析(下载过程中的块选取策略)
BT源代码学习心得(十五):客户端源代码分析(下载过程中的块选取策略) 发信人: wolfenstein (NeverSayNever), 个人文集标题: BT源代码学习心得(十五):客户端源代 ...
flume学习（四）：Flume Interceptors的使用
对于flume拦截器,我的理解是:在app(应用程序日志)和 source 之间的,对app日志进行拦截处理的.也即在日志进入到source之前,对日志进行一些包装.清新过滤等等动作. 官方上提供的已 ...
flume学习之一是什么，主要应用在什么场景
1.Flume是什么? Flume是Cloudera开发的一个高可用的,高可靠的,分布式的海量日志采集.聚合和传输的系统,Flume支持在日志系统中定制各类数据发送方,用于收集数据.数据源可定制.可扩 ...
hive 修改分桶数分桶表_hive 学习之路（五)、Hive的分区表与分桶表
Hive将表划分为分区(partition)表和分桶(bucket)表. 分区可以让数据的部分查询变得更快,也就是说,在加载数据的时候可以指定加载某一部分数据,并不是全量的数据. 分桶表通常是在原始数 ...
大数据学习之路-Hive
Hive 1. Hive基本概念 1.1 什么是Hive 1.2 Hive的优缺点 1.2.1 优点 1.2.2 缺点 1.3 Hive架构原理 1.4 Hive和数据库比较 1.4.1 查询语言 ...
五万字 | Hive知识体系保姆级教程
文档目录如下: Hive涉及的知识点如下图所示,本文将逐一讲解: 本文较长,获取本文完整PDF文档,请扫码关注公众号[五分钟学大数据],后台发送:hive pdf,即可下载带目录的完整版Hive文档: ...

flume学习（五）：使用hive来分析flume收集的日志数据

flume学习（五）：使用hive来分析flume收集的日志数据相关推荐

最新文章

热门文章