Design and Implementation of Log Analysis System Based on Hadoop/CloudBase/MySQL

ZHUO Haiyi

1

卓海艺(1987-)，男，硕士，主要研究方向：云计算、网络安全

ZHAO Wenshen

1

赵文深(1954-)，男，副教授，主要研究方向：智能信息系统、云计算

LV Yuqin

1

吕玉琴，女，教授，主要研究方向：通信协议，网络信息安全

HOU Bin

1

侯宾，男，讲师，主要研究方向：云计算，网络安全

1、School of Electronic Engineering, Beijing University of Posts and Telecommunications, Beijing 100876

Abstract：With the rapid development of SNS and mobile internet, the network log data of many internet companies and institutions is becoming bigger and bigger. The problem of how to processing large-scale log data quickly and deriving useful information for better serveice become a urgent problem which need to be solved. Cloud computing and data warehouse technology provides a way to solve this problem. Hadoop is open source framework which is developed by Apache and is widely used in massive data processing. CloudBase is a data warehouse software which is based on Hadoop. It can be used for large scale data analysis,. It is very convenient to query and analyse the large-scale log data by using the ANSI SQL. We have designed and implemented a distributed network log analysis system based on Hadoop and CloudBase and MySQL. It fetchs the network log data regularly from the backend server and processes it, then puts processing results into MySQL for diagram showing on the web page. Such a system can facilitate enterprises and institutions for understanding of user needs and then providing better services for customers. We demonstrate the effectiveness and the worth of our system by evaluating its performance.