python链接hbase模块_HBase实战（1）：使用Python连接Hbase数据库

来源于 https://blog.csdn.net/duan_zhihua/java/article/details/80622166

使用Python连接Hbase数据库

1，Hbase下载。

下载地址：http://hbase.apache.org/downloads.html

2，本地Hbase安装

root@master:/usr/local/setup_tools# tar -zxvf hbase-2.0.0-bin.tar.gz

root@master:/usr/local/setup_tools# mv hbase-2.0.0 /usr/local/

root@master:/usr/local/setup_tools# cd /usr/local

root@master:/usr/local# ls | grep hbase

hbase-2.0.0

root@master:/usr/local#

root@master:/usr/local/hbase-2.0.0# vi /etc/profile

export HBASE_HOME=/usr/local/hbase-2.0.0

export PATH=.:$PATH:$JAVA_HOME/bin:$SCALA_HOME/bin:$HADOOP_HOME/bin:$SPARK_HOME/bin:$HIVE_HOME/bin:$FLUME_HOME/bin:$ZOOKEEPER_HOME/bin:$KAFKA_HOME/bin:$IDEA_HOME/bin:$eclipse_HOME:$MAVEN_HOME/bin:$ALLUXIO_HOME/bin:$HBASE_HOME/bin

root@master:/usr/local/hbase-2.0.0# source /etc/profile

3，配置

修改hbase-site.xml，设置存储数据的根目录。

root@master:/usr/local/hbase-2.0.0/conf# vi hbase-site.xml

hbase.rootdir

file:///usr/local/hbase-2.0.0/data

4，启动hbase

root@master:/usr/local/hbase-2.0.0# cd bin

root@master:/usr/local/hbase-2.0.0/bin# ls

considerAsDead.sh hbase hbase-config.cmd hbase-jruby master-backup.sh replication start-hbase.sh zookeepers.sh

draining_servers.rb hbase-cleanup.sh hbase-config.sh hirb.rb region_mover.rb rolling-restart.sh stop-hbase.cmd

get-active-master.rb hbase.cmd hbase-daemon.sh local-master-backup.sh regionservers.sh shutdown_regionserver.rb stop-hbase.sh

graceful_stop.sh hbase-common.sh hbase-daemons.sh local-regionservers.sh region_status.rb start-hbase.cmd test

root@master:/usr/local/hbase-2.0.0/bin# start-hbase.sh

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/usr/local/hbase-2.0.0/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/local/alluxio-1.7.0-hadoop-2.6/client/alluxio-1.7.0-client.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

running master, logging to /usr/local/hbase-2.0.0/logs/hbase-root-master-master.out

root@master:/usr/local/hbase-2.0.0/bin# jps

2757 Jps

2685 HMaster

5，使用hbase shell

root@master:/usr/local/hbase-2.0.0/bin# hbase shell

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/usr/local/hbase-2.0.0/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/local/alluxio-1.7.0-hadoop-2.6/client/alluxio-1.7.0-client.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

HBase Shell

Use "help" to get list of supported commands.

Use "exit" to quit this interactive shell.

Version 2.0.0, r7483b111e4da77adbfc8062b3b22cbe7c2cb91c1, Sun Apr 22 20:26:55 PDT 2018

Took 0.0044 seconds

hbase(main):001:0>

hbase(main):003:0> version

2.0.0, r7483b111e4da77adbfc8062b3b22cbe7c2cb91c1, Sun Apr 22 20:26:55 PDT 2018

Took 0.0054 seconds

hbase(main):004:0>

启动hbase thrift服务。

root@master:/usr/local/hbase-2.0.0/bin# hbase-daemon.sh start thrift

running thrift, logging to /usr/local/hbase-2.0.0/logs/hbase-root-thrift-master.out

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/usr/local/hbase-2.0.0/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/local/alluxio-1.7.0-hadoop-2.6/client/alluxio-1.7.0-client.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

root@master:/usr/local/hbase-2.0.0/bin# jps

3332 Jps

3254 ThriftServer

2685 HMaster

root@master:/usr/local/hbase-2.0.0/bin#

二，使用Python连接Hbase。

1，建立Python开发环境 ---虚拟环境。

先进行检查。

root@master:/usr/local/hbase-2.0.0/bin# pip

The program 'pip' is currently not installed. You can install it by typing:

apt-get install python-pip

root@master:/usr/local/hbase-2.0.0/bin# python

Python 2.7.6 (default, Mar 22 2014, 22:59:56)

[GCC 4.8.2] on linux2

Type "help", "copyright", "credits" or "license" for more information.

>>> exit()

root@master:/usr/local/hbase-2.0.0/bin#

安装pip。

root@master:/usr/local/hbase-2.0.0/bin# apt-get install python-pip

Reading package lists... Done

Building dependency tree

Reading state information... Done

The following packages were automatically installed and are no longer required:

diffstat hardening-includes libapt-pkg-perl libarchive-zip-perl

libautodie-perl libclass-accessor-perl libemail-valid-perl libio-pty-perl

libio-string-perl libipc-run-perl libipc-system-simple-perl

liblist-moreutils-perl libnet-dns-perl libnet-domain-tld-perl libnet-ip-perl

libparse-debianchangelog-perl libperlio-gzip-perl libsub-identify-perl

libsub-name-perl libtext-levenshtein-perl patchutils t1utils

Use 'apt-get autoremove' to remove them.

The following extra packages will be installed:

python-chardet-whl python-colorama python-colorama-whl python-distlib

python-distlib-whl python-html5lib python-html5lib-whl python-pip-whl

python-pkg-resources python-requests-whl python-setuptools

python-setuptools-whl python-six-whl python-urllib3-whl python-wheel

Suggested packages:

python-genshi python-distribute python-distribute-doc

Recommended packages:

python-dev-all

The following NEW packages will be installed:

python-chardet-whl python-colorama python-colorama-whl python-distlib

python-distlib-whl python-html5lib python-html5lib-whl python-pip

python-pip-whl python-requests-whl python-setuptools python-setuptools-whl

python-six-whl python-urllib3-whl python-wheel

The following packages will be upgraded:

python-pkg-resources

1 upgraded, 15 newly installed, 0 to remove and 637 not upgraded.

Need to get 1,744 kB of archives.

After this operation, 4,184 kB of additional disk space will be used.

Do you want to continue? [Y/n] y

Get:1 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/universe python-chardet-whl all 2.2.1-2~ubuntu1 [170 kB]

Get:2 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/universe python-chardet-whl all 2.2.1-2~ubuntu1 [170 kB]

1% [2 python-chardet-whl 14.0 kB/170 kB 8%]

Get:3 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/universe python-chardet-whl all 2.2.1-2~ubuntu1 [170 kB]

Get:4 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/universe python-chardet-whl all 2.2.1-2~ubuntu1 [170 kB]

2% [4 python-chardet-whl 38.1 kB/170 kB 22%]

6% [4 python-chardet-whl 99.9 kB/170 kB 59%]

Get:5 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/universe python-colorama all 0.2.5-0.1ubuntu2 [18.4 kB]

Get:6 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/universe python-colorama-whl all 0.2.5-0.1ubuntu2 [18.2 kB]

Get:7 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/universe python-distlib all 0.1.8-1ubuntu1 [113 kB]

Get:8 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/universe python-distlib-whl all 0.1.8-1ubuntu1 [140 kB]

Get:9 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main python-html5lib all 0.999-3~ubuntu1 [83.5 kB]

Get:10 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main python-html5lib-whl all 0.999-3~ubuntu1 [109 kB]

Get:11 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main python-six-whl all 1.5.2-1ubuntu1 [10.5 kB]

Get:12 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main python-urllib3-whl all 1.7.1-1ubuntu4 [64.0 kB]

Get:13 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main python-requests-whl all 2.2.1-1ubuntu0.3 [227 kB]

Get:14 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main python-setuptools-whl all 3.3-1ubuntu2 [244 kB]

Get:15 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/universe python-pip-whl all 1.5.4-1ubuntu4 [111 kB]

Get:16 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main python-pkg-resources all 3.3-1ubuntu2 [61.9 kB]

Get:17 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main python-setuptools all 3.3-1ubuntu2 [230 kB]

Get:18 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/universe python-pip all 1.5.4-1ubuntu4 [97.3 kB]

Get:19 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main python-wheel all 0.24.0-1~ubuntu1 [44.7 kB]

Fetched 1,723 kB in 9min 30s (3,019 B/s)

Selecting previously unselected package python-chardet-whl.

(Reading database ... 156494 files and directories currently installed.)

Preparing to unpack .../python-chardet-whl_2.2.1-2~ubuntu1_all.deb ...

Unpacking python-chardet-whl (2.2.1-2~ubuntu1) ...

Selecting previously unselected package python-colorama.

Preparing to unpack .../python-colorama_0.2.5-0.1ubuntu2_all.deb ...

Unpacking python-colorama (0.2.5-0.1ubuntu2) ...

Selecting previously unselected package python-colorama-whl.

Preparing to unpack .../python-colorama-whl_0.2.5-0.1ubuntu2_all.deb ...

Unpacking python-colorama-whl (0.2.5-0.1ubuntu2) ...

Selecting previously unselected package python-distlib.

Preparing to unpack .../python-distlib_0.1.8-1ubuntu1_all.deb ...

Unpacking python-distlib (0.1.8-1ubuntu1) ...

Selecting previously unselected package python-distlib-whl.

Preparing to unpack .../python-distlib-whl_0.1.8-1ubuntu1_all.deb ...

Unpacking python-distlib-whl (0.1.8-1ubuntu1) ...

Selecting previously unselected package python-html5lib.

Preparing to unpack .../python-html5lib_0.999-3~ubuntu1_all.deb ...

Unpacking python-html5lib (0.999-3~ubuntu1) ...

Selecting previously unselected package python-html5lib-whl.

Preparing to unpack .../python-html5lib-whl_0.999-3~ubuntu1_all.deb ...

Unpacking python-html5lib-whl (0.999-3~ubuntu1) ...

Selecting previously unselected package python-six-whl.

Preparing to unpack .../python-six-whl_1.5.2-1ubuntu1_all.deb ...

Unpacking python-six-whl (1.5.2-1ubuntu1) ...

Selecting previously unselected package python-urllib3-whl.

Preparing to unpack .../python-urllib3-whl_1.7.1-1ubuntu4_all.deb ...

Unpacking python-urllib3-whl (1.7.1-1ubuntu4) ...

Selecting previously unselected package python-requests-whl.

Preparing to unpack .../python-requests-whl_2.2.1-1ubuntu0.3_all.deb ...

Unpacking python-requests-whl (2.2.1-1ubuntu0.3) ...

Selecting previously unselected package python-setuptools-whl.

Preparing to unpack .../python-setuptools-whl_3.3-1ubuntu2_all.deb ...

Unpacking python-setuptools-whl (3.3-1ubuntu2) ...

Selecting previously unselected package python-pip-whl.

Preparing to unpack .../python-pip-whl_1.5.4-1ubuntu4_all.deb ...

Unpacking python-pip-whl (1.5.4-1ubuntu4) ...

Preparing to unpack .../python-pkg-resources_3.3-1ubuntu2_all.deb ...

Unpacking python-pkg-resources (3.3-1ubuntu2) over (3.3-1ubuntu1) ...

Selecting previously unselected package python-setuptools.

Preparing to unpack .../python-setuptools_3.3-1ubuntu2_all.deb ...

Unpacking python-setuptools (3.3-1ubuntu2) ...

Selecting previously unselected package python-pip.

Preparing to unpack .../python-pip_1.5.4-1ubuntu4_all.deb ...

Unpacking python-pip (1.5.4-1ubuntu4) ...

Selecting previously unselected package python-wheel.

Preparing to unpack .../python-wheel_0.24.0-1~ubuntu1_all.deb ...

Unpacking python-wheel (0.24.0-1~ubuntu1) ...

Processing triggers for man-db (2.6.7.1-1ubuntu1) ...

Setting up python-chardet-whl (2.2.1-2~ubuntu1) ...

Setting up python-colorama (0.2.5-0.1ubuntu2) ...

Setting up python-colorama-whl (0.2.5-0.1ubuntu2) ...

Setting up python-distlib (0.1.8-1ubuntu1) ...

Setting up python-distlib-whl (0.1.8-1ubuntu1) ...

Setting up python-html5lib (0.999-3~ubuntu1) ...

Setting up python-html5lib-whl (0.999-3~ubuntu1) ...

Setting up python-six-whl (1.5.2-1ubuntu1) ...

Setting up python-urllib3-whl (1.7.1-1ubuntu4) ...

Setting up python-requests-whl (2.2.1-1ubuntu0.3) ...

Setting up python-setuptools-whl (3.3-1ubuntu2) ...

Setting up python-pip-whl (1.5.4-1ubuntu4) ...

Setting up python-pkg-resources (3.3-1ubuntu2) ...

Setting up python-setuptools (3.3-1ubuntu2) ...

Setting up python-pip (1.5.4-1ubuntu4) ...

Setting up python-wheel (0.24.0-1~ubuntu1) ...

root@master:/usr/local/hbase-2.0.0/bin#

Pip安装好之后，安装virtualenv软件，让操作系统支持虚拟环境，可以得到命令virtualenv或者pyvenv命令。

root@master:/usr/local/hbase-2.0.0/bin# pip install virtualenv

Downloading/unpacking virtualenv

Downloading virtualenv-16.0.0-py2.py3-none-any.whl (1.9MB): 1.9MB downloaded

Installing collected packages: virtualenv

Successfully installed virtualenv

Cleaning up...

root@master:/usr/local/hbase-2.0.0/bin#

创建虚拟环境

root@master:/usr/local# mkdir myPythonEnv

root@master:/usr/local# cd myPythonEnv

root@master:/usr/local/myPythonEnv# pwd

/usr/local/myPythonEnv

root@master:/usr/local/myPythonEnv# Pyvenv project-env

Pyvenv: command not found

root@master:/usr/local/myPythonEnv# virtualenv project-env

New python executable in /usr/local/myPythonEnv/project-env/bin/python

Installing setuptools, pip, wheel...done.

root@master:/usr/local/myPythonEnv#

进入虚拟环境：

root@master:/usr/local/myPythonEnv/project-env/bin# source /usr/local/myPythonEnv/project-env/bin/activate

(project-env) root@master:/usr/local/myPythonEnv/project-env/bin#

安装Thrift依赖包

(project-env) root@master:/usr/local/myPythonEnv/project-env/bin# pip install thrift

Collecting thrift

/usr/local/myPythonEnv/project-env/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:339: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings

SNIMissingWarning

/usr/local/myPythonEnv/project-env/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:137: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings

InsecurePlatformWarning

Downloading https://files.pythonhosted.org/packages/c6/b4/510617906f8e0c5660e7d96fbc5585113f83ad547a3989b80297ac72a74c/thrift-0.11.0.tar.gz (52kB)

100% |████████████████████████████████| 61kB 22kB/s

Collecting six>=1.7.2 (from thrift)

Downloading https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl

Building wheels for collected packages: thrift

Running setup.py bdist_wheel for thrift ... done

Stored in directory: /root/.cache/pip/wheels/be/36/81/0f93ba89a1cb7887c91937948519840a72c0ffdd57cac0ae8f

Successfully built thrift

Installing collected packages: six, thrift

Successfully installed six-1.11.0 thrift-0.11.0

安装hbase-thrift依赖包

(project-env) root@master:/usr/local/myPythonEnv/project-env/bin# pip install hbase-thrift

Collecting hbase-thrift

SNIMissingWarning

InsecurePlatformWarning

Downloading https://files.pythonhosted.org/packages/89/f7/dbb6c764bb909ed361c255828701228d8c9867d541cfef84127e6f3704cc/hbase-thrift-0.20.4.tar.gz

Requirement already satisfied: Thrift in /usr/local/myPythonEnv/project-env/lib/python2.7/site-packages (from hbase-thrift) (0.11.0)

Requirement already satisfied: six>=1.7.2 in /usr/local/myPythonEnv/project-env/lib/python2.7/site-packages (from Thrift->hbase-thrift) (1.11.0)

Building wheels for collected packages: hbase-thrift

Running setup.py bdist_wheel for hbase-thrift ... done

Stored in directory: /root/.cache/pip/wheels/fe/51/f2/afb7b010cd97910aa0b651d492735a38ed69a93a817444904e

Successfully built hbase-thrift

Installing collected packages: hbase-thrift

Successfully installed hbase-thrift-0.20.4

(project-env) root@master:/usr/local/myPythonEnv/project-env/bin#

Python连接HBASE的测试程序

(project-env) root@master:/usr/local/noc_python_201806# vi queryTest.py

from thrift import Thrift

from thrift.transport import TSocket

from thrift.transport import TTransport

from thrift.protocol import TBinaryProtocol

from hbase import Hbase

from hbase.ttypes import *

transport = TSocket.TSocket('localhost', 9090)

transport = TTransport.TBufferedTransport(transport)

protocol = TBinaryProtocol.TBinaryProtocol(transport)

client = Hbase.Client(protocol)

transport.open()

print client.getTableNames()

运行结果为，查询出表：

(project-env) root@master:/usr/local/noc_python_201806# python queryTest.py

['test', 'testHbase']

(project-env) root@master:/usr/local/noc_python_201806#

Hbase中的表

hbase(main):025:0> list

TABLE

test

testHbase

2 row(s)

Took 0.0188 seconds

=> ["test", "testHbase"]

hbase(main):026:0>

至此，Python连接Hbase已经成功！

python链接hbase模块_HBase实战（1）：使用Python连接Hbase数据库相关推荐

云计算Python自动化运维开发实战三、python文件类型
为什么80%的码农都做不了架构师?>>> 云计算Python自动化运维开发实战三.python文件类型导语: python常用的有3种文件类型 1. 源代码 py ...
CrazyWing：Python自动化运维开发实战八、Python数据类型之字符串
导语: 字符串是Python中最常用的数据类型,它是编程语言中表示文本的数据类型. 使用引号('或")来创建字符串. 创建字符串只要为变量分配一个值即可. 例如: var1 = 'Hell ...
python之sys模块详解_(转)python之os,sys模块详解
python之sys模块详解原文:http://www.cnblogs.com/cherishry/p/5725184.html sys模块功能多,我们这里介绍一些比较实用的功能,相信你会喜欢的,和 ...
python利用写模块_使用C++编写python扩展模块
简介长话短说,这里说的扩展Python功能与直接用其它语言写一个动态链接库,然后让Python来调用有点不一样(虽然本质是一样的).而是指使用Python本身提供的API,使用C++来对Python ...
用于生成随机数的python标准库模块是_详解Python基础random模块随机数的生成
详解Python基础random模块随机数的生成来源:中文源码网浏览: 次日期:2019年11月5日 [下载文档: 详解Python基础random模块随机数的生成.txt ] ( ...
python中re模块的span,详解Python正则表达式re模块
正则是处理字符串最常用的方法,我们编码中到处可见正则的身影. 正则大同小异,python 中的正则跟其他语言相比略有差异: 1.替换字符串时,替换的字符串可以是一个函数 2.split 函数可以指定分 ...
Python Web开发框架之Django篇——二、Django连接MySQL数据库以及建表的操作
二.Django连接MySQL数据库以及建表的操作准备工作:安装Python访问MySQL的模块一.修改project同名目录下面的__init__.py文件二.修改project同名目录下面的 ...
python能和wincc一起用_python通过ado连接wincc数据库
python通过ado连接wincc数据库注意点: 1,python需要安装32位,64位测试没通过,具体原因不明 2,若使用Provider=WinCCOLEDBProvider.1需要安装WIN ...
python爬虫使用模块_10分钟教你Python爬虫（下）--爬虫的基本模块与简单的实战...
本文来源于公众号[程序猿声],作者向柯玮前言各位看客老爷们,新年好.小玮又来啦.这次给大家带来的是爬虫系列的第二课---爬虫的基本模块与简单的实战. 说到爬虫的基本模块,不知道大家之前有没有了解过 ...

python链接hbase模块_HBase实战（1）：使用Python连接Hbase数据库

python链接hbase模块_HBase实战（1）：使用Python连接Hbase数据库相关推荐

最新文章

热门文章