扫描器

扫描技术。这种技术类似于数据库系统中的游标（cursor), 并利用到了HBase 提供的底层顺序存储的数据结构。

扫描操作的工作方式类似于迭代器，用户无需调用scan() 方法创建实例。只需要调用 Table 的 getScanner() 方法，此方法返回真正的扫描器（scanner）实例的同时，用户也可以使用它迭代获取数据。

注意：要确保尽早始放扫描器实例，一个打开的扫描器会占用不少服务器资源，累计多了会占用大量的堆空间。当使用完 ResultScanner 之后应调用它的 (Scan) close 方法，同时应该把 close 方法放到 try / finally 块中，以保证其在迭代获取数据过程中出现异常和错误时，仍能执行 close。

Table getScanner 方法

 /*** Returns a scanner on the current table as specified by the {@link Scan}* object.* Note that the passed {@link Scan}'s start row and caching properties* maybe changed.** @param scan A configured {@link Scan} object.* @return A scanner.* @throws IOException if a remote or network exception occurs.* @since 0.20.0*/ResultScanner getScanner(Scan scan) throws IOException;/*** Gets a scanner on the current table for the given family.** @param family The column family to scan.* @return A scanner.* @throws IOException if a remote or network exception occurs.* @since 0.20.0*/ResultScanner getScanner(byte[] family) throws IOException;/*** Gets a scanner on the current table for the given family and qualifier.** @param family The column family to scan.* @param qualifier The column qualifier to scan.* @return A scanner.* @throws IOException if a remote or network exception occurs.* @since 0.20.0*/ResultScanner getScanner(byte[] family, byte[] qualifier) throws IOException;

Scan 的构造器

  /*** Create a Scan operation across all rows.*/public Scan() {}/*** @deprecated use {@code new Scan().withStartRow(startRow).setFilter(filter)} instead.*/@Deprecatedpublic Scan(byte[] startRow, Filter filter) {this(startRow);this.filter = filter;}/*** Create a Scan operation starting at the specified row.* <p>* If the specified row does not exist, the Scanner will start from the next closest row after the* specified row.* @param startRow row to start scanner at or after* @deprecated use {@code new Scan().withStartRow(startRow)} instead.*/@Deprecatedpublic Scan(byte[] startRow) {setStartRow(startRow);}/*** Create a Scan operation for the range of rows specified.* @param startRow row to start scanner at or after (inclusive)* @param stopRow row to stop scanner before (exclusive)* @deprecated use {@code new Scan().withStartRow(startRow).withStopRow(stopRow)} instead.*/@Deprecatedpublic Scan(byte[] startRow, byte[] stopRow) {setStartRow(startRow);setStopRow(stopRow);}//拷贝模式public Scan(Scan scan) throws IOException {startRow = scan.getStartRow();includeStartRow = scan.includeStartRow();stopRow  = scan.getStopRow();includeStopRow = scan.includeStopRow();maxVersions = scan.getMaxVersions();batch = scan.getBatch();storeLimit = scan.getMaxResultsPerColumnFamily();storeOffset = scan.getRowOffsetPerColumnFamily();caching = scan.getCaching();maxResultSize = scan.getMaxResultSize();cacheBlocks = scan.getCacheBlocks();filter = scan.getFilter(); // clone?loadColumnFamiliesOnDemand = scan.getLoadColumnFamiliesOnDemandValue();consistency = scan.getConsistency();this.setIsolationLevel(scan.getIsolationLevel());reversed = scan.isReversed();asyncPrefetch = scan.isAsyncPrefetch();small = scan.isSmall();allowPartialResults = scan.getAllowPartialResults();tr = scan.getTimeRange(); // TimeRange is immutableMap<byte[], NavigableSet<byte[]>> fams = scan.getFamilyMap();for (Map.Entry<byte[],NavigableSet<byte[]>> entry : fams.entrySet()) {byte [] fam = entry.getKey();NavigableSet<byte[]> cols = entry.getValue();if (cols != null && cols.size() > 0) {for (byte[] col : cols) {addColumn(fam, col);}} else {addFamily(fam);}}for (Map.Entry<String, byte[]> attr : scan.getAttributesMap().entrySet()) {setAttribute(attr.getKey(), attr.getValue());}for (Map.Entry<byte[], TimeRange> entry : scan.getColumnFamilyTimeRange().entrySet()) {TimeRange tr = entry.getValue();setColumnFamilyTimeRange(entry.getKey(), tr.getMin(), tr.getMax());}this.mvccReadPoint = scan.getMvccReadPoint();this.limit = scan.getLimit();this.needCursorResult = scan.isNeedCursorResult();setPriority(scan.getPriority());}

Scan返回值 ResultScanner类

扫描操作不会通过一次RPC请求返回所有匹配的行，而是以行为单位进行返回。ResultScanner 把扫描器转换为类似的get 操作，它将每一行数据封装成一个Result实例，并将所有的Result 实例放入一个迭代器中。

Result next() throws IOException
Result[] next(int nbRows) throws IOException
void close()

Scan 的优化点

扫描器缓存：

resultScanner的每一次 next 调用都会为每行数据生成一个单独的RPC请求。即使使用next(int nbRows)方法，该方法仅仅是在客户端循环地调用next()方法。

因此可以利用扫描器缓存，让一次RPC请求获取多行数据。（默认不开启）

相关设置

void setScannerCaching(int scannerCaching)
int getScannerCaching()

批量参数

缓存是面向行一级的操作，而批量则是面向列一级的操作。批量可以让用户选择每一次 ResultScanner 实例的 next() 操作要去会取回多少列。

  /*** Set the maximum number of cells to return for each call to next(). Callers should be aware* that this is not equivalent to calling {@link #setAllowPartialResults(boolean)}.* If you don't allow partial results, the number of cells in each Result must equal to your* batch setting unless it is the last Result for current row. So this method is helpful in paging* queries. If you just want to prevent OOM at client, use setAllowPartialResults(true) is better.* @param batch the maximum number of values* @see Result#mayHaveMoreCellsInRow()*/public Scan setBatch(int batch) {if (this.hasFilter() && this.filter.hasFilterRow()) {throw new IncompatibleFilterException("Cannot set batch on a scan using a filter" +" that returns true for filter.hasFilterRow");}this.batch = batch;return this;}

当用户想尽量提高和利用系统性能时，需要为这两个参数选择一个合适的组合。

只有当用户使用批量模式后，行内（intra-row）扫描功能才会启用。

两个参数共同作用

示例

数据样例

hbase(main):006:0> scan 'test3',{VERSIONS=>3}
ROW                            COLUMN+CELL                                                                           ce_shi1                       column=author:name, timestamp=1587558488841, value=zhouyuqin                          ce_shi1                       column=author:name, timestamp=1587402132957, value=nicholas                           ce_shi1                       column=author:name, timestamp=1587402040153, value=nicholas                           ce_shi1                       column=author:nickname, timestamp=1587402040153, value=lee                            ce_shi2                       column=author:name, timestamp=1587402132957, value=spark                              ce_shi2                       column=author:name, timestamp=1587402040153, value=spark                              ce_shi2                       column=author:nickname, timestamp=1587402132957, value=hadoop                         ce_shi2                       column=author:nickname, timestamp=1587402040153, value=hadoop                         ce_shi3                       column=author:age, timestamp=1587558488841, value=12                                  ce_shi3                       column=author:name, timestamp=1587558488841, value=sunzhenhua                         test33                        column=author:name, timestamp=1587402133000, value=sunzhenhua                         test33                        column=author:name, timestamp=1587402040188, value=sunzhenhua                         test33                        column=author:name, timestamp=1587400015581, value=sunzhenhua
4 row(s)
Took 0.0846 seconds

代码：

package hbase_2;import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.*;
import org.apache.hadoop.hbase.util.Bytes;/*** Created by szh on 2020/4/22.* @author szh*/
public class Hbase_BasicScan {public static void main(String[] args) throws Exception{Configuration conf = HBaseConfiguration.create();conf.set("hbase.zookeeper.quorum", "cdh-manager,cdh-node1,cdh-node2");conf.set("hbase.zookeeper.property.clientPort", "2181");Connection conn = ConnectionFactory.createConnection(conf);TableName tableName = TableName.valueOf("test3");Table table = conn.getTable(tableName);//设置客户端缓存大小Scan scan = new Scan();scan.setMaxVersions(3);ResultScanner scanner = table.getScanner(scan);for(Result res : scanner){System.out.println(res);}scanner.close();System.out.println("=============================");System.out.println("=============================");System.out.println("=============================");//TimeRange 默认只返回一个版本
//        * Get versions of columns only within the specified timestamp range,
//        * [minStamp, maxStamp).  Note, default maximum versions to return is 1.  If
//        * your time range spans more than one version and you want all versions
//        * returned, up the number of versions beyond the default.Scan scan2 = new Scan();/*** Set the number of rows for caching that will be passed to scanners.* If not set, the Configuration setting {@link HConstants#HBASE_CLIENT_SCANNER_CACHING} will* apply.* Higher caching values will enable faster scanners but will use more memory.* @param caching the number of rows for caching*/scan2.setCaching(10); //设置客户端一次取数据缓存数据的多少scan2.addFamily(Bytes.toBytes("author"));scan2.setTimeRange(1587402040153L,1587558488841L);ResultScanner scanner2 = table.getScanner(scan2);for(Result res : scanner2){System.out.println(res);}scanner2.close();System.out.println("=============================");System.out.println("=============================");System.out.println("=============================");Scan scan3 =// 不推荐使用//new Scan(Bytes.toBytes("ce_shi1"),Bytes.toBytes("ce_shi9"));new Scan().withStartRow(Bytes.toBytes("ce_shi1")).withStopRow(Bytes.toBytes("ce_shi9"));/*** Set the number of rows for caching that will be passed to scanners.* If not set, the Configuration setting {@link HConstants#HBASE_CLIENT_SCANNER_CACHING} will* apply.* Higher caching values will enable faster scanners but will use more memory.* @param caching the number of rows for caching*/scan3.setCaching(10); //设置客户端一次取数据缓存数据的多少ResultScanner scanner3 = table.getScanner(scan3);for(Result res : scanner3){System.out.println(res);}scanner3.close();System.out.println("=============================");System.out.println("=============================");System.out.println("=============================");Scan scan4 =// 不推荐使用//new Scan(Bytes.toBytes("ce_shi1"),Bytes.toBytes("ce_shi9"));new Scan().withStartRow(Bytes.toBytes("ce_shi1")).withStopRow(Bytes.toBytes("ce_shi9"));/*** Set the number of rows for caching that will be passed to scanners.* If not set, the Configuration setting {@link HConstants#HBASE_CLIENT_SCANNER_CACHING} will* apply.* Higher caching values will enable faster scanners but will use more memory.* @param caching the number of rows for caching*/scan4.setCaching(10); //设置客户端一次取数据缓存数据的多少scan4.setBatch(1);ResultScanner scanner4 = table.getScanner(scan4);for(Result res : scanner4){System.out.println(res);}scanner4.close();table.close();}}

输出

package hbase_2;import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.*;
import org.apache.hadoop.hbase.util.Bytes;/*** Created by szh on 2020/4/22.* @author szh*/
public class Hbase_BasicScan {public static void main(String[] args) throws Exception{Configuration conf = HBaseConfiguration.create();conf.set("hbase.zookeeper.quorum", "cdh-manager,cdh-node1,cdh-node2");conf.set("hbase.zookeeper.property.clientPort", "2181");Connection conn = ConnectionFactory.createConnection(conf);TableName tableName = TableName.valueOf("test3");Table table = conn.getTable(tableName);//设置客户端缓存大小Scan scan = new Scan();scan.setMaxVersions(3);ResultScanner scanner = table.getScanner(scan);for(Result res : scanner){System.out.println(res);}scanner.close();System.out.println("=============================");System.out.println("=============================");System.out.println("=============================");//TimeRange 默认只返回一个版本
//        * Get versions of columns only within the specified timestamp range,
//        * [minStamp, maxStamp).  Note, default maximum versions to return is 1.  If
//        * your time range spans more than one version and you want all versions
//        * returned, up the number of versions beyond the default.Scan scan2 = new Scan();/*** Set the number of rows for caching that will be passed to scanners.* If not set, the Configuration setting {@link HConstants#HBASE_CLIENT_SCANNER_CACHING} will* apply.* Higher caching values will enable faster scanners but will use more memory.* @param caching the number of rows for caching*/scan2.setCaching(10); //设置客户端一次取数据缓存数据的多少scan2.addFamily(Bytes.toBytes("author"));scan2.setTimeRange(1587402040153L,1587558488841L);ResultScanner scanner2 = table.getScanner(scan2);for(Result res : scanner2){System.out.println(res);}scanner2.close();System.out.println("=============================");System.out.println("=============================");System.out.println("=============================");Scan scan3 =// 不推荐使用//new Scan(Bytes.toBytes("ce_shi1"),Bytes.toBytes("ce_shi9"));new Scan().withStartRow(Bytes.toBytes("ce_shi1")).withStopRow(Bytes.toBytes("ce_shi9"));/*** Set the number of rows for caching that will be passed to scanners.* If not set, the Configuration setting {@link HConstants#HBASE_CLIENT_SCANNER_CACHING} will* apply.* Higher caching values will enable faster scanners but will use more memory.* @param caching the number of rows for caching*/scan3.setCaching(10); //设置客户端一次取数据缓存数据的多少ResultScanner scanner3 = table.getScanner(scan3);for(Result res : scanner3){System.out.println(res);}scanner3.close();System.out.println("=============================");System.out.println("=============================");System.out.println("=============================");Scan scan4 =// 不推荐使用//new Scan(Bytes.toBytes("ce_shi1"),Bytes.toBytes("ce_shi9"));new Scan().withStartRow(Bytes.toBytes("ce_shi1")).withStopRow(Bytes.toBytes("ce_shi9"));/*** Set the number of rows for caching that will be passed to scanners.* If not set, the Configuration setting {@link HConstants#HBASE_CLIENT_SCANNER_CACHING} will* apply.* Higher caching values will enable faster scanners but will use more memory.* @param caching the number of rows for caching*/scan4.setCaching(10); //设置客户端一次取数据缓存数据的多少scan4.setBatch(1);ResultScanner scanner4 = table.getScanner(scan4);for(Result res : scanner4){System.out.println(res);}scanner4.close();table.close();}}

HBase_HBase2.0+ Java API 操作指南 (三) 扫描器Scan相关推荐

协议开发中移动CMPP2.0协议API（三）
协议开发中移动CMPP2.0协议API(三) 云网(jimzj@21cn.com) 接上篇... 五.发送接口对于API来说,最重要的一部分就是去做发送数据了.通过SOCKET套接字与网关相连接后 ...
Nachos5.0 java版本前三个proj设计报告
Reference nachos-java Task1.1 Join 包括后面几篇,有proj1的5个task讲解 Nachos Project2思路.代码操作系统nachoes一些问题与解决方法 ...
再谈RDD、DataFrame、DataSet关系以及相互转换（JAVA API）
Spark提供了三种主要的与数据相关的API: RDD DataFrame DataSet 三者图示下面详细介绍下各自的特点: RDD 主要描述:RDD是Spark提供的最主要的一个抽象概念(Res ...
【hadoop2.6.0】利用Hadoop的 Java API
Hadoop2.6.0的所有Java API都在 http://hadoop.apache.org/docs/r2.6.0/api/overview-summary.html 里. 下面实现一个利用J ...
java search 不能使用方法_ElasticSearch实战系列三: ElasticSearch的JAVA API使用教程
前言在上一篇中介绍了ElasticSearch实战系列二: ElasticSearch的DSL语句使用教程---图文详解,本篇文章就来讲解下 ElasticSearch 6.x官方Java API的 ...
Elasticsearch8.0版本中Elasticsearch Java API Client客户端的基本使用方法
关于Elasticsearch Java API Client客户端如何连接以及如何对索引和文档进行基本的增删改查操作请查看我的上一篇博文:Elasticsearch RestHighLevelCli ...
Kafka系列三 java API操作
使用java API操作kafka 1.pom.xml <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xs ...
Windows下Libvirt Java API使用教程(三)- TLS认证访问和动态链接文件依赖
之前已经介绍过了libvirt api的上手使用方式: <Windows下Libvirt Java API使用教程(二)- 接口使用说明> <Windows下Libvirt Java ...
用Java API实现HDFS操作（三）问题汇总
说明:整理博主用Java API实现HDFS操作时遇到的问题问题为搭建分布式操作环境–在ubuntu16中安装IDEA2021.tomcat9.0.hadoop2.7,建议分配磁盘空间30G及以上 ...

HBase_HBase2.0+ Java API 操作指南 (三) 扫描器Scan

扫描器

Table getScanner 方法

Scan 的构造器

Scan返回值 ResultScanner类

Scan 的优化点

HBase_HBase2.0+ Java API 操作指南 (三) 扫描器Scan相关推荐

最新文章

热门文章