拿hbase基准测试列子来分析哪些线程使用比较高的cpu，环境是linux，基准测试命令：

hbase org.apache.hadoop.hbase.PerformanceEvaluation  --rows=500000 --nomapred --presplit=5 --writeToWAL=true randomWrite 5

首先查看占用cpu最高的进程和线程id，执行命令：

[ocnosql@server01 logs]$ ps Hh -eo pid,tid,pcpu | sort -nk3 |tail

12404 12429 0.6

13230 13291 0.6

14117 14178 1.7

14117 14172 1.9

14117 14294 4.6

14117 14295 4.6

14117 14296 4.6

14117 14293 4.7

14117 14297 4.7

14117 14173 8.0

获取线程14297十六进制，后面会在jstack里面用到

[ocnosql@server01 logs]$ printf "%x\n" 14297

37d9

然后根据jstack查看cpu使用为4.7%的线程id14297，进程id14117，获取jstack信息：

[ocnosql@server01 logs]$ jstack 14117 > jstack.log

然后vi jstack.log，查找线程id为37d9的线程

"TestClient-4" #40 prio=5 os_prio=0 tid=0x00007fefc0539000 nid=0x37d9 runnable [0x00007fefa73f9000]

java.lang.Thread.State: RUNNABLE

at org.apache.hadoop.hbase.util.Bytes$LexicographicalComparerHolder$UnsafeComparer.compareTo(Bytes.java:1276)

at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:1187)

at org.apache.hadoop.hbase.util.Bytes$ByteArrayComparator.compare(Bytes.java:153)

at org.apache.hadoop.hbase.util.Bytes$ByteArrayComparator.compare(Bytes.java:142)

at java.util.concurrent.ConcurrentSkipListMap.cpr(ConcurrentSkipListMap.java:655)

at java.util.concurrent.ConcurrentSkipListMap.findNear(ConcurrentSkipListMap.java:1247)

at java.util.concurrent.ConcurrentSkipListMap.getNear(ConcurrentSkipListMap.java:1268)

at java.util.concurrent.ConcurrentSkipListMap.floorEntry(ConcurrentSkipListMap.java:2153)

at org.apache.hadoop.hbase.client.MetaCache.getCachedLocation(MetaCache.java:74)

at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getCachedLocation(ConnectionManager.java:1313)

at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1157)

at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1109)

at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:369)

at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:320)

at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:198)

at org.apache.hadoop.hbase.client.BufferedMutatorImpl.doMutate(BufferedMutatorImpl.java:141)

at org.apache.hadoop.hbase.client.BufferedMutatorImpl.mutate(BufferedMutatorImpl.java:98)

- locked <0x00000000c6ccc800> (a org.apache.hadoop.hbase.client.BufferedMutatorImpl)

at org.apache.hadoop.hbase.PerformanceEvaluation$RandomWriteTest.testRow(PerformanceEvaluation.java:1384)

at org.apache.hadoop.hbase.PerformanceEvaluation$Test.testTimed(PerformanceEvaluation.java:1071)

at org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:1053)

at org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1612)

at org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:410)

at org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:405)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

可以看到此线程正在执行put操作，是基准测试中的一个client，符合我们的预期。