Elasticsearch压测工具esrally使用笔记
2024-04-25 02:40:06
Elasticsearch压测工具esrally使用笔记
一、环境配置
方法一 使用docker
docker pull elastic/rally
docker run elastic/rally list tracks
docker run elastic/rally --track=nested --test-mode --pipeline=benchmark-only --target-hosts=*****:9200
数据集无法导入 弃用
2022.7.23 当时对docker的理解不够深入，没有使用此方法。
方法二
1. 创建新用户esrally
adduser esrally
passwd esrally
授权
[root@cdh-slave1 bin]# whereis sudoers
sudoers: /etc/sudoers /etc/sudoers.d /usr/share/man/man5/sudoers.5.gz
[root@cdh-slave1 bin]# ls -l /etc/sudoers
-r--r-----. 1 root root 3938 Jun 7 2017 /etc/sudoers
[root@cdh-slave1 bin]# chmod -v u+w /etc/sudoers
mode of ‘/etc/sudoers’ changed from 0440 (r--r-----) to 0640 (rw-r-----)
输入命令 nano /etc/sudoers 修改sudoers文件，添加新用户信息：
## Allow root to run any commands anywhere
root ALL=(ALL) ALL
esrally ALL=(ALL) ALL #这个是新用户
收回权限
[root@cdh-slave1 bin]# chmod -v u-w /etc/sudoers
mode of ‘/etc/sudoers’ changed from 0640 (rw-r-----) to 0440 (r--r-----)
2. 安装git1.9
yum install curl curl-devel zlib-devel openssl-devel perl cpio expat-devel gettext-devel gcc perl-ExtUtils-MakeMaker autoconf
cd /usr/src
wget https://www.kernel.org/pub/software/scm/git/git-1.9.4.tar.gz
tar xzf git-1.9.4.tar.gz
cd git-1.9.4
autoconf
./configure
make
make install
echo "export PATH=$PATH:/usr/local/bin/" >> /etc/bashrc
source /etc/bashrc
3.配置好JAVA_HOME
https://blog.csdn.net/qq_41058067/article/details/113375312
4. python3.8及esrally
yum install bzip2-devel libffi-devel
安装python3.8 https://blog.csdn.net/qq_41058067/article/details/112209231
su esrally
pip38 install esrally -i https://pypi.tuna.tsinghua.edu.cn/simple
sudo ln -s /usr/local/python/bin/esrally /usr/bin/esrally
或者
/home/esrally/.local/bin/esrally
urllib3==1.25
pip38 uninstall urllib3
pip38 install urllib3==1.25
5. 配置
- 修改rally.ini文件
nano \home\esrally\.rally\rally.ini
[meta]
config.version = 17[system]
env.name = local[node]
root.dir = /home/esrally/.rally/benchmarks
src.root.dir = /home/esrally/.rally/benchmarks/src[source]
remote.repo.url = https://github.com/elasticsearch/elasticsearch
elasticsearch.src.subdir = elasticsearch[benchmarks]
local.dataset.cache = /home/esrally/.rally/benchmarks/data[reporting]
datastore.type = in-memory
datastore.host =
datastore.port =
datastore.secure = False
datastore.user =
datastore.password =[tracks]
default.url = https://gitee.com/xzhp33p/rally-tracks[teams]
default.url = https://gitee.com/guapian/rally-teams[defaults]
preserve_benchmark_candidate = false[distributions]
release.cache = true
远程仓库地址:
https://gitee.com/mirrors/elasticsearch.git (建议不改)
赛道配置:
https://gitee.com/xzhp33p/rally-tracks （建议不改）
https://gitee.com/guapian/rally-teams
- 手动上传数据集到/home/esrally/.rally/benchmarks/data
geonames、geopoint：都是和地理位置相关的，如果需要测试ES在地理位置处理的性能可以选用
http_logs：是http_server的，如果要测服务器日志、redis日志、apache日志可以选用 说明：esrally
自带的测试数据即为 rally_track 文件夹中的内容，主要包括:Geonames(geonames): for evaluating the performance of structured data.
Geopoint(geopoint): for evaluating the performance of geo queries.
Percolator(percolator): for evaluating the performance of percolation
queries. PMC(pmc): for evaluating the performance of full text search.
NYC taxis(nyc_taxis): for evaluating the performance for highly
structured data. Nested(nested): for evaluating the performance for
nested documents. Logging(logging): for evaluating the performance of
(Web) server logs. noaa(noaa): for evaluating the performance of range
fields.
- git clone tracks和teams到对应位置
/home/esrally/.rally/benchmarks/tracks/default
/home/esrally/.rally/benchmarks/teams/default
TEST
切换到非root用户
su esrally
esrally race --offline --track=nested --distribution-version=7.1.0 --test-mode
第一次运行会下载es7.1.0，等待下载完成。
[esrally@cdh-slave1 .rally]$ esrally race --offline --track=nested --distribution-version=7.1.0 --test-mode____ ____/ __ \____ _/ / /_ __/ /_/ / __ `/ / / / / // _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, //____/[WARNING] Local changes in [/home/esrally/.rally/benchmarks/tracks/default] prevent tracks update from remote. Please commit your changes.
[WARNING] Local changes in [/home/esrally/.rally/benchmarks/teams/default] prevent teams update from remote. Please commit your changes.
[INFO] Preparing for race ...
[WARNING] Local changes in [/home/esrally/.rally/benchmarks/teams/default] prevent teams update from remote. Please commit your changes.
[WARNING] Local changes in [/home/esrally/.rally/benchmarks/tracks/default] prevent tracks update from remote. Please commit your changes.
[INFO] Decompressing track data from [/home/esrally/.rally/benchmarks/data/nested/documents-1k.json.bz2] to [/home/esrally/.rally/benchmarks/data/nested/documents-1k.json] ... [OK]
[INFO] Preparing file offset table for [/home/esrally/.rally/benchmarks/data/nested/documents-1k.json] ... [OK]
[INFO] Racing on track [nested], challenge [nested-search-challenge] and car ['defaults'] with version [7.1.0].Running delete-index [100% done]
Running create-index [100% done]
Running check-cluster-health [100% done]
Running index-append [100% done]
Running refresh-after-index [100% done]
Running force-merge [100% done]
Running refresh-after-force-merge [100% done]
Running wait-until-merges-finish [100% done]
Running randomized-nested-queries [100% done]
Running randomized-term-queries [100% done]
Running randomized-sorted-term-queries [100% done]
Running match-all [100% done]
Running nested-date-histo [100% done]
Running randomized-nested-queries-with-inner-hits_default [100% done]
Running randomized-nested-queries-with-inner-hits_default_big_size [100% done]------------------------------------------------------_______ __ _____/ ____(_)___ ____ _/ / / ___/_________ ________/ /_ / / __ \/ __ `/ / \__ \/ ___/ __ \/ ___/ _ \/ __/ / / / / / /_/ / / ___/ / /__/ /_/ / / / __/
/_/ /_/_/ /_/\__,_/_/ /____/\___/\____/_/ \___/
------------------------------------------------------
| 100th percentile service time | nested-date-histo | 3.94686 | ms |
| error rate | nested-date-histo | 0 | % |
| Min Throughput | randomized-nested-queries-with-inner-hits_default | 288.94 | ops/s |
| Median Throughput | randomized-nested-queries-with-inner-hits_default | 288.94 | ops/s |
| Max Throughput | randomized-nested-queries-with-inner-hits_default | 288.94 | ops/s |
| error rate | nested-date-histo | 0 | % |
| Min Throughput | randomized-nested-queries-with-inner-hits_default | 288.94 | ops/s |
| Median Throughput | randomized-nested-queries-with-inner-hits_default | 288.94 | ops/s |
| Max Throughput | randomized-nested-queries-with-inner-hits_default | 288.94 | ops/s |
| 50th percentile latency | randomized-nested-queries-with-inner-hits_default | 25.3766 | ms |
| 100th percentile latency | randomized-nested-queries-with-inner-hits_default | 27.743 | ms |
| 50th percentile service time | randomized-nested-queries-with-inner-hits_default | 2.97431 | ms |
| 100th percentile service time | randomized-nested-queries-with-inner-hits_default | 3.39379 | ms |
| error rate | randomized-nested-queries-with-inner-hits_default | 0 | % |
| Min Throughput | randomized-nested-queries-with-inner-hits_default_big_size | 441.74 | ops/s |
| Median Throughput | randomized-nested-queries-with-inner-hits_default_big_size | 441.74 | ops/s |
| Max Throughput | randomized-nested-queries-with-inner-hits_default_big_size | 441.74 | ops/s |
| 50th percentile latency | randomized-nested-queries-with-inner-hits_default_big_size | 16.3593 | ms |
| 100th percentile latency | randomized-nested-queries-with-inner-hits_default_big_size | 18.1637 | ms |
| 50th percentile service time | randomized-nested-queries-with-inner-hits_default_big_size | 2.93516 | ms |
| 100th percentile service time | randomized-nested-queries-with-inner-hits_default_big_size | 3.07303 | ms |
| error rate | randomized-nested-queries-with-inner-hits_default_big_size | 0 | % |[WARNING] Error rate is 100.0 for operation 'index-append'. Please check the logs.--------------------------------
[INFO] SUCCESS (took 25 seconds)
--------------------------------
二、使用
相关名词：
Race:压力测试比作Race，就是比赛的意思
Track:赛道的意思。es在track的标准和策略去执行比赛
cars:去执行的标准。比如各种配置的车去ES上测试。
–distribution-version 针对 es 7.1.0 版本进行压力测试（测试已有集群不用写）
esrally race --distribution-version=7.1.0
–test-mode 用1000条文档进行测试
esrally race --test-mode
–pipelines 流水线、流程
[esrally@cdh-slave1 .rally]$ esrally list pipelines--user-tag="version:2.4.6" 加标签方便对比____ ____/ __ \____ _/ / /_ __/ /_/ / __ `/ / / / / // _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, //____/Available pipelines:Name Description
----------------------- ----------------------------------------------------------------------------------------------
from-sources Builds and provisions Elasticsearch, runs a benchmark and reports results.
from-sources-complete Builds and provisions Elasticsearch, runs a benchmark and reports results [deprecated].
from-sources-skip-build Provisions Elasticsearch (skips the build), runs a benchmark and reports results [deprecated].
from-distribution Downloads an Elasticsearch distribution, provisions it, runs a benchmark and reports results.
benchmark-only Assumes an already running Elasticsearch instance, runs a benchmark and reports results-------------------------------
[INFO] SUCCESS (took 3 seconds)
-------------------------------
from-distribution：默认的，是指esrally在官方打好的包下载下来，解压运行 from-sources-complete：支持从源码本地编译、打包再运行，对于ES开发人员有用
benchmark-only：自定义集群时需要使用的pipeline，只进行压测而不去管理ES实例，测试已有集群，也是最常用的。
from-sources-skip-build：与from-sources-complete相互呼应，利用源码方式但是跳过编译、打包
–offline 离线测试
esrally race --offline --track=nested --distribution-version=7.1.0 --test-mode
测试已有集群
https://cloud.tencent.com/developer/article/1595367
esrally race --pipeline=benchmark-only --target-hosts=127.0.0.1:9200 --cluster-health=yellow --track=nested --challenge=append-no-conflicts --test-mode --report-file=result.csv --report-format=csv
Tips:
–cluster-health=yellow 默认 esrally 会检查集群状态，非 green 状态会直接退出。添加该参数可以避免该情况
–challenge=append-no-conflicts 默认 schedule中指定了该 challenge 中按顺序执行 index-append、force-merge、index-stats、node-stats、default、term、range、hourly_agg、scroll 等 9 个task，其中每个 task 都指定了 一个 operation。
–report-file=result.csv 结果存入csv文件
加密：
esrally race --target-hosts=10.5.5.10:9243,10.5.5.11:9243,10.5.5.12:9243 --pipeline=benchmark-only --client-options="use_ssl:true,verify_certs:true,basic_auth_user:'elastic',basic_auth_password:'changeme'"
Tips: esrally 默认在每次压测是会删除已有的索引后再重新创建索引，如果你不想这样，可以在每个 index 的配置中设置
auto-managed 为 false，具体文档在这里。
通过这个参数，你就可以单独压测查询性能了，而不用每次都要先经过漫长的导入数据的过程。
比较每回race数据
[esrally@cdh-slave1 .rally]$ esrally list races____ ____/ __ \____ _/ / /_ __/ /_/ / __ `/ / / / / // _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, //____/Recent races:Race ID Race Timestamp Track Track Parameters Challenge Car User Tags Track Revision Team Revision
------------------------------------ ---------------- ------- ------------------ ----------------------- -------- ----------- ---------------- ---------------
d8a3ab2c-164b-46f3-8fd7-57ac6114d2bc 20210203T025730Z nested nested-search-challenge defaults d463f63 e608ca7
486a8b4a-57d0-48e5-b335-f7d7f621ab71 20210203T024736Z nested nested-search-challenge defaults d463f63 e608ca7
6a4099ba-cffa-45c3-b5ec-7be465fe41c9 20210203T023316Z nested nested-search-challenge defaults--------------------------------
[INFO] SUCCESS (took 44 seconds)
--------------------------------
[esrally@cdh-slave1 .rally]$ esrally compare --baseline=d8a3ab2c-164b-46f3-8fd7-57ac6114d2bc --contender=486a8b4a-57d0-48e5-b335-f7d7f621ab71____ ____/ __ \____ _/ / /_ __/ /_/ / __ `/ / / / / // _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, //____/Comparing baselineRace ID: d8a3ab2c-164b-46f3-8fd7-57ac6114d2bcRace timestamp: 2021-02-03 02:57:30Challenge: nested-search-challengeCar: defaultswith contenderRace ID: 486a8b4a-57d0-48e5-b335-f7d7f621ab71Race timestamp: 2021-02-03 02:47:36Challenge: nested-search-challengeCar: defaults------------------------------------------------------_______ __ _____/ ____(_)___ ____ _/ / / ___/_________ ________/ /_ / / __ \/ __ `/ / \__ \/ ___/ __ \/ ___/ _ \/ __/ / / / / / /_/ / / ___/ / /__/ /_/ / / / __/
/_/ /_/_/ /_/\__,_/_/ /____/\___/\____/_/ \___/
------------------------------------------------------| Metric | Task | Baseline | Contender | Diff | Unit |
|--------------------------------------------------------------:|-----------------------------------------------------------:|------------:|------------:|---------:|-------:|
| Cumulative indexing time of primary shards | | 0 | 0 | 0 | min |
| Cumulative indexing time of primary shards | | 0 | [175/1551] 0 | min |
| Min cumulative indexing time across primary shard | | 0 | 0 | 0 | min |
| Median cumulative indexing time across primary shard | | 0 | 0 | 0 | min |
| Max cumulative indexing time across primary shard | | 0 | 0 | 0 | min |
| Cumulative indexing throttle time of primary shards | | 0 | 0 | 0 | min |
| Min cumulative indexing throttle time across primary shard | | 0 | 0 | 0 | min |
| Median cumulative indexing throttle time across primary shard | | 0 | 0 | 0 | min |
| Max cumulative indexing throttle time across primary shard | | 0 | 0 | 0 | min |
| Cumulative merge time of primary shards | | 0 | 0 | 0 | min |
| Cumulative merge count of primary shards | | 0 | 0 | 0 | |
| Min cumulative merge time across primary shard | | 0 | 0 | 0 | min |
| Median cumulative merge time across primary shard | | 0 | 0 | 0 | min |
| Max cumulative merge time across primary shard | | 0 | 0 | 0 | min |
| Cumulative merge throttle time of primary shards | | 0 | 0 | 0 | min |
| Min cumulative merge throttle time across primary shard | | 0 | 0 | 0 | min |
| Median cumulative merge throttle time across primary shard | | 0 | 0 | 0 | min |
| Max cumulative merge throttle time across primary shard | | 0 | 0 | 0 | min |
| Cumulative refresh time of primary shards | | 0 | 0 | 0 | min |
| Cumulative refresh count of primary shards | | 4 | 4 | 0 | |
| Min cumulative refresh time across primary shard | | 0 | 0 | 0 | | [138/1551]
| Min cumulative refresh time across primary shard | | 0 | 0 | 0 | min |
| Median cumulative refresh time across primary shard | | 0 | 0 | 0 | min |
| Max cumulative refresh time across primary shard | | 0 | 0 | 0 | min |
| Cumulative flush time of primary shards | | 0 | 0 | 0 | min |
| Cumulative flush count of primary shards | | 0 | 0 | 0 | |
| Min cumulative flush time across primary shard | | 0 | 0 | 0 | min |
| Median cumulative flush time across primary shard | | 0 | 0 | 0 | min |
| Max cumulative flush time across primary shard | | 0 | 0 | 0 | min |
| Total Young Gen GC time | | 0.037 | 0.042 | 0.005 | s |
| Total Young Gen GC count | | 2 | 2 | 0 | |
| Total Old Gen GC time | | 0.031 | 0.033 | 0.002 | s |
| Total Old Gen GC count | | 1 | 1 | 0 | |
| Store size | | 2.14204e-07 | 2.14204e-07 | 0 | GB |
| Translog size | | 1.02445e-07 | 1.02445e-07 | 0 | GB |
| Heap used for segments | | 0 | 0 | 0 | MB |
| Heap used for doc values | | 0 | 0 | 0 | MB |
| Heap used for terms | | 0 | 0 | 0 | MB |
| Heap used for norms | | 0 | 0 | 0 | MB |
| Heap used for points | | 0 | 0 | 0 | MB |
| Heap used for points | | 0 | [101/1551] 0 | MB |
| Heap used for stored fields | | 0 | 0 | 0 | MB |
| Segment count | | 0 | 0 | 0 | |
| Min Throughput | index-append | 1999.18 | 2083.69 | 84.5111 | docs/s |
| Median Throughput | index-append | 1999.18 | 2083.69 | 84.5111 | docs/s |
| Max Throughput | index-append | 1999.18 | 2083.69 | 84.5111 | docs/s |
| 50th percentile latency | index-append | 495.615 | 455.084 | -40.5311 | ms |
| 100th percentile latency | index-append | 502.17 | 472.47 | -29.7004 | ms |
| 50th percentile service time | index-append | 495.615 | 455.084 | -40.5311 | ms |
| 100th percentile service time | index-append | 502.17 | 472.47 | -29.7004 | ms |
| error rate | index-append | 100 | 100 | 0 | % |
| Min Throughput | randomized-nested-queries | 82.8584 | 84.1334 | 1.275 | ops/s |
| Median Throughput | randomized-nested-queries | 82.8584 | 84.1334 | 1.275 | ops/s |
| Max Throughput | randomized-nested-queries | 82.8584 | 84.1334 | 1.275 | ops/s |
| 50th percentile latency | randomized-nested-queries | 94.6767 | 92.4562 | -2.22055 | ms |
| 100th percentile latency | randomized-nested-queries | 97.0203 | 94.8462 | -2.17409 | ms |
| 50th percentile service time | randomized-nested-queries | 4.63714 | 4.95024 | 0.3131 | ms |
| 100th percentile service time | randomized-nested-queries | 5.23386 | 5.36543 | 0.13158 | ms |
| error rate | randomized-nested-queries | 0 | 0 | 0 | % |
| Min Throughput | randomized-term-queries | 455.55 | 499.262 | 0 | % | [64/1551]
| Min Throughput | randomized-term-queries | 455.55 | 499.262 | 43.7127 | ops/s |
| Median Throughput | randomized-term-queries | 455.55 | 499.262 | 43.7127 | ops/s |
| Max Throughput | randomized-term-queries | 455.55 | 499.262 | 43.7127 | ops/s |
| 50th percentile latency | randomized-term-queries | 15.9268 | 14.8665 | -1.06028 | ms |
| 100th percentile latency | randomized-term-queries | 17.7799 | 17.2582 | -0.52164 | ms |
| 50th percentile service time | randomized-term-queries | 2.8336 | 2.8089 | -0.0247 | ms |
| 100th percentile service time | randomized-term-queries | 2.89962 | 3.10157 | 0.20195 | ms |
| error rate | randomized-term-queries | 0 | 0 | 0 | % |
| Min Throughput | randomized-sorted-term-queries | 232.031 | 238.16 | 6.12974 | ops/s |
| Median Throughput | randomized-sorted-term-queries | 232.031 | 238.16 | 6.12974 | ops/s |
| Max Throughput | randomized-sorted-term-queries | 232.031 | 238.16 | 6.12974 | ops/s |
| 50th percentile latency | randomized-sorted-term-queries | 33.1484 | 31.329 | -1.81942 | ms |
| 100th percentile latency | randomized-sorted-term-queries | 34.5927 | 33.6738 | -0.91884 | ms |
| 50th percentile service time | randomized-sorted-term-queries | 2.46165 | 2.3668 | -0.09485 | ms |
| 100th percentile service time | randomized-sorted-term-queries | 2.94664 | 2.65031 | -0.29632 | ms |
| error rate | randomized-sorted-term-queries | 0 | 0 | 0 | % |
| Min Throughput | match-all | 406.548 | 410.071 | 3.52263 | ops/s |
| Median Throughput | match-all | 406.548 | 410.071 | 3.52263 | ops/s |
| Max Throughput | match-all | 406.548 | 410.071 | 3.52263 | ops/s |
| Max Throughput | match-all | 406.548 | 41 |3.52263 | ops/s |
| 50th percentile latency | match-all | 18.6788 | 17.6785 | -1.00034 | ms |
| 100th percentile latency | match-all | 20.7883 | 19.1982 | -1.59011 | ms |
| 50th percentile service time | match-all | 2.96051 | 2.57572 | -0.38479 | ms |
| 100th percentile service time | match-all | 3.14092 | 2.7214 | -0.41952 | ms |
| error rate | match-all | 0 | 0 | 0 | % |
| Min Throughput | nested-date-histo | 171.843 | 186.983 | 15.1396 | ops/s |
| Median Throughput | nested-date-histo | 171.843 | 186.983 | 15.1396 | ops/s |
| Max Throughput | nested-date-histo | 171.843 | 186.983 | 15.1396 | ops/s |
| 50th percentile latency | nested-date-histo | 45.1057 | 41.4101 | -3.69562 | ms |
| 100th percentile latency | nested-date-histo | 46.8601 | 44.099 | -2.76112 | ms |
| 50th percentile service time | nested-date-histo | 2.86964 | 3.08447 | 0.21483 | ms |
| 100th percentile service time | nested-date-histo | 3.11078 | 3.24104 | 0.13026 | ms |
| error rate | nested-date-histo | 0 | 0 | 0 | % |
| Min Throughput | randomized-nested-queries-with-inner-hits_default | 298.047 | 280.197 | -17.85 | ops/s |
| Median Throughput | randomized-nested-queries-with-inner-hits_default | 298.047 | 280.197 | -17.85 | ops/s |
| Max Throughput | randomized-nested-queries-with-inner-hits_default | 298.047 | 280.197 | -17.85 | ops/s |
| 50th percentile latency | randomized-nested-queries-with-inner-hits_default | 25.2046 | 27.0777 | 1.87311 | ms |
| 100th percentile latency | randomized-nested-queries-with-inner-hits_default | 27.313 | 29.2122 | 1.89925 | ms |
| 50th percentile service time | randomized-nested-queries-with-inner-hits_default | 2.98297 | 3.00676 | 0 | % | [0/1551]
| Min Throughput | randomized-nested-queries-with-inner-hits_default | 298.047 | 280.197 | -17.85 | ops/s |
| Median Throughput | randomized-nested-queries-with-inner-hits_default | 298.047 | 280.197 | -17.85 | ops/s |
| Max Throughput | randomized-nested-queries-with-inner-hits_default | 298.047 | 280.197 | -17.85 | ops/s |
| 50th percentile latency | randomized-nested-queries-with-inner-hits_default | 25.2046 | 27.0777 | 1.87311 | ms |
| 100th percentile latency | randomized-nested-queries-with-inner-hits_default | 27.313 | 29.2122 | 1.89925 | ms |
| 50th percentile service time | randomized-nested-queries-with-inner-hits_default | 2.98297 | 3.00676 | 0.02379 | ms |
| 100th percentile service time | randomized-nested-queries-with-inner-hits_default | 3.10269 | 3.17378 | 0.07109 | ms |
| error rate | randomized-nested-queries-with-inner-hits_default | 0 | 0 | 0 | % |
| Min Throughput | randomized-nested-queries-with-inner-hits_default_big_size | 449.653 | 480.95 | 31.2972 | ops/s |
| Median Throughput | randomized-nested-queries-with-inner-hits_default_big_size | 449.653 | 480.95 | 31.2972 | ops/s |
| Max Throughput | randomized-nested-queries-with-inner-hits_default_big_size | 449.653 | 480.95 | 31.2972 | ops/s |
| 50th percentile latency | randomized-nested-queries-with-inner-hits_default_big_size | 15.4197 | 15.3532 | -0.06647 | ms |
| 100th percentile latency | randomized-nested-queries-with-inner-hits_default_big_size | 17.452 | 17.1373 | -0.31471 | ms |
| 50th percentile service time | randomized-nested-queries-with-inner-hits_default_big_size | 2.88213 | 2.75255 | -0.12958 | ms |
| 100th percentile service time | randomized-nested-queries-with-inner-hits_default_big_size | 2.93574 | 3.9976 | 1.06186 | ms |
| error rate | randomized-nested-queries-with-inner-hits_default_big_size | 0 | 0 | 0 | % |-------------------------------
[INFO] SUCCESS (took 2 seconds)
-------------------------------
自定义track
https://esrally.readthedocs.io/en/latest/adding_tracks.html
三、结果
–user-tag=“version:2.4.6” 加标签方便对比
- 可以通过日志查看到每次输出的结果报告，路径为~/.rally/logs
- 可以通过json查看到每次输出的结果报告，路径为~/.rally/benchmarks/races
自定义报告
将压测结果保存为文件，并设定文件的格式
–report-file：压测结果导出为文件
–report-format：设定导出文件的格式，具体格式有markdown和csv
esrally --report-file=result.csv --report-format=csv
Metric：具体指标
Task：操作名称
Value：数值
Unit：单位
注意：在Task当中，有空和具体操作名称的区分，如果是空的则表示总计的结果，如果有具体操作名称的则是记录该次操作的结果
Mertic具体指标详解 第一、time：Index | Merge | Refresh | Flush => 对应操作花费的总时间
第二、GC：Young | Total => GC花费的时间
第三、Index size | Total written => 最终索引总大小和实际写入处理的大小
第四、Heap：doc values | terms | norms => 堆内存占用情况
第五、Operation：
Throughput => 当前操作的吞吐情况
Latency => 当前操作的延迟时长（请求到达ES，然后从ES返回的时间，包含了排队的时间）
Service Time => 当前操作的服务时间（ES开始执行请求到结束的时间） Cumulative indexing time of primary
shards：主分片的累计索引时间，用于索引的累积时间
Cumulative indexing time across primary shards：跨主分片的累计索引时间，跨主碎片建立索引的最小、中值和最大累计时间
Cumulative indexing throttle time across primary shards：跨主分片的累计索引节流时间，已限制索引的累积时间
Cumulative merge time of primary shards：主分片的累计合并时间
Cumulative merge count of primary shards：主分片的累计合并个数
Cumulative merge time across primary shards：跨主分片的累计合并时间
Cumulative merge throttle time of primary shards：跨主分片的累计节流时间
Cumulative refresh time of primary shards：主分片的累计刷新时间
Cumulative refresh count of primary shards：主分片的累计刷新次数
Cumulative refresh time across primary shards：跨主分片的累计刷新时间
Cumulative flush time of primary shards：主分片的累计冲洗时间
Cumulative flush count of primary shards：主分片的累计冲洗次数
Cumulative flush time across primary shards：跨主分片的累计冲洗时间
Total Young Gen GC：整个集群中Young Gen GC的总运行时间
Total Old Gen GC：整个集群中Old Gen GC的总运行时间
Store size：索引大小（不包括事务日志），以字节为单位
Translog size：事务日志的大小（以字节为单位）
Index size：索引大小
Total written：总共写入
Heap used for *：堆用于相应项目的字节数
Segment count：细分总数
Throughput：吞吐量
Latency：延迟，提交请求到接收完整响应之间的时间段。它还包括等待时间，即请求花费等待直到准备好由Elasticsearch服务的时间。
Service Time：服务时间，ES处理请求的时间
error rate：错误率，错误响应相对于响应总数的比率。Python Elasticsearch客户端抛出的任何异常都被认为是错误的（例如HTTP响应代码4xx，5xx或网络错误（网络不可达））。
yangning@yangning-VirtualBox:~$ esrally race --offline --track=geonames --distribution-version=6.8.2____ ____/ __ \____ _/ / /_ __/ /_/ / __ `/ / / / / // _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, //____/[INFO] Preparing for race ...
[INFO] Downloading Elasticsearch 6.8.2 (65.3 MB total size) [100%]
[INFO] Decompressing track data from [/home/yangning/.rally/benchmarks/data/geonames/documents-2.json.bz2] to [/home/yangning/.rally/benchmarks/data/geonames/documents-2.json] (resulting size: [3.30] GB) ... [OK]
[INFO] Preparing file offset table for [/home/yangning/.rally/benchmarks/data/geonames/documents-2.json] ... [OK]
[INFO] Racing on track [geonames], challenge [append-no-conflicts] and car ['defaults'] with version [6.8.2].Running delete-index [100% done]
Running create-index [100% done]
Running check-cluster-health [100% done]
Running index-append [100% done]
Running refresh-after-index [100% done]
Running force-merge [100% done]
Running refresh-after-force-merge [100% done]
Running wait-until-merges-finish [100% done]
Running index-stats [100% done]
Running node-stats [100% done]
Running default [100% done]
Running term [100% done]
Running phrase [100% done]
Running country_agg_uncached [100% done]
Running country_agg_cached [100% done]
Running scroll [100% done]
Running expression [100% done]
Running painless_static [100% done]
Running painless_dynamic [100% done]
Running large_terms [100% done]
Running large_filtered_terms [100% done]
Running large_prohibited_terms [100% done]------------------------------------------------------_______ __ _____/ ____(_)___ ____ _/ / / ___/_________ ________/ /_ / / __ \/ __ `/ / \__ \/ ___/ __ \/ ___/ _ \/ __/ / / / / / /_/ / / ___/ / /__/ /_/ / / / __/
/_/ /_/_/ /_/\__,_/_/ /____/\___/\____/_/ \___/
------------------------------------------------------| Metric | Task | Value | Unit |
|---------------------------------------------------------------:|-----------------------:|------------:|--------:|
| Cumulative indexing time of primary shards | | 25.8505 | min |
| Min cumulative indexing time across primary shards | | 4.99313 | min |
| Median cumulative indexing time across primary shards | | 5.11263 | min |
| Max cumulative indexing time across primary shards | | 5.45208 | min |
| Cumulative indexing throttle time of primary shards | | 0 | min |
| Min cumulative indexing throttle time across primary shards | | 0 | min |
| Median cumulative indexing throttle time across primary shards | | 0 | min |
| Max cumulative indexing throttle time across primary shards | | 0 | min |
| Cumulative merge time of primary shards | | 87.8742 | min |
| Cumulative merge count of primary shards | | 672 | |
| Min cumulative merge time across primary shards | | 16.4657 | min |
| Median cumulative merge time across primary shards | | 17.144 | min |
| Max cumulative merge time across primary shards | | 18.8787 | min |
| Cumulative merge throttle time of primary shards | | 1.10953 | min |
| Min cumulative merge throttle time across primary shards | | 0.167183 | min |
| Median cumulative merge throttle time across primary shards | | 0.2269 | min |
| Max cumulative merge throttle time across primary shards | | 0.26805 | min |
| Cumulative refresh time of primary shards | | 11.8221 | min |
| Cumulative refresh count of primary shards | | 6213 | |
| Min cumulative refresh time across primary shards | | 2.23455 | min |
| Median cumulative refresh time across primary shards | | 2.36508 | min |
| Max cumulative refresh time across primary shards | | 2.49413 | min |
| Cumulative flush time of primary shards | | 1.5109 | min |
| Cumulative flush count of primary shards | | 10 | |
| Min cumulative flush time across primary shards | | 0.201467 | min |
| Median cumulative flush time across primary shards | | 0.256433 | min |
| Max cumulative flush time across primary shards | | 0.457683 | min |
| Total Young Gen GC time | | 56.735 | s |
| Total Young Gen GC count | | 4287 | |
| Total Old Gen GC time | | 10.787 | s |
| Total Old Gen GC count | | 43 | |
| Store size | | 2.99647 | GB |
| Translog size | | 2.68815 | GB |
| Heap used for segments | | 18.4159 | MB |
| Heap used for doc values | | 0.0331612 | MB |
| Heap used for terms | | 17.2505 | MB |
| Heap used for norms | | 0.0654907 | MB |
| Heap used for points | | 0.26512 | MB |
| Heap used for stored fields | | 0.801659 | MB |
| Segment count | | 87 | |
| Min Throughput | index-append | 4680.17 | docs/s |
| Median Throughput | index-append | 5328.9 | docs/s |
| Max Throughput | index-append | 5950.68 | docs/s |
| 50th percentile latency | index-append | 5854.25 | ms |
| 90th percentile latency | index-append | 10795.8 | ms |
| 99th percentile latency | index-append | 14857.5 | ms |
| 99.9th percentile latency | index-append | 17891.4 | ms |
| 100th percentile latency | index-append | 18959.8 | ms |
| 50th percentile service time | index-append | 5854.25 | ms |
| 90th percentile service time | index-append | 10795.8 | ms |
| 99th percentile service time | index-append | 14857.5 | ms |
| 99.9th percentile service time | index-append | 17891.4 | ms |
| 100th percentile service time | index-append | 18959.8 | ms |
| error rate | index-append | 0 | % |
| Min Throughput | index-stats | 89.97 | ops/s |
| Median Throughput | index-stats | 90.03 | ops/s |
| Max Throughput | index-stats | 90.08 | ops/s |
| 50th percentile latency | index-stats | 5.98434 | ms |
| 90th percentile latency | index-stats | 9.74006 | ms |
| 99th percentile latency | index-stats | 23.4297 | ms |
| 99.9th percentile latency | index-stats | 35.446 | ms |
| 100th percentile latency | index-stats | 41.5802 | ms |
| 50th percentile service time | index-stats | 3.86547 | ms |
| 90th percentile service time | index-stats | 5.49984 | ms |
| 99th percentile service time | index-stats | 11.1157 | ms |
| 99.9th percentile service time | index-stats | 18.2248 | ms |
| 100th percentile service time | index-stats | 27.0017 | ms |
| error rate | index-stats | 0 | % |
| Min Throughput | node-stats | 89.66 | ops/s |
| Median Throughput | node-stats | 90.06 | ops/s |
| Max Throughput | node-stats | 90.19 | ops/s |
| 50th percentile latency | node-stats | 6.41868 | ms |
| 90th percentile latency | node-stats | 20.9814 | ms |
| 99th percentile latency | node-stats | 401.347 | ms |
| 99.9th percentile latency | node-stats | 445.676 | ms |
| 100th percentile latency | node-stats | 451.803 | ms |
| 50th percentile service time | node-stats | 3.92881 | ms |
| 90th percentile service time | node-stats | 5.53411 | ms |
| 99th percentile service time | node-stats | 10.0591 | ms |
| 99.9th percentile service time | node-stats | 20.6017 | ms |
| 100th percentile service time | node-stats | 24.2616 | ms |
| error rate | node-stats | 0 | % |
| Min Throughput | default | 49.63 | ops/s |
| Median Throughput | default | 50.02 | ops/s |
| Max Throughput | default | 50.03 | ops/s |
| 50th percentile latency | default | 13.0493 | ms |
| 90th percentile latency | default | 17.3717 | ms |
| 99th percentile latency | default | 54.1576 | ms |
| 99.9th percentile latency | default | 108.512 | ms |
| 100th percentile latency | default | 109.845 | ms |
| 50th percentile service time | default | 10.65 | ms |
| 90th percentile service time | default | 13.6786 | ms |
| 99th percentile service time | default | 18.0701 | ms |
| 99.9th percentile service time | default | 20.1515 | ms |
| 100th percentile service time | default | 21.8792 | ms |
| error rate | default | 0 | % |
| Min Throughput | term | 99.97 | ops/s |
| Median Throughput | term | 100.05 | ops/s |
| Max Throughput | term | 100.08 | ops/s |
| 50th percentile latency | term | 5.24344 | ms |
| 90th percentile latency | term | 7.87807 | ms |
| 99th percentile latency | term | 12.8584 | ms |
| 99.9th percentile latency | term | 23.3176 | ms |
| 100th percentile latency | term | 30.1219 | ms |
| 50th percentile service time | term | 3.13181 | ms |
| 90th percentile service time | term | 4.02719 | ms |
| 99th percentile service time | term | 7.54958 | ms |
| 99.9th percentile service time | term | 11.7217 | ms |
| 100th percentile service time | term | 12.7254 | ms |
| error rate | term | 0 | % |
| Min Throughput | phrase | 109.9 | ops/s |
| Median Throughput | phrase | 110.05 | ops/s |
| Max Throughput | phrase | 110.08 | ops/s |
| 50th percentile latency | phrase | 5.0993 | ms |
| 90th percentile latency | phrase | 9.3446 | ms |
| 99th percentile latency | phrase | 29.6138 | ms |
| 99.9th percentile latency | phrase | 40.1349 | ms |
| 100th percentile latency | phrase | 40.5104 | ms |
| 50th percentile service time | phrase | 3.01219 | ms |
| 90th percentile service time | phrase | 4.52979 | ms |
| 99th percentile service time | phrase | 9.46821 | ms |
| 99.9th percentile service time | phrase | 29.843 | ms |
| 100th percentile service time | phrase | 35.0802 | ms |
| error rate | phrase | 0 | % |
| Min Throughput | country_agg_uncached | 1.79 | ops/s |
| Median Throughput | country_agg_uncached | 1.8 | ops/s |
| Max Throughput | country_agg_uncached | 1.82 | ops/s |
| 50th percentile latency | country_agg_uncached | 70227.9 | ms |
| 90th percentile latency | country_agg_uncached | 79653.1 | ms |
| 99th percentile latency | country_agg_uncached | 81779.4 | ms |
| 100th percentile latency | country_agg_uncached | 82026.3 | ms |
| 50th percentile service time | country_agg_uncached | 526.428 | ms |
| 90th percentile service time | country_agg_uncached | 582.676 | ms |
| 99th percentile service time | country_agg_uncached | 685.807 | ms |
| 100th percentile service time | country_agg_uncached | 1006.34 | ms |
| error rate | country_agg_uncached | 0 | % |
| Min Throughput | country_agg_cached | 96.46 | ops/s |
| Median Throughput | country_agg_cached | 100.02 | ops/s |
| Max Throughput | country_agg_cached | 100.05 | ops/s |
| 50th percentile latency | country_agg_cached | 4.08942 | ms |
| 90th percentile latency | country_agg_cached | 10.168 | ms |
| 99th percentile latency | country_agg_cached | 436.561 | ms |
| 99.9th percentile latency | country_agg_cached | 499.803 | ms |
| 100th percentile latency | country_agg_cached | 507.331 | ms |
| 50th percentile service time | country_agg_cached | 2.12405 | ms |
| 90th percentile service time | country_agg_cached | 3.14351 | ms |
| 99th percentile service time | country_agg_cached | 7.70186 | ms |
| 99.9th percentile service time | country_agg_cached | 36.3529 | ms |
| 100th percentile service time | country_agg_cached | 506.219 | ms |
| error rate | country_agg_cached | 0 | % |
| Min Throughput | scroll | 2.97 | pages/s |
| Median Throughput | scroll | 2.98 | pages/s |
| Max Throughput | scroll | 2.98 | pages/s |
| 50th percentile latency | scroll | 1.79326e+06 | ms |
| 90th percentile latency | scroll | 2.0706e+06 | ms |
| 99th percentile latency | scroll | 2.13392e+06 | ms |
| 100th percentile latency | scroll | 2.14113e+06 | ms |
| 50th percentile service time | scroll | 8337.15 | ms |
| 90th percentile service time | scroll | 8685.36 | ms |
| 99th percentile service time | scroll | 8993.97 | ms |
| 100th percentile service time | scroll | 9020.14 | ms |
| error rate | scroll | 0 | % |
| Min Throughput | expression | 0.9 | ops/s |
| Median Throughput | expression | 0.92 | ops/s |
| Max Throughput | expression | 0.92 | ops/s |
| 50th percentile latency | expression | 148849 | ms |
| 90th percentile latency | expression | 171746 | ms |
| 99th percentile latency | expression | 176660 | ms |
| 100th percentile latency | expression | 177193 | ms |
| 50th percentile service time | expression | 1031.74 | ms |
| 90th percentile service time | expression | 1167.51 | ms |
| 99th percentile service time | expression | 1270.27 | ms |
| 100th percentile service time | expression | 1282.5 | ms |
| error rate | expression | 0 | % |
| Min Throughput | painless_static | 0.88 | ops/s |
| Median Throughput | painless_static | 0.89 | ops/s |
| Max Throughput | painless_static | 0.89 | ops/s |
| 50th percentile latency | painless_static | 115945 | ms |
| 90th percentile latency | painless_static | 134898 | ms |
| 99th percentile latency | painless_static | 139326 | ms |
| 100th percentile latency | painless_static | 139700 | ms |
| 50th percentile service time | painless_static | 1124.37 | ms |
| 90th percentile service time | painless_static | 1254.64 | ms |
| 99th percentile service time | painless_static | 1513.41 | ms |
| 100th percentile service time | painless_static | 1554.25 | ms |
| error rate | painless_static | 0 | % |
| Min Throughput | painless_dynamic | 0.86 | ops/s |
| Median Throughput | painless_dynamic | 0.89 | ops/s |
| Max Throughput | painless_dynamic | 0.91 | ops/s |
| 50th percentile latency | painless_dynamic | 114057 | ms |
| 90th percentile latency | painless_dynamic | 142725 | ms |
| 99th percentile latency | painless_dynamic | 146948 | ms |
| 100th percentile latency | painless_dynamic | 147415 | ms |
| 50th percentile service time | painless_dynamic | 1151.78 | ms |
| 90th percentile service time | painless_dynamic | 1584.52 | ms |
| 99th percentile service time | painless_dynamic | 2168.97 | ms |
| 100th percentile service time | painless_dynamic | 2273.89 | ms |
| error rate | painless_dynamic | 0 | % |
| Min Throughput | large_terms | 1.1 | ops/s |
| Median Throughput | large_terms | 1.1 | ops/s |
| Max Throughput | large_terms | 1.1 | ops/s |
| 50th percentile latency | large_terms | 664.162 | ms |
| 90th percentile latency | large_terms | 779.05 | ms |
| 99th percentile latency | large_terms | 862.6 | ms |
| 100th percentile latency | large_terms | 870.042 | ms |
| 50th percentile service time | large_terms | 644.53 | ms |
| 90th percentile service time | large_terms | 758.667 | ms |
| 99th percentile service time | large_terms | 843.01 | ms |
| 100th percentile service time | large_terms | 851.3 | ms |
| error rate | large_terms | 0 | % |
| Min Throughput | large_filtered_terms | 1.1 | ops/s |
| Median Throughput | large_filtered_terms | 1.1 | ops/s |
| Max Throughput | large_filtered_terms | 1.1 | ops/s |
| 50th percentile latency | large_filtered_terms | 694.78 | ms |
| 90th percentile latency | large_filtered_terms | 786.346 | ms |
| 99th percentile latency | large_filtered_terms | 838.045 | ms |
| 100th percentile latency | large_filtered_terms | 859.325 | ms |
| 50th percentile service time | large_filtered_terms | 673.595 | ms |
| 90th percentile service time | large_filtered_terms | 763.597 | ms |
| 99th percentile service time | large_filtered_terms | 821.495 | ms |
| 100th percentile service time | large_filtered_terms | 843.153 | ms |
| error rate | large_filtered_terms | 0 | % |
| Min Throughput | large_prohibited_terms | 1.1 | ops/s |
| Median Throughput | large_prohibited_terms | 1.1 | ops/s |
| Max Throughput | large_prohibited_terms | 1.1 | ops/s |
| 50th percentile latency | large_prohibited_terms | 672.762 | ms |
| 90th percentile latency | large_prohibited_terms | 754.787 | ms |
| 99th percentile latency | large_prohibited_terms | 792.064 | ms |
| 100th percentile latency | large_prohibited_terms | 801.729 | ms |
| 50th percentile service time | large_prohibited_terms | 648.942 | ms |
| 90th percentile service time | large_prohibited_terms | 734.24 | ms |
| 99th percentile service time | large_prohibited_terms | 772.429 | ms |
| 100th percentile service time | large_prohibited_terms | 785.69 | ms |
| error rate | large_prohibited_terms | 0 | % |----------------------------------
[INFO] SUCCESS (took 7219 seconds)
----------------------------------Elasticsearch压测工具esrally使用笔记相关推荐
- Elasticsearch压测工具Rally安装
一.硬件要求 在负载生成器机器上使用SSD,如果运行批量索引基准测试,Rally会从磁盘上读取一个或多个数据文件.通常会配置多个客户端,每个客户端读取数据文件的一部分.这种随机访问模式对于旋转磁盘来说 ...
- Elasticsearch - 压测方案之 esrally 简介
文章目录 为什么要压测? 如何进行压测? 入门 简介 快速入门 相关术语 track car race Tournament Pipeline 压测流程 压测结果分析 实战 问题一 问题二 问题三 进 ...
- Elasticsearch 压测方案之 es rally 简介
rally文档:http://esrally.readthedocs.io/en/latest/quickstart.html 由于 Elasticsearch(后文简称es) 的简单易用及其在大数据 ...
- 基于python的压测工具_Python Locust性能测试简介及框架实践
Locust(俗称 蝗虫), 一个轻量级的开源压测工具,用Python编写.使用 Python 代码定义用户行为,也可以仿真百万个用户: Locust 非常简单易用,是分布式,用户负载测试工具.Loc ...
- Http压测工具wrk使用指南【转】
用过了很多压测工具,却一直没找到中意的那款.最近试了wrk感觉不错,写下这份使用指南给自己备忘用,如果能帮到你,那也很好. 安装 wrk支持大多数类UNIX系统,不支持windows.需要操作系统支持 ...
- Http压测工具wrk使用指南
用过了很多压测工具,却一直没找到中意的那款.最近试了wrk感觉不错,写下这份使用指南给自己备忘用,如果能帮到你,那也很好. 安装 wrk支持大多数类UNIX系统,不支持windows.需要操作系统支持 ...
- 精准容量、秒级弹性，压测工具 + SAE 方案如何成功突破传统大促难关？
作者 | 代序 阿里云云原生技术团队 本文整理自<Serverless 技术公开课>,"Serverless"公众号后台回复"入门",即可获取系列文 ...
- python服务端多进程压测工具
本文描述一个python实现的多进程压测工具,这个压测工具的特点如下: 多进程 在大多数情况下,压测一般适用于IO密集型场景(如访问接口并等待返回),在这种场景下多线程多进程的区分并不明显(详情请参见 ...
- jmeter constant timer 如何添加_阿里巴巴在开源压测工具 JMeter 上的实践和优化
Apache JMeter 是 Apache 旗下的开源压测工具,创建于 1999 年初,迄今已有超过 20 年历史.JMeter 功能丰富,社区(用户群体)庞大,是主流开源压测工具之一. 性能测试通 ...
最新文章
- Vimium - 让Chrome高效工作的神器
- python自动访问网页_Python自动化 selenium 网页自动化-在访问一个网页时弹出的浏览器窗口，我该如何处理？...
- I2C原理及应用实例
- 阿德莱德大学计算机科学学士收费,2017阿德莱德大学研究生学费
- 关于document.cookie的使用
- Debug程序的使用
- Google Fuchsia 对中国操作系统的启示 | 畅言
- Spring Data + Thymeleaf 3 + Bootstrap 4 实现分页器
- linux多进程子进程继承,Linux-fork调用后，父进程的线程是否会被子进程继承？
- 泛微为芯片行业搭建数字化平台：研发、生产、销售、文档一体管理
- Android设置图标背景透明
- linux下搭建DNS域名解析服务器
- 【Unity3d Shader】景深效果
- L1-012 计算指数 (5 分) C语言
- 【学习总结】企业信息化管理之数据管理发展与探索之路
- 计算机组成原理3.6高速缓冲存储器Cache
- 基于Android的智慧校园的设计与实现
- 西门子S7-1500PLC大型程序，各种FB块PTO控制20多个轴，5台S7-1200PLC智能IO通讯，ModbusRTU通讯轮询
- 百度一下，你就知道.1
- 前端架构，前端工程化
热门文章
- MacPorts卡在正在运行软件包脚本解决方法
- 梦幻西游只知道服务器怎么找回,买回八年前的梦幻号，只为找回当年的美好
- uni的numberbox怎么用_模拟器最强BIOS篇，如何使用uni-bios
- 分享个龙门神途脚本辅助工具，相当于一个云端的安卓模拟器
- ionic知识系列：Could not remove dir ‘/data/data/io.ionic.starter/code_cache/.ll/‘
- EasyX教程（一）
- 支付宝支付，当面付二维码支付
- windows10 家庭版 Hyper-V 虚拟机服务开启与应用 （一）
- PS图片无法保存ICO格式解决方法
- 从 DP 到 DDP 到 apex