M102: MongoDB for DBAs chapter 3 performance学习记录

运行环境

操作系统：windows 10 家庭中文版
Mongodb ：Mongodb 3.4

Mongodb安装路径：E:>MongoDB\Server\3.4\bin\
Mongodb存储路径：E:>MongoDB\data

课后习题

3.1题目

Download Handouts:

performance__m102_performance_reorg_537a56c08bb48b7c467a20d3.zip

Start a mongod server instance (if you still have a replica set, that would work too).

Next, download the handout and run:

mongo --shell localhost/performance performance.jshomework.init()

Build an index on the “active” and “tstamp” fields. You can verify that you’ve done your job with

db.sensor_readings.getIndexes()

When you are done, run:

homework.a()

and enter the numeric result below (no spaces).

Note: if you would like to try different indexes, you can use db.sensor_readings.dropIndexes() to drop your old index before creating a new one. (For this problem you will only need one index beyond the _id index which is present by default.)

解答

下载指定的performance__m102_performance_reorg_537a56c08bb48b7c467a20d3.zip文件，解压后放在E:\MongoDB目录下，启动我的mongod服务：

C:\Users\Shinelon>e:
E:\>MongoDB\Server\3.4\bin\mongod.exe --dbpath MongoDB\data

按要求通过js脚本启动mongo数据库

E:\>MongoDB\Server\3.4\bin\mongo.exe --shell localhost/performance MongoDB\performance.js
MongoDB shell version v3.4.6
connecting to: mongodb://localhost/performance
MongoDB server version: 3.4.6
type "help" for help
Server has startup warnings:
2018-04-08T20:18:31.157-0700 I CONTROL  [initandlisten]
2018-04-08T20:18:31.157-0700 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2018-04-08T20:18:31.157-0700 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2018-04-08T20:18:31.158-0700 I CONTROL  [initandlisten]
>

初始化homework脚本：

> homework.init()
{"connectionId" : 2,"n" : 0,"syncMillis" : 0,"writtenTo" : null,"err" : null,"ok" : 1
}
still working...
{"connectionId" : 2,"updatedExisting" : true,"n" : 20000,"syncMillis" : 0,"writtenTo" : null,"err" : null,"ok" : 1
}
count: 20000

先查看sensor_readings集合中索引情况：

> db.sensor_readings.getIndexes()
[{"v" : 2,"key" : {"_id" : 1},"name" : "_id_","ns" : "performance.sensor_readings"}
]

结果显示只在_id字段上有一个默认的索引，下面按题目要求在”active” 和”tstamp”字段创建一个复合索引：

> db.sensor_readings.ensureIndex({"active" : 1 , "tstamp" : 1})
{"createdCollectionAutomatically" : false,"numIndexesBefore" : 1,"numIndexesAfter" : 2,"ok" : 1
}

再次查看集合中索引状态：

> db.sensor_readings.getIndexes()
[{"v" : 2,"key" : {"_id" : 1},"name" : "_id_","ns" : "performance.sensor_readings"},{"v" : 2,"key" : {"active" : 1,"tstamp" : 1},"name" : "active_1_tstamp_1","ns" : "performance.sensor_readings"}
]

发现所要求索引已经生成，下面执行homework脚本生成答案：

> homework.a()
6

3.2题目

For this homework, you will need to use version 3.2 or later of MongoDB.

In a mongo shell run homework.b(). This will run in an infinite loop printing some output as it runs various statements against the server.

We’ll now imagine that on this system a user has complained of slowness and we suspect there is a slow operation running. Find the slow operation and terminate it.

In order to do this, you’ll want to open a second window (or tab) and there, run a second instance of the mongo shell, with something like:

$ mongo --shell localhost/performance performance.js

Keep the other shell with homework.b() going while this is happening. Once you have eliminated the slow operation, run (on your second tab):

homework.c()

and enter the output below. Once you have it right and are ready to move on, ctrl-c (terminate) the shell that is still running the homework.b() function.

解答

按题目要求在mongo中执行homework脚本：

> homework.b()
simulating a workload in this shell. after completing this homework, you can
stop this shell with ctrl-c. let it run in the meantime...
(you will want to open another mongo shell to do your work in.)note: this function changes the indexes on db.sensor_readings. so if you goback to the previous homework, drop and recreate indexes (or run homework.init() again)info: 1 operations were in progress when homework.b() begins.if you have a replica set more than zero is normal as the replicas willquery the primary.looping...
connecting to: mongodb://localhost/performance
connecting to: mongodb://localhost/performance
MongoDB server version: 3.4.6
MongoDB server version: 3.4.6

此时需要再开启一个mongo服务：

E:\>MongoDB\Server\3.4\bin\mongo.exe --shell localhost/performance MongoDB\performance.js
MongoDB shell version v3.4.6
connecting to: mongodb://localhost/performance
MongoDB server version: 3.4.6
type "help" for help
Server has startup warnings:
2018-04-08T20:18:31.157-0700 I CONTROL  [initandlisten]
2018-04-08T20:18:31.157-0700 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2018-04-08T20:18:31.157-0700 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2018-04-08T20:18:31.158-0700 I CONTROL  [initandlisten]

在新的mongo端执行homework脚本，但我执行的时候生成的数字会变化：

> homework.c()
12
> homework.c()
12
> homework.c()
8
> homework.c()
9
> homework.c()
9

强制终止homework.b()：

do you want to kill the current op(s) on the server? (y/n): y
2018-04-09T11:54:01.243+0800 E QUERY    [thread1] TypeError: err.hasWriteErrors is not a function :
DBCollection.prototype.updateMany@src/mongo/shell/crud_api.js:629:13
homework.b@MongoDB\performance.js:124:1
@(shell):1:1
> 2018-04-09T11:54:01.243+0800 I CONTROL  [thread4] shutting down with code:0

再次执行homework.c():

> homework.c()
not seeing the expected activity, is homework.b() really running in a separate shell? try again.

正确答案是12，但我实验时结果在变化，混对的

3.3题目

Download Handouts:

products__homework_m102_week3_5518c233d8ca39395277dfbf.zip

Download and extract the json file in products.zip

Then perform the following in the terminal (or at the command prompt):

mongoimport --drop -d pcat -c products products.json

If that looks somewhat familiar, that’s because it’s (nearly) the same command you used to import the pcat.products collection for Homework 2.1, with the only difference in the command being that it will drop the collection if it’s already present. This version of the collection, however, contains the state of the collection as it would exist once you’ve solved all of the homework of chapter 2.

Next, go into the pcat database.

mongo pcat

Create an index on the products collection for the field, “for”.

After creating the index, do a find() for products that work with an “ac3” phone (“ac3” is present in the “for” field).

Q1: How many products match this query?
Q2: Run the same query, but this time do an explain(). How many documents were examined?
Q3: Does the explain() output indicate that an index was used?

Check all that apply:

解答

按题目要求下载products__homework_m102_week3_5518c233d8ca39395277dfbf.zip文件并解压到E:\MongoDB路径下，启动mongod服务（我在上个实验把它关了）：

E:\>MongoDB\Server\3.4\bin\mongod.exe --dbpath MongoDB\data

按题目要求导入products.json文件，它将删除老的pcat库，并新建一个pcat库：

E:\>MongoDB\Server\3.4\bin\mongoimport.exe --drop -d pcat -c products MongoDB\products.json
2018-04-09T14:28:39.570+0800    connected to: localhost
2018-04-09T14:28:39.601+0800    dropping: pcat.products
2018-04-09T14:28:39.748+0800    imported 12 documents

进入新建的pcat库：

E:\>MongoDB\Server\3.4\bin\mongo.exe pcat
MongoDB shell version v3.4.6
connecting to: mongodb://127.0.0.1:27017/pcat
MongoDB server version: 3.4.6
Server has startup warnings:
2018-04-08T23:28:33.383-0700 I CONTROL  [initandlisten]
2018-04-08T23:28:33.384-0700 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2018-04-08T23:28:33.384-0700 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2018-04-08T23:28:33.385-0700 I CONTROL  [initandlisten]
>

按要求在products库的for字段创建索引：

> db.products.ensureIndex({"for" : 1})
{"createdCollectionAutomatically" : false,"numIndexesBefore" : 1,"numIndexesAfter" : 2,"ok" : 1
}

执行查询操作，查看for字段为”ac3”的数据：

> db.products.find({"for" : "ac3"})
{ "_id" : ObjectId("507d95d5719dbef170f15bfd"), "available" : true, "color" : "red", "for" : "ac3", "name" : "AC3 Case Red", "price" : 12, "type" : [ "accessory", "case" ], "warranty_years" : 0.25 }
{ "_id" : ObjectId("507d95d5719dbef170f15bfb"), "for" : [ "ac3", "ac7", "ac9", "qp7", "qp8", "qp9" ], "name" : "Phone Extended Warranty", "price" : 38, "type" : "warranty", "warranty_years" : 2 }
{ "_id" : ObjectId("507d95d5719dbef170f15bf9"), "for" : [ "ac3", "ac7", "ac9" ], "name" : "AC3 Series Charger", "price" : 19, "type" : [ "accessory", "charger" ], "warranty_years" : 0.25 }
{ "_id" : ObjectId("507d95d5719dbef170f15bfc"), "available" : false, "color" : "black", "for" : "ac3", "name" : "AC3 Case Black", "price" : 12.5, "type" : [ "accessory", "case" ], "warranty_years" : 0.25 }

可以数出来是4个，再添加explain()，查看结果：

> db.products.find({"for" : "ac3"}).explain()
{"queryPlanner" : {"plannerVersion" : 1,"namespace" : "pcat.products","indexFilterSet" : false,"parsedQuery" : {"for" : {"$eq" : "ac3"}},"winningPlan" : {"stage" : "FETCH","inputStage" : {"stage" : "IXSCAN","keyPattern" : {"for" : 1},"indexName" : "for_1","isMultiKey" : true,"multiKeyPaths" : {"for" : ["for"]},"isUnique" : false,"isSparse" : false,"isPartial" : false,"indexVersion" : 2,"direction" : "forward","indexBounds" : {"for" : ["[\"ac3\", \"ac3\"]"]}}},"rejectedPlans" : [ ]},"serverInfo" : {"host" : "DESKTOP-MP9NVQ7","port" : 27017,"version" : "3.4.6","gitVersion" : "c55eb86ef46ee7aede3b1e2a5d184a7df4bfb5b5"},"ok" : 1
}

观察到explain.queryPlanner.indexFilterSet的属性是false，说明没使用索引来过滤，那么执行计划审查的文件数应该跟查询总数一致，为4

观察到explain.queryPlanner.winningPlan.inputStage.stage的属性是”IXSCAN”，即index scanning，说明使用了索引。
含义如下：

 COLLSCAN 全表扫描        IXSCAN 索引扫描          FETCH 根据索引去检索文档 SHARD_MERGE 合并分片结果

3.4题目

Which of the following are available in WiredTiger but not in MMAPv1? Check all that apply.

Check all that apply:

Document level locking
Data compression
Indexes
Collection level locking
Covered Queries

解答

根据官方文档的描述：

MMAPv1介绍：

MMAPv1 is MongoDB’s original storage engine based on memory mapped files. It excels at workloads with high volume inserts, reads, and in-place updates.

而自从3.2版本开始，mongodb的默认存储引擎从MMAPv1变为了WiredTiger

而WiredTiger的优势有：

Document Level Concurrency(文档级并发性)

WiredTiger针对写入操作使用文档级并发控制。因此，多个客户端可以同时修改集合中的不同文档。

对于大多数读写操作，WiredTiger使用乐观并发控制。WiredTiger仅在全局，数据库和集合级别使用意向锁。当存储引擎检测到两个操作之间发生冲突时，会引发写冲突，导致MongoDB透明地重试该操作。

一些全球性操作（通常涉及多个数据库的短期操作）仍需要全局“实例范围”锁定。其他一些操作（如删除集合）仍然需要独占数据库锁定。
Snapshots and Checkpoints（快照和检查点）

在写入磁盘时，WiredTiger将所有数据文件中的所有数据以快照方式写入磁盘。现在持久的数据充当数据文件中的检查点。检查点确保数据文件在最后一个检查点之前保持一致;即检查点可以充当恢复点。

在版本3.6中更改：MongoDB配置WiredTiger以60秒为间隔创建检查点（即将快照数据写入磁盘）。

当WiredTiger的元数据表被原子更新为引用新的检查点时，新的检查点变得可访问且永久。一旦可以访问新的检查点，WiredTiger将从旧检查点释放页面。

使用WiredTiger，即使没有日志记录，MongoDB也可以从上一个检查点恢复;但是，要恢复上次检查点后所做的更改，请使用日记功能运行。
Journal（日志）

WiredTiger使用提前写入事务日志与检查点相结合来确保数据持久性。

WiredTiger日志使用快速压缩库进行压缩。要指定备用压缩算法或不压缩，请使用storage.wiredTiger.engineConfig.journalCompressor设置。
Compression（压缩）

借助WiredTiger，MongoDB支持所有集合和索引的压缩。压缩最大限度地减少存储使用，但需要增加额外的CPU。

默认情况下，WiredTiger使用快速压缩库对所有集合进行块压缩，并对所有索引进行前缀压缩。

对于集合，也可以使用zlib进行块压缩。要指定备用压缩算法或不压缩，请使用storage.wiredTiger.collectionConfig.blockCompressor设置。

对于索引，要禁用前缀压缩，请使用storage.wiredTiger.indexConfig.prefixCompression设置。

在收集和索引创建过程中，压缩设置也可以按收集和按索引进行配置。请参阅指定存储引擎选项和db.collection.createIndex（）storageEngine选项。

对于大多数工作负载，默认的压缩设置可以平衡存储效率和处理需求。
Memory Use（内存使用)

借助WiredTiger，MongoDB既利用WiredTiger内部缓存，又利用文件系统缓存。

从3.4开始，默认情况下，WiredTiger内部缓存将使用以下两者中的较大者：
```
（RAM - 1 GB）的50％，或256 MB
```
默认情况下，WiredTiger对所有索引使用Snappy块压缩，对所有索引使用前缀压缩。压缩默认值可以在全局级别配置，也可以在收集和索引创建期间按照每个集合和每个索引进行设置。

WiredTiger内部缓存中的数据与磁盘格式中的数据使用不同的表示形式：
- 文件系统缓存中的数据与磁盘上的格式相同，包括对数据文件进行任何压缩的好处。操作系统使用文件系统缓存来减少磁盘I / O。
- 加载到WiredTiger内部缓存中的索引具有与磁盘格式不同的数据表示形式，但仍可以利用索引前缀压缩来降低RAM使用率。索引前缀压缩可从索引字段中删除重复共享前缀。
- WiredTiger内部缓存中的收集数据是未压缩的，并使用磁盘格式中的不同表示。块压缩可以提供显着的磁盘存储节省，但是数据必须是未压缩的才能被服务器操纵。
通过文件系统缓存，MongoDB自动使用WiredTiger缓存或其他进程未使用的所有空闲内存。

所以，满足条件的是：

Document level locking 文档级锁
Data compression 数据压缩