PostgreSQL Heap Only Tuple - HOT (降低UPDATE引入的索引写IO放大)
2024-04-05 17:06:28
标签
PostgreSQL , Heap Only Tuple , HOT
背景
PostgreSQL目前默认的存储引擎在更新记录时，会在堆内产生一条新版本，旧版本在不需要使用后VACUUM回收，回收旧版本前，需要先回收所有关联这个版本的所有索引POINT。
PG的索引的KEY为索引字段或表达式的值，VALUE为行号。
8.3以前，每个TUPLE版本（行号）都有对应的索引POINT，因此更新的放大比较大。
8.3开始，引入了HOT的概念，当更新记录时，如果能满足两个条件时，通过HEAP PAGE内部LINK来串起所有TUPLE版本，因此索引不变。
HOT必须满足如下两个条件：
Necessary Condition A: UPDATE does not change any of the index keys Necessary Condition B: The new version should fit in the same old block – HOT chains can not cross block boundary. 1、索引字段的值不变。(其中任意一个索引字段的值发生了变化，则所有索引都需要新增版本) 2、新的版本与旧的版本在同一个HEAP PAGE中。
10以后，可以使用二级索引来解决更新引入的索引放大问题：
《PostgreSQL 10.0 preview 性能增强 - 间接索引(secondary index)》
HOT 实例解说
1、创建测试表，写入10条测试数据
postgres=# create table a(id int, c1 int, c2 int, c3 int);
CREATE TABLE
postgres=# insert into a select generate_series(1,10), random()*100, random()*100, random()*100;
INSERT 0 10
2、创建索引
postgres=# create index idx_a_1 on a (id);
CREATE INDEX
postgres=# create index idx_a_2 on a (c1);
CREATE INDEX
postgres=# create index idx_a_3 on a (c2);
CREATE INDEX
3、通过pageinspect插件观察索引页内容
postgres=# SELECT * FROM bt_metap('idx_a_1'); magic | version | root | level | fastroot | fastlevel
--------+---------+------+-------+----------+----------- 340322 | 2 | 1 | 0 | 1 | 0
(1 row) postgres=# SELECT * FROM bt_metap('idx_a_2'); magic | version | root | level | fastroot | fastlevel
--------+---------+------+-------+----------+----------- 340322 | 2 | 1 | 0 | 1 | 0
(1 row) postgres=# SELECT * FROM bt_metap('idx_a_3'); magic | version | root | level | fastroot | fastlevel
--------+---------+------+-------+----------+----------- 340322 | 2 | 1 | 0 | 1 | 0
(1 row)
postgres=# SELECT * FROM bt_page_items('idx_a_1',1); itemoffset | ctid | itemlen | nulls | vars | data
------------+--------+---------+-------+------+------------------------- 1 | (0,1) | 16 | f | f | 01 00 00 00 00 00 00 00 2 | (0,2) | 16 | f | f | 02 00 00 00 00 00 00 00 3 | (0,3) | 16 | f | f | 03 00 00 00 00 00 00 00 4 | (0,4) | 16 | f | f | 04 00 00 00 00 00 00 00 5 | (0,5) | 16 | f | f | 05 00 00 00 00 00 00 00 6 | (0,6) | 16 | f | f | 06 00 00 00 00 00 00 00 7 | (0,7) | 16 | f | f | 07 00 00 00 00 00 00 00 8 | (0,8) | 16 | f | f | 08 00 00 00 00 00 00 00 9 | (0,9) | 16 | f | f | 09 00 00 00 00 00 00 00 10 | (0,10) | 16 | f | f | 0a 00 00 00 00 00 00 00
(10 rows) postgres=# SELECT * FROM bt_page_items('idx_a_2',1); itemoffset | ctid | itemlen | nulls | vars | data
------------+--------+---------+-------+------+------------------------- 1 | (0,5) | 16 | f | f | 05 00 00 00 00 00 00 00 2 | (0,7) | 16 | f | f | 06 00 00 00 00 00 00 00 3 | (0,1) | 16 | f | f | 0d 00 00 00 00 00 00 00 4 | (0,9) | 16 | f | f | 0e 00 00 00 00 00 00 00 5 | (0,4) | 16 | f | f | 20 00 00 00 00 00 00 00 6 | (0,3) | 16 | f | f | 24 00 00 00 00 00 00 00 7 | (0,6) | 16 | f | f | 28 00 00 00 00 00 00 00 8 | (0,10) | 16 | f | f | 2a 00 00 00 00 00 00 00 9 | (0,2) | 16 | f | f | 3f 00 00 00 00 00 00 00 10 | (0,8) | 16 | f | f | 55 00 00 00 00 00 00 00
(10 rows) postgres=# SELECT * FROM bt_page_items('idx_a_3',1); itemoffset | ctid | itemlen | nulls | vars | data
------------+--------+---------+-------+------+------------------------- 1 | (0,6) | 16 | f | f | 09 00 00 00 00 00 00 00 2 | (0,5) | 16 | f | f | 12 00 00 00 00 00 00 00 3 | (0,8) | 16 | f | f | 13 00 00 00 00 00 00 00 4 | (0,1) | 16 | f | f | 14 00 00 00 00 00 00 00 5 | (0,4) | 16 | f | f | 22 00 00 00 00 00 00 00 6 | (0,10) | 16 | f | f | 2b 00 00 00 00 00 00 00 7 | (0,3) | 16 | f | f | 30 00 00 00 00 00 00 00 8 | (0,9) | 16 | f | f | 33 00 00 00 00 00 00 00 9 | (0,7) | 16 | f | f | 46 00 00 00 00 00 00 00 10 | (0,2) | 16 | f | f | 4b 00 00 00 00 00 00 00
(10 rows)
4、HOT更新，（更新的字段上没有索引，并且新的版本记录在同一个HEAP PAGE上）
postgres=# update a set c3=c3+1 where id=1;
UPDATE 1
postgres=# SELECT * FROM bt_page_items('idx_a_1',1); itemoffset | ctid | itemlen | nulls | vars | data
------------+--------+---------+-------+------+------------------------- 1 | (0,1) | 16 | f | f | 01 00 00 00 00 00 00 00 2 | (0,2) | 16 | f | f | 02 00 00 00 00 00 00 00 3 | (0,3) | 16 | f | f | 03 00 00 00 00 00 00 00 4 | (0,4) | 16 | f | f | 04 00 00 00 00 00 00 00 5 | (0,5) | 16 | f | f | 05 00 00 00 00 00 00 00 6 | (0,6) | 16 | f | f | 06 00 00 00 00 00 00 00 7 | (0,7) | 16 | f | f | 07 00 00 00 00 00 00 00 8 | (0,8) | 16 | f | f | 08 00 00 00 00 00 00 00 9 | (0,9) | 16 | f | f | 09 00 00 00 00 00 00 00 10 | (0,10) | 16 | f | f | 0a 00 00 00 00 00 00 00
(10 rows) postgres=# SELECT * FROM bt_page_items('idx_a_2',1); itemoffset | ctid | itemlen | nulls | vars | data
------------+--------+---------+-------+------+------------------------- 1 | (0,5) | 16 | f | f | 05 00 00 00 00 00 00 00 2 | (0,7) | 16 | f | f | 06 00 00 00 00 00 00 00 3 | (0,1) | 16 | f | f | 0d 00 00 00 00 00 00 00 4 | (0,9) | 16 | f | f | 0e 00 00 00 00 00 00 00 5 | (0,4) | 16 | f | f | 20 00 00 00 00 00 00 00 6 | (0,3) | 16 | f | f | 24 00 00 00 00 00 00 00 7 | (0,6) | 16 | f | f | 28 00 00 00 00 00 00 00 8 | (0,10) | 16 | f | f | 2a 00 00 00 00 00 00 00 9 | (0,2) | 16 | f | f | 3f 00 00 00 00 00 00 00 10 | (0,8) | 16 | f | f | 55 00 00 00 00 00 00 00
(10 rows) postgres=# SELECT * FROM bt_page_items('idx_a_3',1); itemoffset | ctid | itemlen | nulls | vars | data
------------+--------+---------+-------+------+------------------------- 1 | (0,6) | 16 | f | f | 09 00 00 00 00 00 00 00 2 | (0,5) | 16 | f | f | 12 00 00 00 00 00 00 00 3 | (0,8) | 16 | f | f | 13 00 00 00 00 00 00 00 4 | (0,1) | 16 | f | f | 14 00 00 00 00 00 00 00 5 | (0,4) | 16 | f | f | 22 00 00 00 00 00 00 00 6 | (0,10) | 16 | f | f | 2b 00 00 00 00 00 00 00 7 | (0,3) | 16 | f | f | 30 00 00 00 00 00 00 00 8 | (0,9) | 16 | f | f | 33 00 00 00 00 00 00 00 9 | (0,7) | 16 | f | f | 46 00 00 00 00 00 00 00 10 | (0,2) | 16 | f | f | 4b 00 00 00 00 00 00 00
(10 rows)
postgres=# update a set c3=c3 where id=1 returning ctid,*; ctid | id | c1 | c2 | c3
--------+----+----+----+---- (0,13) | 1 | 13 | 20 | 15
(1 row) UPDATE 1
postgres=# SELECT * FROM bt_page_items('idx_a_1',1); itemoffset | ctid | itemlen | nulls | vars | data
------------+--------+---------+-------+------+------------------------- 1 | (0,1) | 16 | f | f | 01 00 00 00 00 00 00 00 2 | (0,2) | 16 | f | f | 02 00 00 00 00 00 00 00 3 | (0,3) | 16 | f | f | 03 00 00 00 00 00 00 00 4 | (0,4) | 16 | f | f | 04 00 00 00 00 00 00 00 5 | (0,5) | 16 | f | f | 05 00 00 00 00 00 00 00 6 | (0,6) | 16 | f | f | 06 00 00 00 00 00 00 00 7 | (0,7) | 16 | f | f | 07 00 00 00 00 00 00 00 8 | (0,8) | 16 | f | f | 08 00 00 00 00 00 00 00 9 | (0,9) | 16 | f | f | 09 00 00 00 00 00 00 00 10 | (0,10) | 16 | f | f | 0a 00 00 00 00 00 00 00
(10 rows) postgres=# SELECT * FROM bt_page_items('idx_a_2',1); itemoffset | ctid | itemlen | nulls | vars | data
------------+--------+---------+-------+------+------------------------- 1 | (0,5) | 16 | f | f | 05 00 00 00 00 00 00 00 2 | (0,7) | 16 | f | f | 06 00 00 00 00 00 00 00 3 | (0,1) | 16 | f | f | 0d 00 00 00 00 00 00 00 4 | (0,9) | 16 | f | f | 0e 00 00 00 00 00 00 00 5 | (0,4) | 16 | f | f | 20 00 00 00 00 00 00 00 6 | (0,3) | 16 | f | f | 24 00 00 00 00 00 00 00 7 | (0,6) | 16 | f | f | 28 00 00 00 00 00 00 00 8 | (0,10) | 16 | f | f | 2a 00 00 00 00 00 00 00 9 | (0,2) | 16 | f | f | 3f 00 00 00 00 00 00 00 10 | (0,8) | 16 | f | f | 55 00 00 00 00 00 00 00
(10 rows) postgres=# SELECT * FROM bt_page_items('idx_a_3',1); itemoffset | ctid | itemlen | nulls | vars | data
------------+--------+---------+-------+------+------------------------- 1 | (0,6) | 16 | f | f | 09 00 00 00 00 00 00 00 2 | (0,5) | 16 | f | f | 12 00 00 00 00 00 00 00 3 | (0,8) | 16 | f | f | 13 00 00 00 00 00 00 00 4 | (0,1) | 16 | f | f | 14 00 00 00 00 00 00 00 5 | (0,4) | 16 | f | f | 22 00 00 00 00 00 00 00 6 | (0,10) | 16 | f | f | 2b 00 00 00 00 00 00 00 7 | (0,3) | 16 | f | f | 30 00 00 00 00 00 00 00 8 | (0,9) | 16 | f | f | 33 00 00 00 00 00 00 00 9 | (0,7) | 16 | f | f | 46 00 00 00 00 00 00 00 10 | (0,2) | 16 | f | f | 4b 00 00 00 00 00 00 00
(10 rows)
postgres=# update a set c3=c3 where id=1;
UPDATE 1
postgres=# SELECT * FROM bt_page_items('idx_a_1',1); itemoffset | ctid | itemlen | nulls | vars | data
------------+--------+---------+-------+------+------------------------- 1 | (0,1) | 16 | f | f | 01 00 00 00 00 00 00 00 2 | (0,2) | 16 | f | f | 02 00 00 00 00 00 00 00 3 | (0,3) | 16 | f | f | 03 00 00 00 00 00 00 00 4 | (0,4) | 16 | f | f | 04 00 00 00 00 00 00 00 5 | (0,5) | 16 | f | f | 05 00 00 00 00 00 00 00 6 | (0,6) | 16 | f | f | 06 00 00 00 00 00 00 00 7 | (0,7) | 16 | f | f | 07 00 00 00 00 00 00 00 8 | (0,8) | 16 | f | f | 08 00 00 00 00 00 00 00 9 | (0,9) | 16 | f | f | 09 00 00 00 00 00 00 00 10 | (0,10) | 16 | f | f | 0a 00 00 00 00 00 00 00
(10 rows) postgres=# SELECT * FROM bt_page_items('idx_a_2',1); itemoffset | ctid | itemlen | nulls | vars | data
------------+--------+---------+-------+------+------------------------- 1 | (0,5) | 16 | f | f | 05 00 00 00 00 00 00 00 2 | (0,7) | 16 | f | f | 06 00 00 00 00 00 00 00 3 | (0,1) | 16 | f | f | 0d 00 00 00 00 00 00 00 4 | (0,9) | 16 | f | f | 0e 00 00 00 00 00 00 00 5 | (0,4) | 16 | f | f | 20 00 00 00 00 00 00 00 6 | (0,3) | 16 | f | f | 24 00 00 00 00 00 00 00 7 | (0,6) | 16 | f | f | 28 00 00 00 00 00 00 00 8 | (0,10) | 16 | f | f | 2a 00 00 00 00 00 00 00 9 | (0,2) | 16 | f | f | 3f 00 00 00 00 00 00 00 10 | (0,8) | 16 | f | f | 55 00 00 00 00 00 00 00
(10 rows) postgres=# SELECT * FROM bt_page_items('idx_a_3',1); itemoffset | ctid | itemlen | nulls | vars | data
------------+--------+---------+-------+------+------------------------- 1 | (0,6) | 16 | f | f | 09 00 00 00 00 00 00 00 2 | (0,5) | 16 | f | f | 12 00 00 00 00 00 00 00 3 | (0,8) | 16 | f | f | 13 00 00 00 00 00 00 00 4 | (0,1) | 16 | f | f | 14 00 00 00 00 00 00 00 5 | (0,4) | 16 | f | f | 22 00 00 00 00 00 00 00 6 | (0,10) | 16 | f | f | 2b 00 00 00 00 00 00 00 7 | (0,3) | 16 | f | f | 30 00 00 00 00 00 00 00 8 | (0,9) | 16 | f | f | 33 00 00 00 00 00 00 00 9 | (0,7) | 16 | f | f | 46 00 00 00 00 00 00 00 10 | (0,2) | 16 | f | f | 4b 00 00 00 00 00 00 00
(10 rows)
5、HOT更新，有索引的字段被更新，但是值不变。并且新版本在同一个PAGE里。
postgres=# update a set c2=c2 where id=1 returning ctid,*; ctid | id | c1 | c2 | c3
--------+----+----+----+---- (0,14) | 1 | 13 | 20 | 15
(1 row) UPDATE 1
postgres=# SELECT * FROM bt_page_items('idx_a_1',1); itemoffset | ctid | itemlen | nulls | vars | data
------------+--------+---------+-------+------+------------------------- 1 | (0,1) | 16 | f | f | 01 00 00 00 00 00 00 00 2 | (0,2) | 16 | f | f | 02 00 00 00 00 00 00 00 3 | (0,3) | 16 | f | f | 03 00 00 00 00 00 00 00 4 | (0,4) | 16 | f | f | 04 00 00 00 00 00 00 00 5 | (0,5) | 16 | f | f | 05 00 00 00 00 00 00 00 6 | (0,6) | 16 | f | f | 06 00 00 00 00 00 00 00 7 | (0,7) | 16 | f | f | 07 00 00 00 00 00 00 00 8 | (0,8) | 16 | f | f | 08 00 00 00 00 00 00 00 9 | (0,9) | 16 | f | f | 09 00 00 00 00 00 00 00 10 | (0,10) | 16 | f | f | 0a 00 00 00 00 00 00 00
(10 rows) postgres=# SELECT * FROM bt_page_items('idx_a_2',1); itemoffset | ctid | itemlen | nulls | vars | data
------------+--------+---------+-------+------+------------------------- 1 | (0,5) | 16 | f | f | 05 00 00 00 00 00 00 00 2 | (0,7) | 16 | f | f | 06 00 00 00 00 00 00 00 3 | (0,1) | 16 | f | f | 0d 00 00 00 00 00 00 00 4 | (0,9) | 16 | f | f | 0e 00 00 00 00 00 00 00 5 | (0,4) | 16 | f | f | 20 00 00 00 00 00 00 00 6 | (0,3) | 16 | f | f | 24 00 00 00 00 00 00 00 7 | (0,6) | 16 | f | f | 28 00 00 00 00 00 00 00 8 | (0,10) | 16 | f | f | 2a 00 00 00 00 00 00 00 9 | (0,2) | 16 | f | f | 3f 00 00 00 00 00 00 00 10 | (0,8) | 16 | f | f | 55 00 00 00 00 00 00 00
(10 rows) postgres=# SELECT * FROM bt_page_items('idx_a_3',1); itemoffset | ctid | itemlen | nulls | vars | data
------------+--------+---------+-------+------+------------------------- 1 | (0,6) | 16 | f | f | 09 00 00 00 00 00 00 00 2 | (0,5) | 16 | f | f | 12 00 00 00 00 00 00 00 3 | (0,8) | 16 | f | f | 13 00 00 00 00 00 00 00 4 | (0,1) | 16 | f | f | 14 00 00 00 00 00 00 00 5 | (0,4) | 16 | f | f | 22 00 00 00 00 00 00 00 6 | (0,10) | 16 | f | f | 2b 00 00 00 00 00 00 00 7 | (0,3) | 16 | f | f | 30 00 00 00 00 00 00 00 8 | (0,9) | 16 | f | f | 33 00 00 00 00 00 00 00 9 | (0,7) | 16 | f | f | 46 00 00 00 00 00 00 00 10 | (0,2) | 16 | f | f | 4b 00 00 00 00 00 00 00
(10 rows)
以上HOT更新，所有索引都没有发生变化。
6、NON-HOT更新，更新了索引字段的值，所有索引都发生了变化，至少发生了3个索引IO。
postgres=# update a set c2=c2+1 where id=1 returning ctid,*; ctid | id | c1 | c2 | c3
--------+----+----+----+---- (0,15) | 1 | 13 | 21 | 15
(1 row) UPDATE 1
postgres=# SELECT * FROM bt_page_items('idx_a_1',1); itemoffset | ctid | itemlen | nulls | vars | data
------------+--------+---------+-------+------+------------------------- 1 | (0,15) | 16 | f | f | 01 00 00 00 00 00 00 00 2 | (0,1) | 16 | f | f | 01 00 00 00 00 00 00 00 3 | (0,2) | 16 | f | f | 02 00 00 00 00 00 00 00 4 | (0,3) | 16 | f | f | 03 00 00 00 00 00 00 00 5 | (0,4) | 16 | f | f | 04 00 00 00 00 00 00 00 6 | (0,5) | 16 | f | f | 05 00 00 00 00 00 00 00 7 | (0,6) | 16 | f | f | 06 00 00 00 00 00 00 00 8 | (0,7) | 16 | f | f | 07 00 00 00 00 00 00 00 9 | (0,8) | 16 | f | f | 08 00 00 00 00 00 00 00 10 | (0,9) | 16 | f | f | 09 00 00 00 00 00 00 00 11 | (0,10) | 16 | f | f | 0a 00 00 00 00 00 00 00
(11 rows) postgres=# SELECT * FROM bt_page_items('idx_a_2',1); itemoffset | ctid | itemlen | nulls | vars | data
------------+--------+---------+-------+------+------------------------- 1 | (0,5) | 16 | f | f | 05 00 00 00 00 00 00 00 2 | (0,7) | 16 | f | f | 06 00 00 00 00 00 00 00 3 | (0,15) | 16 | f | f | 0d 00 00 00 00 00 00 00 4 | (0,1) | 16 | f | f | 0d 00 00 00 00 00 00 00 5 | (0,9) | 16 | f | f | 0e 00 00 00 00 00 00 00 6 | (0,4) | 16 | f | f | 20 00 00 00 00 00 00 00 7 | (0,3) | 16 | f | f | 24 00 00 00 00 00 00 00 8 | (0,6) | 16 | f | f | 28 00 00 00 00 00 00 00 9 | (0,10) | 16 | f | f | 2a 00 00 00 00 00 00 00 10 | (0,2) | 16 | f | f | 3f 00 00 00 00 00 00 00 11 | (0,8) | 16 | f | f | 55 00 00 00 00 00 00 00
(11 rows) postgres=# SELECT * FROM bt_page_items('idx_a_3',1); itemoffset | ctid | itemlen | nulls | vars | data
------------+--------+---------+-------+------+------------------------- 1 | (0,6) | 16 | f | f | 09 00 00 00 00 00 00 00 2 | (0,5) | 16 | f | f | 12 00 00 00 00 00 00 00 3 | (0,8) | 16 | f | f | 13 00 00 00 00 00 00 00 4 | (0,1) | 16 | f | f | 14 00 00 00 00 00 00 00 5 | (0,15) | 16 | f | f | 15 00 00 00 00 00 00 00 6 | (0,4) | 16 | f | f | 22 00 00 00 00 00 00 00 7 | (0,10) | 16 | f | f | 2b 00 00 00 00 00 00 00 8 | (0,3) | 16 | f | f | 30 00 00 00 00 00 00 00 9 | (0,9) | 16 | f | f | 33 00 00 00 00 00 00 00 10 | (0,7) | 16 | f | f | 46 00 00 00 00 00 00 00 11 | (0,2) | 16 | f | f | 4b 00 00 00 00 00 00 00
(11 rows)
7、垃圾回收，首先回收索引垃圾版本，最后回收表的垃圾版本。
postgres=# vacuum verbose a;
INFO: vacuuming "public.a"
INFO: scanned index "idx_a_1" to remove 1 row versions
DETAIL: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
INFO: scanned index "idx_a_2" to remove 1 row versions
DETAIL: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
INFO: scanned index "idx_a_3" to remove 1 row versions
DETAIL: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
INFO: "a": removed 1 row versions in 1 pages
DETAIL: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
INFO: index "idx_a_1" now contains 10 row versions in 2 pages
DETAIL: 1 index row versions were removed.
0 index pages have been deleted, 0 are currently reusable.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: index "idx_a_2" now contains 10 row versions in 2 pages
DETAIL: 1 index row versions were removed.
0 index pages have been deleted, 0 are currently reusable.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: index "idx_a_3" now contains 10 row versions in 2 pages
DETAIL: 1 index row versions were removed.
0 index pages have been deleted, 0 are currently reusable.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: "a": found 5 removable, 10 nonremovable row versions in 1 out of 1 pages
DETAIL: 0 dead row versions cannot be removed yet, oldest xmin: 716311280
There were 4 unused item pointers.
Skipped 0 pages due to buffer pins, 0 frozen pages.
0 pages are entirely empty.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
VACUUM
8、垃圾回收后，索引的垃圾版本被清除。
postgres=# SELECT * FROM bt_page_items('idx_a_1',1); itemoffset | ctid | itemlen | nulls | vars | data
------------+--------+---------+-------+------+------------------------- 1 | (0,15) | 16 | f | f | 01 00 00 00 00 00 00 00 2 | (0,2) | 16 | f | f | 02 00 00 00 00 00 00 00 3 | (0,3) | 16 | f | f | 03 00 00 00 00 00 00 00 4 | (0,4) | 16 | f | f | 04 00 00 00 00 00 00 00 5 | (0,5) | 16 | f | f | 05 00 00 00 00 00 00 00 6 | (0,6) | 16 | f | f | 06 00 00 00 00 00 00 00 7 | (0,7) | 16 | f | f | 07 00 00 00 00 00 00 00 8 | (0,8) | 16 | f | f | 08 00 00 00 00 00 00 00 9 | (0,9) | 16 | f | f | 09 00 00 00 00 00 00 00 10 | (0,10) | 16 | f | f | 0a 00 00 00 00 00 00 00
(10 rows) postgres=# SELECT * FROM bt_page_items('idx_a_2',1); itemoffset | ctid | itemlen | nulls | vars | data
------------+--------+---------+-------+------+------------------------- 1 | (0,5) | 16 | f | f | 05 00 00 00 00 00 00 00 2 | (0,7) | 16 | f | f | 06 00 00 00 00 00 00 00 3 | (0,15) | 16 | f | f | 0d 00 00 00 00 00 00 00 4 | (0,9) | 16 | f | f | 0e 00 00 00 00 00 00 00 5 | (0,4) | 16 | f | f | 20 00 00 00 00 00 00 00 6 | (0,3) | 16 | f | f | 24 00 00 00 00 00 00 00 7 | (0,6) | 16 | f | f | 28 00 00 00 00 00 00 00 8 | (0,10) | 16 | f | f | 2a 00 00 00 00 00 00 00 9 | (0,2) | 16 | f | f | 3f 00 00 00 00 00 00 00 10 | (0,8) | 16 | f | f | 55 00 00 00 00 00 00 00
(10 rows) postgres=# SELECT * FROM bt_page_items('idx_a_3',1); itemoffset | ctid | itemlen | nulls | vars | data
------------+--------+---------+-------+------+------------------------- 1 | (0,6) | 16 | f | f | 09 00 00 00 00 00 00 00 2 | (0,5) | 16 | f | f | 12 00 00 00 00 00 00 00 3 | (0,8) | 16 | f | f | 13 00 00 00 00 00 00 00 4 | (0,15) | 16 | f | f | 15 00 00 00 00 00 00 00 5 | (0,4) | 16 | f | f | 22 00 00 00 00 00 00 00 6 | (0,10) | 16 | f | f | 2b 00 00 00 00 00 00 00 7 | (0,3) | 16 | f | f | 30 00 00 00 00 00 00 00 8 | (0,9) | 16 | f | f | 33 00 00 00 00 00 00 00 9 | (0,7) | 16 | f | f | 46 00 00 00 00 00 00 00 10 | (0,2) | 16 | f | f | 4b 00 00 00 00 00 00 00
(10 rows)
性能测试
1、创建测试表32个索引。
do language plpgsql $$
declare sql text;
begin sql := 'create table a (id int primary key,'; for i in 1..32 loop sql := sql||'c'||i||' int default random()*1000,'; end loop; sql := rtrim(sql,','); sql := sql||') with (fillfactor=80)'; execute sql; for i in 2..32 loop execute 'create index idx_a_c'||i||' on a (c'||i||')'; end loop;
end;
$$;
2、写入1000万记录
insert into a (id) select generate_series(1,10000000);
3、non-hot更新
vi test_non_hot.sql
\set id random(1,10000000)
update a set c2=c2+random()*100-100 where id=:id;
4、HOT更新
vi test_hot1.sql
\set id random(1,10000000)
update a set c1=c1+random()*100-100 where id=:id; vi test_hot2.sql
\set id random(1,10000000)
update a set c2=c2 where id=:id;
5、性能对比
5.1、HOT
pgbench -M prepared -n -r -P 1 -f ./test_hot1.sql -c 28 -j 28 -T 120 transaction type: ./test_hot1.sql
scaling factor: 1
query mode: prepared
number of clients: 28
number of threads: 28
duration: 120 s
number of transactions actually processed: 9139010
latency average = 0.368 ms
latency stddev = 0.187 ms
tps = 76157.798606 (including connections establishing)
tps = 76174.469712 (excluding connections establishing)
script statistics: - statement latencies in milliseconds: 0.001 \set id random(1,10000000) 0.366 update a set c1=c1+random()*100-100 where id=:id; Total DISK READ : 0.00 B/s | Total DISK WRITE : 13.14 M/s
Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 13.82 M/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
45828 be/4 postgres 0.00 B/s 23.39 K/s 0.00 % 1.06 % postgres: postgres postgres 127.0.0.1(41326) UPDATE
45810 be/4 postgres 0.00 B/s 23.39 K/s 0.00 % 0.99 % postgres: postgres postgres 127.0.0.1(41290) UPDATE
45821 be/4 postgres 0.00 B/s 23.39 K/s 0.00 % 0.98 % postgres: postgres postgres 127.0.0.1(41312) idle
45820 be/4 postgres 0.00 B/s 7.80 K/s 0.00 % 0.96 % postgres: postgres postgres 127.0.0.1(41310) UPDATE
45822 be/4 postgres 0.00 B/s 0.00 B/s 0.00 % 0.96 % postgres: postgres postgres 127.0.0.1(41314) UPDATE
45819 be/4 postgres 0.00 B/s 7.80 K/s 0.00 % 0.94 % postgres: postgres postgres 127.0.0.1(41308) UPDATE
45806 be/4 postgres 0.00 B/s 7.80 K/s 0.00 % 0.92 % postgres: postgres postgres 127.0.0.1(41282) idle
45824 be/4 postgres 0.00 B/s 7.80 K/s 0.00 % 0.90 % postgres: postgres postgres 127.0.0.1(41318) UPDATE
45827 be/4 postgres 0.00 B/s 54.58 K/s 0.00 % 0.89 % postgres: postgres postgres 127.0.0.1(41324) UPDATE
45814 be/4 postgres 0.00 B/s 0.00 B/s 0.00 % 0.89 % postgres: postgres postgres 127.0.0.1(41298) UPDATE
45818 be/4 postgres 0.00 B/s 7.80 K/s 0.00 % 0.88 % postgres: postgres postgres 127.0.0.1(41306) idle
45823 be/4 postgres 0.00 B/s 15.59 K/s 0.00 % 0.87 % postgres: postgres postgres 127.0.0.1(41316) UPDATE
45805 be/4 postgres 0.00 B/s 0.00 B/s 0.00 % 0.87 % postgres: postgres postgres 127.0.0.1(41280) UPDATE
45826 be/4 postgres 0.00 B/s 7.80 K/s 0.00 % 0.86 % postgres: postgres postgres 127.0.0.1(41322) UPDATE
45809 be/4 postgres 0.00 B/s 0.00 B/s 0.00 % 0.86 % postgres: postgres postgres 127.0.0.1(41288) UPDATE
45808 be/4 postgres 0.00 B/s 23.39 K/s 0.00 % 0.85 % postgres: postgres postgres 127.0.0.1(41286) UPDATE
45825 be/4 postgres 0.00 B/s 0.00 B/s 0.00 % 0.85 % postgres: postgres postgres 127.0.0.1(41320) UPDATE
45804 be/4 postgres 0.00 B/s 23.39 K/s 0.00 % 0.84 % postgres: postgres postgres 127.0.0.1(41278) UPDATE
49040 be/4 postgres 0.00 B/s 12.85 M/s 0.00 % 0.84 % postgres: wal writer process
45816 be/4 postgres 0.00 B/s 7.80 K/s 0.00 % 0.66 % postgres: postgres postgres 127.0.0.1(41302) idle
45829 be/4 postgres 0.00 B/s 0.00 B/s 0.00 % 0.64 % postgres: postgres postgres 127.0.0.1(41328) UPDATE
45803 be/4 postgres 0.00 B/s 0.00 B/s 0.00 % 0.62 % postgres: postgres postgres 127.0.0.1(41276) BIND
45795 be/4 postgres 0.00 B/s 0.00 B/s 0.00 % 0.59 % postgres: postgres postgres 127.0.0.1(41274) UPDATE
45807 be/4 postgres 0.00 B/s 7.80 K/s 0.00 % 0.58 % postgres: postgres postgres 127.0.0.1(41284) UPDATE
45812 be/4 postgres 0.00 B/s 15.59 K/s 0.00 % 0.56 % postgres: postgres postgres 127.0.0.1(41294) UPDATE
45811 be/4 postgres 0.00 B/s 15.59 K/s 0.00 % 0.55 % postgres: postgres postgres 127.0.0.1(41292) UPDATE
45817 be/4 postgres 0.00 B/s 7.80 K/s 0.00 % 0.55 % postgres: postgres postgres 127.0.0.1(41304) UPDATE
45815 be/4 postgres 0.00 B/s 7.80 K/s 0.00 % 0.55 % postgres: postgres postgres 127.0.0.1(41300) UPDATE
45813 be/4 postgres 0.00 B/s 0.00 B/s 0.00 % 0.51 % postgres: postgres postgres 127.0.0.1(41296) UPDATE
5.2、NON-HOT
pgbench -M prepared -n -r -P 1 -f ./test_non_hot.sql -c 28 -j 28 -T 120 transaction type: ./test_non_hot.sql
scaling factor: 1
query mode: prepared
number of clients: 28
number of threads: 28
duration: 120 s
number of transactions actually processed: 6472445
latency average = 0.519 ms
latency stddev = 0.707 ms
tps = 53922.273197 (including connections establishing)
tps = 53933.908671 (excluding connections establishing)
script statistics: - statement latencies in milliseconds: 0.002 \set id random(1,10000000) 0.517 update a set c2=c2+random()*100-100 where id=:id; Total DISK READ : 0.00 B/s | Total DISK WRITE : 191.66 M/s
Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 142.11 M/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
49040 be/4 postgres 0.00 B/s 136.56 M/s 0.00 % 7.17 % postgres: wal writer process
45997 be/4 postgres 0.00 B/s 1918.73 K/s 0.00 % 1.60 % postgres: postgres postgres 127.0.0.1(41384) BIND
45983 be/4 postgres 0.00 B/s 1903.25 K/s 0.00 % 1.56 % postgres: postgres postgres 127.0.0.1(41356) UPDATE
45977 be/4 postgres 0.00 B/s 1829.75 K/s 0.00 % 1.54 % postgres: postgres postgres 127.0.0.1(41344) UPDATE
45984 be/4 postgres 0.00 B/s 1918.73 K/s 0.00 % 1.53 % postgres: postgres postgres 127.0.0.1(41358) UPDATE
45985 be/4 postgres 0.00 B/s 1616.99 K/s 0.00 % 1.50 % postgres: postgres postgres 127.0.0.1(41360) UPDATE
45986 be/4 postgres 0.00 B/s 1748.52 K/s 0.00 % 1.49 % postgres: postgres postgres 127.0.0.1(41362) UPDATE
45995 be/4 postgres 0.00 B/s 1616.99 K/s 0.00 % 1.47 % postgres: postgres postgres 127.0.0.1(41380) UPDATE
45988 be/4 postgres 0.00 B/s 1910.99 K/s 0.00 % 1.46 % postgres: postgres postgres 127.0.0.1(41366) UPDATE
45979 be/4 postgres 0.00 B/s 1763.99 K/s 0.00 % 1.46 % postgres: postgres postgres 127.0.0.1(41348) UPDATE
45976 be/4 postgres 0.00 B/s 1918.73 K/s 0.00 % 1.45 % postgres: postgres postgres 127.0.0.1(41342) UPDATE
45982 be/4 postgres 0.00 B/s 1887.78 K/s 0.00 % 1.43 % postgres: postgres postgres 127.0.0.1(41354) UPDATE
45987 be/4 postgres 0.00 B/s 2019.31 K/s 0.00 % 1.42 % postgres: postgres postgres 127.0.0.1(41364) UPDATE
45992 be/4 postgres 0.00 B/s 1616.99 K/s 0.00 % 1.40 % postgres: postgres postgres 127.0.0.1(41374) UPDATE
45994 be/4 postgres 0.00 B/s 1941.94 K/s 0.00 % 1.38 % postgres: postgres postgres 127.0.0.1(41378) UPDATE
45990 be/4 postgres 0.00 B/s 1794.94 K/s 0.00 % 1.36 % postgres: postgres postgres 127.0.0.1(41370) UPDATE
45975 be/4 postgres 0.00 B/s 1934.20 K/s 0.00 % 1.35 % postgres: postgres postgres 127.0.0.1(41340) UPDATE
45974 be/4 postgres 0.00 B/s 1910.99 K/s 0.00 % 1.35 % postgres: postgres postgres 127.0.0.1(41338) UPDATE
45980 be/4 postgres 0.00 B/s 2003.83 K/s 0.00 % 1.28 % postgres: postgres postgres 127.0.0.1(41350) UPDATE
45991 be/4 postgres 0.00 B/s 1748.52 K/s 0.00 % 1.13 % postgres: postgres postgres 127.0.0.1(41372) UPDATE
45996 be/4 postgres 0.00 B/s 2003.83 K/s 0.00 % 1.04 % postgres: postgres postgres 127.0.0.1(41382) UPDATE
45993 be/4 postgres 0.00 B/s 1918.73 K/s 0.00 % 1.02 % postgres: postgres postgres 127.0.0.1(41376) UPDATE
45972 be/4 postgres 0.00 B/s 1903.25 K/s 0.00 % 1.02 % postgres: postgres postgres 127.0.0.1(41334) UPDATE
45978 be/4 postgres 0.00 B/s 1740.78 K/s 0.00 % 1.02 % postgres: postgres postgres 127.0.0.1(41346) UPDATE
45998 be/4 postgres 0.00 B/s 1841.36 K/s 0.00 % 1.02 % postgres: postgres postgres 127.0.0.1(41386) UPDATE
45981 be/4 postgres 0.00 B/s 1818.15 K/s 0.00 % 1.01 % postgres: postgres postgres 127.0.0.1(41352) UPDATE
45989 be/4 postgres 0.00 B/s 1895.52 K/s 0.00 % 1.01 % postgres: postgres postgres 127.0.0.1(41368) UPDATE
45973 be/4 postgres 0.00 B/s 1941.94 K/s 0.00 % 0.99 % postgres: postgres postgres 127.0.0.1(41336) idle
45961 be/4 postgres 0.00 B/s 1872.31 K/s 0.00 % 0.96 % postgres: postgres postgres 127.0.0.1(41332) UPDATE
49039 be/4 postgres 0.00 B/s 4.37 M/s 0.00 % 0.00 % postgres: writer process
49036 be/4 postgres 0.00 B/s 3.87 K/s 0.00 % 0.00 % postgres: logger process
使用HOT技术，使得TPS从 53922 提升到了 76157 。IO资源消耗从 192MB/s 降低到了 14MB/s 。
小结
HOT、二级索引、zheap存储引擎，都可以解决更新引入的索引放大问题。
本文介绍了HOT，HOT必须满足如下两个条件：
Necessary Condition A: UPDATE does not change any of the index keys Necessary Condition B: The new version should fit in the same old block – HOT chains can not cross block boundary. 1、索引字段的值不变。(其中任意一个索引字段的值发生了变化，则所有索引都需要新增版本) 2、新的版本与旧的版本在同一个HEAP PAGE中。
参考
《PostgreSQL 11 preview - Surjective indexes - 索引HOT增强（表达式）update评估》
HOT Inside - The Technical Architecture
《PostgreSQL 10.0 preview 性能增强 - 间接索引(secondary index)》
src/backend/access/heap/README.HOT
PostgreSQL Heap Only Tuple - HOT (降低UPDATE引入的索引写IO放大)相关推荐
- 为什么引入文件索引节点能大大加快目录检索速度
首先弄清楚什么是索引结点(inode)? 一般来说,面试不会问 inode .但是 inode 是一个重要概念,是理解 Unix/Linux 文件系统和硬盘储存的基础. 理解inode,要从文件储存说 ...
- PostgreSQL TID及tuple slot
1)postgresql默认存储的是堆表,数据按行存储在heap page中.行记录除了存储字段的值外还会存储对应的ctid即行号,表示在哪个页第几个记录. 2)进行select的时候也可以直接指定c ...
- PostgreSQL 自定义复合类型(composite type) deform引入的额外开销
标签 PostgreSQL , UDT , 自定义类型 背景 PG 允许用户自定义复合类型,自定义复合类型由多个类型组成,使用时会引入一些deform的开销. 例子 postgres=# create ...
- postgresql数据库迁移技巧（降低版本迁移到GreenPlum pg11-pg8）
工具:navicat12 步骤: 1.创建scheme 2.导出源scheme的结构sql 3.在目标数据库执行源scheme结构sql中创建序列部分 4.直接操作拷贝scheme所有表(使用生成sq ...
- 在 HTML 中引入 vue.js 写页面
突然说要写两个页面(只有两个页面,不是一个完整的项目..),有点懵,不知道从哪下手,而且只对 vue 熟悉那么一丢丢,其他框架不是很熟悉.但是没办法鸭,只能硬着头皮去做了!在这里做一下笔记. 一.前期 ...
- 前端项目引入js文件写绝对目录还是相对目录_flask入门（三）静态文件
前言 上篇文章讲到 flask 的模板文件如何使用,印象模糊的朋友可以回顾一下flask入门 (二)(不用写代码的前端!) 今天的主题 - flask 和静态文件结合的使用技巧. 静态文件概念 先来简 ...
- PostgreSQL 查询所有数据库大小，表大小，索引大小，以及表空间大小
1. 查询数据库大小 -- 查询单个数据库大小 select pg_size_pretty(pg_database_size('postgres')) as size;-- 查询所有数据库大小 sel ...
- 阿里数据库内核月报导航
数据库内核月报 是阿里云RDS-数据库内核组维护的一个关于数据库的技术博客,上面的文章质量是有保证的.不过有一点不太友好的是,这个网站首页非常简洁,只是按照时间月份来归档文章,并没有按照类别来分,这样 ...
- 阿里内核数据库文章-目录
/monthly/2022/07 http://mysql.taobao.org/monthly/2022/07/01/ MySQL · 行业洞察 · 为什么游戏行业喜欢用PolarDB h ...
- HBase 2.0版本正式发布
1.HBase 2.0版本正式发布 关于HBase版本定义 HBase从1.0.0版本开始,在版本定义上正式遵循了Semantic Versioning规范: 一个版本号,由三部分组成: MAJOR. ...
最新文章
- 【黑客浅析】像黑客一样思考
- 通知 | 2017年清华-青岛数据科学研究院“RONG”奖学金即将开放申请
- 你真的会用java注解吗？
- 【SQL Join】两个字段关联到同一个表，如何使用left join
- python mysql library popular_python之mysql入门详解（四）
- css背景图片、圆角、盒子阴影、浮动
- JavaScript URL参数读取
- 博客园开始对X++语言语法高亮的支持
- 系数矩阵为方阵的线性方程组解的情况
- php mixed约束,Mixed Content: 混合被动/显示内容和混合活动内容
- 如何用python做模型_python的模型如何使用？
- gentoo Portage使用
- 综述｜重邮高新波等最新《少样本目标检测算法》
- 2022美国大学生数学建模竞赛C题摘要及总结
- python文本字词分割及词库云
- 格式化后如何修复损坏的JPEG文件?
- element table radio 单选表格实现
- 数数字（找规律+模拟）
- 简易网络直播聊天室源码+自适应手机版正规企业财经直播聊天室系统
- android DevAppsDirect开源项目