PostgreSQL 10 新特性, 流式接收端在线压缩redo

背景

虽然现在磁盘已经很廉价，大多数时候不需要压缩了。

但是在一些嵌入式系统，或者一些未扩容的产品环境中，压缩还是非常有必要的。

特别是数据变化大、或者数据增量大的场景，例如物联网(IoT)，每天单库可能新增TB级的增量。

比如《PostgreSQL 如何潇洒的处理每天上百TB的数据增量》指的就是IoT场景。

这样的场景，数据库的归档日志也会很大，这个时候就体现压缩的重要性了。

CPU和网络带宽的权衡选择

压缩势必会增加CPU的开销，我们可以选择是在数据库的服务端压缩，还是在客户端压缩。

在数据库的服务端压缩，增加的是数据库服务器的CPU运算开销，好处是网络传输的数据变少了。

而在客户端压缩的话，对数据库服务器没有额外的CPU需求，但是网络传输的数据量则是压缩前的量。

用户可以根据实际情况选择。

流式接收与在线压缩的结合

PostgreSQL很早以前的版本就提供了一种流式接收redo的方法，通过在客户端使用pg_receivexlog命令，与普通应用程序一样，连接PostgreSQL数据库，可以通过PG的流式协议实时接收来自数据库的REDO。

另一个命令是pg_basebackup，可以流式接收数据库的所有文件（包括数据文件，REDO等），常被用于备份。

10.0对pg_basebackup也有一个改进，请参考：

《元旦技术大礼包 - 2017金秋将要发布的PostgreSQL 10.0已装备了哪些核武器？》

流式接收与在线压缩结合，解决了IoT场景REDO归档日志体积庞大的问题。

pg_receivexlog支持通过开关控制是否需要开启压缩、以及选择压缩级别。

pg_receivexlog启动时，自动扫描存放归档文件的目标目录，选择断点续传的位置，然后向PostgreSQL数据库请求相应位置为起点的REDO。

pg_receivexlog开启压缩后，依旧支持同步反馈模式，所以它依旧可以作为一个同步副本。

关于PostgreSQL数据库的多副本同步，以及应用场景请参考

《PostgreSQL 9.6 同步多副本与 remote_apply事务同步级别》

《PostgreSQL 金融行业高可用和容灾解决方案》

《元旦技术大礼包 - 2017金秋将要发布的PostgreSQL 10.0已装备了哪些核武器？》

正文

http://paquier.xyz/postgresql-2/postgres-10-pgreceivexlog-compression/

https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=cada1af31d769a6b607018d68894f2c879ff275f

One of my personal areas of work lately is in finding ways to improve the user experience with WAL archiving and pg_receivexlog.

A couple of experiments have been done, and one of them has finished as a patch for upstream Postgres, in the shape of this commit:

commit: cada1af31d769a6b607018d68894f2c879ff275f
author: Magnus Hagander <magnus@hagander.net>
date: Tue, 17 Jan 2017 12:10:26 +0100
Add compression support to pg_receivexlog  Author: Michael Paquier, review and small changes by me

Combined with replication slots, pg_receivexlog is a nice way to ensure that there is no hole in WAL segments. Compared to the archive_command itself, any failure handling in case of successive failures in archiving a completed segment is easier as there is no need to tweak the parameter of the archive_command or the script used in the command itself to avoid a bloat in pg_xlog, resulting in a crash of Postgres if the partition holding this folder gets full. Any failure handling can happen from a remote position, and there is no need to have a superuser to do this work, only a user with replication rights is enough to drop a slot and unlock the situation. Note though that enforcing the recycling of past segments requires a checkpoint to happen.

The commit above has added a way to compression on-the-fly with zlib WAL records and to store them in .gz files, one for each segment. In those days where disk is cheaper than CPU, compression is not a big deal for many users and they are fine to afford more space to store the same amount of history. However, in cases where Postgres is embedded in a system and the amount of space allowed is controlled it may be a big deal to be able to retain more history using the same amount of space, particularly knowing that a WAL segment compressed with zlib is 3 to 4 times smaller.

The compression option can be activated with a new option switch called --compress, with which can be specified a number from 0 to 9, 0 meaning no compression and 9 the highest level of compression. Note that level 9 is a huge CPU eater and that in an INSERT-only load the compression of each segment may not be able to follow with the WAL generation, resulting in pg_receivexlog complaining that a segment it is requesting has already been removed by a backend checkpoint or, if a replication slot is used, resulting in a crash of the Postgres instance because of pg_xlog getting full.

$ pg_receivexlog --compress=1 -D /path/to/logs/ --verbose   pg_receivexlog: starting log streaming at 0/1000000 (timeline 1)
pg_receivexlog: finished segment at 0/2000000 (timeline 1)
pg_receivexlog: finished segment at 0/3000000 (timeline 1)
[...]

And this generates many gzip-ready files.

$ ls /path/to/logs/   000000010000000000000001.gz
000000010000000000000002.gz
[...]
000000010000000000000027.gz.partial

--synchronous works as well with the compression support and makes sure that the compressed files, even if not completed segments, are still available. Backup and history files are compressed as well.

Another thing to note is that at startup phase, pg_receivexlog scans the directory it writes the WAL data into for existing segments that are on it and decides based on that from which position it needs to continue working on. The committed patch is smart enough to make a difference between compressed, non-compressed, and even partial segments so it is perfectly fine to mix compression or not and keep the same range of segments saved.

参考

http://paquier.xyz/postgresql-2/postgres-10-pgreceivexlog-compression/

https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=cada1af31d769a6b607018d68894f2c879ff275f

《元旦技术大礼包 - 2017金秋将要发布的PostgreSQL 10.0已装备了哪些核武器？》

《PostgreSQL 9.6 同步多副本与 remote_apply事务同步级别》

《PostgreSQL 金融行业高可用和容灾解决方案》