pg_receivewal实践

https://www.cnblogs.com/kuang17/p/10868556.html

测试从pg_receivewal的日志中恢复从库为主库：

主从配置async模式，配置pg_receivewal接收日志pg_receivewal -D /dbaas/pg/data/pg_receivewal_data -v -h 10.9.10.202

主插入1000万数据，当插入一半时，停止从库

主库插完数据，停止

将pg_receivewal的日志拷贝到从库/dbaas/pg/data/pg_receivewal_data下

修改从库recovery.conf文件，添加内容：

restore_command = 'cp /dbaas/pg/data/pg_receivewal_data/%f "%p"'

启动数据库，数据库成功恢复到10000000行。

另外一种用法：

restore_command = 'cp /dbaas/pg/data/pg_receivewal_data/%f "%p"'

standby_mode = on

primary_conninfo = 'host=10.9.10.203 port=5432 user=postgres connect_timeout=60'

recovery_target_timeline = 'latest'

trigger_file = '/dbaas/pg/data/.tfile'

这种把恢复和备库设置放一起，会先去从归档恢复，当恢复完成后，直接建立流复制关系，这样就相当方便了，当有归档时，不用重做从库。

测试pg_receivewal的同步机制：

-S slotname

--slot=slotname

Require pg_receivewal to use an existing replication slot (see Section 26.2.6). When this option is used, pg_receivewal will report a flush position to the server, indicating when each segment has been synchronized to disk so that the server can remove that segment if it is not otherwise needed.

When the replication client of pg_receivewal is configured on the server as a synchronous standby, then using a replication slot will report the flush position to the server, but only when a WAL file is closed. Therefore, that configuration will cause transactions on the primary to wait for a long time and effectively not work satisfactorily. The option --synchronous (see below) must be specified in addition to make this work correctly.

--synchronous

Flush the WAL data to disk immediately after it has been received. Also send a status packet back to the server immediately after flushing, regardless of --status-interval.

This option should be specified if the replication client of pg_receivewal is configured on the server as a synchronous standby, to ensure that timely feedback is sent to the server.

在这个说明中，使用了--slot=for_pgreceivewal --synchronous后，理论上是有点像sync同步一样，数据库会等待接收端进行回放，回放结束，再进行commit。

1）使slot和synchronous参数：

数据库端：

修改postgresql.conf参数synchronous_standby_name=‘pg_receivewal’

重启数据库

postgres=# select pg_create_physical_replication_slot('for_pgreceivewal');

pg_create_physical_replication_slot

-------------------------------------

(for_pgreceivewal,)

(1 row)

postgres=# select pg_get_replication_slots();

pg_get_replication_slots

----------------------------------------

(for_pgreceivewal,,physical,,f,f,,,,,)

(1 row)

postgres=# select pg_get_replication_slots();

pg_get_replication_slots

--------------------------------------------------------

(for_pgreceivewal,,physical,,f,t,629488,,,0/9DD77B00,)

(1 row)

接收端：

pg_receivewal -D /dbaas/pg/data/pg_receivewal_data --slot=for_pgreceivewal --synchronous -v -h 10.9.10.202

数据库端：

postgres=# select * from pg_stat_replication ;

--------+----------+----------+------------------+-------------+-----------------+-------------+-------------------------------+--------------+-----------+------------+------------+------------+------------+-----------------+-----------------+-----------------+---------------+------------

912703 | 10 | postgres | pg_receivewal | 10.9.10.203 | | 63103 | 2019-05-15 02:52:30.029814+00 | | streaming | 0/D9E44528 | 0/D9E44528 | 0/D9E44528 | | 00:00:00.010972 | 00:00:00.010972 | 00:06:49.964729 | 1 | sync

(1 row)

在数据库端，使用pgbench测试：

[postgres@90027282-dd74-4642-b5dd-359b6cc67aaf /dbaas/pg/data]$ pgbench -c 10 -j 5 -r -T 100 pgbench

starting vacuum...end.

transaction type: <builtin: TPC-B (sort of)>

scaling factor: 100

query mode: simple

number of clients: 10

number of threads: 5

duration: 100 s

number of transactions actually processed: 15290

latency average = 65.426 ms

tps = 152.845128 (including connections establishing)

tps = 152.855112 (excluding connections establishing)

statement latencies in milliseconds:

0.002 \set aid random(1, 100000 * :scale)

0.001 \set bid random(1, 1 * :scale)

0.001 \set tid random(1, 10 * :scale)

0.000 \set delta random(-5000, 5000)

0.046 BEGIN;

0.300 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;

0.111 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;

0.417 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;

2.878 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;

0.092 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);

61.539 END;

2）不使用slot和synchronous参数

数据库端，删除slot

postgres=# select pg_drop_replication_slot('for_pgreceivewal');

pg_drop_replication_slot

--------------------------

(1 row)

postgres=# select pg_get_replication_slots();

pg_get_replication_slots

--------------------------

(0 rows)

接收端：

pg_receivewal -D /dbaas/pg/data/pg_receivewal_data -v -h 10.9.10.202

压测：

[postgres@90027282-dd74-4642-b5dd-359b6cc67aaf ~]$ pgbench -c 10 -j 5 -r -T 100 pgbench

starting vacuum...end.

transaction type: <builtin: TPC-B (sort of)>

scaling factor: 100

query mode: simple

number of clients: 10

number of threads: 5

duration: 100 s

number of transactions actually processed: 27015

latency average = 37.021 ms

tps = 270.120226 (including connections establishing)

tps = 270.138206 (excluding connections establishing)

statement latencies in milliseconds:

0.002 \set aid random(1, 100000 * :scale)

0.001 \set bid random(1, 1 * :scale)

0.001 \set tid random(1, 10 * :scale)

0.000 \set delta random(-5000, 5000)

0.042 BEGIN;

0.242 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;

0.110 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;

0.291 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;

1.778 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;

0.091 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);

34.396 END;

同步模式下，停止pg_receivewal，会导致主hang住：

postgres=# create table test(id int, info varchar(8), crt_time timestamp);

^CCancel request sent

WARNING: canceling wait for synchronous replication due to user request

DETAIL: The transaction has already committed locally, but might not have been replicated to the standby.

CREATE TABLE

pg_receivewal的使用场景思考：

1）当配置高可用集群时，一个备库很危险，容易出现单点故障，当同步节点死掉后，主节点会hang住（这是没有专业高可用工具的情况下，ecox这种高可用工具，会自动处理为单节点可写）。而部署两个同步节点又会浪费资源（不做读写分离的情况下），这个时候，可以创建一个pg_receivewal节点，使用同步模式，当一个同步节点停止后，它可以充当sync节点，给主节点报告LSN。

为了减少网络开销，还可以将pg_receivewal节点设置为主库本地，即起到了归档作用，又起到了同步节点的功能。唯一注意的是，要定期清理日志空间。

2）第二种情况，数据库集群不使用sync模式，想使用更加高性能的async模式。这时，使用pg_receivewal来做日志归档，至少比aync节点同步的日志要多。这样基于他的恢复，也将丢更少的数据。当数据库更新插入极端的大时，pg_receivewal归档的日志，可能还没有归档命令归档的多。归档是到本地，如果服务器挂了，那么归档也没有了，可以使用pgbackrest或者rync工具将归档同步到其他服务器。

用更加精炼的话说：

1）避免单点故障。这点比较鸡肋，某些高级的高可用工具本身就能解决单点故障。

2）用来做日志归档，归档到块级别，且能归档到远端。比archive_command要更加多（默认归档是完成一个wal文件才归档一个，还未写满的wal是不会归档的）。主从之间保持sync模式，那没必要用pg_receivewal来归档；如果是async模式追求性能，pg_receivewal也不能配成同步来影响性能啊（让步：将pg_receivewal放到本地，这样减少网络传输开销，保持同步，能否接受，答案是同步下性能还是很差，而且差距很大，下面是测试数据）。

postgres=# select * from pg_stat_replication ;

--------+----------+----------+------------------+-------------+-----------------+-------------+-------------------------------+--------------+-----------+------------+------------+------------+------------+-----------------+-----------------+-----------------+---------------+------------

745643 | 10 | <strong>postgres | pg_receivewal | ::1 </strong> | | 58465 | 2019-05-16 03:53:06.628954+00 | | streaming | 1/CB059FC8 | 1/CB059FC8 | 1/CB059FC8 | | 00:00:00.191089 | 00:00:00.191089 | 00:00:32.187898 | 1 | <strong>sync</strong>

668818 | 10 | postgres | walreceiver | 10.9.10.203 | | 42007 | 2019-05-16 03:37:21.983067+00 | | streaming | 1/CB059FC8 | 1/CB059FC8 | 1/CB059FC8 | 1/CB059FC8 | | | | 0 | async

(2 rows)

(END)

在本地同步性能:

[postgres@90027282-dd74-4642-b5dd-359b6cc67aaf /dbaas/pg]$ pgbench -c 10 -j 5 -r -T 100 pgbench

starting vacuum...end.

transaction type: <builtin: TPC-B (sort of)>

scaling factor: 100

query mode: simple

number of clients: 10

number of threads: 5

duration: 100 s

number of transactions actually processed: 17313

latency average = 57.785 ms

tps = 173.054591 (including connections establishing)

tps = 173.066373 (excluding connections establishing)

statement latencies in milliseconds:

0.002 \set aid random(1, 100000 * :scale)

0.001 \set bid random(1, 1 * :scale)

0.001 \set tid random(1, 10 * :scale)

0.000 \set delta random(-5000, 5000)

0.048 BEGIN;

0.268 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;

0.111 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;

0.457 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;

2.694 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;

0.092 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);

53.995 END;

异地同步性能更低：

[postgres@90027282-dd74-4642-b5dd-359b6cc67aaf /dbaas/pg]$ pgbench -c 10 -j 5 -r -T 100 pgbench

starting vacuum...end.

transaction type: <builtin: TPC-B (sort of)>

scaling factor: 100

query mode: simple

number of clients: 10

number of threads: 5

duration: 100 s

number of transactions actually processed: 14289

latency average = 70.130 ms

tps = 142.591560 (including connections establishing)

tps = 142.601721 (excluding connections establishing)

statement latencies in milliseconds:

0.002 \set aid random(1, 100000 * :scale)

0.001 \set bid random(1, 1 * :scale)

0.001 \set tid random(1, 10 * :scale)

0.000 \set delta random(-5000, 5000)

0.045 BEGIN;

0.244 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;

0.111 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;

0.388 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;

2.975 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;

0.091 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);

65.978 END;

异地异步，性能高很多：

[postgres@90027282-dd74-4642-b5dd-359b6cc67aaf /dbaas/pg]$ pgbench -c 10 -j 5 -r -T 100 pgbench

starting vacuum...end.

transaction type: <builtin: TPC-B (sort of)>

scaling factor: 100

query mode: simple

number of clients: 10

number of threads: 5

duration: 100 s

number of transactions actually processed: 26333

latency average = 37.987 ms

tps = 263.246388 (including connections establishing)

tps = 263.264004 (excluding connections establishing)

statement latencies in milliseconds:

0.002 \set aid random(1, 100000 * :scale)

0.001 \set bid random(1, 1 * :scale)

0.001 \set tid random(1, 10 * :scale)

0.000 \set delta random(-5000, 5000)

0.042 BEGIN;

0.228 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;

0.109 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;

0.310 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;

1.764 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;

0.090 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);

35.349 END;

本地异步呢，更快：

[postgres@90027282-dd74-4642-b5dd-359b6cc67aaf /dbaas/pg]$ pgbench -c 10 -j 5 -r -T 100 pgbench

starting vacuum...end.

transaction type: <builtin: TPC-B (sort of)>

scaling factor: 100

query mode: simple

number of clients: 10

number of threads: 5

duration: 100 s

number of transactions actually processed: 29572

latency average = 33.821 ms

tps = 295.670234 (including connections establishing)

tps = 295.691390 (excluding connections establishing)

statement latencies in milliseconds:

0.002 \set aid random(1, 100000 * :scale)

0.001 \set bid random(1, 1 * :scale)

0.001 \set tid random(1, 10 * :scale)

0.000 \set delta random(-5000, 5000)

0.042 BEGIN;

0.219 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;

0.108 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;

0.253 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;

1.592 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;

0.091 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);

31.412 END;

接下来要思考的场景：

那如果遇到一个落后很久的备库，想不重建的情况下恢复。有归档日志，我们应该怎么做？

1）先从归档恢复，再建立同步？不行，恢复后，日志线就+1了，不能再和主建立同步了。

2）将归档日志，从库缺少的地方开始，拷贝到主库pg_wal下，再进行启动从库。待验证。

3）直接在从库的recovery.conf文件配置如下，先从归档恢复，再恢复流复制关系，这样最靠谱：

restore_command = 'cp /dbaas/pg/data/pg_receivewal_data/%f "%p"'

standby_mode = on

primary_conninfo = 'host=10.9.10.203 port=5432 user=postgres connect_timeout=60'

recovery_target_timeline = 'latest'

trigger_file = '/dbaas/pg/data/.tfile'

严以律己、宽以待人

pg_receivewal实践相关推荐

Golang TDD实践报告：快速排序Quick Sort
Golang TDD实践报告:快速排序Quick Sort [阅读时间:约5分钟] 0.项目需求 1.编写符合项目输入输出的一个测试 2.尝试运行测试 3.先使用最少的代码来让失败的测试先跑起来 4. ...
编写高性能Java代码的最佳实践
编写高性能Java代码的最佳实践摘要:本文首先介绍了负载测试.基于APM工具的应用程序和服务器监控,随后介绍了编写高性能Java代码的一些最佳实践.最后研究了JVM特定的调优技巧.数据库端的优化和架 ...
【K8S实践】关于Kubernetes Dashboard的实践学习
前言: 陆陆续续,学k8s一个月了,从开始看docker容器书,视频,怎么都无法上手:到后来闷着头实践地傻瓜式操作:到今天整个完全串通好!感觉到学习也是一段深刻的成长经历! 过程: 这个过程中使用的工 ...
提高期——SLA实践指导秘籍
前言: 第二遍看提高期的SLA文档,开始提高期的第一个项目. 感受: 这个秘籍和自己看的<番茄工作法图解>非常像,写作风格和内容量上. 从大脑的功能分析,为什么要这样做.我们的学习方法符合 ...
Spring Data JPA 五分钟快速入门和实践
Spring Data JPA(类似于Java Web 中的 DAO) 操作声明持久层的接口(Repository) 三个核心接口: CrudRepository PagingAndSortingRe ...
TVM优化c++部署实践
TVM优化c++部署实践使用TVM导入神经网络模型: 模型支持pytorch , tensorflow , onnx, caffe 等.平时pytorch用的多,这里给一种pytorch的导入方式. ...
Dockerfile构建实践
Dockerfile构建实践本文介绍了用于构建有效图像的推荐最佳实践和方法. Docker通过从一个Dockerfile文本文件中读取指令来自动构建映像,该文本文件按顺序包含构建给定映像所需的所有命 ...
Tensor基础实践
Tensor基础实践飞桨(PaddlePaddle,以下简称Paddle)和其他深度学习框架一样,使用Tensor来表示数据,在神经网络中传递的数据均为Tensor. Tensor可以将其理解为多维 ...
MindSpore模型精度调优实践
MindSpore模型精度调优实践引论:在模型的开发过程中,精度达不到预期常常让人头疼.为了帮助用户解决模型调试调优的问题,为MindSpore量身定做了可视化调试调优组件:MindInsight. ...

pg_receivewal实践

pg_receivewal实践

pg_receivewal实践相关推荐

最新文章

热门文章