greenplum segment down 实例
由于数据分析师语句问题，导致segment好几个节点down
那么需要恢复segment
gpstop -r -M fast
此处省略一万字
20170629:14:38:30:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-Stopping master standby host 192.168.102.68 mode=fast
20170629:14:38:32:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-Successfully shutdown standby process on 192.168.102.68
20170629:14:38:33:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-Commencing parallel primary segment instance shutdown, please wait...
20170629:14:38:33:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-0.00% of jobs completed
20170629:14:38:43:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-12.50% of jobs completed
20170629:14:38:53:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-93.75% of jobs completed
20170629:14:39:03:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-100.00% of jobs completed
20170629:14:39:03:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-Commencing parallel mirror segment instance shutdown, please wait...
20170629:14:39:03:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-0.00% of jobs completed
20170629:14:39:13:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-100.00% of jobs completed
20170629:14:39:13:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-----------------------------------------------------
20170629:14:39:13:041991 gpstop:192-168-102-67:gpadmin-[INFO]:- Segments stopped successfully = 126
20170629:14:39:13:041991 gpstop:192-168-102-67:gpadmin-[INFO]:- Segments with errors during stop = 0
20170629:14:39:13:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-
20170629:14:39:13:041991 gpstop:192-168-102-67:gpadmin-[WARNING]:-Segments that are currently marked down in configuration = 2 <<<<<<<<
20170629:14:39:13:041991 gpstop:192-168-102-67:gpadmin-[INFO]:- (stop was still attempted on these segments)
20170629:14:39:13:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-----------------------------------------------------
20170629:14:39:13:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-Successfully shutdown 126 of 128 segment instances
20170629:14:39:13:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-Database successfully shutdown with no errors reported
20170629:14:39:13:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-Cleaning up leftover gpmmon process
20170629:14:39:13:041991 gpstop:192-168-102-67:gpadmin-[WARNING]:-No leftover gpmmon process found
20170629:14:39:13:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-Cleaning up leftover gpsmon processes
20170629:14:39:14:041991 gpstop:192-168-102-67:gpadmin-[WARNING]:-No leftover gpsmon processes on some hosts. not attempting forceful termination on these hosts
20170629:14:39:14:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-Cleaning up leftover shared memory
20170629:14:39:20:041991 gpstop:192-168-102-67:gpadmin-[INFO]:-Restarting System...
按正常步骤进行recovery
gprecoverseg
20170629:14:54:28:043652 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Starting gprecoverseg with args:
20170629:14:54:29:043652 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.4.1 build 2'
20170629:14:54:29:043652 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.4.1 build 2) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on Feb 10 2015 14:15:10'
20170629:14:54:29:043652 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Checking if segments are ready
20170629:14:54:29:043652 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Obtaining Segment details from master...
20170629:14:54:34:043652 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Obtaining Segment details from master...
20170629:14:54:37:043652 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Performing persistent table check
20170629:14:54:50:043652 gprecoverseg:192-168-102-67:gpadmin-[ERROR]:-Persistent table check gp_persistent_relation_node <=> filesystem failed on host 192-168-102-71:50001.
20170629:14:54:50:043652 gprecoverseg:192-168-102-67:gpadmin-[ERROR]:-Persistent table check gp_persistent_relation_node <=> filesystem failed on host 192-168-102-78:50000.
20170629:14:54:50:043652 gprecoverseg:192-168-102-67:gpadmin-[CRITICAL]:-gprecoverseg failed. (Reason='Persistent tables check failed. Please fix the persistent tables issues before running recoverseg') exiting...
查阅文档，我的版本postgres (Greenplum Database) 4.3.4.1 build 2
添加--skip-persistent-check参数再次进行recovery
gprecoverseg --skip-persistent-check
0170629:14:59:25:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Starting gprecoverseg with args: --skip-persistent-check
20170629:14:59:25:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.4.1 build 2'
20170629:14:59:25:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.4.1 build 2) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on Feb 10 2015 14:15:10'
20170629:14:59:26:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Checking if segments are ready
20170629:14:59:26:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Obtaining Segment details from master...
20170629:14:59:30:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Obtaining Segment details from master...
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Skipping persistent table check
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Greenplum instance recovery parameters
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:----------------------------------------------------------
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Recovery type = Standard
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:----------------------------------------------------------
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Recovery 1 of 2
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:----------------------------------------------------------
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Synchronization mode = Incremental
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Failed instance host = 192-168-102-70
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Failed instance address = 10-102-103-70if2
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Failed instance directory = /datastore_01/primary/gpseg5
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Failed instance port = 40001
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Failed instance replication port = 41001
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Failed instance da directory = /datastore_01/primary/schemas/da/gpseg5
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Failed instance dm directory = /datastore_01/primary/schemas/dm/gpseg5
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Failed instance ims directory = /datastore_01/primary/schemas/ims/gpseg5
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Failed instance openreport directory = /datastore_01/primary/schemas/openreport/gpseg5
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Failed instance report directory = /datastore_01/primary/schemas/report/gpseg5
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Recovery Source instance host = 192-168-102-71
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Recovery Source instance address = 10-102-104-71if3
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Recovery Source instance directory = /datastore_01/mirror/gpseg5
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Recovery Source instance port = 50001
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Recovery Source instance replication port = 51001
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Recovery Source instance da directory = /datastore_01/mirror/schemas/da/gpseg5
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Recovery Source instance dm directory = /datastore_01/mirror/schemas/dm/gpseg5
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Recovery Source instance ims directory = /datastore_01/mirror/schemas/ims/gpseg5
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Recovery Source instance openreport directory = /datastore_01/mirror/schemas/openreport/gpseg5
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Recovery Source instance report directory = /datastore_01/mirror/schemas/report/gpseg5
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Recovery Target = in-place
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:----------------------------------------------------------
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Recovery 2 of 2
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:----------------------------------------------------------
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Synchronization mode = Incremental
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Failed instance host = 192-168-102-77
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Failed instance address = 10-102-102-77if1
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Failed instance directory = /datastore_01/primary/gpseg32
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Failed instance port = 40000
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Failed instance replication port = 41000
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Failed instance da directory = /datastore_01/primary/schemas/da/gpseg32
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Failed instance dm directory = /datastore_01/primary/schemas/dm/gpseg32
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Failed instance ims directory = /datastore_01/primary/schemas/ims/gpseg32
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Failed instance openreport directory = /datastore_01/primary/schemas/openreport/gpseg32
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Failed instance report directory = /datastore_01/primary/schemas/report/gpseg32
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Recovery Source instance host = 192-168-102-78
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Recovery Source instance address = 10-102-103-78if2
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Recovery Source instance directory = /datastore_01/mirror/gpseg32
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Recovery Source instance port = 50000
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Recovery Source instance replication port = 51000
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Recovery Source instance da directory = /datastore_01/mirror/schemas/da/gpseg32
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Recovery Source instance dm directory = /datastore_01/mirror/schemas/dm/gpseg32
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Recovery Source instance ims directory = /datastore_01/mirror/schemas/ims/gpseg32
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Recovery Source instance openreport directory = /datastore_01/mirror/schemas/openreport/gpseg32
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Recovery Source instance report directory = /datastore_01/mirror/schemas/report/gpseg32
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:- Recovery Target = in-place
20170629:14:59:34:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:----------------------------------------------------------
Continue with segment recovery procedure Yy|Nn (default=N):
> y
20170629:15:00:15:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-2 segment(s) to recover
20170629:15:00:15:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Ensuring 2 failed segment(s) are stopped
20170629:15:00:16:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Ensuring that shared memory is cleaned up for stopped segments
updating flat files
20170629:15:00:23:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Updating configuration with new mirrors
20170629:15:00:24:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Updating mirrors
.
20170629:15:00:25:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Starting mirrors
20170629:15:00:25:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Commencing parallel primary and mirror segment instance startup, please wait...
....
20170629:15:00:29:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Process results...
20170629:15:00:29:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Updating configuration to mark mirrors up
20170629:15:00:29:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Updating primaries
20170629:15:00:29:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Commencing parallel primary conversion of 2 segments, please wait...
................................................................................................................................................................................................................................................................................................................................................................................................................................................................
20170629:15:07:58:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Process results...
20170629:15:07:58:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Done updating primaries
20170629:15:07:58:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-******************************************************************
20170629:15:07:58:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Updating segments for resynchronization is completed.
20170629:15:07:58:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-For segments updated successfully, resynchronization will continue in the background.
20170629:15:07:58:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-
20170629:15:07:58:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-Use gpstate -s to check the resynchronization progress.
20170629:15:07:58:044869 gprecoverseg:192-168-102-67:gpadmin-[INFO]:-******************************************************************
最后进行role对调
gprecoverseg -r
greenplum segment down 实例相关推荐
- Greenplum segment级问题的排查 - gp_session_role=utility (含csvlog日志格式解读)
标签 PostgreSQL , Greenplum , segment , utility 背景 有些时候,我们在使用Greenplum时,你会发现明明连接数是够的,但是报错的时候segment连接满 ...
- Greenplum Segment节点掉线恢复介绍
1. 背景 Greenplum版本:6.13.0 问题: Segment节点异常关机,恢复Segment节点并恢复Mirror节点状态. 如图所示:sdw1节点掉线. 2. 解决方法: 2.1 查看M ...
- greenplum segment恢复的过程
#此时已经知道坏了两个SEGMENT 在启动命令里加上-R以限制模式启动 [gpadmin1@hadoop1 ~]$ gpstart -R 20101027:14:11:55:gpstart:h ...
- 故障分析 | Greenplum Segment 故障处理
作者:杨文 DBA,负责客户项目的需求与维护,会点数据库,不限于MySQL.Redis.Cassandra.GreenPlum.ClickHouse.Elastic.TDSQL等等. 本文来源:原创投 ...
- Deepgreen Greenplum 高可用（一） - Segment节点故障转移
尚书中云:惟事事,乃其有备,有备无患.这教导我们做事一定要有准备,做事尚且如此,在企事业单位发展中处于基础地位的数据仓库软件在运行过程中,何尝不需要有备无患呢? 今天别的不表,主要来谈谈企业级数据仓库 ...
- Greenplum介绍
一.Greenplum的发展历史 Greenplum的发展可以分为下面6个阶段: 图 1 Greenplum时间线 Postgres关系型数据库.Postgres是UC Berkeley开发的关系型数 ...
- Greenplum——升级的分布式PostgresSQL
Greenplum数据库基于PostgreSQL开源技术.本质上讲,它是多个PostgreSQL实例一起充当一个数据库管理系统.Greenplum以PostgreSQL 8.2.15为基础构建,在SQ ...
- greenplum数据库集群的备份与恢复看这一篇就够啦！
针对greenplum-db-6.1.0的数据库集群的备份与恢复方案 使用gpbackup和gprestore的GPDB的备份方案 1.前言 2.Greenplum备份恢复发展历程 3.备份方案的优劣 ...
- 【GP】Greenplum入门解析（一）
(图片来源于网络,侵删) 注:本章内容主要来源于网络各种渠道,侵删 一.Greenplum概述 [1]Greenplum简介 Greenplum(以下简称GP)是一款开源数据仓库.基于开源的Postg ...
- Greenplum简介
Greenplum: http://greenplum.org/ 原来是个商业产品,后来开源. 从Slogan看: 是个数据库 着眼于数据仓库 主要在于大规模并行 基于强大的PostgreSQL,Po ...
最新文章
- iOS开发之网络编程--使用NSURLConnection实现大文件断点续传下载
- python 今日头条 微头条_头条号运营技巧：如何玩转微头条？
- linux每日命令(36): wc命令
- Android Navigation Drawer(导航抽屉)
- linux c glob使用(文件路径模式查找函数)
- 算法导论之字符串匹配
- TJA1050只能RX不能TX
- 一步一步教你使用AgileEAS.NET基础类库进行应用开发-WinForm应用篇-实现字典的打印...
- 雷蛇用户计算机不满足系统要求,我的RazerBook13已经升级Windows11了
- iSPRINT：Google 最高能的创新加速课程，清华老师都来给点赞！
- VSCode使用VIM复制内容到系统剪切板
- Android 系统签名打包方法
- 微信小程序之蓝牙打印
- 编程语言：8086汇编中int 16h接收alt+方向键
- 基本函数依赖和候选键_给定一组字母表示的函数依赖集，怎样确定候选键？
- rfm模型python_Python pandas RFM模型应用实例详解
- Neo4j 图数据科学应用 - 图目录和图投影（二）
- 基于SpringBoot HII健身房app
- 黑马C++学习总结之对象的初始化和清理
- cv曲线面积的意义_耳机的瀑布图真的有意义吗？关于耳机的瀑布图，你需要知道的。...