今天磁盘链路故障，登陆系统后发现数据库已经宕机了，日志记录是IO error

操作系统日志

[11640320.581749] sd 12:0:0:2: [sdi] Write Protect is off
[11640320.581751] sd 12:0:0:2: [sdi] Mode Sense: 6b 00 00 08
[11640320.582015] sd 12:0:0:2: [sdi] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[11640320.582126] sd 12:0:0:1: [sdh] Very big device. Trying to use READ CAPACITY(16).
[11640320.591970] sdi: sdi1
[11640350.851141] qla2xxx [0000:82:00.0]-801c:12: Abort command issued nexus=12:0:0 -- 1 2002.
[11640350.852099] sd 12:0:0:0: [sdd] Attached SCSI disk
[11640351.851404] qla2xxx [0000:82:00.0]-801c:12: Abort command issued nexus=12:0:0 -- 1 2002.
[11640351.851604] qla2xxx [0000:82:00.0]-801c:12: Abort command issued nexus=12:0:1 -- 1 2002.
[11640351.851796] qla2xxx [0000:82:00.0]-801c:12: Abort command issued nexus=12:0:2 -- 1 2002.
[11640351.857535] sdh: unknown partition table
[11640351.857882] sd 12:0:0:1: [sdh] Very big device. Trying to use READ CAPACITY(16).
[11640351.858692] sd 12:0:0:1: [sdh] Attached SCSI disk
[11640381.939887] qla2xxx [0000:82:00.0]-801c:12: Abort command issued nexus=12:0:2 -- 1 2002.
[11640381.940712] sd 12:0:0:2: [sdi] Attached SCSI disk
[11640390.879505] qla2xxx [0000:82:00.0]-801c:12: Abort command issued nexus=12:0:2 -- 1 2002.
[11640421.856372] qla2xxx [0000:82:00.0]-801c:12: Abort command issued nexus=12:0:2 -- 1 2002.
[11643411.613773] EXT4-fs (dm-2): error count since last fsck: 9
[11643411.613778] EXT4-fs (dm-2): initial error at time 1511349539: ext4_writepages:2414
[11643411.613781] EXT4-fs (dm-2): last error at time 1511349539: ext4_journal_check_start:56
[11646750.394697] rport-12:0-0: blocked FC remote port time out: removing target and saving binding
[11646750.396193] sd 12:0:0:0: [sdd] Synchronizing SCSI cache
[11646750.396215] sd 12:0:0:0: [sdd] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[11646750.411178] sd 12:0:0:1: [sdh] Synchronizing SCSI cache
[11646750.411194] sd 12:0:0:1: [sdh] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[11646750.412116] sd 12:0:0:2: [sdi] Synchronizing SCSI cache
[11646750.412129] sd 12:0:0:2: [sdi] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[11646767.708015] scsi 12:0:0:0: Direct-Access HITACHI OPEN-V 8301 PQ: 0 ANSI: 3
[11646767.708735] sd 12:0:0:0: Attached scsi generic sg4 type 0
[11646767.709158] sd 12:0:0:0: [sdd] 209715200 512-byte logical blocks: (107 GB/100 GiB)
[11646767.709436] sd 12:0:0:0: [sdd] Write Protect is off
[11646767.709439] sd 12:0:0:0: [sdd] Mode Sense: 6b 00 00 08
[11646767.709641] sd 12:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[11646767.710477] scsi 12:0:0:1: Direct-Access HITACHI OPEN-V 8301 PQ: 0 ANSI: 3
[11646767.711081] sdd: unknown partition table
[11646767.711311] sd 12:0:0:1: Attached scsi generic sg5 type 0
[11646767.711792] sd 12:0:0:0: [sdd] Attached SCSI disk
[11646767.711819] scsi 12:0:0:2: Direct-Access HITACHI OPEN-V 8301 PQ: 0 ANSI: 3
[11646767.712207] sd 12:0:0:1: [sde] Very big device. Trying to use READ CAPACITY(16).
[11646767.712345] sd 12:0:0:1: [sde] 4294967296 512-byte logical blocks: (2.19 TB/2.00 TiB)
[11646767.712620] sd 12:0:0:2: Attached scsi generic sg7 type 0
[11646767.712724] sd 12:0:0:1: [sde] Write Protect is off
[11646767.712726] sd 12:0:0:1: [sde] Mode Sense: 6b 00 00 08
[11646767.712858] sd 12:0:0:2: [sdg] 1048576000 512-byte logical blocks: (536 GB/500 GiB)
[11646767.712952] sd 12:0:0:2: [sdg] Write Protect is off
[11646767.712954] sd 12:0:0:2: [sdg] Mode Sense: 6b 00 00 08
[11646767.713040] sd 12:0:0:1: [sde] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[11646767.713179] sd 12:0:0:2: [sdg] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[11646767.713546] sd 12:0:0:1: [sde] Very big device. Trying to use READ CAPACITY(16).
[11646767.714450] sde: unknown partition table
[11646767.714800] sd 12:0:0:1: [sde] Very big device. Trying to use READ CAPACITY(16).
[11646767.715479] sd 12:0:0:1: [sde] Attached SCSI disk
[11646767.721519] sdg: sdg1
[11646767.722228] sd 12:0:0:2: [sdg] Attached SCSI disk
[11646797.741521] qla2xxx [0000:82:00.0]-801c:12: Abort command issued nexus=12:0:0 -- 1 2002.
[11646828.766379] qla2xxx [0000:82:00.0]-801c:12: Abort command issued nexus=12:0:0 -- 1 2002.
[11654134.042047] rport-12:0-0: blocked FC remote port time out: removing target and saving binding
[11654134.042715] sd 12:0:0:0: [sdd] Synchronizing SCSI cache
[11654134.042735] sd 12:0:0:0: [sdd] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[11654134.056655] sd 12:0:0:1: [sde] Synchronizing SCSI cache
[11654134.056686] sd 12:0:0:1: [sde] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[11654134.058829] sd 12:0:0:2: [sdg] Synchronizing SCSI cache
[11654134.058841] sd 12:0:0:2: [sdg] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[11654269.600683] scsi 12:0:0:0: Direct-Access HITACHI OPEN-V 8301 PQ: 0 ANSI: 3
[11654269.601393] sd 12:0:0:0: Attached scsi generic sg4 type 0
[11654269.601821] sd 12:0:0:0: [sdd] 209715200 512-byte logical blocks: (107 GB/100 GiB)
[11654269.601913] sd 12:0:0:0: [sdd] Write Protect is off
[11654269.601916] sd 12:0:0:0: [sdd] Mode Sense: 6b 00 00 08
[11654269.602185] sd 12:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[11654269.603154] scsi 12:0:0:1: Direct-Access HITACHI OPEN-V 8301 PQ: 0 ANSI: 3
[11654269.603706] sdd: unknown partition table
[11654269.603891] sd 12:0:0:1: Attached scsi generic sg5 type 0
[11654269.604352] sd 12:0:0:0: [sdd] Attached SCSI disk
[11654269.604379] scsi 12:0:0:2: Direct-Access HITACHI OPEN-V 8301 PQ: 0 ANSI: 3
[11654269.605063] sd 12:0:0:2: Attached scsi generic sg7 type 0
[11654269.605339] sd 12:0:0:2: [sdi] 1048576000 512-byte logical blocks: (536 GB/500 GiB)
[11654269.605452] sd 12:0:0:2: [sdi] Write Protect is off
[11654269.605453] sd 12:0:0:2: [sdi] Mode Sense: 6b 00 00 08
[11654269.605653] sd 12:0:0:2: [sdi] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[11654269.614459] sdi: sdi1
[11654269.615149] sd 12:0:0:2: [sdi] Attached SCSI disk
[11654300.115015] qla2xxx [0000:82:00.0]-801c:12: Abort command issued nexus=12:0:0 -- 1 2002.
[11654301.115185] qla2xxx [0000:82:00.0]-801c:12: Abort command issued nexus=12:0:1 -- 1 2002.
[11654301.115487] qla2xxx [0000:82:00.0]-801c:12: Abort command issued nexus=12:0:2 -- 1 2002.
[11654302.117764] sd 12:0:0:1: [sdh] Very big device. Trying to use READ CAPACITY(16).
[11654302.117994] sd 12:0:0:1: [sdh] 4294967296 512-byte logical blocks: (2.19 TB/2.00 TiB)
[11654302.118219] sd 12:0:0:1: [sdh] Write Protect is off
[11654302.118222] sd 12:0:0:1: [sdh] Mode Sense: 6b 00 00 08
[11654302.118611] sd 12:0:0:1: [sdh] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[11654302.119111] sd 12:0:0:1: [sdh] Very big device. Trying to use READ CAPACITY(16).
[11654302.120087] sdh: unknown partition table
[11654302.120377] sd 12:0:0:1: [sdh] Very big device. Trying to use READ CAPACITY(16).
[11654302.121181] sd 12:0:0:1: [sdh] Attached SCSI disk
[11654333.122856] qla2xxx [0000:82:00.0]-801c:12: Abort command issued nexus=12:0:2 -- 1 2002.
[11655264.634648] qla2xxx [0000:82:00.0]-801c:12: Abort command issued nexus=12:0:1 -- 1 2002.
[11655295.595455] qla2xxx [0000:82:00.0]-801c:12: Abort command issued nexus=12:0:2 -- 1 2002.
[11729876.334629] EXT4-fs (dm-2): error count since last fsck: 9
[11729876.334635] EXT4-fs (dm-2): initial error at time 1511349539: ext4_writepages:2414
[11729876.334638] EXT4-fs (dm-2): last error at time 1511349539: ext4_journal_check_start:56
[11816341.072955] EXT4-fs (dm-2): error count since last fsck: 9
[11816341.072960] EXT4-fs (dm-2): initial error at time 1511349539: ext4_writepages:2414

里面有fs，hba卡，system的报错，定位链路问题，告诉存储的大神。。。。

喝完咖啡后，重启系统，问题解决。

tips：由于用的是日立的存储，建议用新版的多路径软件，规划好磁盘分完区后，多重启几遍系统，刷新配置后看日志，没有问题才投入生产环境

转载于:https://www.cnblogs.com/automng/p/7991377.html

linux磁盘链路故障相关推荐

  1. php 文件写入磁盘错误,Linux磁盘读写故障的通常处理流程

    问题 使用Linux作为服务器操作系统的时候 维护少不需要重新启动的优势非常明显 但有时候会遇到莫明其妙的故障 例如数据无法写入,执行某程序报错 这个时候建议检查一下,是否是磁盘子系统的故障 例如SC ...

  2. linux 磁盘挂载故障,LVM讲解和磁盘挂载故障

    LVM是 Logical Volume Manager(逻辑卷管理)的简写,它是Linux环境下对磁盘分区进行管理的一种机制,使硬盘不必使用分区也能被简单地重新划分大小. 首先我们先准备磁盘分区,模拟 ...

  3. linux磁盘空间df-h出不出来,linux 故障：df -h统计磁盘空间占用太多，但又du -h找不到大的文件...

    用lsof / | grep -i delete 从根目录定位打开的被删除的文件 如果定位到某文件占用空间很大 主要是因为我们在删除这个日志文件的时候是用rm -rf *.log这样的命令删除的,删除 ...

  4. linux mysql io压力大_MySQL性能调优(四) Linux 磁盘IO

    1. IO处理过程 磁盘IO经常会成为系统的一个瓶颈,特别是对于运行数据库的系统而言.数据从磁盘读取到内存,在到CPU缓存和寄存器,然后进行处理,最后写回磁盘,中间要经过很多的过程,下图是一个以wri ...

  5. Linux磁盘管理——df、du、磁盘分区、格式化、挂载、LVM

    2019独角兽企业重金招聘Python工程师标准>>> Linux磁盘管理 一. df命令 df命令参数功能:检查文件系统的磁盘空间占用情况.可以利用该命令来获取硬盘被占用了多少空间 ...

  6. linux 磁盘资源管理以及IO

    柱面:Cylinder :一个环形磁道的面Cylinder =Track,早期的磁盘一个面上有1024个磁柱面:现在磁盘(服务器上的磁盘)做的很大,一个磁面上可以达到十几万的柱面数. 磁道:Track ...

  7. Linux的性能故障的含义,Linux排查性能故障的方法

    好了,鉴于我们的Linux发行版现已安装,许多方面看起来似乎根本不"好".今天我们将介绍为安装的Linux排查性能故障的方法.虽然今天的这篇文章明显倾向于Ubuntu,但我们探讨的 ...

  8. linux mysql io压力大_MySQL 调优基础(四) Linux 磁盘IO_MySQL

    1. IO处理过程 磁盘IO经常会成为系统的一个瓶颈,特别是对于运行数据库的系统而言.数据从磁盘读取到内存,在到CPU缓存和寄存器,然后进行处理,最后写回磁盘,中间要经过很多的过程,下图是一个以wri ...

  9. Linux磁盘术语描述

    文章目录 Linux磁盘术语描述 什么是磁盘 软盘(Floppy Disk) 硬盘 硬盘的接口类型 IDE SATA SCSI 光纤通道 SAS 非DOS分区 主分区 扩展分区 分区格式 FAT16 ...

最新文章

  1. 软件架构自学笔记——非功能特性
  2. Socket粘包问题
  3. Vue菜鸟之路: 封装上传组件
  4. Intger To Roman
  5. 猿类如何捕获少女心--难以琢磨的try-catch
  6. JavaScript自学笔记（1）---表单验证，let和const,JSON文件
  7. kafka zookeeper java_简单搭建kafka + zookeeper，附简单Java生产和消费客户端
  8. java重定向并保持同一个会话_当会话在Java Web应用程序中过期时，如何重定向到登录页面？...
  9. css的重置和原子类的使用
  10. 在.net中序列化读写xml方法的总结
  11. 大学计算机基础模拟系统2014综合第三套,秋第三套计算机一级试题.doc
  12. 全站仪数据导入电脑_怎么把全站仪的数据导到电脑上来，并且成图？
  13. 手足之爱，平生一人：他们是中国历史上感情最好的一对兄弟 （苏轼苏辙，邓林武邓林飞）
  14. 官网下载历史版本的tomcat
  15. java 俄罗斯方块原理,俄罗斯方块设计原理
  16. 移动平均线rolling()与加权移动平均线ewm()
  17. 如何调节肠道菌群？常见天然物质、益生菌、益生元的介绍
  18. People seldom do what they believe in. They do what is convenient, then repent.
  19. 有什么好用的财务分析软件？
  20. 微机原理与接口技术系列笔记（一）

热门文章

  1. JavaScript 中 apply 、call 的详解
  2. BZOJ 1004: [HNOI2008]Cards [Polya 生成函数DP]
  3. 自定义UITabBar
  4. 纠正网上Mac 上使用Hbuilder运行夜神游模拟器，Hbuilder找不到模拟器
  5. HTML+CSS制作旋转的loading效果
  6. c++字符数组整数转换中文大写金额的形式_如何对PHP日期数组进行排序
  7. DOCX%20是什么格式
  8. 基金投资需要注意什么？
  9. 过本命年有什么讲究吗？
  10. 高人，不断地在营销， 有收入就是高人