There are 7 missing blocks. The following files may be corrupted

断电、hdfs web提示blocks missing

[root@nufront-master-01 ~]# hdfs fsck /
Connecting to namenode via http://nufront-master-02:50070/fsck?ugi=root&path=%2F
FSCK started by root (auth:SIMPLE) from /172.16.34.121 for path / at Fri May 21 16:36:27 CST 2021/tmp/hive/root/62665454-411d-4de2-b177-d5daca2cba98/hive_2021-05-21_12-06-58_867_4007960590733956456-1/-mr-10004/HashTable-Stage-5/Stage-5.tar.gz: CORRUPT blockpool BP-1719687234-172.16.34.123-1607417729793 block blk_1076309041/tmp/hive/root/62665454-411d-4de2-b177-d5daca2cba98/hive_2021-05-21_12-06-58_867_4007960590733956456-1/-mr-10004/HashTable-Stage-5/Stage-5.tar.gz: CORRUPT 1 blocks of total size 8702 B.
/tmp/hive/root/62665454-411d-4de2-b177-d5daca2cba98/hive_2021-05-21_12-06-58_867_4007960590733956456-1/-mr-10010/d77543dd-435e-4ef7-b87f-5383c7959499/map.xml: CORRUPT blockpool BP-1719687234-172.16.34.123-1607417729793 block blk_1076309042/tmp/hive/root/62665454-411d-4de2-b177-d5daca2cba98/hive_2021-05-21_12-06-58_867_4007960590733956456-1/-mr-10010/d77543dd-435e-4ef7-b87f-5383c7959499/map.xml: CORRUPT 1 blocks of total size 9518 B.
/tmp/logs/root/logs/application_1621214977998_5719/nufront-worker-03_8041: CORRUPT blockpool BP-1719687234-172.16.34.123-1607417729793 block blk_1076309087/tmp/logs/root/logs/application_1621214977998_5719/nufront-worker-03_8041: CORRUPT 1 blocks of total size 155671 B.
/user/history/done/2021/05/21/000005/job_1621214977998_5719-1621570859432-root-insert+into+table+village...t1.area_id%3Dt2.id+%28-1621570872686-1-0-SUCCEEDED-root.users.root-1621570865414.jhist: CORRUPT blockpool BP-1719687234-172.16.34.123-1607417729793 block blk_1076309084/user/history/done/2021/05/21/000005/job_1621214977998_5719-1621570859432-root-insert+into+table+village...t1.area_id%3Dt2.id+%28-1621570872686-1-0-SUCCEEDED-root.users.root-1621570865414.jhist: CORRUPT 1 blocks of total size 17998 B.
/user/history/done/2021/05/21/000005/job_1621214977998_5719_conf.xml: CORRUPT blockpool BP-1719687234-172.16.34.123-1607417729793 block blk_1076309085/user/history/done/2021/05/21/000005/job_1621214977998_5719_conf.xml: CORRUPT 1 blocks of total size 374485 B.
/user/hive/warehouse/villagemonitor.db/dws_nanxiong_routerflow/partday=20200929/.hive-staging_hive_2021-05-21_12-06-58_867_4007960590733956456-1/-ext-10001/tmpstats-0: CORRUPT blockpool BP-1719687234-172.16.34.123-1607417729793 block blk_1076309081/user/hive/warehouse/villagemonitor.db/dws_nanxiong_routerflow/partday=20200929/.hive-staging_hive_2021-05-21_12-06-58_867_4007960590733956456-1/-ext-10001/tmpstats-0: CORRUPT 1 blocks of total size 111 B.
/user/hive/warehouse/villagemonitor.db/dws_nanxiong_routerflow/partday=20200929/.hive-staging_hive_2021-05-21_12-06-58_867_4007960590733956456-1/_tmp.-ext-10000/000000_0: CORRUPT blockpool BP-1719687234-172.16.34.123-1607417729793 block blk_1076309080/user/hive/warehouse/villagemonitor.db/dws_nanxiong_routerflow/partday=20200929/.hive-staging_hive_2021-05-21_12-06-58_867_4007960590733956456-1/_tmp.-ext-10000/000000_0: CORRUPT 1 blocks of total size 3837 B.
Status: CORRUPTNumber of data-nodes:  3Number of racks:               1Total dirs:                    20757Total symlinks:                0Replicated Blocks:Total size:    119664216828 BTotal files:   45863 (Files currently being written: 6)Total blocks (validated):      46465 (avg. block size 2575362 B) (Total open file blocks (not validated): 4)********************************UNDER MIN REPL'D BLOCKS:      7 (0.015065103 %)dfs.namenode.replication.min: 1CORRUPT FILES:        7CORRUPT BLOCKS:       7CORRUPT SIZE:         570322 B********************************Minimally replicated blocks:   46458 (99.98493 %)Over-replicated blocks:        0 (0.0 %)Under-replicated blocks:       0 (0.0 %)Mis-replicated blocks:         0 (0.0 %)Default replication factor:    3Average block replication:     2.999505Missing blocks:                0Corrupt blocks:                7Missing replicas:              0 (0.0 %)Blocks queued for replication: 0Erasure Coded Block Groups:Total size:    0 BTotal files:   0Total block groups (validated):        0Minimally erasure-coded block groups:  0Over-erasure-coded block groups:       0Under-erasure-coded block groups:      0Unsatisfactory placement block groups: 0Average block group size:      0.0Missing block groups:          0Corrupt block groups:          0Missing internal blocks:       0Blocks queued for replication: 0
FSCK ended at Fri May 21 16:36:28 CST 2021 in 837 millisecondsThe filesystem under path '/' is CORRUPT
[root@nufront-master-01 ~]# hdfs fsck -list-corruptfileblocks
Connecting to namenode via http://nufront-master-02:50070/fsck?ugi=root&listcorruptfileblocks=1&path=%2F
The list of corrupt files under path '/' are:
blk_1076309042  /tmp/hive/root/62665454-411d-4de2-b177-d5daca2cba98/hive_2021-05-21_12-06-58_867_4007960590733956456-1/-mr-10010/d77543dd-435e-4ef7-b87f-5383c7959499/map.xml
blk_1076309080  /user/hive/warehouse/villagemonitor.db/dws_nanxiong_routerflow/partday=20200929/.hive-staging_hive_2021-05-21_12-06-58_867_4007960590733956456-1/_tmp.-ext-10000/000000_0
blk_1076309081  /user/hive/warehouse/villagemonitor.db/dws_nanxiong_routerflow/partday=20200929/.hive-staging_hive_2021-05-21_12-06-58_867_4007960590733956456-1/-ext-10001/tmpstats-0
blk_1076309084  /user/history/done/2021/05/21/000005/job_1621214977998_5719-1621570859432-root-insert+into+table+village...t1.area_id%3Dt2.id+%28-1621570872686-1-0-SUCCEEDED-root.users.root-1621570865414.jhist
blk_1076309085  /user/history/done/2021/05/21/000005/job_1621214977998_5719_conf.xml
blk_1076309087  /tmp/logs/root/logs/application_1621214977998_5719/nufront-worker-03_8041
blk_1076309041  /tmp/hive/root/62665454-411d-4de2-b177-d5daca2cba98/hive_2021-05-21_12-06-58_867_4007960590733956456-1/-mr-10004/HashTable-Stage-5/Stage-5.tar.gz
The filesystem under path '/' has 7 CORRUPT files
[root@nufront-master-01 ~]# hdfs fsck -delete /xxx

解决方法：

如果文件不重要，可以直接删除此文件(hdfs fsck -delete)；或删除后重新复制一份到集群中
如果不能删除，需要从上面命令中找到发生在哪台机器上，然后到此机器上查看日志。

常用命令介绍：

1、查看节点、hdfs、丢失的数据块命令：hadoop dfsadmin -report

2、查看文件系统的健康状况：hdfs fsck --<path> [-options]

[root@nufront-master-01 ~]# hdfs fsck -help
Usage: hdfs fsck <path> [-list-corruptfileblocks | [-move | -delete | -openforwrite] [-files [-blocks [-locations | -racks | -replicaDetails | -upgradedomains]]]] [-includeSnapshots] [-showprogress] [-storagepolicies] [-maintenance] [-blockId <blk_Id>] [-replicate]<path>  start checking from this path-move   move corrupted files to /lost+found-delete delete corrupted files-files  print out files being checked-openforwrite   print out files opened for write-includeSnapshots       include snapshot data if the given path indicates a snapshottable directory or there are snapshottable directories under it-list-corruptfileblocks print out list of missing blocks and files they belong to-files -blocks  print out block report-files -blocks -locations       print out locations for every block-files -blocks -racks   print out network topology for data-node locations-files -blocks -replicaDetails  print out each replica details-files -blocks -upgradedomains  print out upgrade domains for every block-storagepolicies        print out storage policy summary for the blocks-maintenance    print out maintenance state node details-showprogress   show progress in output. Default is OFF (no progress)-blockId        print out which file this blockId belongs to, locations (nodes, racks) of this block, and other diagnostics info (under replicated, corrupted or not, etc)-replicate initiate replication work to make mis-replicatedblocks satisfy block placement policy

hadoop fsck / 用这个命令可以检查整个文件系统的健康状况，但是要注意它不会主动恢复备份缺失的block，这个是由NameNode单独的线程异步处理的。

3、手动修复 hdfs debug

[root@nufront-master-01 ~]# hdfs debug --help
Usage: hdfs debug <command> [arguments]These commands are for advanced users only.Incorrect usages may result in data loss. Use at your own risk.verifyMeta -meta <metadata-file> [-block <block-file>]
computeMeta -block <block-file> -out <output-metadata-file>
recoverLease -path <path> [-retries <num-retries>]
[root@nufront-master-01 ~]# hdfs debug recoverLease -path /xxx -retries 10

4、如果hadoop不能自动恢复，则只能删除 corrupted blocks;

hdfs fsck -delete

参考：https://blog.csdn.net/lingbo229/article/details/81128316

https://blog.51cto.com/xiaolanlan/2071642

https://blog.csdn.net/qq_40337206/article/details/99900310

https://www.cnblogs.com/tesla-turing/p/11487899.html

https://hadoop.apache.org/docs/r3.2.0/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html