Hadoop BUG和采坑集锦—阿里云ECS配置Hadoop, datanode一直显示localhost, 下载文件失败

问题描述

 我个人的阿里云主机，在上面搭建Hadoop，主要配置如下：

/etc/hosts文件内容：

~etc/hadoop/hdfs-site.xml文件内容

为了方便大家复制粘贴下面是源码

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License. See accompanying LICENSE file.
--><!-- Put site-specific property overrides in this file. -->
<configuration><property><name>dfs.replication</name><value>1</value></property><property><name>dfs.datanode.http.address</name><value>aliyun:50075</value></property><property><name>dfs.datanode.address</name><value>aliyun:50010</value></property>
</configuration>

~etc/hadoop/core-site.xml文件内容：

为了方便大家复制粘贴下面是源码

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License. See accompanying LICENSE file.
--><!-- Put site-specific property overrides in this file. -->
<configuration><property><name>fs.defaultFS</name><value>hdfs://aliyun:9001</value></property><property><name>hadoop.tmp.dir</name><value>/opt/hadoop/hadoop-2.10.1/tmp/</value></property>
</configuration>

启动 datanode和namenode之后,然后访问结果：

还有下面浏览文件的时候准备 download的时候：

百思不得其解：因为在上面的hdfs-site.xml设置dfs.datanode.http.address为aliyun:50075寻思着为什么还是localhost不是aliyun呢？

然后就开始外网上冲浪，外海也去冲浪了，都说要配置hosts文件，我配置了，而且也生效（可以ping自己的域名）了，

然后问一个大佬，他告诉我，你修改一下主机名试试
然后他就让我改一下hostname试试改成aliyun，试试呗，

然后重启 datanode和namenode，其他的都没改动，你猜怎么着？

TMD，全体起立，成了。

然后我就寻思这不科学啊，这尼玛一定要将dfs.datanode.http.address设置成hostname这就离谱，约束性也太强了，而且我配置的时候是根据官方配置文件理解配置的，

这明明写的是Server address 不是hostname（个人认为这个是文档的bug）。然后我寻思着他们好像还有个配置
这里可以指定hostname，后面解释写的是默认是主机当前的hostname，但是也不对啊，我上面查看hostname的时候返回的是iZ2ze5ru3jzfp5m5bmaganZ，但是Hadoop显示的却是localhost（个人认为是bug），
为了举一反三，我就寻思着既然和hostname有关，刚好dfs.datanode.hostname可以指订hostname，并且将hostname先设置回iZ2ze5ru3jzfp5m5bmaganZ，再在hdfs-site.xml添加dfs.datanode.hostname设置看看会不会生效，操作步骤如下：

还原hostname
在hdfs-site.xml添加配置后如下：

为了方便大家复制粘贴下面是源码

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License. See accompanying LICENSE file.
--><!-- Put site-specific property overrides in this file. -->
<configuration><property><name>dfs.replication</name><value>1</value></property><property><name>dfs.datanode.hostname</name><value>aliyun</value></property><property><name>dfs.datanode.http.address</name><value>aliyun:50075</value></property><property><name>dfs.datanode.address</name><value>aliyun:50010</value></property>
</configuration>

然后重启datanode和namenode你猜怎么着，
卧槽绝了，

举一反三成功。

点击文件下载失败

因为在阿里云上部署的，只能部署内网ip，我上面的hosts的文件配置映射aliyun也是映射的内网，不然根本启动不起来，所以这里你点击download的时候，就会失败，出现下面的画面

因为aliyun上面只能配置内网，所以这里自动跳转的时候的域名也是hadoop上面配置的aliyun:50075,但是你本地没办法解析这个域名，所以要想成功就需要你自己在自己本地的电脑上的hosts文件上面添加

这样就成了，或者你直接

这样就妥了。