win10 安装 hadoop 3.3.1报错 Unable to load native-hadoop library

安装好hadoop后, 配置了HADOOP_HOME, Path

命令行执行 hdfs dfs -ls /

报错

2022-10-24 21:00:12,070 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:793)
at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:1215)
at org.apache.hadoop.fs.FileUtil.list(FileUtil.java:1420)
at org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:601)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1972)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:2014)
at org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:761)
at org.apache.hadoop.fs.shell.PathData.getDirectoryContents(PathData.java:269)
at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:439)
at org.apache.hadoop.fs.shell.Ls.processPathArgument(Ls.java:242)
at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:286)
at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:270)
at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:121)
at org.apache.hadoop.fs.shell.Command.run(Command.java:177)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:327)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:390)

解决方案

网上的很多方法,有点乱,找篇很细致的讲解放这里, 其实主要是没有启动hadoop

Spark | Hadoop - Exception in thread “main” java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z - Spark by {Examples}

下载winutils.exe , 参考下面这个链接,内部比较新

GitHub - kontext-tech/winutils: winutils.exe hadoop.dll and hdfs.dll binaries for hadoop windows

下载

winutils.exe, hadoop.dll 放到 %HADOOP_HOME%\bin 目录, 有时可能需要将 hadoop.dll 放到 C:/Windows/System32 目录, 我验证了一下, 只需要放在bin目录即可, 不可以仅放system32,仅放这里没效果, 放 %HADOOP_HOME%\bin 目录很重要, 再将重启命令行执行

hdfs dfs -ls /

仍然报错

C:\Users\HUAWEI>hdfs dfs -ls /
Found 21 items
2022-10-24 21:07:59,378 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: $Recycle.Bin
2022-10-24 21:07:59,422 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: $WinREAgent
2022-10-24 21:07:59,459 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: Config.Msi
2022-10-24 21:07:59,493 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: Config.Msi
2022-10-24 21:07:59,527 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: Documents and Settings
2022-10-24 21:07:59,562 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: DumpStack.log.tmp
2022-10-24 21:07:59,595 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: DumpStack.log.tmp
2022-10-24 21:07:59,629 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: Intel
2022-10-24 21:07:59,665 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: PerfLogs
2022-10-24 21:07:59,702 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: PerfLogs
2022-10-24 21:07:59,737 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: Program Files
2022-10-24 21:07:59,771 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: Program Files (x86)
2022-10-24 21:07:59,808 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: ProgramData
2022-10-24 21:07:59,845 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: Python310
2022-10-24 21:07:59,882 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: Recovery
2022-10-24 21:07:59,918 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: Recycler
2022-10-24 21:07:59,957 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: System Volume Information
2022-10-24 21:07:59,994 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: System Volume Information
2022-10-24 21:08:00,032 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: Users
2022-10-24 21:08:00,069 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: Windows
2022-10-24 21:08:00,106 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: Windows.old
2022-10-24 21:08:00,145 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: hiberfil.sys
2022-10-24 21:08:00,182 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: hiberfil.sys
2022-10-24 21:08:00,216 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: jdk1.8.0_333
2022-10-24 21:08:00,255 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: pagefile.sys
2022-10-24 21:08:00,293 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: pagefile.sys
2022-10-24 21:08:00,328 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: swapfile.sys
2022-10-24 21:08:00,364 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: swapfile.sys
drwxrwx---   - SYSTEM           SYSTEM                 4096 2022-10-08 11:20 /$Recycle.Bin
drwxrwx---   - Administrators   Administrators            0 2022-10-17 10:01 /$WinREAgent
2022-10-24 21:08:00,401 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: Config.Msi
2022-10-24 21:08:00,440 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: Config.Msi
2022-10-24 21:08:00,478 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: Config.Msi
drw-rw-rw-   -                                            0 1970-01-01 08:00 /Config.Msi
d-wx-wx--x   - SYSTEM           SYSTEM                 4096 2022-10-19 14:37 /Documents and Settings
2022-10-24 21:08:00,514 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: DumpStack.log.tmp
2022-10-24 21:08:00,551 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: DumpStack.log.tmp
2022-10-24 21:08:00,587 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: DumpStack.log.tmp
-rw-rw-rw-   1                                         8192 1970-01-01 08:00 /DumpStack.log.tmp
drwxrwx---   - Administrators   SYSTEM                    0 2022-10-24 20:50 /Intel
2022-10-24 21:08:00,623 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: PerfLogs
2022-10-24 21:08:00,658 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: PerfLogs
2022-10-24 21:08:00,695 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: PerfLogs
drw-rw-rw-   -                                            0 1970-01-01 08:00 /PerfLogs
drwxrwx---   - TrustedInstaller TrustedInstaller      12288 2022-10-24 17:24 /Program Files
drwxrwx---   - TrustedInstaller TrustedInstaller       8192 2022-10-24 19:46 /Program Files (x86)
drwxrwx---   - SYSTEM           SYSTEM                 8192 2022-10-24 19:46 /ProgramData
drwxrwx---   - Administrators   SYSTEM                 4096 2022-10-17 15:09 /Python310
drwxrwx---   - SYSTEM           SYSTEM                    0 2022-10-19 14:37 /Recovery
drwxrwx---   - SYSTEM           SYSTEM                    0 2022-10-09 03:01 /Recycler
2022-10-24 21:08:00,738 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: System Volume Information
2022-10-24 21:08:00,774 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: System Volume Information
2022-10-24 21:08:00,813 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: System Volume Information
drw-rw-rw-   -                                         4096 1970-01-01 08:00 /System Volume Information
drwxrwxr-x   - SYSTEM           SYSTEM                 4096 2022-10-19 14:37 /Users
drwxrwx---   - TrustedInstaller TrustedInstaller      16384 2022-10-19 14:42 /Windows
drwxrwx---   - SYSTEM           SYSTEM                 4096 2022-10-19 14:42 /Windows.old
2022-10-24 21:08:00,852 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: hiberfil.sys
2022-10-24 21:08:00,887 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: hiberfil.sys
2022-10-24 21:08:00,924 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: hiberfil.sys
-rw-rw-rw-   1                                   6807244800 1970-01-01 08:00 /hiberfil.sys
d---------   - HUAWEI           None                   4096 2022-10-24 16:54 /jdk1.8.0_333
2022-10-24 21:08:00,961 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: pagefile.sys
2022-10-24 21:08:00,998 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: pagefile.sys
2022-10-24 21:08:01,035 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: pagefile.sys
-rw-rw-rw-   1                                   4710393856 1970-01-01 08:00 /pagefile.sys
2022-10-24 21:08:01,072 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: swapfile.sys
2022-10-24 21:08:01,107 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: swapfile.sys
2022-10-24 21:08:01,145 WARN nativeio.NativeIO: NativeIO.getStat error (2): 系统找不到指定的文件。-- file path: swapfile.sys
-rw-rw-rw-   1                                     16777216 1970-01-01 08:00 /swapfile.sys

这里面的原因就可以理解了,找不到目录因为没有按教程开始hadoop

参考教程设置一遍 Hadoop2OnWindows - HADOOP2 - Apache Software Foundation

由于我设置了HADOOP_HOME及path

所以这个文件hadoop-env.cmd的修改就没有按教程来,而是这样

set HADOOP_PREFIX=%HADOOP_HOME%
set HADOOP_CONF_DIR=%HADOOP_PREFIX%\etc\hadoop
set YARN_CONF_DIR=%HADOOP_CONF_DIR%

根据需要, 需要修改数据存储目录的,修改一下

如

core-site.xml

<configuration><property><name>hadoop.tmp.dir</name><value>/D:/hadoop-3.3.1/data/tmp</value></property><property><name>dfs.name.dir</name><value>/D:/hadoop-3.3.1/data/name</value></property><property><name>fs.default.name</name><value>hdfs://localhost:9000</value></property>
</configuration>

hdfs-site.xml

<configuration><property><name>dfs.replication</name><value>1</value></property><property><name>dfs.data.dir</name><value>/D:/hadoop-3.3.1/data/dfsdata</value></property>
</configuration>

mapred-site.xml

<configuration><property><name>mapreduce.job.user.name</name><value>%USERNAME%</value></property><property><name>mapreduce.framework.name</name><value>yarn</value></property><property><name>yarn.apps.stagingDir</name><value>/D:/hadoop-3.3.1/data/%USERNAME%/staging</value></property><property><name>mapreduce.jobtracker.address</name><value>local</value></property></configuration>

yarn-site.xml

<configuration><property><name>yarn.server.resourcemanager.address</name><value>0.0.0.0:8020</value></property><property><name>yarn.server.resourcemanager.application.expiry.interval</name><value>60000</value></property><property><name>yarn.server.nodemanager.address</name><value>0.0.0.0:45454</value></property><property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value></property><property><name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name><value>org.apache.hadoop.mapred.ShuffleHandler</value></property><property><name>yarn.server.nodemanager.remote-app-log-dir</name><value>/D:/hadoop-3.3.1/data/app-logs</value></property><property><name>yarn.nodemanager.log-dirs</name><value>/D:/hadoop-3.3.1/data/userlogs</value></property><property><name>yarn.server.mapreduce-appmanager.attempt-listener.bindAddress</name><value>0.0.0.0</value></property><property><name>yarn.server.mapreduce-appmanager.client-service.bindAddress</name><value>0.0.0.0</value></property><property><name>yarn.log-aggregation-enable</name><value>true</value></property><property><name>yarn.log-aggregation.retain-seconds</name><value>-1</value></property><property><name>yarn.application.classpath</name><value>%HADOOP_CONF_DIR%,%HADOOP_COMMON_HOME%/share/hadoop/common/*,%HADOOP_COMMON_HOME%/share/hadoop/common/lib/*,%HADOOP_HDFS_HOME%/share/hadoop/hdfs/*,%HADOOP_HDFS_HOME%/share/hadoop/hdfs/lib/*,%HADOOP_MAPRED_HOME%/share/hadoop/mapreduce/*,%HADOOP_MAPRED_HOME%/share/hadoop/mapreduce/lib/*,%HADOOP_YARN_HOME%/share/hadoop/yarn/*,%HADOOP_YARN_HOME%/share/hadoop/yarn/lib/*</value></property>
</configuration>

修改完成后, 初始化

命令行中运行

%HADOOP_HOME%\etc\hadoop\hadoop-env.cmd

执行完成,没有任何输出

可以通过打印 echo %HADOOP_PREFIX% 检查是否配置正确

然后接着初始化

%HADOOP_PREFIX%\bin\hdfs namenode -format

运行成功没有报错, 此时数据目录之tmp目录应该有内容

运行命令

%HADOOP_PREFIX%\sbin\start-dfs.cmd 启动hadoop

启动成功,验证

cd %HADOOP_PREFIX%

上传一个文件

hdfs dfs -put README.txt /

检查文件是否上传成功

hdfs dfs -ls /

文件上传成功, 表示可用

win10 安装 hadoop 3.3.1报错 Unable to load native-hadoop library相关推荐

【解决】win10下emqx启动报错Unable to load emulator DLL、node.db_role = EMQX_NODE__DB_ROLE = core
问题描述1: 启动时报错:win10下emqx启动报错 Unable to load emulator DLL D:\Applications_installed\emqx-5.0.4-windows ...
react-native---配置reactnative报红，run-Android时候报错unable to load script from assets 'index.android.bundl
react-native-配置reactnative报红,run-Android时候报错unable to load script from assets 'index.android.bundle ...
linux oracle 失败怎么办,Linux开机报错unable to load selinux policy怎么办？
Linux在开机的时候提示unable to load selinux policy错误,导致系统无法进入,通常是由2个情况所导致的,下面小编就给大家介绍下Linux开机报错unable to loa ...
win10安装elasticSearch8.1.2，报错解决方案
安装步骤可以参考<win10部署安装Elasticsearch8.1.2> 1.报错:此时不应有 \java\jdk1.8.0_20 解决方案: 在java环境变量都设置正确情况下,有可能 ...
React Native 红屏报错 Unable to load script from assets
出现上面的报错,先确认rn服务是否运行成功有的情况下是启动失败,或是启动后意外关闭了,或是端口不一致. 这里记录一下另外一种情况: 如果是从GITHUB上面新CLONE下来的项目 ,可能 andro ...
Genymotion-启动报错Unable to load VirtualBox engine....
今天下午看到Genymotion提示升级,升级到了genymotion-2.6.0,虚机升级到了VirtualBox-5.0.12-104815-Win的版本. genymotion无法启动了--.真 ...
Android 报错Failed to load native library: XXXX_so
产生原因: cpu架构不支持 so库没有打包到Android下解决方案: 第一种可以通过命令adb shell cat /proc/cpuinfo 来查看cpu详细信息,排查架构不支持的问题我这里 ...
如何解决运行Spark-shell，出现报错Unable to load native-hadoop library for your platform的问题
问题:启动spark后,运行bin/spark-shell会出现一个警告且交互界面显示时间非常长(有很长的延迟时间)(如图) WARN util.NativeCodeLoader: Unable to ...
python 安装gdal成功，但是导入报错：DLL load failed 找不到指定的模块
最近基于Python3.8安装gdal,报错如题,尝试了很多方法,包括各种替换gdal版本,升级对应的依赖包版本一直没得到解决. 最后看到两个很有参考价值的文章: 1.关于python环境配置几个常见 ...
win10 x64下安装oracle 12c出现[INS-30131]报错的解决方案
win10 x64下安装oracle 12c出现[INS-30131]报错的解决方案参考文章: (1)win10 x64下安装oracle 12c出现[INS-30131]报错的解决方案 (2)ht ...

win10 安装 hadoop 3.3.1报错 Unable to load native-hadoop library

win10 安装 hadoop 3.3.1报错 Unable to load native-hadoop library相关推荐

最新文章

热门文章