在mac上安装hadoop伪分布式
换了macbook pro之后,要重新安装hadoop,但是mac上的jdk跟windows上的不同,导致折腾了挺久的,现在分享出来,希望对大家有用。
一:下载jdk
选择最新版本下载,地址:http://www.oracle.com/technetwork/java/javase/downloads/index.html
安装完成之后,打开终端,输入java -version ,出现类似如下说明安装成功。
java version "1.8.0"
Java(TM) SE Runtime Environment (build 1.8.0-b132)
Java HotSpot(TM) 64-Bit Server VM (build 25.0-b70, mixed mode)
二:配置hadoop
下载hadoop,自己可到官网下载稳定版本,不用下载最新的,因为最新的可能不稳定。
配置hadoop 里面conf文件夹四个文件(hadoop-env.sh,core-site.xml,mapred-site.xml,hdfs-site.xml)
下载完hadoop之后,把它解压到你想存放的文件夹,然后进入hadoop的conf目录
1.配置hadopp-env.sh
打开该文件之后找到
#export JAVA_HOME=
#export HADOOP_HEAPSIZE=2000
#export HADOOP_OPTS=-server
修改为:
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_79.jdk/Contents/Homeexport HADOOP_HEAPSIZE=2000#export HADOOP_OPTS=-serverexport HADOOP_OPTS="-Djava.security.krb5.realm=OX.AC.UK -Djava.security.krb5.kdc=kdc0.ox.ac.uk:kdc1.ox.ac.uk"
即去掉前面的解释符#
特别注意:
JAVA_HOME=的配置是类似上面的目录,百度上有些文章写到是在终端输入whereis java出现的目录,这是不对的。
因为mac的jdk安装在根目录的Library文件夹下面。
2.配置core-site.xml
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>hadoop.tmp.dir</name><value>/Users/username/Documents/apache/hadoop-2.6.0/hadoop_tmp</value><description>A base for other temporary directories.</description></property> <property><name>fs.default.name</name><value>hdfs://localhost:9000</value> </property> <property><name>mapred.job.tracker</name><value>hdfs://localhost:9001</value> </property> <property><name>dfs.replication</name><value>1</value> </property> </configuration>
3.配置mapred-site.xml
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>mapred.job.tracker</name> <value>localhost:9001</value> </property> </configuration>
4.配置hdfs-site.xml
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>dfs.replication</name> <value>1</value> </property> </configuration>
三:配置ssh,无密码登录
1.mac 上已经ssh了,在终端输入ssh-keygen -t rsa命令,碰到需要输入密码的直接按enter健即可。出现如下成功
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /Users/jia/.ssh/id_rsa.
Your public key has been saved in /Users/jia/.ssh/id_rsa.pub.
The key fingerprint is:
d4:85:aa:83:ae:db:50:48:0c:5b:dd:80:bb:fa:26:a7 jia@JIAS-MacBook-Pro.local
The key's randomart image is:
+--[ RSA 2048]----+
|. .o.o .. |
| =. . . ... |
|. o. ... |
| ... .. |
| .... .S |
| ... o |
| ... . |
|o oo. |
|E*+o. |
+-----------------+
在终端输入cd .ssh 进入.ssh目录,输入命令。
cp id_rsa.pub authorized_keys
即可。
2.在mac上输入
ssh localhost
如果出现
ssh: connect to host localhost port 22: Connection refused
表示当前用户没有权限。这个可能是系统为安全考虑,默认设置的。
更改设置如下:进入system preference --> sharing --> 勾选remote login,并设置allow access for all users。再次输入“ssh localhost",再输入密码并确认之后,可以看到ssh成功
如果出现
No such file or directory...
1.启动sshd服务:
sudo launchctl load -w /System/Library/LaunchDaemons/ssh.plist
2.查看服务的启动,筛选:
sudo launchctl list | grep ssh
四:配置环境变量
suodo vim /etc/profile 添加: export HADOOP_HOME=/Users/username/Documents/apache/hadoop-2.6.0 export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
五:启动hadoop
1.进入hadoop文件夹,用如下命令格式化:
bin/hadoop namenode -format
出现如下,说明成功
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = JIAS-MacBook-Pro.local/192.168.1.3
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 0.20.2
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
Re-format filesystem in /tmp/hadoop-jia/dfs/name ? (Y or N) Y
14/07/14 13:55:17 INFO namenode.FSNamesystem: fsOwner=jia,staff,everyone,localaccounts,_appserverusr,admin,_appserveradm,_lpadmin,com.apple.sharepoint.group.1,_appstore,_lpoperator,_developer,com.apple.access_screensharing,com.apple.access_ssh
14/07/14 13:55:17 INFO namenode.FSNamesystem: supergroup=supergroup
14/07/14 13:55:17 INFO namenode.FSNamesystem: isPermissionEnabled=true
14/07/14 13:55:17 INFO common.Storage: Image file of size 93 saved in 0 seconds.
14/07/14 13:55:17 INFO common.Storage: Storage directory /tmp/hadoop-jia/dfs/name has been successfully formatted.
14/07/14 13:55:17 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at JIAS-MacBook-Pro.local/192.168.1.3
************************************************************/
2.启动hadoop守护进程
bin/start-all.sh
3.停止hadoop守护进程
bin/stop-all.sh
六:错误分析
1.之前运行时总出现如下错误
cat: Documents/hadoop-0.20.2/conf/slaves: No such file or directory cat: Documents/hadoop-0.20.2/conf/masters: No such file or directory
其实这是在配置mahout的环境时加上了下面这句环境变量导致的错误
export HADOOP_CONF_DIR=Documents/hadoop-0.20.2/conf
之前我电脑的环境变量为(在终端输入命令:open /etc/ , 找的profile文件,打开,配置好环境变量在覆盖之前的)
export HADOOP_HOME=Documents/hadoop-0.20.0 export MAHOUT_HOME=Documents/hadoop-0.20.2/mahout-distribution-0.9 export MAVEN_HOME=Documents/apache-maven-3.2.2export PATH=$PATH:$HADOOP_HOME/bin:$MAHOUT_HOME/bin:$MAVEN_HOME/binexport HADOOP_CONF_DIR=Documents/hadoop-0.20.2/conf export MAHOUT_CONF_DIR=Documents/hadoop-0.20.2/mahout-distribution-0.9/confexport classpath=$classpath:$MAHOUT_HOME/lib:$HADOOP_CONF_DIR:$MAHOUT_CONF_DIR
把环境配置改为如下就可以正常运行了
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0.jdk/Contents/Home
export HADOOP_HOME=Documents/hadoop-0.20.0
export MAHOUT_HOME=Documents/hadoop-0.20.2/mahout-distribution-0.9
export MAVEN_HOME=Documents/apache-maven-3.2.2
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$MAVEN_HOME/bin:$MAHOUT_HOME/bin
export MAHOUT_CONF_DIR=Documents/hadoop-0.20.2/mahout-distribution-0.9/conf
export classpath=$classpath:$JAVA_HOME/lib:$MAHOUT_HOME/lib:$MAHOUT_CONF_DIR
红色部分做好加进去
既去掉
export HADOOP_CONF_DIR=Documents/hadoop-0.20.2/conf
注意:
修改/etc/profile上的环境配置文件之后,用如下命令使其生效
source /etc/profile
2.如果需要自己编译hadoop源码的话,编译过程可能会出现org.apache.maven.plugin.MojoExecutionException的错误
Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:2.6.2:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: 'protoc --version' did not return a version -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:2.6.2:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: 'protoc --version' did not return a versionat org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:606)at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356) Caused by: org.apache.maven.plugin.MojoExecutionException: org.apache.maven.plugin.MojoExecutionException: 'protoc --version' did not return a versionat org.apache.hadoop.maven.plugin.protoc.ProtocMojo.execute(ProtocMojo.java:105)at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)... 20 more Caused by: org.apache.maven.plugin.MojoExecutionException: 'protoc --version' did not return a versionat org.apache.hadoop.maven.plugin.protoc.ProtocMojo.execute(ProtocMojo.java:68)... 22 more
下载protobuf-2.4.1.jar
http://vdisk.weibo.com/s/tYlk6JrNUYTY
安装protobuf
tar xzf protobuf-2.4.1.tar.gz cd protobuf-2.4.1 ./configure make sudo make install sudo ldconfig
在重新编译
3.namenode没启动
解决方法:先可视化之后再启动
bin/hadoop namenode -format
转载于:https://www.cnblogs.com/aijianiula/p/3842581.html
在mac上安装hadoop伪分布式相关推荐
- Windows 7 64位系统上搭建Hadoop伪分布式环境(很详细)
在开始配置前,我们先了解Hadoop的三种运行模式. Hadoop的三种运行模式 独立(或本地)模式:无需运行任何守护进程,所有程序都在同一个JVM上执行.在独立模式下测试和调试MapReduce程序 ...
- CentOS7下安装Hadoop伪分布式
前提条件 拥有CentOS7服务器版环境 软件版本说明 jdk1.8 hadoop2.7.3 虚拟机环境准备 通网络 能ping通外网,例如: ping baidu.com 如果ping不通,可以修改 ...
- 在Mac上安装Hadoop HA 高可
**Mac安装Hadoop HA高可 在Mac上安装Hadoop HA 高可用 配置环境: HA & Federation介绍 Hadoop 1.x与Hadoop 2.x Hadoop 2.x ...
- Mac安装hadoop伪分布式
目录 一. 下载jdk 二. 配置Hadoop 三. 配置ssh,无密码登录 四. 启动hadoop 五. 环境配置 一. 下载jdk 选择最新版本下载,地址:http://www.oracle.co ...
- Mac上安装hadoop
1.安装JAVA--之后会运行jar包,肯定需要java的运行环境 查看java版本: [html] view plaincopy java -version -------------------- ...
- 安装hadoop伪分布式模式(Single Node Cluster)
目的 本文档介绍如何去安装单节点hadoop集群,以便你可以的了解和使用hadoop的HDFS和MapReduce. 环境: os: CentOS release 6.5 (Final) ip: 17 ...
- 在Mac上安装Hadoop
对我这个之前从未接触过*nix的用户来说,使用命令行来做一系列的事情还是废了一番功夫.特写这个记录,以做备份. 获取Java 我的Mac运行的操作系统是OS X 10.7 Lion,之前已经安装过Ja ...
- 安装hadoop伪分布式集群心得
1.hadoop安装: https://www.cnblogs.com/news1997/p/9522717.html https://blog.csdn.net/y12345678904/artic ...
- centOS 7下Hadoop伪分布式搭建
记录笔记,尝试在centOS下安装Hadoop伪分布式(之前是在Ubuntu下安装的),查漏补缺. 步骤 第一步:先把在window系统中下载好的关于Hadoop.jdk压缩包用Xftp传输到cent ...
最新文章
- Java学习_day001
- uva10401Injured Queen Problem(递推)
- vCenter功能基本介绍
- Qcom LK阶段如何使用ADC介绍
- rm、shutdown、磁盘挂载、vi使用方法
- python前端调用后端模型_前端调用后端的方法(基于restful接口的mvc架构)
- 【IDEA】idea中maven项目Resource目录下的资源文件无法编译
- 后端接口都测试什么?怎么测?
- springcloud的中文文档地址和中国社区
- 一个小小的抽奖活动测试脚本(python2.7)
- 如何学习angular.js
- linux调度器(五)——进程管理与CFS
- 十五届恩智浦智能车-四十天做四轮-调车日记
- echart 表格_市政工程表格不会填?市政工程1000个表格模板,一键即可套用
- java递归下降计算四则运算_用递归下降方法实现算术表达式解析器
- linux 压缩文件 zip,linux zip命令压缩zip文件详解
- [代码审计]DuxCMS 2.0审计
- css样式字体文本汇总
- 【软件】网梭浏览器v2.4.7 思路
- oscp——five86: 2