Alex 的 Hadoop 菜鸟教程: 第18课 用Http的方式访问HDFS - HttpFs 教程
2024-06-07 14:57:43
原文地址： http://blog.csdn.net/nsrainbow/article/details/43678237 最新课程请关注原作者博客，获得更好的显示体验
声明
- 本文基于Centos 6.x + CDH 5.x
HttpFs 有啥用
HttpFs可以干这两件事情
- 通过HttpFs你可以在浏览器里面管理HDFS上的文件
- HttpFs还提供了一套REST 风格的API可以用来管理HDFS
其实很简单的一个东西嘛，但是很实用
安装HttpFs
在集群里面找一台可以访问hdfs的机器安装HttpFs
$ sudo yum install hadoop-httpfs配置
编辑/etc/hadoop/conf/core-site.xml
<property>
<name>hadoop.proxyuser.httpfs.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.httpfs.groups</name>
<value>*</value>
</property> 这边是定义可以使用httpfs的用户组和host，写*就是不限制
配置好之后重启hadoop
启动HttpFs
$ sudo service hadoop-httpfs start使用HttpFs
打开浏览器访问 http://host2:14000/webhdfs/v1?op=LISTSTATUS&user.name=httpfs 可以看到
{"FileStatuses": {"FileStatus": [{"pathSuffix": "hbase","type": "DIRECTORY","length": 0,"owner": "hbase","group": "hadoop","permission": "755","accessTime": 0,"modificationTime": 1423446940595,"blockSize": 0,"replication": 0},{"pathSuffix": "tmp","type": "DIRECTORY","length": 0,"owner": "hdfs","group": "hadoop","permission": "1777","accessTime": 0,"modificationTime": 1423122488037,"blockSize": 0,"replication": 0},{"pathSuffix": "user","type": "DIRECTORY","length": 0,"owner": "hdfs","group": "hadoop","permission": "755","accessTime": 0,"modificationTime": 1423529997937,"blockSize": 0,"replication": 0},{"pathSuffix": "var","type": "DIRECTORY","length": 0,"owner": "hdfs","group": "hadoop","permission": "755","accessTime": 0,"modificationTime": 1422945036465,"blockSize": 0,"replication": 0}]}
}这个 &user.name=httpfs 表示用默认用户 httpfs 访问，默认用户是没有密码的。
webhdfs/v1 这是HttpFs的根目录
访问 http://host2:14000/webhdfs/v1/user?op=LISTSTATUS&user.name=httpfs 可以看到
{"FileStatuses": {"FileStatus": [{"pathSuffix": "cloudera","type": "DIRECTORY","length": 0,"owner": "root","group": "hadoop","permission": "755","accessTime": 0,"modificationTime": 1423472508868,"blockSize": 0,"replication": 0},{"pathSuffix": "hdfs","type": "DIRECTORY","length": 0,"owner": "hdfs","group": "hadoop","permission": "700","accessTime": 0,"modificationTime": 1422947019504,"blockSize": 0,"replication": 0},{"pathSuffix": "history","type": "DIRECTORY","length": 0,"owner": "mapred","group": "hadoop","permission": "1777","accessTime": 0,"modificationTime": 1422945692887,"blockSize": 0,"replication": 0},{"pathSuffix": "hive","type": "DIRECTORY","length": 0,"owner": "hive","group": "hadoop","permission": "755","accessTime": 0,"modificationTime": 1423123187569,"blockSize": 0,"replication": 0},{"pathSuffix": "hive_people","type": "DIRECTORY","length": 0,"owner": "root","group": "hadoop","permission": "755","accessTime": 0,"modificationTime": 1423216966453,"blockSize": 0,"replication": 0},{"pathSuffix": "hive_people2","type": "DIRECTORY","length": 0,"owner": "root","group": "hadoop","permission": "755","accessTime": 0,"modificationTime": 1423222237254,"blockSize": 0,"replication": 0},{"pathSuffix": "impala","type": "DIRECTORY","length": 0,"owner": "root","group": "hadoop","permission": "755","accessTime": 0,"modificationTime": 1423475272189,"blockSize": 0,"replication": 0},{"pathSuffix": "root","type": "DIRECTORY","length": 0,"owner": "root","group": "hadoop","permission": "700","accessTime": 0,"modificationTime": 1423221719835,"blockSize": 0,"replication": 0},{"pathSuffix": "spark","type": "DIRECTORY","length": 0,"owner": "spark","group": "spark","permission": "755","accessTime": 0,"modificationTime": 1423530243396,"blockSize": 0,"replication": 0},{"pathSuffix": "sqoop","type": "DIRECTORY","length": 0,"owner": "hdfs","group": "hadoop","permission": "755","accessTime": 0,"modificationTime": 1423127462911,"blockSize": 0,"replication": 0},{"pathSuffix": "test_hive","type": "DIRECTORY","length": 0,"owner": "root","group": "hadoop","permission": "755","accessTime": 0,"modificationTime": 1423215687891,"blockSize": 0,"replication": 0}]}
}很奇怪的是HttpFs的文档很少，更具体的命令要去 WebHDFS的文档里面看 WebHDFS REST API
支持的命令
Operations
- HTTP GET
- OPEN (see FileSystem.open)
- GETFILESTATUS (see FileSystem.getFileStatus)
- LISTSTATUS (see FileSystem.listStatus)
- GETCONTENTSUMMARY (see FileSystem.getContentSummary)
- GETFILECHECKSUM (see FileSystem.getFileChecksum)
- GETHOMEDIRECTORY (see FileSystem.getHomeDirectory)
- GETDELEGATIONTOKEN (see FileSystem.getDelegationToken)
- HTTP PUT
- CREATE (see FileSystem.create)
- MKDIRS (see FileSystem.mkdirs)
- RENAME (see FileSystem.rename)
- SETREPLICATION (see FileSystem.setReplication)
- SETOWNER (see FileSystem.setOwner)
- SETPERMISSION (see FileSystem.setPermission)
- SETTIMES (see FileSystem.setTimes)
- RENEWDELEGATIONTOKEN (see DistributedFileSystem.renewDelegationToken)
- CANCELDELEGATIONTOKEN (see DistributedFileSystem.cancelDelegationToken)
- HTTP POST
- APPEND (see FileSystem.append)
- HTTP DELETE
- DELETE (see FileSystem.delete)
建立文件夹
尝试建立一个叫 abc 的文件夹
[root@host2 hadoop-httpfs]# curl -i -X PUT "http://host2:14000/webhdfs/v1/user/abc?op=MKDIRS&user.name=httpfs"
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Set-Cookie: hadoop.auth="u=httpfs&p=httpfs&t=simple&e=1423573951025&s=Ab44ha1Slg1f4xCrK+x4R/s1eMY="; Path=/; Expires=Tue, 10-Feb-2015 13:12:31 GMT; HttpOnly
Content-Type: application/json
Transfer-Encoding: chunked
Date: Tue, 10 Feb 2015 03:12:36 GMT{"boolean":true}然后用服务器上的hdfs dfs -ls 命令看下结果
[root@host2 conf]# hdfs dfs -ls /user
Found 12 items
drwxr-xr-x - httpfs hadoop 0 2015-02-10 11:12 /user/abc
drwxr-xr-x - root hadoop 0 2015-02-09 17:01 /user/cloudera
drwx------ - hdfs hadoop 0 2015-02-03 15:03 /user/hdfs
drwxrwxrwt - mapred hadoop 0 2015-02-03 14:41 /user/history
drwxr-xr-x - hive hadoop 0 2015-02-05 15:59 /user/hive
drwxr-xr-x - root hadoop 0 2015-02-06 18:02 /user/hive_people
drwxr-xr-x - root hadoop 0 2015-02-06 19:30 /user/hive_people2
drwxr-xr-x - root hadoop 0 2015-02-09 17:47 /user/impala
drwx------ - root hadoop 0 2015-02-06 19:21 /user/root
drwxr-xr-x - spark spark 0 2015-02-10 09:04 /user/spark
drwxr-xr-x - hdfs hadoop 0 2015-02-05 17:11 /user/sqoop
drwxr-xr-x - root hadoop 0 2015-02-06 17:41 /user/test_hive可以看到建立了一个属于 httpfs 的文件夹 abc
打开文件
从后台上传一个文本文件 test.txt 到 /user/abc 目录下，内容是
Hello World!用httpfs访问
[root@host2 hadoop-httpfs]# curl -i -X GET "http://host2:14000/webhdfs/v1/user/abc/test.txt?op=OPEN&user.name=httpfs"
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Set-Cookie: hadoop.auth="u=httpfs&p=httpfs&t=simple&e=1423574166943&s=JTxqIJUsblVBeHVuTs6JCV2UbBs="; Path=/; Expires=Tue, 10-Feb-2015 13:16:06 GMT; HttpOnly
Content-Type: application/octet-stream
Content-Length: 13
Date: Tue, 10 Feb 2015 03:16:07 GMTHello World!Alex 的 Hadoop 菜鸟教程: 第18课 用Http的方式访问HDFS - HttpFs 教程相关推荐
- Alex 的 Hadoop 菜鸟教程: 第5课 YARN 安装以及helloworld (基于centos的CDH)
原帖地址:http://blog.csdn.net/nsrainbow/article/details/36627675 新老MapReduce的比较 说到YARN肯定要先说下老的MapReduce ...
- Alex 的 Hadoop 菜鸟教程: 第10课 Hive 安装和使用教程
原帖地址: http://blog.csdn.net/nsrainbow/article/details/41748863 最新课程请关注原作者博客 声明 本文基于Centos 6.x + CDH 5 ...
- appinventor HTML5,App Inventor编程教程-第18课-创建动画应用
本章将讨论另一类应用--包含简单动画(会移动的物体)的应用.你将学习用App Inventor创建二维游戏的基本知识,包括熟练使用精灵组件,以及处理像两个物体碰撞这样的事件. 当在电脑屏幕上看到一个平 ...
- 三维实景沙盘数字模拟沙盘M3DGIS系统开发教程第18课
上一节我们实现了模型的移动控制.这次我们来实现模型的材质控制,首先我们找一个模型.在3dmax中如下: 可以看到这个模型很复杂.分成了很多层.我们先不管它.导入SDK后如下图: 有贴图还是比较漂亮了. ...
- Hadoop通过路径和和链接访问HDFS
如果既想在Hadoop服务器本地可以通过绝对路径如"/user/hadoop"方式访问hdfs,也想通过"hdfs://local host:9000/user/hado ...
- 第18课 Altium Designer20(AD20)+VESC6.4实战教程：DRV8301布局及注意事项（北冥有鱼）
第18课 Altium Designer20(AD20)+VESC6.4实战教程:DRV8301布局及注意事项(北冥有鱼)
- NeHe OpenGL教程 第二十三课：球面映射
转自[翻译]NeHe OpenGL 教程 前言 声明,此 NeHe OpenGL教程系列文章由51博客yarin翻译(2010-08-19),本博客为转载并稍加整理与修改.对NeHe的OpenGL管线 ...
- Lance老师UI系列教程第三课-QQ登录注册界面的实现（android-2012最新版）
分类: android UI教程2012-08-06 22:37 3731人阅读 评论(6) 收藏 举报 uiandroidqqlayoutbutton UI系列教程第三课:腾讯登录注册界面的实现 今 ...
- Linux教程 第十一课 Linux进程管理及作业控制(几本没看懂)
Linux教程 第十一课 Linux进程管理及作业控制(几本没看懂) ----------------------------------------------------------------- ...
- 《计算机网络教程》(微课版 第五版)第四章 网络层与网络互连
<计算机网络教程>(微课版 第五版)第四章 网络层与网络互连 课后习题及答案 1.网络层向上提供的服务有哪两种?试比较其优缺点. 解答:面向连接的虚电路服务和无连接的数据报服务. 2.请简 ...
最新文章
- maven 插件的应用
- 抽象工厂模式_常用设计模式--抽象工厂模式
- 如何在私有链部署智能合约
- arm汇编解析—qnnpack卷积实现
- git初使用（本地创建后第一次提交到git）
- [讨论帖] 程序员如何赚外快？
- 西安工业学院计算机系王翊,西安文理学院艺术学院
- Ambari安装client报错OSError:[Error 17] File exists
- 那些兼职中你不知道的事
- fedora下安装python
- 2019PASS发布以来第一次更新，快点击查看！
- 前照灯检测仪_前照灯检测仪
- maven插件安装与使用
- 数字图像处理基础-第三章图像预处理
- html编辑器怎么设置为excel,excel2013宏编辑器的设置方法教程
- 各位大神，有没有类似于百度云软件开始时的设置向导的例子呀
- 计算机中保存和另存为,电脑另存为在哪里
- VS2019 MFC模式下如何调用控制台并用cprintf在控制台打印
- CMake问题：The CXX compiler identification is unknown
- wipe、root、底包、rsd是什么意思？小白入门释义