Apache Knox安装测试
Apache Knox安装测试
- 一、Knox简介
- 一、下载Knox
- 二、安装Hadoop和Knox
- 三、读写HDFS
一、Knox简介
Announcing Apache Knox 1.6.1!
REST API and Application Gateway for the Apache Hadoop Ecosystem
The Apache Knox™ Gateway is an Application Gateway for interacting with the REST APIs and UIs
of Apache Hadoop deployments.
The Knox Gateway provides a single access point for all REST and HTTP interactions with Apache Hadoop
clusters.Knox delivers three groups of user facing services:
官网介绍:
Apache Knox网关是一个为集群中的Apache Hadoop服务提供单点身份验证和访问的系统。目标是为用户(即访问集群数据和执行作业的用户)和操作员(即控制访问和管理集群的用户)简化Hadoop安全。网关作为服务器(或服务器集群)运行,提供对一个或多个Hadoop集群的集中访问。一般来说,网关的目标如下:
- 为Hadoop REST API提供外围安全,使Hadoop安全更易于设置和使用
- 在外围提供身份验证和令牌验证
- 实现与企业和云身份管理系统的身份验证集成
- 在外围提供服务级别授权
- 公开聚合Hadoop集群的REST API的单个URL层次结构
- 限制访问Hadoop集群所需的网络端点(以及防火墙漏洞)
- 对潜在攻击者隐藏内部Hadoop群集拓扑
运行逻辑图:
一、下载Knox
- 下载地址:下载Knox链接
二、安装Hadoop和Knox
- Hadoop安装可参考博文:Hadoop2.7.3全分布式环境搭建
- 安装Knox
- 注意:需要使用普通用户安装,如omm
- 新建用户组和用户,如:
groupadd knoxgroup useradd -g knoxgroup omm passwd omm
- 新建用户组和用户,如:
- 可参考官网:Quick Start
- 使用omm用户登录服务器,默认安装在/home/omm下
- 上传knox1.6.1.zip包到虚拟机中
- 解压安装:
unzip knox1.6.1.zip
- 环境配置(可选)
- 启动LDAP,进入到解压之后的目录/home/omm/knox1.6.1下执行命令:
bin/ldap.sh start
- 创建Master密钥,执行如下命令:
bin/knoxcli.sh create-master
,回车,输入密钥密码 - 启动Knox服务,执行:
bin/gateway.sh start
- 检查LDAP和knox服务是否正常启动,执行:
jps
,结果如下:
- 注意:需要使用普通用户安装,如omm
三、读写HDFS
通过网关gateway在WebHDFS上调用LISTSTATUS操作,命令如下:
执行命令:
curl -i -k -u guest:guest-password -X GET 'https://localhost:8443/gateway/sandbox/webhdfs/v1/?op=LISTSTATUS'
结果如图:
文字结果:
HTTP/1.1 200 OK Date: Tue, 20 Sep 2022 13:55:54 GMT Set-Cookie: KNOXSESSIONID=node01jvpkpq1sedtp1xcfgvzi9ok721.node0; Path=/gateway/sandbox; Secure; HttpOnly Expires: Thu, 01 Jan 1970 00:00:00 GMT Set-Cookie: rememberMe=deleteMe; Path=/gateway/sandbox; Max-Age=0; Expires=Mon, 19-Sep-2022 13:55:54 GMT; SameSite=lax Cache-Control: no-cache Expires: Tue, 20 Sep 2022 13:55:54 GMT Date: Tue, 20 Sep 2022 13:55:54 GMT Pragma: no-cache Expires: Tue, 20 Sep 2022 13:55:54 GMT Date: Tue, 20 Sep 2022 13:55:54 GMT Pragma: no-cache Content-Type: application/json;charset=utf-8 Server: Jetty(6.1.26) Transfer-Encoding: chunked{"FileStatuses": {"FileStatus": [{"accessTime": 0,"blockSize": 0,"childrenNum": 0,"fileId": 17828,"group": "supergroup","length": 0,"modificationTime": 1655327421392,"owner": "root","pathSuffix": "elogs","permission": "755","replication": 0,"storagePolicy": 0,"type": "DIRECTORY"},{"accessTime": 0,"blockSize": 0,"childrenNum": 3,"fileId": 17453,"group": "supergroup","length": 0,"modificationTime": 1655487609271,"owner": "root","pathSuffix": "flumeLogs","permission": "755","replication": 0,"storagePolicy": 0,"type": "DIRECTORY"},{"accessTime": 0,"blockSize": 0,"childrenNum": 9,"fileId": 16936,"group": "supergroup","length": 0,"modificationTime": 1655436743899,"owner": "root","pathSuffix": "hbase","permission": "757","replication": 0,"storagePolicy": 0,"type": "DIRECTORY"},{"accessTime": 0,"blockSize": 0,"childrenNum": 0,"fileId": 17590,"group": "supergroup","length": 0,"modificationTime": 1655281104031,"owner": "suben","pathSuffix": "hdfsdemo","permission": "755","replication": 0,"storagePolicy": 0,"type": "DIRECTORY"},{"accessTime": 0,"blockSize": 0,"childrenNum": 6,"fileId": 16644,"group": "supergroup","length": 0,"modificationTime": 1656012489237,"owner": "root","pathSuffix": "inputs","permission": "755","replication": 0,"storagePolicy": 0,"type": "DIRECTORY"},{"accessTime": 0,"blockSize": 0,"childrenNum": 2,"fileId": 16658,"group": "supergroup","length": 0,"modificationTime": 1653646793954,"owner": "root","pathSuffix": "output","permission": "755","replication": 0,"storagePolicy": 0,"type": "DIRECTORY"},{"accessTime": 0,"blockSize": 0,"childrenNum": 4,"fileId": 17375,"group": "supergroup","length": 0,"modificationTime": 1655463859092,"owner": "root","pathSuffix": "outputs","permission": "755","replication": 0,"storagePolicy": 0,"type": "DIRECTORY"},{"accessTime": 0,"blockSize": 0,"childrenNum": 1,"fileId": 17570,"group": "supergroup","length": 0,"modificationTime": 1654919667421,"owner": "root","pathSuffix": "rdbms","permission": "755","replication": 0,"storagePolicy": 0,"type": "DIRECTORY"},{"accessTime": 0,"blockSize": 0,"childrenNum": 0,"fileId": 16641,"group": "supergroup","length": 0,"modificationTime": 1653636523487,"owner": "root","pathSuffix": "sparkhistory","permission": "755","replication": 0,"storagePolicy": 0,"type": "DIRECTORY"},{"accessTime": 0,"blockSize": 0,"childrenNum": 30,"fileId": 16642,"group": "supergroup","length": 0,"modificationTime": 1656024159488,"owner": "root","pathSuffix": "sparklogs","permission": "755","replication": 0,"storagePolicy": 0,"type": "DIRECTORY"},{"accessTime": 0,"blockSize": 0,"childrenNum": 1,"fileId": 18294,"group": "supergroup","length": 0,"modificationTime": 1656035964378,"owner": "root","pathSuffix": "testDataSet","permission": "755","replication": 0,"storagePolicy": 0,"type": "DIRECTORY"},{"accessTime": 0,"blockSize": 0,"childrenNum": 3,"fileId": 16575,"group": "supergroup","length": 0,"modificationTime": 1653628122500,"owner": "root","pathSuffix": "tmp","permission": "733","replication": 0,"storagePolicy": 0,"type": "DIRECTORY"},{"accessTime": 0,"blockSize": 0,"childrenNum": 2,"fileId": 16582,"group": "supergroup","length": 0,"modificationTime": 1654752608696,"owner": "root","pathSuffix": "user","permission": "757","replication": 0,"storagePolicy": 0,"type": "DIRECTORY"}]} }
通过网关gateway在上传文件到HDFS上:
执行create操作命令:
curl -i -k -u guest:guest-password -X PUT 'https://localhost:8443/gateway/sandbox/webhdfs/v1/tmp/LICENSE?op=CREATE'
生成结果:
HTTP/1.1 307 Temporary Redirect Date: Tue, 20 Sep 2022 15:03:21 GMT Set-Cookie: KNOXSESSIONID=node01eqe641021389b3zv3vxzhjw411.node0; Path=/gateway/sandbox; Secure; HttpOnly Expires: Thu, 01 Jan 1970 00:00:00 GMT Set-Cookie: rememberMe=deleteMe; Path=/gateway/sandbox; Max-Age=0; Expires=Mon, 19-Sep-2022 15:03:21 GMT; SameSite=lax Cache-Control: no-cache Expires: Tue, 20 Sep 2022 15:03:21 GMT Date: Tue, 20 Sep 2022 15:03:21 GMT Pragma: no-cache Expires: Tue, 20 Sep 2022 15:03:21 GMT Date: Tue, 20 Sep 2022 15:03:21 GMT Pragma: no-cache Content-Type: application/octet-stream Location: https://localhost:8443/gateway/sandbox/webhdfs/data/v1/webhdfs/v1/tmp/word?_=AAAACAAAABAAAABwTpIK-W64x8ISPJNY9JvX3XuoTVo-OXwLrz64napWpWGTq-oww25mTd4fgPehTjUXmd2_FKZ37LwpehOpiUBZtO788j4AEKIBv30d6HApTLika55LfTqUIugsgKw_bl6G8jgBfxQa-W2-qiB5xRzijYnrs0NYYveBgVnpS8lKlV1y-RJZFJjrPw Server: Jetty(6.1.26) Content-Length: 0
把上面结果内容中的
Location
的结果单独复制出来,后面上传文件时需要使用到,如下:https://localhost:8443/gateway/sandbox/webhdfs/data/v1/webhdfs/v1/tmp/word?_=AAAACAAAABAAAABwTpIK-W64x8ISPJNY9JvX3XuoTVo-OXwLrz64napWpWGTq-oww25mTd4fgPehTjUXmd2_FKZ37LwpehOpiUBZtO788j4AEKIBv30d6HApTLika55LfTqUIugsgKw_bl6G8jgBfxQa-W2-qiB5xRzijYnrs0NYYveBgVnpS8lKlV1y-RJZFJjrPw
执行上传Put操作上传本地文件word.txt(如没有请自行创建)到HDFS上:
curl -i -k -u guest:guest-password -T /home/omm/word.txt -X PUT 'https://localhost:8443/gateway/sandbox/webhdfs/data/v1/webhdfs/v1/tmp/word?_=AAAACAAAABAAAABwTpIK-W64x8ISPJNY9JvX3XuoTVo-OXwLrz64napWpWGTq-oww25mTd4fgPehTjUXmd2_FKZ37LwpehOpiUBZtO788j4AEKIBv30d6HApTLika55LfTqUIugsgKw_bl6G8jgBfxQa-W2-qiB5xRzijYnrs0NYYveBgVnpS8lKlV1y-RJZFJjrPw'
生成结果
通过网关gateway在查看HDFS上的文件:
执行open操作,命令:
curl -i -k -u guest:guest-password -X GET 'https://localhost:8443/gateway/sandbox/webhdfs/v1/tmp/LICENSE?op=OPEN'
结果:
将
Location
对应的值复制出来,如下:https://localhost:8443/gateway/sandbox/webhdfs/data/v1/webhdfs/v1/tmp/word?_=AAAACAAAABAAAABwvGlR2yaXxjIdF-OhvfiGfk-Vbyc5rBHj44sz4sFRAeuNC-Gde7WIc7cXuJgRAGKKRuTfQYooqjnuhMycxSifYsYnT9zVEZsS_2O_rUCDpmqanxQkJDUSiOzpbbY8K0ihByeDeIBwmTKQ1gXq_nxY3KpH5sB2lOMEOtQsEisJ0F0VsfejKtInRQ
执行Get操作查看文件,命令:
curl -i -k -u guest:guest-password -X GET 'https://localhost:8443/gateway/sandbox/webhdfs/data/v1/webhdfs/v1/tmp/word?_=AAAACAAAABAAAABwvGlR2yaXxjIdF-OhvfiGfk-Vbyc5rBHj44sz4sFRAeuNC-Gde7WIc7cXuJgRAGKKRuTfQYooqjnuhMycxSifYsYnT9zVEZsS_2O_rUCDpmqanxQkJDUSiOzpbbY8K0ihByeDeIBwmTKQ1gXq_nxY3KpH5sB2lOMEOtQsEisJ0F0VsfejKtInRQ'
结果:
当然可以使用hdfs命令查看/tmp目录下是不是多出了一个/tmp/word的文件,如下所示:
查看下内容:hdfs dfs -cat /tmp/word
,结果和上述的结果一样,具体如下所示:
至此,简单的安装测试apache knox就到此结束,该篇博文参考了knox官网教程,在验证过程中记录下来,当作是练习使用吧,希望对你有所帮助~~~
Apache Knox安装测试相关推荐
- apache knox 安装使用步骤
2. knox 源码 git clone https://gitbox.apache.org/repos/asf/knox.git http://knox.apache.org/ 启动: http:/ ...
- Apache Drill安装测试
简介 Apache Drill是Google Dremel 的开源实现.可以查询 local files.HDFS.Amazon S3.HBase.MongoDB.Hive.RDBMS.Kafka.O ...
- 2020全网最全Apache Knox实战总结
本文针对knox_1.0.0进行总结,过程中可能会穿插其他版本的异同 一.环境准备 运行环境:Java1.8 支持Hadoop 3.x 下载knox1.0.0 https://cwiki.apache ...
- Apache Maven 安装与配置-修改源
Maven配置,强大的Java包管理器 Maven介绍 Apache Maven 下载 Apache Maven 安装 要求 启动 解压到适当的文件夹 添加环境变量 修改配置 启动测试 Maven介绍 ...
- RHEL5+PXE+DHCP+Apache+Kickstart安装RHCE5.1 i386实验室环境
此博文出自:http://andrewyu.blog.51cto.com/1604432/1100567 使用RHEL5+PXE+DHCP+Apache+Kickstart安装RHCE5.1 i386 ...
- apache 源代码安装
1.下载源代码<?XML:NAMESPACE PREFIX = O /> 去官网http://www.apache.org/下载最新稳定版的源代码 本案例中下载的是httpd-2.2. ...
- Apache ab压力测试工具
查看apache是否安装 # rpm -qa httpd httpd-2.2.3-63.el5.centos (在apache 版本2以后,apache全部改名为httpd) ab压力测试工具是apa ...
- apache 编译安装php mysql_编译安装APACHE+PHP+MYSQL
adminjun qq:279872 网络技术研究会群:8070045 中国linux公社论坛群:4478487 其实这是我用RouterOS和redhat linux进行pppoe服务器的配置及在r ...
- Redis集群方案,Codis安装测试
Redis集群方案,Codis安装测试 1,关于豌豆荚开源的Codis Codis是豌豆荚使用Go和C语言开发.以代理的方式实现的一个Redis分布式集群解决方案,且完全兼容Twemproxy.Twe ...
最新文章
- 【动态规划专题】最长上升子序列模型
- 使用nginx阻止别人域名绑定到自己的IP上
- 【原生js】js动态添加dom,如何绑定事件
- 数值分析(part1)--拉格朗日插值
- .NET 开源配置组件 AgileConfig 初体验
- 一份好的商业计划书该怎么写
- 杨强教授新番!破解机器学习数据孤岛和数据保护难题
- c语言贪吃蛇作业报告,贪吃蛇设计报告
- 基于【国基北盛】云基础架构平台软件搭建openstack私有云平台(先电V2.4版本)
- 分布式会议系统中的服务器,多功能会议系统分配服务器质量过硬
- 外卖分销分佣小程序外卖cps小程序返利系统源码分享
- Angular cdk 学习之 drag-drop
- 小马哥---高仿苹果6 plus 主板型号Q33 6582芯片刷机拆机主板图与开机界面图 镜面后壳
- 美食杰(个人主页编辑资料)
- lol手游之任务进度条精准计算
- 不容错过, Paintinglite轻量级Sqlite3框架
- Python脚本——信手拈来的自动化测试工具
- 基于JavaSwing开发画图软件 课程设计 小作业 大作业源码
- C语言讲义——错误处理
- sql如何把一个数据库的表复制到另一个数据库中【转载】