0027-如何在CDH集群启用Kerberos
2019独角兽企业重金招聘Python工程师标准>>>
1.文档编写目的
本文档讲述如何在CDH集群启用及配置Kerberos,您将学习到以下知识:
1.如何安装及配置KDC服务
2.如何通过CDH启用Kerberos
3.如何登录Kerberos并访问Hadoop相关服务
文档主要分为以下几步:
1.安装及配置KDC服务
2.CDH集群启用Kerberos
3.Kerberos使用
这篇文档将重点介绍如何在CDH集群启用及配置Kerberos,并基于以下假设:
1.CDH集群运行正常
2.集群未启用Kerberos
3.MySQL 5.1.73
以下是本次测试环境,但不是本操作手册的必需环境:
1.操作系统:CentOS 6.5
2.CDH和CM版本为5.12.0
3.采用root用户进行操作
2.KDC服务安装及配置
本文档中将KDC服务安装在Cloudera Manager Server所在服务器上(KDC服务可根据自己需要安装在其他服务器)
1.在Cloudera Manager服务器上安装KDC服务
[root@ip-172-31-6-148~]# yum -y install krb5-serverkrb5-libs krb5-auth-dialog krb5-workstation
2.修改/etc/krb5.conf配置
[root@ip-172-31-6-148 fayson_r]# vim /etc/krb5.conf
[logging]default = FILE:/var/log/krb5libs.logkdc = FILE:/var/log/krb5kdc.logadmin_server = FILE:/var/log/kadmind.log[libdefaults]default_realm = FAYSON.COMdns_lookup_realm = falsedns_lookup_kdc = falseticket_lifetime = 24hrenew_lifetime = 7dforwardable = true[realms]FAYSON.COM = {kdc = ip-172-31-6-148.fayson.comadmin_server = ip-172-31-6-148.fayson.com}[domain_realm].ip-172-31-6-148.fayson.com = FAYSON.COMip-172-31-6-148.fayson.com = FAYSON.COM
标红部分为需要修改的信息。
3.修改/var/kerberos/krb5kdc/kadm5.acl配置
[root@ip-172-31-6-148~]# vim /var/kerberos/krb5kdc/kadm5.acl
*/admin@FAYSON.COM *
4.修改/var/kerberos/krb5kdc/kdc.conf配置
[root@ip-172-31-6-148 ~]# vim /var/kerberos/krb5kdc/kdc.conf
[kdcdefaults]kdc_ports = 88kdc_tcp_ports = 88[realms]FAYSON.COM= {#master_key_type = aes256-ctsmax_renewable_life= 7d 0h 0m 0sacl_file = /var/kerberos/krb5kdc/kadm5.acldict_file = /usr/share/dict/wordsadmin_keytab = /var/kerberos/krb5kdc/kadm5.keytabsupported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal des-hmac-sha1:n
ormal des-cbc-md5:normal des-cbc-crc:normal}
标红部分为需要修改的配置。
5.创建Kerberos数据库
[root@ip-172-31-6-148 ~]# kdb5_util create –r FAYSON.COM -s
Loading random data
Initializing database '/var/kerberos/krb5kdc/principal' for realm 'FAYSON.COM',
master key name 'K/M@FAYSON.COM'
You will be prompted for the database Master Password.
It is important that you NOT FORGET this password.
Enter KDC database master key:
Re-enter KDC database master key to verify:
此处需要输入Kerberos数据库的密码。
6.创建Kerberos的管理账号
[root@ip-172-31-6-148 ~]# kadmin.local
Authenticating as principal fayson/admin@CLOUDERA.COM with password.
kadmin.local: addprinc admin/admin@FAYSON.COM
WARNING: no policy specified for admin/admin@FAYSON.COM; defaulting to no policy
Enter password for principal "admin/admin@FAYSON.COM":
Re-enter password for principal "admin/admin@FAYSON.COM":
Principal "admin/admin@FAYSON.COM" created.
kadmin.local: exit
[root@ip-172-31-6-148 ~]#
标红部分为Kerberos管理员账号,需要输入管理员密码。
7.将Kerberos服务添加到自启动服务,并启动krb5kdc和kadmin服务
[root@ip-172-31-6-148~]# chkconfig krb5kdc on
[root@ip-172-31-6-148 ~]# chkconfig kadmin on
[root@ip-172-31-6-148 ~]# service krb5kdc start
Starting Kerberos 5 KDC: [ OK ]
[root@ip-172-31-6-148 ~]# service kadmin start
Starting Kerberos 5 Admin Server: [ OK ]
[root@ip-172-31-6-148 ~]#
8.测试Kerberos的管理员账号
[root@ip-172-31-6-148 ~]# kinit admin/admin@FAYSON.COM
Password for admin/admin@FAYSON.COM:
[root@ip-172-31-6-148 ~]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: admin/admin@FAYSON.COMValid starting Expires Service principal
09/05/17 16:39:17 09/06/17 16:39:17 krbtgt/FAYSON.COM@FAYSON.COMrenew until 09/12/17 16:39:17
[root@ip-172-31-6-148 ~]#
9.为集群安装所有Kerberos客户端,包括Cloudera Manager
[root@ip-172-31-6-148 cdh-shell-master]# yum -y install krb5-libs krb5-workstation
10.在Cloudera Manager Server服务器上安装额外的包
[root@ip-172-31-6-148cdh-shell-master]# yum -y install openldap-clients
11.将KDC Server上的krb5.conf文件拷贝到所有Kerberos客户端
[root@ip-172-31-6-148cdh-shell-master]# scp -r /etc/krb5.conf root@172.31.5.190:/etc/
此处使用脚本进行拷贝
[root@ip-172-31-6-148cdh-shell-master]# sh b.sh node.list /etc/krb5.conf /etc/
krb5.conf 100% 451 0.4KB/s 00:00
krb5.conf 100% 451 0.4KB/s 00:00
krb5.conf 100% 451 0.4KB/s 00:00
krb5.conf 100% 451 0.4KB/s 00:00
[root@ip-172-31-6-148 cdh-shell-master]#
3.CDH集群启用Kerberos
1.在KDC中给Cloudera Manager添加管理员账号
[root@ip-172-31-6-148 cdh-shell-bak]# kadmin.local
Authenticating as principal admin/admin@FAYSON.COM with password.
kadmin.local: addprinc cloudera-scm/admin@FAYSON.COM
WARNING: no policy specified for cloudera-scm/admin@FAYSON.COM; defaulting to no policy
Enter password for principal "cloudera-scm/admin@FAYSON.COM":
Re-enter password for principal "cloudera-scm/admin@FAYSON.COM":
Principal "cloudera-scm/admin@FAYSON.COM" created.
kadmin.local: exit
[root@ip-172-31-6-148 cdh-shell-bak]#
2.进入Cloudera Manager的“管理”-> “安全”界面
3.选择“启用Kerberos”,进入如下界面
确保如下列出的所有检查项都已完成
4.点击“继续”,配置相关的KDC信息,包括类型、KDC服务器、KDC Realm、加密类型以及待创建的Service Principal(hdfs,yarn,,hbase,hive等)的更新生命期等
5.点击“继续”
6.不建议让Cloudera Manager来管理krb5.conf, 点击“继续”
7.输入Cloudera Manager的Kerbers管理员账号,必须和之前创建的账号一致,点击“继续”
8.等待启用Kerberos完成,点击“继续”
9.点击“继续”
10.勾选重启集群,点击“继续”
11.等待集群重启成功,点击“继续”
至此已成功启用Kerberos。
4.Kerberos使用
使用fayson用户运行MapReduce任务及操作Hive,需要在集群所有节点创建fayson用户。
1.使用kadmin创建一个fayson的principal
[root@ip-172-31-6-148 cdh-shell-bak]# kadmin.local
Authenticating as principal admin/admin@FAYSON.COM with password.
kadmin.local: addprinc fayson@FAYSON.COM
WARNING: no policy specified for fayson@FAYSON.COM; defaulting to no policy
Enter password for principal "fayson@FAYSON.COM":
Re-enter password for principal "fayson@FAYSON.COM":
Principal "fayson@FAYSON.COM" created.
kadmin.local: exit
[root@ip-172-31-6-148 cdh-shell-bak]#
2.使用fayson用户登录Kerberos
[root@ip-172-31-6-148 cdh-shell-bak]# kdestroy
[root@ip-172-31-6-148 cdh-shell-bak]# kinit fayson
Password for fayson@FAYSON.COM:
[root@ip-172-31-6-148 cdh-shell-bak]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: fayson@FAYSON.COMValid starting Expires Service principal
09/05/17 17:19:08 09/06/17 17:19:08 krbtgt/FAYSON.COM@FAYSON.COMrenew until 09/12/17 17:19:08
[root@ip-172-31-6-148 cdh-shell-bak]#
3.运行MapReduce作业
[root@ip-172-31-6-148~]# hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-0.20-mapreduce/hadoop-examples.jar pi 10 1
...
Starting Job
17/09/02 20:10:43 INFO mapreduce.Job: Running job: job_1504383005209_0001
17/09/02 20:10:56 INFO mapreduce.Job: Job job_1504383005209_0001 running in ubermode : false
17/09/02 20:10:56 INFO mapreduce.Job: map0% reduce 0%
17/09/02 20:11:09 INFO mapreduce.Job: map20% reduce 0%
17/09/02 20:11:12 INFO mapreduce.Job: map40% reduce 0%
17/09/02 20:11:13 INFO mapreduce.Job: map50% reduce 0%
17/09/02 20:11:15 INFO mapreduce.Job: map60% reduce 0%
17/09/02 20:11:16 INFO mapreduce.Job: map70% reduce 0%
17/09/02 20:11:19 INFO mapreduce.Job: map80% reduce 0%
17/09/02 20:11:21 INFO mapreduce.Job: map100% reduce 0%
17/09/02 20:11:26 INFO mapreduce.Job: map100% reduce 100%
17/09/02 20:11:26 INFO mapreduce.Job: Job job_1504383005209_0001 completedsuccessfully
4.使用beeline连接hive进行测试
[root@ip-172-31-6-148 cdh-shell-bak]# beeline
Beeline version 1.1.0-cdh5.12.1 by Apache Hive
beeline> !connect jdbc:hive2://localhost:10000/;principal=hive/ip-172-31-6-148.fayson.com@FAYSON.COM
...
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://localhost:10000/> show tables;
...
INFO : OK
+-------------+--+
| tab_name |
+-------------+--+
| test_table |
+-------------+--+
1 row selected (0.194 seconds)
0: jdbc:hive2://localhost:10000/> select * from test_table;
...
INFO : OK
+----------------+----------------+--+
| test_table.s1 | test_table.s2 |
+----------------+----------------+--+
| 4 | lisi |
| 1 | test |
| 2 | fayson |
| 3 | zhangsan |
+----------------+----------------+--+
4 rows selected (0.144 seconds)
0: jdbc:hive2://localhost:10000/>
运行Hive MapReduce作业
0: jdbc:hive2://localhost:10000/> select count(*) from test_table;
...
INFO : OK
+------+--+
| _c0 |
+------+--+
| 4 |
+------+--+
1 row selected (35.779 seconds)
0: jdbc:hive2://localhost:10000/>
5.常见问题
1.使用Kerberos用户身份运行MapReduce作业报错
main : run as user is fayson
main : requested yarn user is fayson
Requested user fayson is not whitelisted and has id 501,whichis below the minimum allowed 1000Failing this attempt. Failing the application.
17/09/02 20:05:04 INFO mapreduce.Job: Counters: 0
Job Finished in 6.184 seconds
java.io.FileNotFoundException: File does not exist:hdfs://ip-172-31-6-148:8020/user/fayson/QuasiMonteCarlo_1504382696029_1308422444/out/reduce-outat org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1266)at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1258)at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1258)at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1820)at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1844)at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:314)at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:354)at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:363)at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethod)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:606)at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethod)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:606)atorg.apache.hadoop.util.RunJar.run(RunJar.java:221)at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
问题原因:是由于Yarn限制了用户id小于10000的用户提交作业;
解决方法:修改Yarn的min.user.id来解决
推荐关注Hadoop实操,第一时间,分享更多Hadoop干货,欢迎转发和分享。
转载于:https://my.oschina.net/u/4016761/blog/2878676
0027-如何在CDH集群启用Kerberos相关推荐
- 【安全】CDH集群开启Kerberos安全认证
文章目录 1.安装kerberos 2.CDH集群开启Kerberos安全认证 1.安装kerberos 参考:https://blog.csdn.net/qq_21383435/article/de ...
- CDH集群禁用kerberos的讲解
CDH集群禁用kerberos 情况: 我为什么会遇到这个需要禁用kerberos的情况,是因为我需要两个集群之前的数据迁移,源集群是有kerberos认证的,新集群也配置了认证,两个集群如果都配置了 ...
- 0110-如何给Kerberos环境下的CDH集群添加Gateway节点
2019独角兽企业重金招聘Python工程师标准>>> Fayson的github: https://github.com/fayson/cdhproject 1.文档编写目的 Ga ...
- 为CDH 5.7集群添加Kerberos身份验证及Sentry权限控制
转载请注明出处:http://www.cnblogs.com/xiaodf/ 4. 为CDH 5集群添加Kerberos身份验证 4.1 安装sentry 1.点击"操作",&qu ...
- CDH集群执行任务报错:User xxx not found
1.基本情况 CDH集群, 节点分别为node1/node2/node3, 已安装Kerberos安全认证, 执行spark任务, 执行模式为spark on yarn 集群模式. 2.报错信息 ...
- 从零搭建生产Hadoop集群(五)——CDH集群修改主机名与IP
从零搭建生产Hadoop集群(五)--CDH集群修改主机名与IP 一.概述 二.完整步骤 1.官网推荐步骤: (1)验证下SSL/TLS认证是否所有服务都有配备,确认创建新的SSL/TLS认证. (2 ...
- CDH集群中YARN的参数配置
为什么80%的码农都做不了架构师?>>> CDH集群中YARN的参数配置 前言:Hadoop 2.0之后,原先的MapReduce不在是简单的离线批处理MR任务的框架,升级为M ...
- CDH集群搭建(CDH 6.0.1)
CDH集群安装步骤 参考文章 安装前准备 配置网络名称 禁用防火墙 设置SELinux模式 启用NTP服务 配置集群ssh免密登录 安装 安装jdk 安装服务和配置TLS 安装服务 配置TLS 安装并 ...
- CDH集群中HDFS单点故障解决方案:HA模式(High Availability)
CDH集群开发HDFS的High Avaliability模式 Step1 什么是单点故障? Step2 HDFS为什么会出现单点故障?它是怎么解决的? Step3 开启HDFS的高可用(High A ...
最新文章
- FPGA(2)基础语法 -- 按键控制led(alway@语句)
- 【Qt】编译QtCreator
- 大家来说说咱们IT人订阅什么周刊和月刊比较好
- 新手该怎样学习网站分析?
- Android——电脑蓝屏重启后,studio无法认出Android环境 setup JDK(缓存!缓存!缓存)...
- soft nms torch测试
- PdfSharp.dll 更改pdf 設置 如不能複製,列印等
- linux数据库服务器的配置与管理,实验七_Linux_MySQL数据库服务器配置与管理
- 英语语法---名词详解
- 农历鸡年(丁酉)正月初一第一博
- 中国十大最美梯田,个个都美如画!
- “嘲羊群众”词条视频惹怒粉丝 百度知道向张艺兴道歉...
- 二叉排序树删除节点_二叉排序树的创建、插入和删除
- python画动态图-python画动态图
- skynet源码阅读4--定时器实现
- 袁玉玮:简介人工智能在基金界的应用现状(四)
- Nvme协议固态装纯净win10系统
- 【web前端期末大作业】html网上在线书城大学生静态网页 大学生html当当书城仿站 网上书城购物网页作业HTML
- 2017年世界500强企业排行榜【完整榜单】2017财富杂志世界五百强排名
- 剖析 OceanConnect 物联网特性