从hive中matastore表中删除分区表

1.引入分区表(需要根据日期对日志进行管理)

/user/hive/warehouse/log_partition/20190101/20190101.log
/user/hive/warehouse/log_partition/20190102/20190102.log
/user/hive/warehouse/log_partition/20190103/20190103.log

2.创建分区表语法

…

create table dept_partition(
deptno int,
dname string,
loc string
)
partitioned by (month string)
row format delimited fields terminated by ‘\t’;…

3.加载数据到分区表中

load data local inpath ‘/home/hadoop/data/dept201901.txt’ into table dept_partition partition(month=‘201901’);

坑一：Failed with exception MetaException(message:For direct MetaStore DB connections, we don’t support retries at the client level.)
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask

https://blog.csdn.net/qq_35732963/article/details/54291970

load data local inpath ‘/home/hadoop/data/dept201902.txt’ into table default.dept_partition partition(month=‘201902’);

load data local inpath ‘/home/hadoop/data/dept201903.txt’ into table default.dept_partition partition(month=‘201903’);

4.进入http://192.168.217.129:50070/查看partition_table分区是否成功

5.查询分区表中数据

5.1 单分区查询

select * from dept_partition where month=‘201901’;

5.2 多分区联合查询

select * from dept_partition where month=‘201901’
union
select * from dept_partition where month=‘201902’
union
select * from dept_partition where month=‘201903’

6 Dele hive tables from matastore

6.1、从元数据删除Hive的表需要得到五个ID.TBL_ID,SD_ID ,CD_ID,PART_ID,SERDE_ID.

a.select * from TBLS \G;

*************************** 6. row ***************************TBL_ID: 26CREATE_TIME: 1546787041DB_ID: 1

LAST_ACCESS_TIME: 0
OWNER: hadoop
RETENTION: 0
SD_ID: 26
TBL_NAME: dept_partition
TBL_TYPE: MANAGED_TABLE
VIEW_EXPANDED_TEXT: NULL
VIEW_ORIGINAL_TEXT: NULL
6 rows in set (0.00 sec)

b.select * from SDS \G;

*************************** 6. row ***************************
SD_ID: 26
CD_ID: 26
INPUT_FORMAT: org.apache.hadoop.mapred.TextInputFormat
IS_COMPRESSED:
IS_STOREDASSUBDIRECTORIES:
LOCATION: hdfs://hadoop001:8020/user/hive/warehouse/dept_partition
NUM_BUCKETS: -1
OUTPUT_FORMAT: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
SERDE_ID: 26
*************************** 7. row ***************************
SD_ID: 27
CD_ID: 26
INPUT_FORMAT: org.apache.hadoop.mapred.TextInputFormat
IS_COMPRESSED:
IS_STOREDASSUBDIRECTORIES:
LOCATION: hdfs://hadoop001:8020/user/hive/warehouse/dept_partition/month=201901
NUM_BUCKETS: -1
OUTPUT_FORMAT: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
SERDE_ID: 27
*************************** 8. row ***************************
SD_ID: 28
CD_ID: 26
INPUT_FORMAT: org.apache.hadoop.mapred.TextInputFormat
IS_COMPRESSED:
IS_STOREDASSUBDIRECTORIES:
LOCATION: hdfs://hadoop001:8020/user/hive/warehouse/dept_partition/month=201902
NUM_BUCKETS: -1
OUTPUT_FORMAT: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
SERDE_ID: 28
*************************** 9. row ***************************
SD_ID: 29
CD_ID: 26
INPUT_FORMAT: org.apache.hadoop.mapred.TextInputFormat
IS_COMPRESSED:
IS_STOREDASSUBDIRECTORIES:
LOCATION: hdfs://hadoop001:8020/user/hive/warehouse/dept_partition/month=201903
NUM_BUCKETS: -1
OUTPUT_FORMAT: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
SERDE_ID: 29
9 rows in set (0.00 sec)

6.2、TBL_ID和SD_ID可以从TBL_ID表得到，CD_ID,SERDE_ID可以从SDS表得到，PART_ID可以从PARTITIONS表中得到。

TBL_ID = 26;
SD_ID = 26;CD_ID = 26;SERDE_ID = 27,28,29

7 写一个脚本

[root@hadoop001 etl]# cat delete_hive_metadata.sh

#! /bin/bash
mysql -uroot -p***** hive -e"
delete FROM PARTITION_PARAMS WHERE PART_ID=$4;
delete FROM PARTITION_KEY_VALS WHERE PART_ID=$4;
delete FROM PARTITIONS WHERE TBL_ID=$1;

delete FROM PARTITION_KEYS WHERE TBL_ID=$1;
delete FROM TABLE_PARAMS WHERE TBL_ID=$1;
delete FROM TBLS WHERE TBL_ID=$1;
delete FROM SERDE_PARAMS WHERE SERDE_ID=$5;
delete FROM SERDES WHERE SERDE_ID=$5;

delete FROM SDS WHERE SD_ID=$2;
delete FROM COLUMNS_V2 WHERE CD_ID=$3;"

…sh drop_hive_metadata.sh…

vim sh drop_hive_metadata.sh

#! /bin/bash
mysql -uroot -p***** hive -e"
delete FROM PARTITION_PARAMS WHERE PART_ID=$4;
delete FROM PARTITION_KEY_VALS WHERE PART_ID=$4;
delete FROM PARTITIONS WHERE TBL_ID=$1;

delete FROM SDS WHERE SD_ID=$2;
delete FROM COLUMNS_V2 WHERE CD_ID=$3;"

sh drop_hive_metadata.sh 26 26 26 1 27

(1)delete from PARTITION_KEY_VALS where part_id=

(2)delete from PARTITION_PARAMS where part_id=

(3)delete from PARTITIONS where tbl_id= 26 and part_id

(4)delete from SDS where SERDE_ID = 30;

hive> select * from dept_partition where month = ‘201901’;

结果为空

遇到的坑二：

select * from dept_partition;
OK
10 ACCOUNTING NEW YORK 201901
20 RESEARCH DALLAS 201901
30 SALES CHICAGO 201901
40 OPERATIONS BOSTON 201901
10 ACCOUNTING NEW YORK 201901
20 RESEARCH DALLAS 201901
30 SALES CHICAGO 201901
40 OPERATIONS BOSTON 201901
10 ACCOUNTING NEW YORK 201901
20 RESEARCH DALLAS 201901
30 SALES CHICAGO 201901
40 OPERATIONS BOSTON 201901
10 ACCOUNTING NEW YORK 201902
20 RESEARCH DALLAS 201902
30 SALES CHICAGO 201902
40 OPERATIONS BOSTON 201902
10 ACCOUNTING NEW YORK 201903
20 RESEARCH DALLAS 201903
30 SALES CHICAGO 201903
40 OPERATIONS BOSTON 201903
Time taken: 0.091 seconds, Fetched: 20 row(s)

原因是忘记删除：delete from SDS where SERDE_ID;

从hive中matastore表中删除分区表相关推荐

查看hive中某个表中的数据、表结构及所在路径
查看hive中action_data_myisam表中的数据.表结构及所在路径 1.客户端进入hive环境:hive 2.查看表数据,鉴于数据量大,这里只显示前五条:select * from act ...
redis将散裂中某个值自增_Redis中hash表中的field的value自增可以用hincrby
Redis HINCRBY命令用于增加存储在字段中存储由增量键哈希的数量.如果键不存在,新的key被哈希创建.如果字段不存在,值被设置为0之前进行操作. 回复整数,字段的增值操作后的值. redis ...
mysql 存储过程新增表,mysql实例存储过程中向表中添加数据
摘要腾兴网为您分享:mysql实例存储过程中向表中添加数据,信和财富,视达网,人人视频,企商理财等软件知识,以及电视猫,office2000,贵阳银行,搜电共享充电宝,雅图电影院,达达跑腿,万圣节 ...
如何从Oracle数据库中的表中获取列名（字段名）列表？
如何从Oracle数据库中的表中获取列名(字段名)列表? 目录如何从Oracle数据库中的表中获取列名(字段名)列表? #示例一 #示例二示例三: #示例一可以获取: table_name:表名 ...
mysql 批量替换所有表_[收藏]批量替换一个数据库中所有表中所有记录
/***********批量替换一个数据库中所有表中所有记录************/ declare @delStr nvarchar(500) set @delStr='这里是要替换的字符' /* ...
php mysql 偶数 id_javascript – 如何从mysql数据库(PHP)中的表中选择所有偶数的帖子ID...
我想从我的 mysql数据库中的表中选择所有偶数的id,然后显示它们.我也想得到所有的帖子与奇怪的id,并显示在别的地方. 我想用PHP这样做,因为这是我使用的服务器端语言. 或者,我必须选择所有帖子 ...
mysql 如何删除重复的行_如何从mysql中的表中删除重复的行
我需要从 mysql中删除表中的重复记录. 所以我有一个表名"employee"字段是empid,empname,empssn 为了获得重复记录我写了一个查询 SELECT COU ...
快速删除数据库中所有表中的数据
今天又学到一招,可以快速删除数据库中所有的用户表中的数据.我是个菜鸟,望各位大神多多指教 select 'truncate table ' + Name + ';' from sysobjects w ...
mysql中关于表的删除和表中数据的删除
1.删除表(drop属于DDL语句) 格式:drop table 表名 2.删除表中的数据(delete属于DML语句,truncate属于DDL语句) 第一种方式:delete from 表名 wh ...

从hive中matastore表中删除分区表

从hive中matastore表中删除分区表相关推荐

最新文章

热门文章