1,准备测试数据

1.1 建库建表

mysql> create database hwdb;

Query OK, 1 row affected (0.34 sec)

mysql>

mysql> show create table t_huawei;

+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Table | Create Table |

| t_huawei | CREATE TABLE `t_huawei` (

`hwid` int(11) DEFAULT NULL,

`content` varchar(4000) DEFAULT NULL,

`TYPE` char(2) DEFAULT NULL,

`creator_id` varchar(30) DEFAULT NULL,

`create_time` datetime DEFAULT NULL

) ENGINE=InnoDB DEFAULT CHARSET=utf8 |

1 row in set (0.00 sec)

mysql>

1.2 准备存储过程

DELIMITER $$

USE `hwdb`$$

DROP PROCEDURE IF EXISTS `pro_insert_hw`$$

CREATE DEFINER=`root`@`localhost` PROCEDURE`pro_insert_hw`(num INT)

BEGIN

DECLARE var1 INT DEFAULT 0;

WHILE var1<num DO

SET var1=var1+1;

INSERT INTO hwdb.t_huawei(hwid,content,TYPE,creator_id,create_time)

VALUES (var1,CONCAT(var1,' - ','Players supposedly said they''d avoidteam facility this summer if Thibodeau remained the '),

0,10013,NOW());

END WHILE;

END$$

DELIMITER ;

mysql> show create table t_huawei;

| Table | Create Table |

| t_huawei | CREATE TABLE `t_huawei` (

`hwid` int(11) DEFAULT NULL,

`content` varchar(4000) DEFAULT NULL,

`TYPE` char(2) DEFAULT NULL,

`creator_id`varchar(30) DEFAULT NULL,

`create_time` datetime DEFAULT NULL

) ENGINE=InnoDB DEFAULT CHARSET=utf8 |

1 row in set (0.00 sec)

mysql>

1.3 开始录入数据

因为要录入千万级别数据，所以用后台mysql进程来录入，如下所示：

nohup /usr/local/mysql/bin/mysql -uroot--password="" -S /usr/local/mysql/mysql.sock -e "select now();truncate table hwdb.t_huawei; call hwdb.pro_insert_hw(10000000); select now();" > /data/call.log &

这里因为测试环境机器性能有限，在录入到7023850条的时候，我终止了进程，开始准备测试工作：

mysql> SELECT COUNT(1) FROM `t_huawei` ;

+----------+

| COUNT(1) |

+----------+

| 7023850 |

+----------+

1 row in set (5.52 sec)

mysql>

2，开始测试

2.1 普通的ORDER BY RAND()测试很慢很慢，如下所示，需要块1分钟的时间

mysql> SELECT SQL_NO_CACHE * FROM `t_huawei` AS t1 ORDER BY RAND() LIMIT 5;

+---------+-----------------------------------------------------------------------------------------------------+------+------------+---------------------+

+---------+-----------------------------------------------------------------------------------------------------+------+------------+---------------------+

| 3743323 | 3743323 - Players supposedlysaid they'd avoid team facility this summer if Thibodeau remained the | 0 | 10013 | 2015-06-08 20:43:01|

| 2418491 | 2418491 - Players supposedlysaid they'd avoid team facility this summer if Thibodeau remained the | 0 | 10013 | 2015-06-08 20:19:31|

| 1224667 | 1224667 - Players supposedlysaid they'd avoid team facility this summer if Thibodeau remained the | 0 | 10013 | 2015-06-08 19:58:35|

| 1639270 | 1639270 - Players supposedlysaid they'd avoid team facility this summer if Thibodeau remained the | 0 | 10013 | 2015-06-08 20:05:50|

| 2756470 | 2756470 - Players supposedlysaid they'd avoid team facility this summer if Thibodeau remained the | 0 | 10013 | 2015-06-08 20:25:28|

+---------+-----------------------------------------------------------------------------------------------------+------+------------+---------------------+

5 rows in set (54.92 sec)

mysql>

原因是：MYSQL的官方手册，里面针对RAND()的提示大概意思就是，在ORDER BY从句里面不能使用RAND()函数，因为这样会导致数据列被多次扫描。但是在MYSQL 3.23版本中，仍然可以通过ORDER BY RAND()来实现随机。

2.2 在JOIN中使用RAND()* (SELECTMAX(hwid) 的方式来实现

mysql> SELECT SQL_NO_CACHE t1.* FROM`t_huawei` AS t1 LEFT JOIN(SELECT ROUND(RAND()* (SELECT MAX(hwid) FROM `t_huawei` )) AS id ) AS t2 ONt1.`hwid`>=t2.id ORDER BY t1.`hwid` ASC LIMIT 5;

+------+-----------------------------------------------------------------------------------------------+------+------------+---------------------+

+------+-----------------------------------------------------------------------------------------------+------+------------+---------------------+

| 1 | 1 - Players supposedly said they'd avoid team facility this summerif Thibodeau remained the | 0 | 10013 | 2015-06-08 19:37:02 |

| 2 | 2 - Players supposedly said they'd avoid team facility this summerif Thibodeau remained the | 0 | 10013 | 2015-06-08 19:37:02 |

| 3 | 3 - Players supposedly said they'd avoid team facility this summerif Thibodeau remained the | 0 | 10013 | 2015-06-08 19:37:02 |

| 4 | 4 - Players supposedly said they'd avoid team facility this summerif Thibodeau remained the | 0 | 10013 | 2015-06-08 19:37:02 |

| 5 | 5 - Players supposedly said they'd avoid team facility this summerif Thibodeau remained the | 0 | 10013 | 2015-06-08 19:37:02 |

+------+-----------------------------------------------------------------------------------------------+------+------------+---------------------+

5 rows in set (50.04 sec)

mysql>

需要50面，比起直接的order by来说少了4秒，但是还是比较耗时的。

2.3 WHERE子句的方式来优化

mysql> SELECT SQL_NO_CACHE t1.* FROMhwdb.`t_huawei` t1 WHERE t1.`hwid` >=(SELECT FLOOR(RAND() * (SELECTMAX(t2.hwid) FROM hwdb.`t_huawei` t2 ) )) ORDER BY t1.hwid LIMIT 5;

+------+--------------------------------------------------------------------------------------------------+------+------------+---------------------+

+------+--------------------------------------------------------------------------------------------------+------+------------+---------------------+

| 1624 | 1624 - Players supposedly saidthey'd avoid team facility this summer if Thibodeau remained the | 0 | 10013 | 2015-06-08 19:37:05|

| 6076 | 6076 - Players supposedly saidthey'd avoid team facility this summer if Thibodeau remained the | 0 | 10013 | 2015-06-08 19:37:11|

| 7988 | 7988 - Players supposedly saidthey'd avoid team facility this summer if Thibodeau remained the | 0 | 10013 | 2015-06-08 19:37:14|

| 8016 | 8016 - Players supposedly saidthey'd avoid team facility this summer if Thibodeau remained the | 0 | 10013 | 2015-06-08 19:37:14|

| 8106 | 8106 - Players supposedly saidthey'd avoid team facility this summer if Thibodeau remained the | 0 | 10013 | 2015-06-08 19:37:14|

+------+--------------------------------------------------------------------------------------------------+------+------------+---------------------+

5 rows in set (7.25 sec)

mysql>

花了7.25秒，比上一种要快的多很多了。

2.4 MAX(hwid)- MIN(hwid)的方式来RAND来操作

mysql> SELECT SQL_NO_CACHE * FROMhwdb.`t_huawei` WHERE hwid >= (SELECT FLOOR(RAND() * ((SELECT MAX(hwid) FROMhwdb.`t_huawei`) - (SELECT MIN(hwid) FROM hwdb.`t_huawei`)) + (SELECT MIN(hwid)FROM hwdb.`t_huawei`))) ORDER BY hwid LIMIT 5;

+------+--------------------------------------------------------------------------------------------------+------+------------+---------------------+

+------+--------------------------------------------------------------------------------------------------+------+------------+---------------------+

| 5438 | 5438 - Players supposedly saidthey'd avoid team facility this summer if Thibodeau remained the | 0 | 10013 | 2015-06-08 19:37:10|

| 5883 | 5883 - Players supposedly saidthey'd avoid team facility this summer if Thibodeau remained the | 0 | 10013 | 2015-06-08 19:37:10|

| 6782 | 6782 - Players supposedly saidthey'd avoid team facility this summer if Thibodeau remained the | 0 | 10013 | 2015-06-08 19:37:12|

| 7797 | 7797 - Players supposedly saidthey'd avoid team facility this summer if Thibodeau remained the | 0 | 10013 | 2015-06-08 19:37:13|

| 8625 | 8625 - Players supposedly saidthey'd avoid team facility this summer if Thibodeau remained the | 0 | 10013 | 2015-06-08 19:37:15|

+------+--------------------------------------------------------------------------------------------------+------+------------+---------------------+

5 rows in set (13.50 sec)

mysql>

MAX(hwid)- MIN(hwid)的方式来RAND来操作需要13.5秒，比where子句花时间要多一些。

3，总结

WHERE子句的方式效率最高，样例sql：SELECTSQL_NO_CACHE t1.* FROM hwdb.`t_huawei` t1 WHERE t1.`hwid` >=(SELECTFLOOR(RAND() * (SELECT MAX(t2.hwid) FROM hwdb.`t_huawei` t2 ) )) ORDER BYt1.hwid LIMIT 5;

MySQL千万级别表数据中提高RAND随机查询的实验相关推荐

mysql千万级大数据SQL查询优化
1.对查询进行优化,应尽量避免全表扫描,首先应考虑在 where 及 order by 涉及的列上建立索引.2.应尽量避免在 where 子句中对字段进行 null 值判断,否则将导致引擎放弃使用索引 ...
MySQL数据库中如何使用rand随机查询记录
以下的文章主要介绍的是MySQL使用rand 随机查询记录效率测试,我们大家一直都以为MySQL数据库随机查询的几条数据,就用以下的东东,其实其实际效率是十分低的,以下就是文章的主要内容. 1.SEL ...
30个mysql千万级大数据SQL查询优化技巧详解
点击上方关注 "终端研发部" 设为"星标",和你一起掌握更多数据库知识文章来自:脚本之家 http://www.jb51.net/article/136701 ...
AutoML 在表数据中的研究与应用
导读:大家好,今天分享的题目是 AutoML 在表数据中的研究与应用.对于 AutoML,大家听到比较多的可能是神经网络结构搜索 ( NAS,Neural Architecture Search ), ...
MySql千万级表性能优化
近期在处理一个千万级表数据的优化措施与总结(不考虑硬件方面): 1)数据库做主从,读写分离 2)表结构把不常用的字段列移入附加表 3)查询语句优化,避免全表扫描,. 4)按需合理增加索引 5)做分区 ...
MySQL千万级别大表如何优化？
当MySQL单表记录数过大时,增删改查性能都会急剧下降,可以参考以下步骤来优化: 单表优化除非单表数据未来会一直不断上涨,否则不要一开始就考虑拆分,拆分会带来逻辑.部署.运维的各种复杂度,一般以整型 ...
面试官邪魅一笑：MySQL千万级别大表，你要如何优化？
作者:码农原文链接:https://segmentfault.com/a/1190000006158186#comment-area 当MySQL单表记录数过大时,增删改查性能都会急剧下降,可以参考 ...
Mysql千万级别数据如何做分页？
后端开发中为了防止⼀次性加载太多数据导致内存.磁盘IO都开销过⼤,经常需要分⻚展示,这个时候就需要⽤到MySQL的LIMIT关键字.但你以为LIMIT分⻚就万事大吉了么,LIMIT在数据量⼤的时候极可 ...
菜鸟mysql四分钟导入千万级别的数据
最近在分析数据时,遇到1000万条csv数据,于是便想着将其导入MySQL进行分析,由于本人比较笨,折腾了一晚上还没搞定,总是遇到各种各样的错误,终于在今天成功导入了这1000万条数据,在此跟大家分享 ...

MySQL千万级别表数据中提高RAND随机查询的实验

1,准备测试数据

1.1 建库建表

1.2 准备存储过程

1.3 开始录入数据

2，开始测试

2.1 普通的ORDER BY RAND()测试很慢很慢，如下所示，需要块1分钟的时间

2.2 在JOIN中使用RAND()* (SELECTMAX(hwid) 的方式来实现

2.3 WHERE子句的方式来优化

2.4 MAX(hwid)- MIN(hwid)的方式来RAND来操作

3，总结

MySQL千万级别表数据中提高RAND随机查询的实验相关推荐

最新文章

热门文章