分布式系统生成唯一主键
2024-05-30 09:59:29
原创文章，转载请注明出处：服务器非业余研究http://blog.csdn.net/erlib 作者Sunface
唯一ID在系统小的时候，随便整都可以，但是系统大了后呢？这个时候如果出现了问题，ID重构就是很大的困难，因此对于任何一个系统，从最初架构时就进行好唯一ID设计是非常重要的，twitter的snowflake就解决了这种需求，实现也还是很简单的，除去配置信息，核心代码就是毫秒级时间41位+机器ID 10位+毫秒内序列12位。
该项目地址为：https://github.com/twitter/snowflake是用Scala实现的。
python版详见开源项目https://github.com/erans/pysnowflake。
核心代码为其IdWorker这个类实现，其原理结构如下，我分别用一个0表示一位，用—分割开部分的作用：
1
|
0---0000000000 0000000000 0000000000 0000000000 0 --- 00000 ---00000 ---000000000000
|
在上面的字符串中，第一位为未使用（实际上也可作为long的符号位），接下来的41位为毫秒级时间，然后5位datacenter标识位，5位机器ID（并不算标识符，实际是为线程标识），然后12位该毫秒内的当前毫秒内的计数，加起来刚好64位，为一个Long型。
这样的好处是，整体上按照时间自增排序，并且整个分布式系统内不会产生ID碰撞（由datacenter和机器ID作区分），并且效率较高，经测试，snowflake每秒能够产生26万ID左右，完全满足需要。
且看其核心代码：
1
|
</pre>
|
2
|
/** Copyright 2010-2012 Twitter, Inc.*/
|
3
|
package com.twitter.service.snowflake
|
4
|
5
|
import com.twitter.ostrich.stats.Stats
|
6
|
import com.twitter.service.snowflake.gen._
|
7
|
import java.util.Random
|
8
|
import com.twitter.logging.Logger
|
9
|
10
|
/**
|
11
|
* An object that generates IDs.
|
12
|
* This is broken into a separate class in case
|
13
|
* we ever want to support multiple worker threads
|
14
|
* per process
|
15
|
*/
|
16
|
class IdWorker(val workerId: Long, val datacenterId: Long, private val reporter: Reporter, var sequence: Long = 0L)
|
17
|
extends Snowflake.Iface {
|
18
|
private[this] def genCounter(agent: String) = {
|
19
|
Stats.incr("ids_generated")
|
20
|
Stats.incr("ids_generated_%s".format(agent))
|
21
|
}
|
22
|
private[this] val exceptionCounter = Stats.getCounter("exceptions")
|
23
|
private[this] val log = Logger.get
|
24
|
private[this] val rand = new Random
|
25
|
26
|
val twepoch = 1288834974657L
|
27
|
28
|
//机器标识位数
|
29
|
30
|
private[this] val workerIdBits = 5L
|
31
|
32
|
//数据中心标识位数
|
33
|
private[this] val datacenterIdBits = 5L
|
34
|
35
|
//机器ID最大值
|
36
|
private[this] val maxWorkerId = -1L ^ (-1L << workerIdBits)
|
37
|
38
|
//数据中心ID最大值
|
39
|
private[this] val maxDatacenterId = -1L ^ (-1L << datacenterIdBits)
|
40
|
41
|
//毫秒内自增位
|
42
|
private[this] val sequenceBits = 12L
|
43
|
44
|
//机器ID偏左移12位
|
45
|
46
|
private[this] val workerIdShift = sequenceBits
|
47
|
48
|
//数据中心ID左移17位
|
49
|
private[this] val datacenterIdShift = sequenceBits + workerIdBits
|
50
|
51
|
//时间毫秒左移22位
|
52
|
private[this] val timestampLeftShift = sequenceBits + workerIdBits + datacenterIdBits
|
53
|
private[this] val sequenceMask = -1L ^ (-1L << sequenceBits)
|
54
|
55
|
private[this] var lastTimestamp = -1L
|
56
|
57
|
// sanity check for workerId
|
58
|
if (workerId > maxWorkerId || workerId < 0) {
|
59
|
exceptionCounter.incr(1)
|
60
|
throw new IllegalArgumentException("worker Id can't be greater than %d or less than 0".format(maxWorkerId))
|
61
|
}
|
62
|
63
|
if (datacenterId > maxDatacenterId || datacenterId < 0) {
|
64
|
exceptionCounter.incr(1)
|
65
|
throw new IllegalArgumentException("datacenter Id can't be greater than %d or less than 0".format(maxDatacenterId))
|
66
|
}
|
67
|
68
|
log.info("worker starting. timestamp left shift %d, datacenter id bits %d, worker id bits %d, sequence bits %d, workerid %d",
|
69
|
timestampLeftShift, datacenterIdBits, workerIdBits, sequenceBits, workerId)
|
70
|
71
|
def get_id(useragent: String): Long = {
|
72
|
if (!validUseragent(useragent)) {
|
73
|
exceptionCounter.incr(1)
|
74
|
throw new InvalidUserAgentError
|
75
|
}
|
76
|
77
|
val id = nextId()
|
78
|
genCounter(useragent)
|
79
|
80
|
reporter.report(new AuditLogEntry(id, useragent, rand.nextLong))
|
81
|
id
|
82
|
}
|
83
|
84
|
def get_worker_id(): Long = workerId
|
85
|
def get_datacenter_id(): Long = datacenterId
|
86
|
def get_timestamp() = System.currentTimeMillis
|
87
|
88
|
protected[snowflake] def nextId(): Long = synchronized {
|
89
|
var timestamp = timeGen()
|
90
|
91
|
//时间错误
|
92
|
93
|
if (timestamp < lastTimestamp) {
|
94
|
exceptionCounter.incr(1)
|
95
|
log.error("clock is moving backwards. Rejecting requests until %d.", lastTimestamp);
|
96
|
throw new InvalidSystemClock("Clock moved backwards. Refusing to generate id for %d milliseconds".format(
|
97
|
lastTimestamp - timestamp))
|
98
|
}
|
99
|
100
|
if (lastTimestamp == timestamp) {
|
101
|
//当前毫秒内，则+1
|
102
|
sequence = (sequence + 1) & sequenceMask
|
103
|
if (sequence == 0) {
|
104
|
//当前毫秒内计数满了，则等待下一秒
|
105
|
timestamp = tilNextMillis(lastTimestamp)
|
106
|
}
|
107
|
} else {
|
108
|
sequence = 0
|
109
|
}
|
110
|
111
|
lastTimestamp = timestamp
|
112
|
//ID偏移组合生成最终的ID，并返回ID
|
113
|
114
|
((timestamp - twepoch) << timestampLeftShift) |
|
115
|
(datacenterId << datacenterIdShift) |
|
116
|
(workerId << workerIdShift) |
|
117
|
sequence
|
118
|
}
|
119
|
120
|
//等待下一个毫秒的到来
|
121
|
122
|
protected def tilNextMillis(lastTimestamp: Long): Long = {
|
123
|
var timestamp = timeGen()
|
124
|
while (timestamp <= lastTimestamp) {
|
125
|
timestamp = timeGen()
|
126
|
}
|
127
|
timestamp
|
128
|
}
|
129
|
130
|
protected def timeGen(): Long = System.currentTimeMillis()
|
131
|
132
|
val AgentParser = """([a-zA-Z][a-zA-Z\-0-9]*)""".r
|
133
|
134
|
def validUseragent(useragent: String): Boolean = useragent match {
|
135
|
case AgentParser(_) => true
|
136
|
case _ => false
|
137
|
}
|
138
|
}
|
139
|
<pre>
|
上述为twitter的实现，下面且看一个Java实现，貌似为淘宝的朋友写的。
1
|
public class IdWorker {
|
2
|
private final long workerId;
|
3
|
private final static long twepoch = 1361753741828L;
|
4
|
private long sequence = 0L;
|
5
|
private final static long workerIdBits = 4L;
|
6
|
public final static long maxWorkerId = -1L ^ -1L << workerIdBits;
|
7
|
private final static long sequenceBits = 10L;
|
8
|
9
|
private final static long workerIdShift = sequenceBits;
|
10
|
private final static long timestampLeftShift = sequenceBits + workerIdBits;
|
11
|
public final static long sequenceMask = -1L ^ -1L << sequenceBits;
|
12
|
13
|
private long lastTimestamp = -1L;
|
14
|
15
|
public IdWorker(final long workerId) {
|
16
|
super();
|
17
|
if (workerId > this.maxWorkerId || workerId < 0) {
|
18
|
throw new IllegalArgumentException(String.format(
|
19
|
"worker Id can't be greater than %d or less than 0",
|
20
|
this.maxWorkerId));
|
21
|
}
|
22
|
this.workerId = workerId;
|
23
|
}
|
24
|
25
|
public synchronized long nextId() {
|
26
|
long timestamp = this.timeGen();
|
27
|
if (this.lastTimestamp == timestamp) {
|
28
|
this.sequence = (this.sequence + 1) & this.sequenceMask;
|
29
|
if (this.sequence == 0) {
|
30
|
System.out.println("###########" + sequenceMask);
|
31
|
timestamp = this.tilNextMillis(this.lastTimestamp);
|
32
|
}
|
33
|
} else {
|
34
|
this.sequence = 0;
|
35
|
}
|
36
|
if (timestamp < this.lastTimestamp) {
|
37
|
try {
|
38
|
throw new Exception(
|
39
|
String.format(
|
40
|
"Clock moved backwards. Refusing to generate id for %d milliseconds",
|
41
|
this.lastTimestamp - timestamp));
|
42
|
} catch (Exception e) {
|
43
|
e.printStackTrace();
|
44
|
}
|
45
|
}
|
46
|
47
|
this.lastTimestamp = timestamp;
|
48
|
long nextId = ((timestamp - twepoch << timestampLeftShift))
|
49
|
| (this.workerId << this.workerIdShift) | (this.sequence);
|
50
|
// System.out.println("timestamp:" + timestamp + ",timestampLeftShift:"
|
51
|
// + timestampLeftShift + ",nextId:" + nextId + ",workerId:"
|
52
|
// + workerId + ",sequence:" + sequence);
|
53
|
return nextId;
|
54
|
}
|
55
|
56
|
private long tilNextMillis(final long lastTimestamp) {
|
57
|
long timestamp = this.timeGen();
|
58
|
while (timestamp <= lastTimestamp) {
|
59
|
timestamp = this.timeGen();
|
60
|
}
|
61
|
return timestamp;
|
62
|
}
|
63
|
64
|
private long timeGen() {
|
65
|
return System.currentTimeMillis();
|
66
|
}
|
67
|
|
68
|
|
69
|
public static void main(String[] args){
|
70
|
IdWorker worker2 = new IdWorker(2);
|
71
|
System.out.println(worker2.nextId());
|
72
|
73
|
|
74
|
}
|
75
|
76
|
}
|
再来看一个php的实现
1
|
<?php
|
2
|
class Idwork
|
3
|
{
|
4
|
const debug = 1;
|
5
|
static $workerId;
|
6
|
static $twepoch = 1361775855078;
|
7
|
static $sequence = 0;
|
8
|
const workerIdBits = 4;
|
9
|
static $maxWorkerId = 15;
|
10
|
const sequenceBits = 10;
|
11
|
static $workerIdShift = 10;
|
12
|
static $timestampLeftShift = 14;
|
13
|
static $sequenceMask = 1023;
|
14
|
private static $lastTimestamp = -1;
|
15
|
16
|
function __construct($workId){
|
17
|
if($workId > self::$maxWorkerId || $workId< 0 )
|
18
|
{
|
19
|
throw new Exception("worker Id can't be greater than 15 or less than 0");
|
20
|
}
|
21
|
self::$workerId=$workId;
|
22
|
23
|
echo 'logdebug->__construct()->self::$workerId:'.self::$workerId;
|
24
|
echo '</br>';
|
25
|
26
|
}
|
27
|
28
|
function timeGen(){
|
29
|
//获得当前时间戳
|
30
|
$time = explode(' ', microtime());
|
31
|
$time2= substr($time[0], 2, 3);
|
32
|
$timestramp = $time[1].$time2;
|
33
|
echo 'logdebug->timeGen()->$timestramp:'.$time[1].$time2;
|
34
|
echo '</br>';
|
35
|
return $time[1].$time2;
|
36
|
}
|
37
|
function tilNextMillis($lastTimestamp) {
|
38
|
$timestamp = $this->timeGen();
|
39
|
while ($timestamp <= $lastTimestamp) {
|
40
|
$timestamp = $this->timeGen();
|
41
|
}
|
42
|
43
|
echo 'logdebug->tilNextMillis()->$timestamp:'.$timestamp;
|
44
|
echo '</br>';
|
45
|
return $timestamp;
|
46
|
}
|
47
|
48
|
function nextId()
|
49
|
{
|
50
|
$timestamp=$this->timeGen();
|
51
|
echo 'logdebug->nextId()->self::$lastTimestamp1:'.self::$lastTimestamp;
|
52
|
echo '</br>';
|
53
|
if(self::$lastTimestamp == $timestamp) {
|
54
|
self::$sequence = (self::$sequence + 1) & self::$sequenceMask;
|
55
|
if (self::$sequence == 0) {
|
56
|
echo "###########".self::$sequenceMask;
|
57
|
$timestamp = $this->tilNextMillis(self::$lastTimestamp);
|
58
|
echo 'logdebug->nextId()->self::$lastTimestamp2:'.self::$lastTimestamp;
|
59
|
echo '</br>';
|
60
|
}
|
61
|
} else {
|
62
|
self::$sequence = 0;
|
63
|
echo 'logdebug->nextId()->self::$sequence:'.self::$sequence;
|
64
|
echo '</br>';
|
65
|
}
|
66
|
if ($timestamp < self::$lastTimestamp) {
|
67
|
throw new Excwption("Clock moved backwards. Refusing to generate id for ".(self::$lastTimestamp-$timestamp)." milliseconds");
|
68
|
}
|
69
|
self::$lastTimestamp = $timestamp;
|
70
|
echo 'logdebug->nextId()->self::$lastTimestamp3:'.self::$lastTimestamp;
|
71
|
echo '</br>';
|
72
|
73
|
echo 'logdebug->nextId()->(($timestamp - self::$twepoch << self::$timestampLeftShift )):'.((sprintf('%.0f', $timestamp) - sprintf('%.0f', self::$twepoch) ));
|
74
|
echo '</br>';
|
75
|
$nextId = ((sprintf('%.0f', $timestamp) - sprintf('%.0f', self::$twepoch) )) | ( self::$workerId << self::$workerIdShift ) | self::$sequence;
|
76
|
echo 'timestamp:'.$timestamp.'-----';
|
77
|
echo 'twepoch:'.sprintf('%.0f', self::$twepoch).'-----';
|
78
|
echo 'timestampLeftShift ='.self::$timestampLeftShift.'-----';
|
79
|
echo 'nextId:'.$nextId.'----';
|
80
|
echo 'workId:'.self::$workerId.'-----';
|
81
|
echo 'workerIdShift:'.self::$workerIdShift.'-----';
|
82
|
return $nextId;
|
83
|
}
|
84
|
85
|
}
|
86
|
$Idwork = new Idwork(1);
|
87
|
$a= $Idwork->nextId();
|
88
|
$Idwork = new Idwork(2);
|
89
|
$a= $Idwork->nextId();
|
90
|
?>
|
分布式系统生成唯一主键相关推荐
- 分布式数据库中全局唯一主键
[相关文章] <分布式数据库中全局唯一主键生成策略的设计与实现> <activiti5.10解决分布式集群部署的主键问题> <分布式环境下数据库主键方案> < ...
- 分布式学习笔记-唯一主键生成方式
分布式如何生成唯一主键 描述 优点 缺点 UUID UUID 通用唯一标识码缩写,其目是让分布式系统中所有元素都有唯一的辨识信息,而且不需要通过中央控制器来指定唯一标识 java.util.UUID. ...
- Ticket 服务: 一种经济的分布式唯一主键生成方案
2019独角兽企业重金招聘Python工程师标准>>> MySQL分库分表早已经不是什么新鲜话题了.甚至已经成了说到MySQL就会说到的话题.在一张表中,MySQL提供了原生的自增主 ...
- 数据库唯一主键如何实现幂等性？
数据库唯一主键的实现主要是利用数据库中主键唯一约束的特性,一般来说唯一主键比较适用于"插入"时的幂等性,其能保证一张表中只能存在一条带该唯一主键的记录. 使用数据库唯一主键完成幂等 ...
- truncate报ORA-02266错“唯一/主键被启用的外部关键字引用”解决方法
今天想truncate一张表,但报ORA-02266错误 想到可能是主键被子表引用,查了一下 解决如下: SQL> truncate table table_name; truncate tab ...
- oracle唯一索引能删除吗,Oracle：ora-02429:无法用于删除强制唯一/主键的索引 解决...
今天打算删除orcale数据库中无用的表空间,发现报错,查资料删除,写个过程留着备用. 1.drop tablespace dldata INCLUDING CONTENTS CASCADE CONS ...
- Java生成唯一主键
一般有时候我们需要生成唯一主键id,如果数据库是mysql我们可以使用主键自增,如果是oracle我们可以创建触发器或者序列,如果不借助数据库我们也可以在java层面自己生成唯一主键. 使用随机数: ...
- Oracle 删除表空间错误 提示：ora-02429:无法删除用于强制唯一/主键的索引
sql>droptablespace xxx ora-01549:表空间非空,请使用INCLUDING CONTENTS 选项 sql>droptablespace xxx INCLUDI ...
- 数仓业务上判断一个表的唯一主键是哪几个字段
平时业务量比较大,一张HIVE表里面的字段也比较多,如何判断所select 的这几个字段是这张表的唯一主键,也就是,选取这几个主键,可以唯一确定只有一行数据 可以通过,select A,B,C fro ...
最新文章
- 《Python和Pygame游戏开发指南》——2.16 pygame.display.update()函数
- ev3pid巡线_PID算法巡线
- oracle 连续次数,如何求字段连续出现的次数？
- Active Directory操作主机的转移 —图形操作
- 二进制包 mysql_二进制包;mysql
- Bootstrap4+MySQL前后端综合实训-Day01-PM【position定位的四种方式、Flex布局语法教程及案例(概念、容器属性、项目属性)、双飞翼布局复习、Bootstrap4 教程】
- 数据库笔记05：创建与管理数据库
- linux stat文件,Linux stat命令：显示文件或文件系统的详细信息
- Windows下MySql主从配置实战教程
- qq发文件大小上限_微信再两个放大招！网友：QQ可以卸载了？
- wraper for bootstrap3.0 + simple_form
- C++求复数的角度_高考数学一轮复习33，复数，常见类型及解题策略
- Python基于迁移学习的猫狗大战实战【图像二分类任务】【实测准确度超过99.5%】
- 手机个人热点连接台式计算机,电脑怎么连接手机个人热点
- 对企业形象识别系统（CIS）的了解
- 什么软件可以测试睡眠质量心率,Beddit：粘在床上就能测试心率的睡眠监测器
- indesign里怎么打根号_indesign 数学符号
- PAT 1068 万绿丛中一点红
- CountDownLatch、CyclicBarrier实战场景分析(附代码)
- SysML 第一讲：SysML简介