關於分布式唯一ID，snowflake的一些思考及改進(完美解決時鍾回撥問題)

1.寫唯一ID生成器的原由

在閱讀工程源碼的時候，發現有一個工具職責生成一個消息ID，方便進行全鏈路的查詢，實現方式特別簡單，核心源碼不過兩行，根據時間戳以及隨機數生成一個ID，這種算法ID在分布式系統中重復的風險就很明顯了。本來以為只是日志打印功能，根據於此在不同系統調用間關聯業務日志而已，不過后來發現此ID需要入庫，看到這里就覺得有些風險了，於是就想着怎么改造它。

String timeString = String.valueOf(System.currentTimeMillis());
return Long.parseLong(timeString.substring(timeString.length() - 8, timeString.length()))
                * RandomUtils.nextInt(1, 9);

2. Twitter snowflake

既然是分布式唯一ID，自然而然想到了Twitter的snowflake算法，在以前的做的部分業務中也用到過它來生成數據庫主鍵ID，不過當時僅限於使用，以及將64 bit的long數字拆分成幾個部分，以保證唯一，對具體實現沒有深入研究。正好借着機會深入下。

snowflake拆分long的示意圖

該ID生成方式，用41位時間戳保存當前時間的毫秒數（69年后一個輪回），十位機器碼，最多可以供一項業務1024台實例，每個毫秒數，12位序列自增，每秒理論上單機環境可生產409.6萬個ID，分享一個官方github的scala實現，twitter-archive/snowflake。

3.關於snowflake的一些思考

1.監視器鎖

此鎖的目的是為了保證在多線程的情況下，只有一個線程進入方法體生成ID，保證並發情況下生成ID的唯一性，如果在競爭激烈情況下，自選鎖+ CAS原子變量的方式或許是更為合理的選擇，可以達到優化部分性能的目的。

源碼中的監視器鎖

2.時間回退問題

時間校准，以及其他因素，可能導致服務器時間回退（時間向前快進不會有問題），如果恰巧回退前生成過一些ID，而時間回退后，生成的ID就有可能重復。官方對於此並沒有給出解決方案，而是簡單的拋錯處理，這樣會造成在時間被追回之前的這段時間服務不可用，顯然我無法接受這一點。

官方的拋錯處理

而對於此的思考是，既然snowflake理論情況下單機可實現每秒409.6萬個ID的生成上限，實際上能想得到的業務都不太可能產生如此高的並發，那么就會存在在過去的一段時間內，有大量的時間戳“被浪費”，達不到該上限，可能在某一毫秒內只生成幾個ID,如果發生了時間回退，這些“被浪費”的資源是不是就能利用起來，而不是拋錯。

被浪費的時間戳

如果在內存中建立一個數組，這個數組設定固定長度，比如說200，這些數組中存儲上一次該位置對應的毫秒數的messageId，那么就能在時間回退到追回時間這段時間內，再至多提供819200（(2^12) *200）個messageId，如果發生時間回退，就只用在上一次messageId進行+1操作，直到系統時間被追回（此段結合后續源碼進行解釋）。

4.改進版的snowflake

1.機器碼生成器 MachineIdService設計及其實現：

public interface MachineIdService {
    /** * 生成MachineId的方法 * * @return machineId 機器碼 * @throws MessageIdException 獲取機器碼可能因為外部因素失敗 */
    Long getMachineId() throws MessageIdException;
}

實現該接口確保一個集群中，每台實例生成不同的machineID，並且MachineID 不能超過(2^10) 1023,具體實現方式，可使用MySQL數據庫，文件描述映射，Redis自增等方式，這里我使用了Redis自增的方式（所以在需要用到該ID生成器的地方需要依賴Redis），具體實現方式如下：

public class RedisMachineIdServiceImpl implements MachineIdService {
<span class="token keyword">private</span> <span class="token keyword">static</span> final String <span class="token constant">MAX_ID</span> <span class="token operator">=</span> <span class="token string">"MAX_ID"</span><span class="token punctuation">;</span>
<span class="token keyword">private</span> <span class="token keyword">static</span> final String <span class="token constant">IP_MACHINE_ID_MAPPING</span> <span class="token operator">=</span> <span class="token string">"IP_MACHINE_ID_MAPPING"</span><span class="token punctuation">;</span>

<span class="token keyword">private</span> RedisTemplate<span class="token operator">&lt;</span>String<span class="token punctuation">,</span> String<span class="token operator">&gt;</span> redisTemplate<span class="token punctuation">;</span>


<span class="token keyword">private</span> String redisKeyPrefix<span class="token punctuation">;</span>

<span class="token comment">//設置RedisTemplate實例</span>
<span class="token keyword">public</span> <span class="token keyword">void</span> <span class="token function">setRedisTemplate</span><span class="token punctuation">(</span><span class="token parameter">RedisTemplate<span class="token operator">&lt;</span>String<span class="token punctuation">,</span> String<span class="token operator">&gt;</span> redisTemplate</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
    <span class="token keyword">this</span><span class="token punctuation">.</span>redisTemplate <span class="token operator">=</span> redisTemplate<span class="token punctuation">;</span>
<span class="token punctuation">}</span>

<span class="token comment">// 設置redisKey前綴，如果多個業務使用同一個Redis集群，使用不同的Redis前綴進行區分</span>
<span class="token keyword">public</span> <span class="token keyword">void</span> <span class="token function">setRedisKeyPrefix</span><span class="token punctuation">(</span><span class="token parameter">String redisKeyPrefix</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
    <span class="token keyword">this</span><span class="token punctuation">.</span>redisKeyPrefix <span class="token operator">=</span> redisKeyPrefix<span class="token punctuation">;</span>
<span class="token punctuation">}</span>

@Override
<span class="token keyword">public</span> Long <span class="token function">getMachineId</span><span class="token punctuation">(</span><span class="token punctuation">)</span> throws MessageIdException <span class="token punctuation">{</span>
    String host<span class="token punctuation">;</span>
    <span class="token keyword">try</span> <span class="token punctuation">{</span>
        <span class="token comment">//獲取本機IP地址</span>
        host <span class="token operator">=</span> InetAddress<span class="token punctuation">.</span><span class="token function">getLocalHost</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">getHostAddress</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
    <span class="token punctuation">}</span> <span class="token keyword">catch</span> <span class="token punctuation">(</span>UnknownHostException e<span class="token punctuation">)</span> <span class="token punctuation">{</span>
        <span class="token keyword">throw</span> <span class="token keyword">new</span> <span class="token class-name">MessageIdException</span><span class="token punctuation">(</span><span class="token string">"Can not get the host!"</span><span class="token punctuation">,</span> e<span class="token punctuation">)</span><span class="token punctuation">;</span>
    <span class="token punctuation">}</span>
    <span class="token keyword">if</span> <span class="token punctuation">(</span>redisTemplate <span class="token operator">==</span> <span class="token keyword">null</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
        <span class="token keyword">throw</span> <span class="token keyword">new</span> <span class="token class-name">MessageIdException</span><span class="token punctuation">(</span><span class="token string">"Can not get the redisTemplate instance!"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
    <span class="token punctuation">}</span>
    <span class="token keyword">if</span> <span class="token punctuation">(</span>redisKeyPrefix <span class="token operator">==</span> <span class="token keyword">null</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
        <span class="token keyword">throw</span> <span class="token keyword">new</span> <span class="token class-name">MessageIdException</span><span class="token punctuation">(</span><span class="token string">"The redis key prefix is null,please set redis key prefix first!"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
    <span class="token punctuation">}</span>
    HashOperations<span class="token operator">&lt;</span>String<span class="token punctuation">,</span> String<span class="token punctuation">,</span> Long<span class="token operator">&gt;</span> hashOperations <span class="token operator">=</span> redisTemplate<span class="token punctuation">.</span><span class="token function">opsForHash</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
    <span class="token comment">//通過IP地址在Redis中的映射，找到本機的MachineId</span>
    Long result <span class="token operator">=</span> hashOperations<span class="token punctuation">.</span><span class="token keyword">get</span><span class="token punctuation">(</span>redisKeyPrefix <span class="token operator">+</span> <span class="token constant">IP_MACHINE_ID_MAPPING</span><span class="token punctuation">,</span> host<span class="token punctuation">)</span><span class="token punctuation">;</span>
    <span class="token keyword">if</span> <span class="token punctuation">(</span>result <span class="token operator">!=</span> <span class="token keyword">null</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
        <span class="token keyword">return</span> result<span class="token punctuation">;</span>
    <span class="token punctuation">}</span>
    <span class="token comment">//如果沒有找到，說明需要對該實例進行新增MachineId，使用Redis的自增函數，生成一個新的MachineId</span>
    Long incrementResult <span class="token operator">=</span> redisTemplate<span class="token punctuation">.</span><span class="token function">opsForValue</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">increment</span><span class="token punctuation">(</span>redisKeyPrefix <span class="token operator">+</span> <span class="token constant">MAX_ID</span><span class="token punctuation">,</span> <span class="token number">1</span>L<span class="token punctuation">)</span><span class="token punctuation">;</span>
    <span class="token keyword">if</span> <span class="token punctuation">(</span>incrementResult <span class="token operator">==</span> <span class="token keyword">null</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
        <span class="token keyword">throw</span> <span class="token keyword">new</span> <span class="token class-name">MessageIdException</span><span class="token punctuation">(</span><span class="token string">"Get the machine id failed,please check the redis environment!"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
    <span class="token punctuation">}</span>
    <span class="token comment">//將生成的MachineId放入Redis中，方便下次查找映射</span>
    hashOperations<span class="token punctuation">.</span><span class="token function">put</span><span class="token punctuation">(</span>redisKeyPrefix <span class="token operator">+</span> <span class="token constant">IP_MACHINE_ID_MAPPING</span><span class="token punctuation">,</span> host<span class="token punctuation">,</span> incrementResult<span class="token punctuation">)</span><span class="token punctuation">;</span>
    <span class="token keyword">return</span> incrementResult<span class="token punctuation">;</span>
<span class="token punctuation">}</span>

}
 

2.MessageIdService設計以及實現

public interface MessageIdService {
<span class="token comment"><span class="hljs-comment">/**
 * 生成一個保證全局唯一的MessageId
 *
 * <span class="hljs-doctag">@return</span> messageId
 */</span></span>
<span class="token keyword"><span class="hljs-function"><span class="hljs-keyword">long</span></span></span><span class="hljs-function"> </span><span class="token function"><span class="hljs-function"><span class="hljs-title">genMessageId</span></span></span><span class="token punctuation"><span class="hljs-function"><span class="hljs-params">(</span></span></span><span class="token punctuation"><span class="hljs-function"><span class="hljs-params">)</span></span></span><span class="token punctuation">;</span>

<span class="token comment"><span class="hljs-comment">/**
 * 初始化方法
 *
 * <span class="hljs-doctag">@throws</span> MessageIdException
 */</span></span>
<span class="token keyword"><span class="hljs-function"><span class="hljs-keyword">void</span></span></span><span class="hljs-function"> </span><span class="token function"><span class="hljs-function"><span class="hljs-title">init</span></span></span><span class="token punctuation"><span class="hljs-function"><span class="hljs-params">(</span></span></span><span class="token punctuation"><span class="hljs-function"><span class="hljs-params">)</span></span></span><span class="hljs-function"> </span><span class="token keyword"><span class="hljs-function"><span class="hljs-keyword">throws</span></span></span><span class="hljs-function"> </span><span class="token class-name"><span class="hljs-function">MessageIdException</span></span><span class="token punctuation">;</span>

}
 

public class MessageIdServiceImpl implements MessageIdService {

    private static final Logger LOGGER = LoggerFactory.getLogger(MessageIdServiceImpl.class);
    //最大的MachineId，1024個
    private static final long MAX_MACHINE_ID = 1023L;
    //AtomicLongArray 環的大小，可保存200毫秒內，每個毫秒數上一次的MessageId，時間回退的時候依賴與此
    private static final int CAPACITY = 200;
    // 時間戳在messageId中左移的位數
    private static final int TIMESTAMP_SHIFT_COUNT = 22;
    // 機器碼在messageId中左移的位數
    private static final int MACHINE_ID_SHIFT_COUNT = 12;
    // 序列號的掩碼 2^12 4096
    private static final long SEQUENCE_MASK = 4095L;

    //messageId ，開始的時間戳，start the world，世界初始之日
    private static long START_THE_WORLD_MILLIS;
    //機器碼變量
    private long machineId;
    // messageId環，解決時間回退的關鍵，亦可在多線程情況下減少毫秒數切換的競爭
    private AtomicLongArray messageIdCycle = new AtomicLongArray(CAPACITY);
    //生成MachineIds的實例
    private MachineIdService machineIdService;

    static {
        try {
            //使用一個固定的時間作為start the world的初始值
            START_THE_WORLD_MILLIS = SimpleDateFormat.getDateTimeInstance().parse("2018-09-13 00:00:00").getTime();
        } catch (ParseException e) {
            throw new RuntimeException("init start the world millis failed", e);
        }
    }

    public void setMachineIdService(MachineIdService machineIdService) {
        this.machineIdService = machineIdService;
    }

    /** * init方法中通過machineIdService 獲取本機的machineId * @throws MessageIdException */
    @Override
    public void init() throws MessageIdException {
        if (machineId == 0L) {
            machineId = machineIdService.getMachineId();
        }
        //獲取的machineId 不能超過最大值
        if (machineId <= 0L || machineId > MAX_MACHINE_ID) {
            throw new MessageIdException("the machine id is out of range,it must between 1 and 1023");
        }
    }
    /** * 核心實現的代碼 */
    @Override
    public long genMessageId() {
        do {
            // 獲取當前時間戳，此時間戳是當前時間減去start the world的毫秒數
            long timestamp = System.currentTimeMillis() - START_THE_WORLD_MILLIS;
            // 獲取當前時間在messageIdCycle 中的下標，用於獲取環中上一個MessageId
            int index = (int)(timestamp % CAPACITY);
            long messageIdInCycle = messageIdCycle.get(index);
            //通過在messageIdCycle 獲取到的messageIdInCycle，計算上一個MessageId的時間戳
            long timestampInCycle = messageIdInCycle >> TIMESTAMP_SHIFT_COUNT;
            // 如果timestampInCycle 並沒有設置時間戳，或時間戳小於當前時間，認為需要設置新的時間戳
            if (messageIdInCycle == 0 || timestampInCycle < timestamp) {
                long messageId = timestamp << TIMESTAMP_SHIFT_COUNT | machineId << MACHINE_ID_SHIFT_COUNT;
                // 使用CAS的方式保證在該條件下，messageId 不被重復
                if (messageIdCycle.compareAndSet(index, messageIdInCycle, messageId)) {
                    return messageId;
                }
                LOGGER.debug("messageId cycle CAS1 failed");
            }
            // 如果當前時間戳與messageIdCycle的時間戳相等，使用環中的序列號+1的方式，生成新的序列號
            // 如果發生了時間回退的情況，（即timestampInCycle > timestamp的情況）那么不能也更新messageIdCycle 的時間戳，使用Cycle中MessageId+1
            if (timestampInCycle >= timestamp) {
                long sequence = messageIdInCycle & SEQUENCE_MASK;
                if (sequence >= SEQUENCE_MASK) {
                    LOGGER.debug("over sequence mask :{}", sequence);
                    continue;
                }
                long messageId = messageIdInCycle + 1L;
                // 使用CAS的方式保證在該條件下，messageId 不被重復
                if (messageIdCycle.compareAndSet(index, messageIdInCycle, messageId)) {
                    return messageId;
                }
                LOGGER.debug("messageId cycle CAS2 failed");
            }
            // 整個生成過程中，采用的spinLock
        } while (true);
    }

}

原文地址：https://www.jianshu.com/p/b1124283fc43