Hystrix 熔斷機制原理

相關配置

circuitBreaker.enabled  是否開啟熔斷
circuitBreaker.requestVolumeThreshold  熔斷最低觸發請求數閾值
circuitBreaker.sleepWindowInMilliseconds  產生熔斷后恢復窗口
circuitBreaker.errorThresholdPercentage  錯誤率閾值
circuitBreaker.forceOpen  強制打開熔斷
circuitBreaker.forceClosed  強制關閉熔斷

狀態圖

執行流程

命令執行前調用circuitBreaker.attemptExecution()，正常情況下會執行返回true，但是如果發生熔斷，則需要通過sleepWindows來進行恢復

public boolean attemptExecution() {
    if (properties.circuitBreakerForceOpen().get()) {
        return false;
    }
    if (properties.circuitBreakerForceClosed().get()) {
        return true;
    }
    if (circuitOpened.get() == -1) {
        return true;
    } else {
        if (isAfterSleepWindow()) {
            if (status.compareAndSet(Status.OPEN, Status.HALF_OPEN)) {
                //only the first request after sleep window should execute
                return true;
            } else {
                return false;
            }
        } else {
            return false;
        }
    }
}

發生熔斷流程

在新版本1.5.12中，會有一個后台線程訂閱metrics流實時計算：

1. 如果沒有達到RequestVolume，則直接返回，不計算是否需要熔斷
2. 如果當前錯誤率大於設置的閾值，則觸發熔斷，狀態由CLOSED切換到OPEN，並設置當前熔斷的時間（用於后續SleepWindows判斷使用）

if (hc.getTotalRequests() < properties.circuitBreakerRequestVolumeThreshold().get()) {
    // we are not past the minimum volume threshold for the stat window,
    // so no change to circuit status.
    // if it was CLOSED, it stays CLOSED
    // if it was half-open, we need to wait for a successful command execution
    // if it was open, we need to wait for sleep window to elapse
} else {
    if (hc.getErrorPercentage() < properties.circuitBreakerErrorThresholdPercentage().get()) {
        //we are not past the minimum error threshold for the stat window,
        // so no change to circuit status.
        // if it was CLOSED, it stays CLOSED
        // if it was half-open, we need to wait for a successful command execution
        // if it was open, we need to wait for sleep window to elapse
    } else {
        // our failure rate is too high, we need to set the state to OPEN
        if (status.compareAndSet(Status.CLOSED, Status.OPEN)) {
            circuitOpened.set(System.currentTimeMillis());
        }
    }
}

熔斷恢復流程

 當發生熔斷，達到SleepWindows指定時間后，則狀態會由OPEN轉換為HALF_OPEN，此時只會讓一個請求通過，如果該請求執行成功，狀態則會轉換為CLOSED，否則會回到OPEN狀態，並且熔斷時間設置為當前時間，重新等待SleepWindows的時間

// 執行成功后調用
public void markSuccess() {
    if (status.compareAndSet(Status.HALF_OPEN, Status.CLOSED)) {
        //This thread wins the race to close the circuit - it resets the stream to start it over from 0
        metrics.resetStream();
        Subscription previousSubscription = activeSubscription.get();
        if (previousSubscription != null) {
            previousSubscription.unsubscribe();
        }
        Subscription newSubscription = subscribeToStream();
        activeSubscription.set(newSubscription);
        circuitOpened.set(-1L);
    }
}

// 執行失敗后調用
public void markNonSuccess() {
    if (status.compareAndSet(Status.HALF_OPEN, Status.OPEN)) {
        //This thread wins the race to re-open the circuit - it resets the start time for the sleep window
        circuitOpened.set(System.currentTimeMillis());
    }
}

參考

1. https://github.com/Netflix/Hystrix/wiki/Configuration
2. https://github.com/Netflix/Hystrix/wiki/How-it-Works#CircuitBreaker
3. HystrixCircuitBreaker源碼