Latency，中文譯作延遲，Tail Latency即尾延遲。
實際生產中的Latency是一種（概率）分布，實際上被描述為百分位數。 延遲可以在 75% 百分位處翻倍，在 99% 之后高出 100 倍。
什么導致了尾延遲
- 磁盤老化。Disk just get slowdown time to time for no reason. The Tail at Store gives more in-depth analysis. Also, disks may degrade significantly when they get old.
- 超時。Failure tolerance and retry is a common design pattern in distributed systems. But one retry is enough to send current request to latency tail. Google SRE Book chapter 21 to 22 discuss it in detail, such as,
- Reduce remaining timeout quota and pass it down each layer of the request processing chain.
- Be aware of the chained retry amplification (layer1 3 retries, layer2 3*3 retries, …).
- 后台任務。Almost every services, from software to even hardware/firmware, have backgroud tasks. Background task may temporarily slowdown the world. The most notorious one is GC (garbage collection，垃圾回收).
- 超負載運行。The customer may be sending you too many/big requests, and upper layer throttling is not working well. Overprovisioned customer VMs may compete with each other resulting slow experience. Some small piece of data may be extremly hot, e.g. many OS images are forked from a small shared base. A large request may be pegging your CPU/network/disk, and make the others queuing up. Or something went wrong, as a dead loop stuck your cpu.
緩解尾延遲
延遲可以分為low、middle和tail。控制和緩解延遲方法總結：
- 緩解low, middle部分：P提供更多資源、削減和並行化任務、消除 “head-of-line” 阻塞和緩存將有所幫助。這是我們應用於橫向擴展分布式系統的常用技術。
- 緩解tail部分：基本思想是hedging。 即使我們已經並行化了服務，最慢的實例也將決定我們的請求何時完成。 您可以使用概率數學對組合延遲分布進行建模。
- 發送比必要更多的請求，只收集最快的返回，有助於減少尾部。Send 2 instread of 1. Send 11 instead of 10 (e.g. in erasure-coding 10 fragment reconstruct read). Send backup requests at 95% percentile latency.
- 金絲雀請求,，i.e. send normal requests but fallback to sending hedged requests if the canary did’t finish in reasonable time.
- 通常，較小的任務分區（微分區）將有助於實現更平滑的延遲分布百分位數。
- 減緩 head-of-line blocking. 少量開銷較大的查詢可能會增加大量並發開銷較低的查詢的延遲。Uniformly smaller tasks partitioning camn help.
- 處理超時
- 首先嘗試a non-block try 讀取（讀取但不等待），然后進行盡力讀取（讀取並等待超時）。
- 當發現超時時，將相關資源標記為known slow。 並告知其他請求繞過這個資源。
- 要設置合適的超時值，我們可以設置為99.9% ，並動態調整它。 任意超時值可能有害。
- 更細粒度的調度，甚至是平衡延遲和成本的管理框架。(e.g. Bing’s Kwiken, also attached below.)
監控
有兩種監控指標：
- Single operation
- Percentile statistics
監控應該能夠：
- 提供可以從用戶請求入口跟蹤到硬件操作的trace id
- 涵蓋每個級別的細分
- 覆蓋容易出問題的地方
有幾個方面需要監控：
- 與故障直接相關的錯誤，例如虛擬機停止/重新啟動
- 直接影響用戶體驗的超時錯誤計數和自動限制
- Operation slowdown
- 典型的硬件性能，如CPU、網絡、磁盤
- 提供從用戶進入的跟蹤、每個級別的細分以及最終到硬件的跟蹤
其他參考資料
- Latency Numbers Every Programmer Should Know: Give you the basic latency numbers of each level of a storage system
- Disks for Data Centers: Google’s disk experience, which covers some aspect of disk tail latency.