NVMe over PCIe最新的NVMe協議是1.3。 在7.2.1講了Command Processing流程。有圖有真相。
This section describes command submission and completion processing. Figure 251 shows the steps that are followed to submit and complete a command. The steps are:
本節描述了命令提交和命令完成處理。圖251列出了提交和完成一個命令的步驟。
- 1. The host places one or more commands for execution in the next free Submission Queue slot(s) in memory. 主機將一個或多個命令放置在位於內存中的提交隊列(SQ)的下一個可用的槽位中執行。
- 2. The host updates the Submission Queue Tail Doorbell register with the new value of the Submission Queue Tail entry pointer. This indicates to the controller that a new command(s) is submitted for processing. Host用SQ尾部指針的新值去更新SQ的TailDB寄存器。這告訴了SSD控制器有一個新的命令被提交需要被處理。
- 3. The controller transfers the command(s) from in the Submission Queue slot(s) into the controller for future execution. Arbitration is the method used to determine the Submission Queue from which the controller starts processing the next candidate command(s), refer to section 4.11. SSD控制器將命令從SQ中轉移到控制器中以供下一步執行。(從哪一個SQ中取出下一條候選命令去執行的仲裁方法，請參見4.11一節。)
- 4. The controller then proceeds with execution of the next command(s). Commands may complete out of order (the order submitted or started execution). 控制器接下來執行下一條命令。命令的執行完成可能是亂序的(與提交或開始執行的時間點無關)。
- 5. After a command has completed execution, the controller places a completion queue entry in the next free slot in the associated Completion Queue. As part of the completion queue entry, the controller indicates the most recent Submission Queue entry that has been consumed by advancing the Submission Queue Head pointer in the completion entry. Each new completion queue entry has a Phase Tag inverted from the previous entry to indicate to the host that this completion queue entry is a new entry. 在命令完成執行之后，SSD控制器將一個完成隊列條目(CQE)放在相關的完成隊列(CQ)的下一個空閑槽位中。作為CQE的一部分，SSD控制器通過修改完成條目的SQ頭指針指示最新的SQE已經被消費了。每一個新的CQE都有一個從前一個條目中反轉的相位標記(Phase Tag), 以向Host表明這個CQE是一個新條目。
- 6. The controller optionally generates an interrupt to the host to indicate that there is a new completion queue entry to consume and process. In the figure, this is shown as an MSI-X interrupt, however, it could also be a pin-based or MSI interrupt. Note that based on interrupt coalescing settings, an interrupt may or may not be generated for each new completion queue entry. 可選地，SSD控制器給Host產生一個中斷，以表明有一個新的CQE已經產生，可以被消費和處理了。在圖中演示的是MSI-X中斷，然而，中斷也可以是基於PIN或者MSI的中斷。注意：基於中斷聯合設置，可能或不能為每一個新的CQE產生一個中斷。
- 7. The host consumes and then processes the new completion queue entries in the Completion Queue. This includes taking any actions based on error conditions indicated. The host continues consuming and processing completion queue entries until it encounters a previously consumed entry with a Phase Tag inverted from the value of the current completion queue entries. Host消費和處理在CQ中放置的新的CQE。包括基於錯誤情況采取的任何操作。Host繼續消耗和處理CQE，直到它遇到以前消費的一個條目的相位標簽(Phase Tag)從當前完成隊列條目(CQEs)的值中反轉。
- 8. The host writes the Completion Queue Head Doorbell register to indicate that the completion queue entry has been consumed. The host may consume many entries before updating the associated Completion Queue Head Doorbell register. Host更新CQ的HeadDB寄存器，表明CQE已經被消費了。在更新相關聯的CQ的HeadDB寄存器之前，Host可能消費了多個CQE。
以上8步，寫得比較具體，稍微優點羅嗦，那么，用更通俗易懂的話總結一下就是：
- 1. Host寫命令到SQ
- 2. Host更新SQ的TailDB, 通知SSD取命令
- 3. SSD收到命令，於是從SQ中取出命令
- 4. SSD執行命令
- 5. 命令執行完成后，SSD往CQ中寫入命令執行結果，同時修改CQ的TailDB
- 6. SSD發短信通知Host命令已經執行完成
- 7. Host收到命令后，到CQ中查看命令完成狀態
- 8. Host處理完CQ中的命令執行結果，更新CQ中的HeadDB, 回復SSD, "命令執行結果已經處理完畢，辛苦啦"
P.S. NVMe over PCIe和我們前面提到的RDMA本質上都是“玩隊列”。 NVMe over PCIe有兩條隊列，一條提交隊列(SQ)和一條完成隊列(CQ)；而RDMA有三條隊列，一條發送隊列(SQ)，一條接收隊列(RQ)和一條完成隊列(CQ)，而一個SQ和一個RQ被稱之為一個QP(隊列對)。
If you don't aim high you will never hit high.