異步設備IO：OVERLAPPED和IOCompletionPort
本文內容為《windows核心編程》第10章內容的總結，僅記錄一些本人感興趣的內容。
1：OVERLAPPED
“overlapped”的意思是執行IO請求的事件與線程執行其他任務的時間是重疊的（overlapped）。
overlapped是執行設備異步IO的基礎。overlapped結構定義如下：
Descriptio：Contains information used in asynchronous (or overlapped) input and output (I/O).保存異步IO信息的結構體。
1 typedef struct _OVERLAPPED { 2 ULONG_PTR Internal; 3 ULONG_PTR InternalHigh; 4 union { 5 struct { 6 DWORD Offset; 7 DWORD OffsetHigh; 8 }; 9 PVOID Pointer; 10 }; 11 HANDLE hEvent; 12 } OVERLAPPED, *LPOVERLAPPED;
Members
- Internal
-
Reserved for operating system use. This member, which specifies a system-dependent status, is valid when the GetOverlappedResult function returns without setting the extended error information to ERROR_IO_PENDING.
系統預留的關鍵字。
- InternalHigh
-
Reserved for operating system use. This member, which specifies the length of the data transferred, is valid when the GetOverlappedResult function returns TRUE.
系統預留的關鍵字。
- Offset
-
The file position at which to start the transfer. The file position is a byte offset from the start of the file. The calling process must set this member before calling the ReadFile or WriteFile function.
This member is used only when the device is a file. Otherwise, this member must be zero.
- OffsetHigh
-
The high-order word of the file position at which to start the transfer.
This member is used only when the device is a file. Otherwise, this member must be zero.
Offset和OffsetHigh參數僅文件設備有效，非文件設備參數應置為0，否則IO請求將會失敗，調用GetLastError會返回ERROR_INVALID_PARAMETER。當設備是文件時，該參數指示訪問文件時應該從哪里進行IO操作
- Pointer
-
Reserved for system use; do not use.
系統預留。
- hEvent
-
A handle to the event that will be set to the signaled state when the operation has been completed. The calling process must set this member either to zero or a valid event handle before calling any overlapped functions. To create an event object, use the CreateEvent function. This function returns a handle that can be used to synchronize simultaneous I/O requests for a device.
Functions such as ReadFile and WriteFile set this handle to the nonsignaled state before they begin an I/O operation. When the operation has completed, the handle is set to the signaled state.
Functions such as GetOverlappedResult and the wait functions reset auto-reset events to the nonsignaled state. Therefore, if you use an auto-reset event, your application can hang if you wait for the operation to complete then call GetOverlappedResult.
IO操作完成的通知事件。在使用IOCP時可用。
如果IO操作是同步執行的，ReadFile和WriteFile將會返回非0值。如果IO操作異步執行，ReadFile和WriteFile將返回False，yongGetLastError將會返回ERROR_IO_PENDING，表示IO請求已經加入隊列，晚些時候完成。
要注意的是，使用ReadFile和WriteFile這樣的異步操作要將ReadFile和WriteFile的參數定義為全局變量，不要定義成局部變量。局部變量被釋放后，ReadFile和WriteFile使用局部變量的地址空間，將會產生無法預料的錯誤。
2：IOCompletionPort
2.1 IO完成端口定義
首先，要明確IO完成端口是一個內核對象，既然是內核對象，按照windows慣例，內部結構是未知的（HADNLDE是指針，外部不可見）。
IO完成端口背后的理論是並發運行的線程的數量必須有一個上限，因為過多的線程會導致Windows內核頻繁在可運行線程里面上下文切換，浙這將占用大量的cpu。一旦可運行線程數量大於可用cpu數量，系統將花時間在執行線程上下文切換，這是並發模型的一個潛在缺點。IO完成端口的創建函數如下：
Description：
Associates an input/output (I/O) completion port with one or more file handles, or it can create an I/O completion port that is not associated with a file handle.
Associating an instance of an opened file with an I/O completion port lets an application receive notification of the completion of asynchronous I/O operations involving that file.
1 HANDLE WINAPI CreateIoCompletionPort( 2 __in HANDLE FileHandle, 3 __in HANDLE ExistingCompletionPort, 4 __in ULONG_PTR CompletionKey, 5 __in DWORD NumberOfConcurrentThreads 6 );
Parameters
- FileHandle
-
A handle to a file opened for overlapped I/O completion. You must specify the FILE_FLAG_OVERLAPPED flag when using the CreateFile function to obtain the handle.
If FileHandle specifies INVALID_HANDLE_VALUE, CreateIoCompletionPort creates an I/O completion port without associating it with a file. In this case, the ExistingCompletionPort parameter must be NULL and the CompletionKey parameter is ignored.
如果CreateIoCompletionPort 用來創建一個IO完成端口，則應配置FileHandle為INVALID_HANDLE_VALUE，ExistingCompletionPort 為NULL ，CompletionKey將被忽略調。
- ExistingCompletionPort
-
A handle to the I/O completion port.
If this parameter specifies an existing completion port, the function associates it with the file specified by the FileHandle parameter. The function returns the handle of the existing completion port; it does not create a new I/O completion port.
If this parameter is NULL, the function creates a new I/O completion port and associates it with the file specified by FileHandle. The function returns the handle to the new I/O completion port.
如果此值為NULL，函數將創建一個新的IO完成端口。如果FileHandle不是INVALID_HANDLE_VALUE，則新的IO完成端口將與FileHandle關聯。
- CompletionKey
-
The per-file completion key that is included in every I/O completion packet for the specified file.
IO完成端口處理完任務后將把此值傳回。
- NumberOfConcurrentThreads
-
The maximum number of threads that the operating system can allow to concurrently process I/O completion packets for the I/O completion port. This parameter is ignored if the ExistingCompletionPort parameter is not NULL.
If this parameter is zero, the system allows as many concurrently running threads as there are processors in the system.
允許IO完成端口關聯的線程並發運行的數量。如果NumberOfConcurrentThreads賦值為0，將默認為系統的cpu數量。
Return Value
If the function succeeds, the return value is the handle to the I/O completion port that is associated with the specified file.
If the function fails, the return value is NULL. To get extended error information, call GetLastError.
CreateIoCompletionPort函數有2個功能，1：創建一個IO完成端口；2：將一個設備與一個IO完成端口關聯。
使用第一個功能時，前3個參數固定傳入INVALID_HANDLE_VALUE、NULL、0，最后一個參數傳入IOCP允許並發線程數量（為0則默認為cpu數量）。
使用第二個功能時，第一個參數傳入設備句柄（包含文件、socket、郵件槽、管道等）、第二個參數傳入IOCP的句柄、第三個參數傳入完成鍵值、第四個參數傳入0。
2.2 IO完成端口周邊架構
IOCP內部維護了設備列表、IO完成隊列、等待線程隊列、已釋放線程隊列、已暫停線程隊列5個表。
設備列表：使用CreateIoCompletionPort關聯設備時，會加入該列表。當設備句柄關閉時，會從設備列表中移除設備信息。
IO完成隊列：當設備的異步IO請求完成時，系統會檢查設備是否與IOCP關聯，如果設備與IOCP關聯，那么系統將該項已完成的IO請求加入IO完成隊列末尾。也可以使用PostQueuedCompletionStatus向IO完成隊列末尾發出請求，進入入隊操作。
等待線程隊列：如果線程調用GetQueuedCompletionStatus去獲取IO完成隊列事件，操作系統將線程切換到睡眠狀態（注意是睡眠狀態，該狀態下不占用CPU），並記錄線程ID到等待隊列。如果IO完成隊列非空，且當前並發線程數量小於最大並發線程數量，則喚醒線程，將喚醒線程的ID從等待線程隊列中移除。值得注意的是，等待線程隊列是后入先出的類型。比如，ABCD4個線程在等待，一個事件到達，D被喚醒，處理完后D又加入線程隊列。當下個事件到達時，依舊是D被喚醒。《win核心》解釋，通過這種后入先出的算法，系統可以將那些未被調度線程（某些線程總不會調度）的內存資源換到磁盤，將其內容湊個高速緩存中清除。
已釋放線程列表：線程調用GetQueuedCompletionStatus得到IO完成隊列事件后，如果線程處於可運行狀態，則信息記錄在已釋放線程隊列。
已暫停線程列表：如果線程調用函數將自己掛起，信息將進入已暫停線程列表中。注意，非GetQueuedCompletionStatus阻塞的線程才進入已暫停線程列表，而GetQueuedCompletionStatus阻塞的線程進入等待線程列表。
2.3 GetQueuedCompletionStatus
Description：Attempts to dequeue an I/O completion packet from the specified I/O completion port. If there is no completion packet queued, the function waits for a pending I/O operation associated with the completion port to complete.
1 BOOL WINAPI GetQueuedCompletionStatus( 2 __in HANDLE CompletionPort, 3 __out LPDWORD lpNumberOfBytes, 4 __out PULONG_PTR lpCompletionKey, 5 __out LPOVERLAPPED* lpOverlapped, 6 __in DWORD dwMilliseconds 7 );
Parameters
- CompletionPort
-
A handle to the completion port. To create a completion port, use the CreateIoCompletionPort function.
- lpNumberOfBytes
-
A pointer to a variable that receives the number of bytes transferred during an I/O operation that has completed.
IO完成端口接收到數據的數據。
- lpCompletionKey
-
A pointer to a variable that receives the completion key value associated with the file handle whose I/O operation has completed. A completion key is a per-file key that is specified in a call to CreateIoCompletionPort.
- lpOverlapped
-
A pointer to a variable that receives the address of the OVERLAPPED structure that was specified when the completed I/O operation was started.
Return Value
If the function dequeues a completion packet for a successful I/O operation from the completion port, the return value is nonzero. The function stores information in the variables pointed to by the lpNumberOfBytes, lpCompletionKey, and lpOverlapped parameters.
如果函數從完成隊列中取出IO事件，返回值非0.並在 lpNumberOfBytes, lpCompletionKey, and lpOverlapped 參數中存儲信息。
If *lpOverlapped is NULL and the function does not dequeue a completion packet from the completion port, the return value is zero. The function does not store information in the variables pointed to by the lpNumberOfBytes and lpCompletionKey parameters. To get extended error information, call GetLastError. If the function did not dequeue a completion packet because the wait timed out, GetLastError returns WAIT_TIMEOUT.
如果*lpOverlapped非NULL，且函數未沖完成端口中取出事件信息，返回值為0。函數不會再lpNumberOfBytes, lpCompletionKey存儲信息。如果函數因時間到達未取出事件，則GetLastError 返回WAIT_TIMEOUT。
If *lpOverlapped is not NULL and the function dequeues a completion packet for a failed I/O operation from the completion port, the return value is zero. The function stores information in the variables pointed to by lpNumberOfBytes, lpCompletionKey, and lpOverlapped. To get extended error information, call GetLastError.
如果*lpOverlapped非NULL，且IO操作失敗，函數取出了IO失敗的事件信息，則返回值是0.lpNumberOfBytes, lpCompletionKey, and lpOverlapped存儲了失敗事件的信息。
2.4 PostQueuedCompletionStatus
Description：Posts an I/O completion packet to an I/O completion port.
1 BOOL WINAPI PostQueuedCompletionStatus( 2 __in HANDLE CompletionPort, 3 __in DWORD dwNumberOfBytesTransferred, 4 __in ULONG_PTR dwCompletionKey, 5 __in LPOVERLAPPED lpOverlapped 6 );
Parameters
- CompletionPort
-
A handle to an I/O completion port to which the I/O completion packet is to be posted.
- dwNumberOfBytesTransferred
-
The value to be returned through the lpNumberOfBytesTransferred parameter of the GetQueuedCompletionStatus function.
- dwCompletionKey
-
The value to be returned through the lpCompletionKey parameter of the GetQueuedCompletionStatus function.
- lpOverlapped
-
The value to be returned through the lpOverlapped parameter of the GetQueuedCompletionStatus function.
Return Value
If the function succeeds, the return value is nonzero.
If the function fails, the return value is zero. To get extended error information, call GetLastError .
PostQueuedCompletionStatus參數和GetQueuedCompletionStatus參數的前4項一一對應。
3 IOCP的封裝類
《win核心》示例程序提供了對IOCP的封裝類，代碼如下：
1 class CIOCP { 2 public: 3 CIOCP(int nMaxConcurrency = -1) { 4 m_hIOCP = NULL; 5 if (nMaxConcurrency != -1) 6 (void) Create(nMaxConcurrency); 7 } 8 9 ~CIOCP() { 10 if (m_hIOCP != NULL) 11 chVERIFY(CloseHandle(m_hIOCP)); 12 } 13 14 //關閉IOCP 15 BOOL Close() { 16 BOOL bResult = CloseHandle(m_hIOCP); 17 m_hIOCP = NULL; 18 return(bResult); 19 } 20 21 //創建IOCP，nMaxConcurrency指定最大線程並發數量，0默認為cpu數量 22 BOOL Create(int nMaxConcurrency = 0) { 23 m_hIOCP = CreateIoCompletionPort(INVALID_HANDLE_VALUE, NULL, 0, nMaxConcurrency); 24 chASSERT(m_hIOCP != NULL); 25 return(m_hIOCP != NULL); 26 } 27 28 //為設備（文件、socket、郵件槽、管道等）關聯一個IOCP 29 BOOL AssociateDevice(HANDLE hDevice, ULONG_PTR CompKey) { 30 BOOL fOk = (CreateIoCompletionPort(hDevice, m_hIOCP, CompKey, 0) == m_hIOCP); 31 chASSERT(fOk); 32 return(fOk); 33 } 34 35 //為設備（文件、socket、郵件槽、管道等）關聯一個IOCP 36 BOOL AssociateSocket(SOCKET hSocket, ULONG_PTR CompKey) { 37 return(AssociateDevice((HANDLE) hSocket, CompKey)); 38 } 39 40 //為iocp傳遞事件通知 41 BOOL PostStatus(ULONG_PTR CompKey, DWORD dwNumBytes = 0, 42 OVERLAPPED* po = NULL) { 43 44 BOOL fOk = PostQueuedCompletionStatus(m_hIOCP, dwNumBytes, CompKey, po); 45 chASSERT(fOk); 46 return(fOk); 47 } 48 49 //從IO完成隊列中獲取事件通知。IO完成隊列無事件時，該函數將阻塞 50 BOOL GetStatus(ULONG_PTR* pCompKey, PDWORD pdwNumBytes, 51 OVERLAPPED** ppo, DWORD dwMilliseconds = INFINITE) { 52 53 return(GetQueuedCompletionStatus(m_hIOCP, pdwNumBytes, 54 pCompKey, ppo, dwMilliseconds)); 55 } 56 57 private: 58 //IOCP句柄 59 HANDLE m_hIOCP; 60 };