64位中使用AWE分配內存

We have already talked about Windows AWE mechanism on 32 bit and how SQL Server utilizes it. Today I would like to go over AWE & related mechanism on 64 bit platforms.

我們已經談過windows AWE 在32bit 中sql server 如何利用。下面是AWE在64位環境的相關性

 

To some people it comes as a surprise that AWE mechanism is still present and actually could be useful on 64 bit platforms. As you remember the mechanism consists of two parts allocating physical memory and mapping it to the given process's VAS. The advantage of allocation mechanism is that once physical memory is allocated operating system can't reclaim it until either the process is terminated or the process frees memory back to the OS. This feature allows an application to control and even avoid paging altogether. Advantage of mapping/unmapping mechanism is that the same physical page could be mapped into different VAS's regions. As you imaging unmapping is not necessary on 64 bit platforms since we have enough VAS to accommodate all of existing physical memory.

 

對於有些人會很驚奇AWE機制任然存在並且在64位中是可用的。這個機制分為2部分分配內存和映射到給定的VAS中。這種分配機制的好處是不會被系統回收，直到進程終止或者進程內存被釋放會操作系統。這特性允許應用程序控制並且避免分頁。映射和非映射機制的好處是相同的物理頁可以被映射到不同VAS空間。你可以想象在64bit中非映射是沒有必要的，因為64bit 有足夠的vas容納所有物理內存。

 

From Operating System theory, OS implements a page table entry, PTE, to describe a mapping of a page in VAS to physical page. Internally physical page is described by page frame number, PFN. Given PFN one can derive complete information about physical page it represents. For example PFN shows to which NUMA node the particular page belongs. OS has a database, collection of PFNs that it manages.  If page in VAS is committed, it has PTE which might or might not point to given PFN.  Conceptually, page that PTE represents can be either in memory or not, for example swapped out to disk. In the former case it is bound to a given PFN and in latter it is not. In its turn, once a physical page is bound to page in VAS, its PFN points back to PTE.

來自操作系統的理論，操作系統引入一個頁表項，PTE，描述了一個在VAS中的頁到物理頁的映射。物理頁被頁幀號描述，PFN。給定的PFN可以導出所有它代表的物理頁。例如PFN顯示了指定的物理也屬於那個NUMA節點。OS有一個數據庫收集了PFN並且管理它。如果頁在VAS中已經被提交了，PTE可能指向了一個給定的PFN。理論上PTE上的頁可能在內存上也可能沒有，如切換到了磁盤上。先前的例子頁綁定了一個PFN，並且之后不綁定了。一旦物理頁被綁定到VAS中，PFN就反向指向到PTE。

 

When OS commits, frees, pages out/in a given PTE or needs to derive some information about it, for example NUMA residency, it has to acquire process's working set lock - to guarantee stability of PTE to PFN binding. This lock is a rather expensive and might hurt scalability of the process. Latter versions of Windows made this lock as light as possible but avoiding still will benefit application's scalability..

當OS提交，釋放，切換一個給定的PTE或者取一些頁的信息。比如NUMA位置，頁不得不請求進程工作集鎖，來保證PTE到PFN之間的穩定性。這個鎖定比較昂貴並且可能會損害進程的擴展性。之后的windows版本會使這個鎖越來越輕量但是不能保證會對程序的擴展性有好處。

 

When allocating physical pages utilizing AWE mechanism we are given a set of PFN entries directly from PFN database - remember that you should not manipulate or modify set of entries you get back  nor can you rely on values you get back. OS is required to take a PFN database lock when allocating PFN entries. Using AWE map mechanism you can map allocated PFN entries to the process's VAS. When mapping occurs PTEs are allocated, bound to PFNs and marked as locked. In this case OS needs to acquire process's working set lock only ones. When mapping regular pages, OS does it on demand and hence will have to acquire both working set and PFN database lock for every page. Since pages are locked in memory, OS will ignore these PTEs during  paging process.

當物理內存通過AWE分配的時候，我們獲取的PFN項直接來至於PFN數據庫——記住你不能刪除或者修改你獲取的項，也不能依賴你獲取的值。當PFN被分配的時候，OS會請求鎖定PFN數據庫。使用AWE映射機制你可以把PFN映射到進程的VAS中。當映射發生PTE就會被分配，綁定到PFN並且標記被鎖定。這個時候OS只需要獲取進程的工作集鎖。當映射常規也的時候，OS按需求，並且會請求工作集鎖和PFN數據庫鎖。因為頁在內存中是鎖定的，在分頁進程的時候OS會忽略這些PTE。

 

On 64 bit platforms it is better to refer to such pages as locked pages - please don't confuse them with pages locked through VirtualLock API. As described above locked pages have two major properties - they are not considered for paging by OS and during allocation they acquire both working set and PFN database lock only ones. 

在64bit平台上， 這些頁被鎖定會更好——不要喝通過VirtualLock API混淆。頁鎖定有2個主要的屬性——他們不會被OS分頁並且在分配的時候只會請求工作集鎖和PFN鎖的一個。

 

The first property has implicit implication on high end hardware such as NUMA. It provides explicit memory residency. Remember that OS commits a page on demand. To allocate physical memory, it will use a node on which a thread touching memory is running.  Latter on, the page can be swapped out by OS. Next time it will be brought up into memory, OS will again allocate physical page from the node a thread touching memory is running on. In this case a node could be completely different  from original one. Such behavior makes hard for applications to rely on page's NUMA residency. Locked pages solve this problem by removing themselves from paging altogether.  Moreover Windows 2003 SP1 introduced a new API - QueryWorkingSetEx. It allows to query extended  information about PTE's PFN.  In order to find out real page residency you should use this API. When pages are locked you need to it only ones. Otherwise you will have to do it periodically since residency of the page can actually change.

 第一個屬性隱約的涉及到了高端的硬件如NUMA。它提供了顯示的內存位置。記住OS按需求提交了頁。分配內存的時候會在線程觸及到的內存節點中分配內存。

 

The second property - taking both working set and PFN's database lock only ones enables applications to perform faster and better scalable ramp up.

 第二個屬性——只要獲取工作集鎖和PFN數據庫鎖一個，可以讓應用程序運行的更快並且有更好的擴展性。

 

On NUMA SQL Server' Buffer Pool marks each allocated page with its node residency. It leverages QueryWorkingSetEx to accomplish it.  Once page is allocated it calls the API to find  out page residency.   It does it only once. Therefore enabling locked pages for SQL server on 64 bit platform  will improve SQL Server ramp up time and will improve performance & scalability over longer period of time. When running SQL Server with locked pages enabled you shouldn't be worried about overall system performance due to memory starvation - SQL Server participates in OS's paging mechanism by listening on OS's memory notification API's and shrinks its working set accordingly.

 在NUMA SQL Server buffer pool 中標記了每個分配也的節點位置。使用QueryWorkingSetEx來完成。一旦頁分配，就會調用api查找出頁所在節點，只運行一次。因此SQL Server 64位平台會提高SQL Server 加速並且會提高性能和可擴展性。當運行sql server鎖定頁你不需要擔心因為內存不足引起的性能問題——sql server 會監聽OS內存通知api參與分頁並且壓縮工作集。

 

Let us summarize now - on 64 bit platform, locked pages, awe mechanism, enable better application's scalability and performance both during ramp up time and over long period of time. However, keep in mind that an application is still required to implement a way of responding to memory pressure to avoid starving the whole system for memory.

總結一下：在64位的平台上，鎖定頁，AWE分配機制，可以讓應用程序有更好的擴展性和性能。但是記住一個應用程序也要增強相應內存壓力的能力來避免整個系統的內存不足。