Chromium內核原理之網絡棧

---

 

查看網絡：以前的：chrome://net-internals#sockets

現在用 chrome://net-export/ 捕獲。用chrome://net-export 去看。

效果，比如看sockets多少個：

 

 

參考

https://dev.chromium.org/for-testers/providing-network-details

https://chromium.googlesource.com/catapult/+/refs/heads/main/netlog_viewer

JeffMony關注

12019.02.13 13:11:19字數 2,846閱讀 1,413

《Chromium內核原理之blink內核工作解密》
《Chromium內核原理之多進程架構》
《Chromium內核原理之進程間通信(IPC)》
《Chromium內核原理之網絡棧》
《Chromium內核原理之網絡棧HTTP Cache》
《Chromium內核原理之Preconnect》
《Chromium內核原理之Prerender》
《Chromium內核原理之cronet獨立化》

1.內核網絡棧概述
2.代碼結構
3.網絡請求剖析（專注於HTTP）

3.1 URLRequest
3.2 URLRequestHttpJob
3.3 HttpNetworkTransaction
3.4 HttpStreamFactory

3.4.1 Proxy解析
3.4.2 連接管理
3.4.3 Host解析
3.4.4 SSL/TLS

1.內核網絡棧概述

網絡堆棧主要是單線程跨平台庫，主要用於資源獲取。它的主要接口是URLRequest和URLRequestContext。 URLRequest，如其名稱所示，表示對URL的請求。 URLRequestContext包含完成URL請求所需的所有關聯上下文，例如cookie，主機解析器，代理解析器，緩存等。許多URLRequest對象可以共享相同的URLRequestContext。盡管磁盤緩存可以使用專用線程，但是大多數網絡對象都不是線程安全的，並且幾個組件（主機解析，證書驗證等）可能使用未連接的工作線程。由於它主要在單個網絡線程上運行，因此不允許阻止網絡線程上的操作。因此，我們使用非阻塞操作和異步回調（通常是CompletionCallback）。網絡堆棧代碼還將大多數操作記錄到NetLog，這允許消費者在內存中記錄所述操作並以用戶友好的格式呈現它以進行調試。

Chromium開發人員編寫了網絡堆棧，以便：

• 允許編碼到跨平台抽象；
• 提供比更高級系統網絡庫（例如WinHTTP或WinINET）更高的控制能力。
  ** 避免系統庫中可能存在的錯誤；
  ** 為性能優化提供更大的機會。

2.代碼結構

• net/base - 獲取一些網絡實用程序，例如主機解析，cookie，網絡變化檢測，SSL。
• net/disk_cache - web resources緩存。
• net/ftp - FTP實現。代碼主要基於舊的HTTP實現。
• net/http - HTTP實現。
• net/ocsp - 不使用系統庫或系統未提供OCSP實施時的OCSP實施。目前僅包含基於NSS的實現。
• net/proxy - 代理（SOCKS和HTTP）配置，解析，腳本提取等。
• net/quic - QUIC實現
• net/socket - TCP套接字，“SSL套接字”和套接字池的跨平台實現。
• net/socket_stream - WebSockets的套接字流。
• net/spdy - HTTP2和SPDY實現。
• net/url_request - URLRequest, URLRequestContext和 URLRequestJob 實現。
• net/websockets - WebSockets實現。

3.網絡請求剖析（專注於HTTP）

 

http_network.jpg

3.1 URLRequest

class URLRequest { public: // Construct a URLRequest for |url|, notifying events to |delegate|. URLRequest(const GURL& url, Delegate* delegate); // Specify the shared state void set_context(URLRequestContext* context); // Start the request. Notifications will be sent to |delegate|. void Start(); // Read data from the request. bool Read(IOBuffer* buf, int max_bytes, int* bytes_read); }; class URLRequest::Delegate { public: // Called after the response has started coming in or an error occurred. virtual void OnResponseStarted(...) = 0; // Called when Read() calls complete. virtual void OnReadCompleted(...) = 0; }; 

當URLRequest啟動時，它首先要做的是決定要創建什么類型的URLRequestJob。主要作業類型是URLRequestHttpJob，用於實現http：//請求。還有其他各種工作，例如URLRequestFileJob（file：//），URLRequestFtpJob（ftp：//），URLRequestDataJob（data：//）等。網絡堆棧將確定滿足請求的相應作業，但它為客戶端提供了兩種自定義作業創建的方法：URLRequest :: Interceptor和URLRequest :: ProtocolFactory。這些是相當多余的，除了URLRequest :: Interceptor的接口更廣泛。隨着工作的進行，它將通知URLRequest，URLRequest將根據需要通知URLRequest :: Delegate。

3.2 URLRequestHttpJob

URLRequestHttpJob將首先識別要為HTTP請求設置的cookie，這需要在請求上下文中查詢CookieMonster。這可以是異步的，因為CookieMonster可能由sqlite數據庫支持。執行此操作后，它將詢問請求上下文的HttpTransactionFactory以創建HttpTransaction。通常，HttpCache將被指定為HttpTransactionFactory。 HttpCache將創建一個HttpCache :: Transaction來處理HTTP請求。 HttpCache :: Transaction將首先檢查HttpCache（它檢查磁盤緩存）以查看緩存條目是否已存在。如果是這樣，這意味着響應已經被緩存，或者此緩存條目已經存在網絡事務，因此只需從該條目中讀取即可。如果緩存條目不存在，那么我們創建它並要求HttpCache的HttpNetworkLayer創建一個HttpNetworkTransaction來為請求提供服務。給HttpNetworkTransaction一個HttpNetworkSession，它包含執行HTTP請求的上下文狀態。其中一些狀態來自URLRequestContext。

3.3 HttpNetworkTransaction

class HttpNetworkSession { ... private: // Shim so we can mock out ClientSockets. ClientSocketFactory* const socket_factory_; // Pointer to URLRequestContext's HostResolver. HostResolver* const host_resolver_; // Reference to URLRequestContext's ProxyService scoped_refptr<ProxyService> proxy_service_; // Contains all the socket pools. ClientSocketPoolManager socket_pool_manager_; // Contains the active SpdySessions. scoped_ptr<SpdySessionPool> spdy_session_pool_; // Handles HttpStream creation. HttpStreamFactory http_stream_factory_; }; 

HttpNetworkTransaction要求HttpStreamFactory創建一個HttpStream。 HttpStreamFactory返回一個HttpStreamRequest，該HttpStreamRequest應該處理確定如何建立連接的所有邏輯，並且一旦建立連接，就用一個HttpStream子類包裝它，該子類調解直接與網絡的通信。

class HttpStream { public: virtual int SendRequest(...) = 0; virtual int ReadResponseHeaders(...) = 0; virtual int ReadResponseBody(...) = 0; ... }; 

目前，只有兩個主要的HttpStream子類：HttpBasicStream和SpdyHttpStream，盡管我們計划為HTTP流水線創建子類。 HttpBasicStream假設它正在直接讀取/寫入套接字。 SpdyHttpStream讀取和寫入SpdyStream。網絡事務將調用流上的方法，並在完成時，將調用回調到HttpCache :: Transaction，它將根據需要通知URLRequestHttpJob和URLRequest。對於HTTP路徑，http請求和響應的生成和解析將由HttpStreamParser處理。對於SPDY路徑，請求和響應解析由SpdyStream和SpdySession處理。根據HTTP響應，HttpNetworkTransaction可能需要執行HTTP身份驗證。這可能涉及重新啟動網絡事務。

3.4 HttpStreamFactory

HttpStreamFactory首先執行代理解析以​​確定是否需要代理。端點設置為URL主機或代理服務器。然后，HttpStreamFactory檢查SpdySessionPool以查看我們是否為此端點提供了可用的SpdySession。如果沒有，則流工廠從適當的池請求“套接字”（TCP /代理/ SSL /等）。如果套接字是SSL套接字，則它檢查NPN是否指示協議（可能是SPDY），如果是，則使用指定的協議。對於SPDY，我們將檢查SpdySession是否已經存在並使用它，如果是這樣，否則我們將從這個SSL套接字創建一個新的SpdySession，並從SpdySession創建一個SpdyStream，我們將SpdyHttpStream包裝起來。對於HTTP，我們將簡單地接受套接字並將其包裝在HttpBasicStream中。

3.4.1 Proxy解析

HttpStreamFactory查詢ProxyService以返回GURL的ProxyInfo。代理服務首先需要檢查它是否具有最新的代理配置。如果沒有，它使用ProxyConfigService向系統查詢當前代理設置。如果代理設置設置為無代理或特定代理，則代理解析很簡單（我們不返回代理或特定代理）。否則，我們需要運行PAC腳本來確定適當的代理（或缺少代理）。如果我們還沒有PAC腳本，那么代理設置將指示我們應該使用WPAD自動檢測，或者將指定自定義PAC URL，我們將使用ProxyScriptFetcher獲取PAC腳本。一旦我們有PAC腳本，我們將通過ProxyResolver執行它。請注意，我們使用填充程序MultiThreadedProxyResolver對象將PAC腳本執行分派給運行ProxyResolverV8實例的線程。這是因為PAC腳本執行可能會阻止主機解析。因此，為了防止一個停滯的PAC腳本執行阻止其他代理解析，我們允許同時執行多個PAC腳本（警告：V8不是線程安全的，所以我們獲取了javascript綁定的鎖，所以當一個V8實例被阻止時主機解析，它釋放鎖定，以便另一個V8實例可以執行PAC腳本來解析不同URL的代理。

3.4.2 連接管理

在HttpStreamRequest確定了適當的端點（URL端點或代理端點）之后，它需要建立連接。它通過識別適當的“套接字”池並從中請求套接字來實現。請注意，“socket”在這里基本上意味着我們可以讀取和寫入的內容，以通過網絡發送數據。 SSL套接字構建在傳輸（TCP）套接字之上，並為用戶加密/解密原始TCP數據。不同的套接字類型還處理不同的連接設置，HTTP / SOCKS代理，SSL握手等。套接字池設計為分層，因此各種連接設置可以分層在其他套接字之上。 HttpStream可以與實際的底層套接字類型無關，因為它只需要讀取和寫入套接字。套接字池執行各種功能 - 它們實現每個代理，每個主機和每個進程限制的連接。目前這些設置為每個代理32個套接字，每個目標主機6個套接字，每個進程256個套接字（未正確實現，但足夠好）。套接字池還從履行中抽象出套接字請求，從而為我們提供套接字的“后期綁定”。套接字請求可以由新連接的套接字或空閑套接字實現（從先前的http事務重用）。

3.4.3 Host解析

請注意，傳輸套接字的連接設置不僅需要傳輸（TCP）握手，還可能需要主機解析。 HostResolverImpl使用getaddrinfo（）來執行主機解析，這是一個阻塞調用，因此解析器會在未連接的工作線程上調用這些調用。通常，主機解析通常涉及DNS解析，但可能涉及非DNS命名空間，例如NetBIOS / WINS。請注意，截至編寫本文時，我們將並發主機分辨率的數量限制為8，但希望優化此值。 HostResolverImpl還包含一個HostCache，它可以緩存多達1000個主機名。

3.4.4 SSL/TLS

SSL套接字需要執行SSL連接設置以及證書驗證。目前，在所有平台上，我們使用NSS的libssl來處理SSL連接邏輯。但是，我們使用特定於平台的API進行證書驗證。我們正在逐步使用證書驗證緩存，它將多個同一證書的證書驗證請求合並到一個證書驗證作業中，並將結果緩存一段時間。

SSLClientSocketNSS大致遵循這一系列事件（忽略Snap Start或基於DNSSEC的證書驗證等高級功能）：

• 調用Connect（）。我們基於SSLConfig指定的配置或預處理器宏來設置NSS的SSL選項。然后我們開始握手。
• 握手完成。假設我們沒有遇到任何錯誤，我們繼續使用CertVerifier驗證服務器的證書。證書驗證可能需要一些時間，因此CertVerifier使用WorkerPool實際調用X509Certificate :: Verify（），這是使用特定於平台的API實現的。

請注意，Chromium有自己的NSS補丁，它支持一些不一定在系統的NSS安裝中的高級功能，例如支持NPN，False Start，Snap Start，OCSP裝訂等。

參考：https://www.chromium.org/developers/design-documents/network-stack

---

最新chrome里面網絡變成了服務： network service。通過mojo調用。

Life of a URLRequest

This document gives an overview of the browser's lower-layers for networking.

Networking in the browser ranges from high level Javascript APIs like fetch(), all the way down to writing encrypted bytes on a socket.

This document assumes that requests for URLs are mediated through the browser's Network Service, and focuses on all the layers below the Network Service, including key points of integration.

It's particularly targeted at people new to the Chrome network stack, but should also be useful for team members who may be experts at some parts of the stack, but are largely unfamiliar with other components. It starts by walking through how a basic request issued by another process works its way through the network stack, and then moves on to discuss how various components plug in.

If you notice any inaccuracies in this document, or feel that things could be better explained, please do not hesitate to submit patches.

Anatomy of the Network Stack

The network stack is located in //net/ in the Chrome repo, and uses the namespace “net”. Whenever a class name in this doc has no namespace, it can generally be assumed it's in //net/ and is in the net namespace.

The top-level network stack object is the URLRequestContext. The context has non-owning pointers to everything needed to create and issue a URLRequest. The context must outlive all requests that use it. Creating a context is a rather complicated process usually managed by URLRequestContextBuilder.

The primary use of the URLRequestContext is to create URLRequest objects using URLRequestContext::CreateRequest(). The URLRequest is the main interface used by direct consumers of the network stack. It manages loading URLs with the http, https, ws, and wss schemes. URLs for other schemes, such as file, filesystem, blob, chrome, and data, are managed completely outside of //net. Each URLRequest tracks a single request across all redirects until an error occurs, it's canceled, or a final response is received, with a (possibly empty) body.

The HttpNetworkSession is another major network stack object. It owns the HttpStreamFactory, the socket pools, and the HTTP/2 and QUIC session pools. It also has non-owning pointers to the network stack objects that more directly deal with sockets.

This document does not mention either of these objects much, but at layers above the HttpStreamFactory, objects often grab their dependencies from the URLRequestContext, while the HttpStreamFactory and layers below it generally get their dependencies from the HttpNetworkSession.

How many “Delegates”?

A URLRequest informs the consumer of important events for a request using two main interfaces: the URLRequest::Delegate interface and the NetworkDelegate interface.

The URLRequest::Delegate interface consists of a small set of callbacks needed to let the embedder drive a request forward. The NetworkDelegate is an object pointed to by the URLRequestContext and shared by all requests, and includes callbacks corresponding to most of the URLRequest::Delegate's callbacks, as well as an assortment of other methods.

The Network Service and Mojo

The network service, which lives in //services/network/, wraps //net/ objects, and provides cross-process network APIs and their implementations for the rest of Chrome. The network service uses the namespace “network” for all its classes. The Mojo interfaces it provides are in the network::mojom namespace. Mojo is Chrome‘s IPC layer. Generally there’s a mojo::Remote proxy object in the consumer‘s process which also implements the network::mojom::Foo interface. When the proxy object’s methods are invoked, it passes the call and all its arguments over a Mojo IPC channel, using a mojo::Receiver, to an implementation of the network::mojom::Foo interface in the network service (the implementation is typically a class named network::Foo), which may be running in another process, another thread in the consumer‘s process, or even the same thread in the consumer’s process.

The network::NetworkService object is singleton that is used by Chrome to create all other network service objects. The primary objects it is used to create are the network::NetworkContexts, each of which owns its own mostly independent URLRequestContext. Chrome has a number of different NetworkContexts, as there is often a need to keep cookies, caches, and socket pools separate for different types of requests, depending on what's making the request. Here are the main NetworkContexts used by Chrome:

• The system NetworkContext, created and owned by Chrome‘s SystemNetworkContextManager, is used for requests that aren’t associated with particular user or Profile. It has no on-disk storage, so loses all state, like cookies, after each browser restart. It has no in-memory http cache, either. SystemNetworkContextManager also sets up global network service preferences.
• Each Chrome Profile, including incognito Profiles, has its own NetworkContext. Except for incognito and guest profiles, these contexts store information in their own on-disk store, which includes cookies and an HTTP cache, among other things. Each of these NetworkContexts is owned by a StoragePartition object in the browser process, and created by a Profile's ProfileNetworkContextService.
• On platforms that support apps, each Profile has a NetworkContext for each app installed on that Profile. As with the main NetworkContext, these may have on-disk data, depending on the Profile and the App.

Life of a Simple URLRequest

A request for data is dispatched from some process, which results in creating a network::URLLoader in the network service (which, on desktop platform, is typically in its own process). The URLLoader then creates a URLRequest to drive the network request. That job first checks the HTTP cache, and then creates a network transaction object, if necessary, to actually fetch the data. That transaction tries to reuse a connection if available. If none is available, it creates a new one. Once it has established a connection, the HTTP request is dispatched, the response read and parsed, and the result returned back up the stack and sent over to the caller.

Of course, it's not quite that simple :-}.

Consider a simple request issued by some process other than the network service‘s process. Suppose it’s an HTTP request, the response is uncompressed, no matching entry in the cache, and there are no idle sockets connected to the server in the socket pool.

Continuing with a “simple” URLRequest, here's a bit more detail on how things work.

Request starts in some (non-network) process

Summary:

• In the browser process, the network::mojom::NetworkContext interface is used to create a network::mojom::URLLoaderFactory.
• A consumer (e.g. the content::ResourceDispatcher for Blink, the content::NavigationURLLoaderImpl for frame navigations, or a network::SimpleURLLoader) passes a network::ResourceRequest object and network::mojom::URLLoaderClient Mojo channel to the network::mojom::URLLoaderFactory, and tells it to create and start a network::mojom::URLLoader.
• Mojo sends the network::ResourceRequest over an IPC pipe to a network::URLLoaderFactory in the network process.

Chrome has a single browser process which handles starting and configuring other processes, tab management, and navigation, among other things, and multiple child processes, which are generally sandboxed and have no network access themselves, apart from the network service (Which either runs in its own process, or potentially in the browser process to conserve RAM). There are multiple types of child processes (renderer, GPU, plugin, network, etc). The renderer processes are the ones that layout webpages and run HTML.

The browser process creates the top level network::mojom::NetworkContext objects. The NetworkContext interface is privileged and can only be accessed from the browser process. The browser process uses it to create network::mojom::URLLoaderFactories, which can then be passed to less privileged processes to allow them to load resources using the NetworkContext. To create a URLLoaderFactory, a network::mojom::URLLoaderFactoryParams object is passed to the NetworkContext to configure fields that other processes are not trusted to set, for security and privacy reasons.

One such field is the net::IsolationInfo field, which includes:

• A net::NetworkIsolationKey, which is used to enforce the privacy sandbox in the network stack, separating network resources used by different sites in order to protect against tracking a user across sites.
• A net::SiteForCookies, which is used to determine which site to send SameSite cookies for. SameSite cookies prevent cross-site attacks by only being accessible when that site is the top-level site.
• How to update these values across redirects.

A consumer, either in the browser process or a child process, that wants to make a network request gets a URLLoaderFactory from the browser process through some manner, assembles a bunch of parameters in the large network::ResourceRequest object, creates a network::mojom::URLLoaderClient Mojo channel for the network::mojom::URLLoader to use to talk back to it, and then passes them all to the URLLoaderFactory, which returns a URLLoader object that it can use to manage the network request.

network::URLLoaderFactory sets up the request in the network service

Summary:

• network::URLLoaderFactory creates a network::URLLoader.
• network::URLLoader uses the network::NetworkContext's URLRequestContext to create and start a URLRequest.

The network::URLLoaderFactory, along with all NetworkContexts and most of the network stack, lives on a single thread in the network service. It gets a reconstituted ResourceRequest object from the network::mojom::URLLoaderFactory Mojo pipe, does some checks to make sure it can service the request, and if so, creates a URLLoader, passing the request and the NetworkContext associated with the URLLoaderFactory.

The URLLoader then calls into the NetworkContext's net::URLRequestContext to create the URLRequest. The URLRequestContext has pointers to all the network stack objects needed to issue the request over the network, such as the cache, cookie store, and host resolver. The URLLoader then calls into the network::ResourceScheduler, which may delay starting the request, based on priority and other activity. Eventually, the ResourceScheduler starts the request.

Check the cache, request an HttpStream

Summary:

• The URLRequest asks the URLRequestJobFactory to create a URLRequestJob, and gets a URLRequestHttpJob.
• The URLRequestHttpJob asks the HttpCache to create an HttpTransaction, and gets an HttpCache::Transaction, assuming caching is enabled.
• The HttpCache::Transaction sees there's no cache entry for the request, and creates an HttpNetworkTransaction.
• The HttpNetworkTransaction calls into the HttpStreamFactory to request an HttpStream.

The URLRequest then calls into the URLRequestJobFactory to create a URLRequestHttpJob, a subclass of URLRequestJob, and then starts it (historically, non-network URL schemes were also disptched through the network stack, so there were a variety of job types.) The URLRequestHttpJob attaches cookies to the request, if needed. Whether or not SameSite cookies are attached depends on the IsolationInfo‘s SiteForCookies, the URL, and the URLRequest’s request_initiator field.

The URLRequestHttpJob calls into the HttpCache to create an HttpCache::Transaction. The cache checks for an entry with the same URL and NetworkIsolationKey. If there's no matching entry, the HttpCache::Transaction will call into the HttpNetworkLayer to create an HttpNetworkTransaction, and transparently wrap it. The HttpNetworkTransaction then calls into the HttpStreamFactory to request an HttpStream to the server.

Create an HttpStream

Summary:

• HttpStreamFactory creates an HttpStreamFactory::Job.
• HttpStreamFactory::Job calls into the TransportClientSocketPool to populate an ClientSocketHandle.
• TransportClientSocketPool has no idle sockets, so it creates a TransportConnectJob and starts it.
• TransportConnectJob creates a StreamSocket and establishes a connection.
• TransportClientSocketPool puts the StreamSocket in the ClientSocketHandle, and calls into HttpStreamFactory::Job.
• HttpStreamFactory::Job creates an HttpBasicStream, which takes ownership of the ClientSocketHandle.
• It returns the HttpBasicStream to the HttpNetworkTransaction.

The HttpStreamFactory::Job creates a ClientSocketHandle to hold a socket, once connected, and passes it into the ClientSocketPoolManager. The ClientSocketPoolManager assembles the TransportSocketParams needed to establish the connection and creates a group name (“host:port”) used to identify sockets that can be used interchangeably.

The ClientSocketPoolManager directs the request to the TransportClientSocketPool, since there‘s no proxy and it’s an HTTP request. The request is forwarded to the pool's ClientSocketPoolBase‘s ClientSocketPoolBaseHelper. If there isn’t already an idle connection, and there are available socket slots, the ClientSocketPoolBaseHelper will create a new TransportConnectJob using the aforementioned params object. This Job will do the actual DNS lookup by calling into the HostResolverImpl, if needed, and then finally establishes a connection.

Once the socket is connected, ownership of the socket is passed to the ClientSocketHandle. The HttpStreamFactory::Job is then informed the connection attempt succeeded, and it then creates an HttpBasicStream, which takes ownership of the ClientSocketHandle. It then passes ownership of the HttpBasicStream back to the HttpNetworkTransaction.

Send request and read the response headers

Summary:

• HttpNetworkTransaction gives the request headers to the HttpBasicStream, and tells it to start the request.
• HttpBasicStream sends the request, and waits for the response.
• The HttpBasicStream sends the response headers back to the HttpNetworkTransaction.
• The response headers are sent up through the URLRequest, to the network::URLLoader.
• They're then sent to the network::mojom::URLLoaderClient via Mojo.

The HttpNetworkTransaction passes the request headers to the HttpBasicStream, which uses an HttpStreamParser to (finally) format the request headers and body (if present) and send them to the server.

The HttpStreamParser waits to receive the response and then parses the HTTP/1.x response headers, and then passes them up through both the HttpNetworkTransaction and HttpCache::Transaction to the URLRequestHttpJob. The URLRequestHttpJob saves any cookies, if needed, and then passes the headers up to the URLRequest and on to the network::URLLoader, which sends the data over a Mojo pipe to the network::mojom::URLLoaderClient, passed in to the URLLoader when it was created.

Response body is read

Summary:

• network::URLLoader creates a raw Mojo data pipe, and passes one end to the network::mojom::URLLoaderClient.
• The URLLoader requests shared memory buffer from the Mojo data pipe.
• The URLLoader tells the URLRequest to write to the memory buffer, and tells the pipe when data has been written to the buffer.
• The last two steps repeat until the request is complete.

Without waiting to hear back from the network::mojom::URLLoaderClient, the network::URLLoader allocates a raw mojo data pipe, and passes the client the read end of the pipe. The URLLoader then grabs an IPC buffer from the pipe, and passes a 64KB body read request down through the URLRequest all the way down to the HttpStreamParser. Once some data is read, possibly less than 64KB, the number of bytes read makes its way back to the URLLoader, which then tells the Mojo pipe the read was complete, and then requests another buffer from the pipe, to continue writing data to. The pipe may apply back pressure, to limit the amount of unconsumed data that can be in shared memory buffers at once. This process repeats until the response body is completely read.

URLRequest is destroyed

Summary:

• When complete, the network::URLLoaderFactory deletes the network::URLLoader, which deletes the URLRequest.
• During destruction, the HttpNetworkTransaction determines if the socket is reusable, and if so, tells the HttpBasicStream to return it to the socket pool.

When the URLRequest informs the network::URLLoader the request is complete, the URLLoader passes the message along to the network::mojom::URLLoaderClient, over its Mojo pipe, before telling the URLLoaderFactory to destroy the URLLoader, which results in destroying the URLRequest and closing all Mojo pipes related to the request.

When the HttpNetworkTransaction is being torn down, it figures out if the socket is reusable. If not, it tells the HttpBasicStream to close the socket. Either way, the ClientSocketHandle returns the socket is then returned to the socket pool, either for reuse or so the socket pool knows it has another free socket slot.

Object Relationships and Ownership

A sample of the object relationships involved in the above process is diagramed here:

There are a couple of points in the above diagram that do not come clear visually:

• The method that generates the filter chain that is hung off the URLRequestJob is declared on URLRequestJob, but the only current implementation of it is on URLRequestHttpJob, so the generation is shown as happening from that class.
• HttpTransactions of different types are layered; i.e. a HttpCache::Transaction contains a pointer to an HttpTransaction, but that pointed-to HttpTransaction generally is an HttpNetworkTransaction.

Additional Topics

HTTP Cache

The HttpCache::Transaction sits between the URLRequestHttpJob and the HttpNetworkTransaction, and implements the HttpTransaction interface, just like the HttpNetworkTransaction. The HttpCache::Transaction checks if a request can be served out of the cache. If a request needs to be revalidated, it handles sending a conditional revalidation request over the network. It may also break a range request into multiple cached and non-cached contiguous chunks, and may issue multiple network requests for a single range URLRequest.

The HttpCache::Transaction uses one of three disk_cache::Backends to actually store the cache's index and files: The in memory backend, the blockfile cache backend, and the simple cache backend. The first is used in incognito. The latter two are both stored on disk, and are used on different platforms.

One important detail is that it has a read/write lock for each URL. The lock technically allows multiple reads at once, but since an HttpCache::Transaction always grabs the lock for writing and reading before downgrading it to a read only lock, all requests for the same URL are effectively done serially. The renderer process merges requests for the same URL in many cases, which mitigates this problem to some extent.

It's also worth noting that each renderer process also has its own in-memory cache, which has no relation to the cache implemented in net/, which lives in the network service.

Cancellation

A consumer can cancel a request at any time by deleting the network::mojom::URLLoader pipe used by the request. This will cause the network::URLLoader to destroy itself and its URLRequest.

When an HttpNetworkTransaction for a cancelled request is being torn down, it figures out if the socket the HttpStream owns can potentially be reused, based on the protocol (HTTP / HTTP/2 / QUIC) and any received headers. If the socket potentially can be reused, an HttpResponseBodyDrainer is created to try and read any remaining body bytes of the HttpStream, if any, before returning the socket to the SocketPool. If this takes too long, or there's an error, the socket is closed instead. Since this all happens at the layer below the cache, any drained bytes are not written to the cache, and as far as the cache layer is concerned, it only has a partial response.

Redirects

The URLRequestHttpJob checks if headers indicate a redirect when it receives them from the next layer down (typically the HttpCache::Transaction). If they indicate a redirect, it tells the cache the response is complete, ignoring the body, so the cache only has the headers. The cache then treats it as a complete entry, even if the headers indicated there will be a body.

The URLRequestHttpJob then checks with the URLRequest if the redirect should be followed. The URLRequest then informs the network::URLLoader about the redirect, which passes information about the redirect to network::mojom::URLLoaderClient, in the consumer process. Whatever issued the original request then checks if the redirect should be followed.

If the redirect should be followed, the URLLoaderClient calls back into the URLLoader over the network::mojom::URLLoader Mojo interface, which tells the URLRequest to follow the redirect. The URLRequest then creates a new URLRequestJob to send the new request. If the URLLoaderClient chooses to cancel the request instead, it can delete the network::mojom::URLLoader pipe, just like the cancellation case discussed above. In either case, the old HttpTransaction is destroyed, and the HttpNetworkTransaction attempts to drain the socket for reuse, as discussed in the previous section.

In some cases, the consumer may choose to handle a redirect itself, like passing off the redirect to a ServiceWorker. In this case, the consumer cancels the request and then calls into some other network::mojom::URLLoaderFactory with the new URL to continue the request.

Filters (gzip, deflate, brotli, etc)

When the URLRequestHttpJob receives headers, it sends a list of all Content-Encoding values to Filter::Factory, which creates a (possibly empty) chain of filters. As body bytes are received, they're passed through the filters at the URLRequestJob layer and the decoded bytes are passed back to the URLRequest::Delegate.

Since this is done above the cache layer, the cache stores the responses prior to decompression. As a result, if files aren‘t compressed over the wire, they aren’t compressed in the cache, either.

Socket Pools

The ClientSocketPoolManager is responsible for assembling the parameters needed to connect a socket, and then sending the request to the right socket pool. Each socket request sent to a socket pool comes with a socket params object, a ClientSocketHandle, and a “group name”. The params object contains all the information a ConnectJob needs to create a connection of a given type, and different types of socket pools take different params types. The ClientSocketHandle will take temporary ownership of a connected socket and return it to the socket pool when done. All connections with the same group name in the same pool can be used to service the same connection requests, so it consists of host, port, protocol, and whether “privacy mode” is enabled for sockets in the goup.

All socket pool classes derive from the ClientSocketPoolBase. The ClientSocketPoolBase handles managing sockets - which requests to create sockets for, which requests get connected sockets first, which sockets belong to which groups, connection limits per group, keeping track of and closing idle sockets, etc. Each ClientSocketPoolBase subclass has its own ConnectJob type, which establishes a connection using the socket params, before the pool hands out the connected socket.

Socket Pool Layering

Some socket pools are layered on top other socket pools. This is done when a “socket” in a higher layer needs to establish a connection in a lower level pool and then take ownership of it as part of its connection process. For example, each socket in the SSLClientSocketPool is layered on top of a socket in the TransportClientSocketPool. There are a couple additional complexities here.

From the perspective of the lower layer pool, all of its sockets that a higher layer pools owns are actively in use, even when the higher layer pool considers them idle. As a result, when a lower layer pool is at its connection limit and needs to make a new connection, it will ask any higher layer pools to close an idle connection if they have one, so it can make a new connection.

Since sockets in the higher layer pool are also in a group in the lower layer pool, they must have their own distinct group name. This is needed so that, for instance, SSL and HTTP connections won't be grouped together in the TcpClientSocketPool, which the SSLClientSocketPool sits on top of.

Socket Pool Class Relationships

The relationships between the important classes in the socket pools is shown diagrammatically for the lowest layer socket pool (TransportSocketPool) below.

The ClientSocketPoolBase is a template class templatized on the class containing the parameters for the appropriate type of socket (in this case TransportSocketParams). It contains a pointer to the ClientSocketPoolBaseHelper, which contains all the type-independent machinery of the socket pool.

When socket pools are initialized, they in turn initialize their templatized ClientSocketPoolBase member with an object with which it should create connect jobs. That object must derive from ClientSocketPoolBase::ConnectJobFactory templatized by the same type as the ClientSocketPoolBase. (In the case of the diagram above, that object is a TransportConnectJobFactory, which derives from ClientSocketPoolBase::ConnectJobFactory.) Internally, that object is wrapped in a type-unsafe wrapper (ClientSocketPoolBase::ConnectJobFactoryAdaptor) so that it can be passed to the initialization of the ClientSocketPoolBaseHelper. This allows the helper to create connect jobs while preserving a type-safe API to the initialization of the socket pool.

SSL

When an SSL connection is needed, the ClientSocketPoolManager assembles the parameters needed both to connect the TCP socket and establish an SSL connection. It then passes them to the SSLClientSocketPool, which creates an SSLConnectJob using them. The SSLConnectJob's first step is to call into the TransportSocketPool to establish a TCP connection.

Once a connection is established by the lower layered pool, the SSLConnectJob then starts SSL negotiation. Once that's done, the SSL socket is passed back to the HttpStreamFactory::Job that initiated the request, and things proceed just as with HTTP. When complete, the socket is returned to the SSLClientSocketPool.

Proxies

Each proxy has its own completely independent set of socket pools. They have their own exclusive TransportSocketPool, their own protocol-specific pool above it, and their own SSLSocketPool above that. HTTPS proxies also have a second SSLSocketPool between the the HttpProxyClientSocketPool and the TransportSocketPool, since they can talk SSL to both the proxy and the destination server, layered on top of each other.

The first step the HttpStreamFactory::Job performs, just before calling into the ClientSocketPoolManager to create a socket, is to pass the URL to the Proxy service to get an ordered list of proxies (if any) that should be tried for that URL. Then when the ClientSocketPoolManager tries to get a socket for the Job, it uses that list of proxies to direct the request to the right socket pool.

Alternate Protocols

HTTP/2 (Formerly SPDY)

HTTP/2 negotation is performed as part of the SSL handshake, so when HttpStreamFactory::Job gets a socket, it may have HTTP/2 negotiated over it as well. When it gets a socket with HTTP/2 negotiated as well, the Job creates a SpdySession using the socket and a SpdyHttpStream on top of the SpdySession. The SpdyHttpStream will be passed to the HttpNetworkTransaction, which drives the stream as usual.

The SpdySession will be shared with other Jobs connecting to the same server, and future Jobs will find the SpdySession before they try to create a connection. HttpServerProperties also tracks which servers supported HTTP/2 when we last talked to them. We only try to establish a single connection to servers we think speak HTTP/2 when multiple HttpStreamFactory::Jobs are trying to connect to them, to avoid wasting resources.

QUIC

QUIC works quite a bit differently from HTTP/2. Servers advertise QUIC support with an “Alternate-Protocol” HTTP header in their responses. HttpServerProperties then tracks servers that have advertised QUIC support.

When a new request comes in to HttpStreamFactory for a connection to a server that has advertised QUIC support in the past, it will create a second HttpStreamFactory::Job for QUIC, which returns an QuicHttpStream on success. The two Jobs (one for QUIC, one for all versions of HTTP) will be raced against each other, and whichever successfully creates an HttpStream first will be used.

As with HTTP/2, once a QUIC connection is established, it will be shared with other Jobs connecting to the same server, and future Jobs will just reuse the existing QUIC session.

Prioritization

URLRequests are assigned a priority on creation. It only comes into play in a couple places:

• The ResourceScheduler lives outside net/, and in some cases, delays starting low priority requests on a per-tab basis.
• DNS lookups are initiated based on the highest priority request for a lookup.
• Socket pools hand out and create sockets based on prioritization. However, when a socket becomes idle, it will be assigned to the highest priority request for the server it‘s connected to, even if there’s a higher priority request to another server that's waiting on a free socket slot.
• HTTP/2 and QUIC both support sending priorities over-the-wire.

At the socket pool layer, sockets are only assigned to socket requests once the socket is connected and SSL is negotiated, if needed. This is done so that if a higher priority request for a group reaches the socket pool before a connection is established, the first usable connection goes to the highest priority socket request.

Non-HTTP Schemes

WebSockets requests (wss:// and ws://) start as HTTP requests with an HTTP upgrade header. Once the handshake completes successfully, the connection is used as a full-duplex communication channel to the server for WebSocket frames, rather than to receive an HTTP response body. WebSockets have their own Mojo interfaces and //net classes, but internally they reuse the full URLRequest machinery up to the point headers are received from the server. Then the connection is handed off to the WebSocket code to manage.

Other schemes typically have their own network::mojom::URLLoaderFactory that is not part of the network service. Standard schemes like file:// and blob:// are handled by the content layer and its dependencies (content::FileURLLoaderFactory and storage::BlobURLLoaderFactory, respectively, for those two schemes). Chrome-specific schemes, like externalfile:// and chrome-extension:// are often handled by a URLLoaderFactory in the chrome layer, though chrome:// itself is actually handled in //content.

data:// URLs are handled a bit differently from other schemes. If a renderer process requests a data:// subresource, the renderer typically decodes it internally, as sending it to an out-of-process URLLoader would be inefficient. Navigations are a bit different. To navigate to a URL, the browser process creates a URLLoader and passes it over to a renderer process. So in the case of a navigation to a data:// URL, a URLLoader is created using a content::DataURLLoaderFactory that lives in the browser process, and then a mojo::Remote for the browser-hosted URLLoader is passed to a renderer proceess.

about:blank is similarly often handled in the renderer, though there is a factory for that used in the case of navigations as well. Other about: URLs are mapped to the corresponding Chrome URLs by the navigation code, rather than having that logic live in a URLLoaderFactory.