ConnectionPoolTimeoutException: Timeout waiting for connection from pool

背景

今天在通過監控系統發現一個錯誤，錯誤如下

org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:316)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:282)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)

動作

1. 起初懷疑是連接池設置過小，故查看代碼對連接池大小的設置

2. 發現連接池設置已經足夠大，我們的業務完全用不到這么多的連接數，故懷疑連接用完沒有被釋放。查看服務器上連接數

netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'

查看代碼， 發現確實沒有釋放連接，如下

HttpResponse response = httpclient.execute(httpPost);

List<String> lines = IOUtils.readLines(response.getEntity().getContent(), "utf-8");
StringBuffer res = new StringBuffer();
for (String line : lines) {
   res.append(line);
}
       
ChatMessage chatMessage = getChatMessage(sendChatMessageDTO, replyMessageInfo, sendMessageesResultBO);
return chatMessage;

解決

即然發現了問題，我們把連接用完歸還到連接池即可。

HttpPost httpPost = new HttpPost(connectUrl);
InputStream inputStream = null;
try {
            
    HttpResponse response = httpclient.execute(httpPost);
    inputStream = response.getEntity().getContent();
    List<String> lines = IOUtils.readLines(inputStream, "utf-8");
} catch (Exception e){
    logger.error("connect error", e);
    if(httpPost != null){
    	httpPost.abort();
     }
} finally {
    if(inputStream != null){
        try {
            inputStream.close();
         } catch (Exception e){
            logger.error("", e);
         }
    }
}

思考(基於版本4.5)

• httpclient 連接池原理是什么樣的呢？

連接池中主要有三個屬性，用於不同狀態的連接

//存放正在使用的連接
private final Set<E> leased;
//存放可用的連接
private final LinkedList<E> available;
//存放 pending 的連接 
private final LinkedList<PoolEntryFuture<E>> pending;

• 默認連接池的大小是多少？

我們查看官方代碼(版本4.5)， 發現其默認總大小為20，每個 route 的默認最大為 2

 public PoolingHttpClientConnectionManager(
        final HttpClientConnectionOperator httpClientConnectionOperator,
        final HttpConnectionFactory<HttpRoute, ManagedHttpClientConnection> connFactory,
        final long timeToLive, final TimeUnit timeUnit) {
        super();
        this.configData = new ConfigData();
        this.pool = new CPool(new InternalConnectionFactory(
                this.configData, connFactory), 2, 20, timeToLive, timeUnit);
        this.pool.setValidateAfterInactivity(2000);
        this.connectionOperator = Args.notNull(httpClientConnectionOperator, "HttpClientConnectionOperator");
        this.isShutDown = new AtomicBoolean(false);
    }

• close 的原理是怎樣呢？

有兩種 close

1. CloseableHttpResponse.close(): 這種方式是將用完的連接放回連接池的可用集合，並不是將連接真正關閉。內部調用的是 release()
2. httpClient.close(): 這種是關閉連接池，將池中的所有資源釋放，即真正關閉連接