JBoss配置解決高並發連接異常問題(轉)

 

這兩天一個項目在做壓力測試的時候，發現只要並發數超過250個，連續測試兩輪就會有連接異常出現，測試輪數越多出現越頻繁，異常日志如下：

 

Caused by: com.caucho.hessian.client.HessianConnectionException: 500: java.io.IOException: Error writing to server
    at com.caucho.hessian.client.HessianURLConnection.sendRequest(HessianURLConnection.java:142)
    at com.caucho.hessian.client.HessianProxy.sendRequest(HessianProxy.java:283)
    at com.caucho.hessian.client.HessianProxy.invoke(HessianProxy.java:170)
    at $Proxy168.sendOpenAcctInfo(Unknown Source)
    at sun.reflect.GeneratedMethodAccessor750.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.springframework.remoting.caucho.HessianClientInterceptor.invoke(HessianClientInterceptor.java:219)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
    at $Proxy169.sendOpenAcctInfo(Unknown Source)
    at com.shine.web.bean.OpenAcctBeanImpl.sendOpenAcctInfo(OpenAcctBeanImpl.java:62)
    ... 32 more
Caused by: java.io.IOException: Error writing to server
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1345)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1339)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:993)
    at com.caucho.hessian.client.HessianURLConnection.sendRequest(HessianURLConnection.java:122)
    ... 43 more
Caused by: java.io.IOException: Error writing to server
    at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:453)
    at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:465)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1047)
    at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:373)
    at com.caucho.hessian.client.HessianURLConnection.sendRequest(HessianURLConnection.java:109)
    ... 43 more

一開始使用Error writing to server去網絡上查找原因，發現基本都無關。

 

后來搜索hessian和spring兼容問題，發現spring2.5.6和hessian4.0.7不兼容。將hessian版本號降低到3.1.3，情況有好轉，但測試10輪之后，異常又出現了。

通過內存監控，先排除虛擬機內存問題。虛擬機內存配置為-Xms1024m -Xmx1024m，監控下來發現實際占用內存不到一半。

然后通過netstat -na監控操作系統端口占用情況，發現端口占用高峰不到500個，這個原因也排除了。（測試服務器已經修改注冊表將TIME_WAIT時間降低到30秒，所以基本不會出現端口占用問題）。

通過CPU監控，確認並發高峰時，CPU占用也不到50%。

種種跡象表明，這些都不是造成連接斷開的原因，那到底瓶頸出現在哪里呢？

於是我們將目光轉向JBoss的配置。

首先確認數據庫的連接池配置，最大連接數設置為50，從前幾輪都可以正常運行來看，數據庫連接應該夠用；

然后確認JBoss的線程池配置，發現默認配置如下：

 

<mbean code="org.jboss.util.threadpool.BasicThreadPool"
   name="jboss.system:service=ThreadPool">
   <attribute name="Name">JBoss System Threads</attribute>
   <attribute name="ThreadGroupName">System Threads</attribute>
   <!-- How long a thread will live without any tasks in MS -->
   <attribute name="KeepAliveTime">60000</attribute>
   <!-- The max number of threads in the pool -->
   <attribute name="MaximumPoolSize">10</attribute>
   <!-- The max number of tasks before the queue is full -->
   <attribute name="MaximumQueueSize">1000</attribute>
   <!-- The behavior of the pool when a task is added and the queue is full.
   abort - a RuntimeException is thrown
   run - the calling thread executes the task
   wait - the calling thread blocks until the queue has room
   discard - the task is silently discarded without being run
   discardOldest - check to see if a task is about to complete and enque
      the new task if possible, else run the task in the calling thread
   -->
   <attribute name="BlockingMode">run</attribute>
</mbean>

搜索了一下相關配置的說明，在進行高並發的時候，建議修改MaximumPoolSize的大小為並發數的125%。

 

由於我們測試的不是持續的並發，因此將線程池大小修改成200先測試了一下，發現並發數在300的時候可以正常運行，又將並發數修改到500，持續測試了6個小時，均沒有發現異常。

現在比較好奇的是，為什么250個並發的時候就能一直不出錯，超過250個並發，就會頻繁出錯，這個值和MaximumPoolSize的參數到底有什么聯系呢？

http://blog.csdn.net/nicholas_lin/article/details/20639481

 http://wenku.baidu.com/link?url=eUQiTt73bQN_XBHVNpAhDnSMYfLdfqQXK1AF5Pp2dhTgBrO4nHaws7rEm8WZY5WVIiOEUaX5UQuuQTNCM9DrsNMjetboto1NnikLSEtzH6S