記錄一次No buffer space available問題

前言：

服務器的接口服務一直是好的，突然有一天，恩，接口服務不通了，看log，發現了這個錯誤：

Caused by: java.net.SocketException: No buffer space available (maximum connections reached?): connect
    at sun.nio.ch.Net.connect0(Native Method)
    at sun.nio.ch.Net.connect(Unknown Source)
    at sun.nio.ch.Net.connect(Unknown Source)
    at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
    at java.nio.channels.SocketChannel.open(Unknown Source)
    at sun.nio.ch.PipeImpl$Initializer$LoopbackConnector.run(Unknown Source)
    ... 36 common frames omitted
2019-07-02 13:10:18 [main] INFO  o.a.c.h.Http11NioProtocol - Pausing ProtocolHandler ["http-nio-8556"]
2019-07-02 13:10:18 [main] ERROR o.a.c.c.Connector - Protocol handler pause failed
java.lang.NullPointerException: null
    at org.apache.tomcat.util.net.AbstractEndpoint.unlockAccept(AbstractEndpoint.java:899)
    at org.apache.tomcat.util.net.AbstractEndpoint.pause(AbstractEndpoint.java:1185)
    at org.apache.coyote.AbstractProtocol.pause(AbstractProtocol.java:612)
    at org.apache.catalina.connector.Connector.pause(Connector.java:944)
    at org.apache.catalina.core.StandardService.stopInternal(StandardService.java:467)
    at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:226)
    at org.apache.catalina.core.StandardServer.stopInternal(StandardServer.java:814)
    at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:226)
    at org.apache.catalina.startup.Tomcat.stop(Tomcat.java:377)
    at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.stopTomcat(TomcatWebServer.java:247)
    at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.stopSilently(TomcatWebServer.java:235)
    at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.start(TomcatWebServer.java:210)
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.startWebServer(ServletWebServerApplicationContext.java:300)
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.finishRefresh(ServletWebServerApplicationContext.java:162)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:553)
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:140)
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:762)
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:398)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:330)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1258)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1246)
    at com.winning.platwebservice.DqmsServiceApplication.main(DqmsServiceApplication.java:10)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:50)
    at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51)

 

解決思路：

1.看到No buffer space available,字面意思是說緩沖區內存不足，於是開始查機器內存：因為我們是window服務器，發現硬盤還有50G，查看運行內存還有10G，虛擬內存還有5G，應該不是這些問題，排除；

2.通過查看time_wait進程發現，pid為8561的有好多time_wait進行，再查詢發現是zabbix_client，聯系了部署zabbix_client的同事，能不能先停一下，發現停下后發現tomcat還是無法啟動，排除；

3.更換端口，無效；

4.調整部署的xms參數，調小調大，無效；

5.百度，百度告訴我們這個問題要修改注冊表信息，要重啟，因為服務器部署了很多別人的應用，沒有考慮這種辦法；

6.重啟tomcat時有報錯：

Caused by: java.net.BindException: Address already in use: connect
	at sun.nio.ch.Net.connect0(Native Method)
	at sun.nio.ch.Net.connect(Unknown Source)
	at sun.nio.ch.Net.connect(Unknown Source)
	at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
	at java.nio.channels.SocketChannel.open(Unknown Source)
	at sun.nio.ch.PipeImpl$Initializer$LoopbackConnector.run(Unknown Source)
	... 37 common frames omitted
2019-07-02 13:10:46 [main] INFO  o.a.c.h.Http11NioProtocol - Pausing ProtocolHandler ["http-nio-8556"]
2019-07-02 13:10:46 [main] INFO  o.a.c.c.StandardService - Stopping service [Tomcat]
2019-07-02 13:10:47 [main] INFO  o.a.c.u.LifecycleBase - The stop() method was called on component [StandardServer[-1]] after stop() had already been called. The second call will be ignored.
2019-07-02 13:10:47 [main] INFO  o.a.c.h.Http11NioProtocol - Stopping ProtocolHandler ["http-nio-8556"]
2019-07-02 13:10:47 [main] INFO  o.a.c.h.Http11NioProtocol - Destroying ProtocolHandler ["http-nio-8556"]
2019-07-02 13:10:47 [main] INFO  o.s.b.a.l.ConditionEvaluationReportLoggingListener - 

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2019-07-02 13:10:47 [main] ERROR o.s.b.d.LoggingFailureAnalysisReporter - 

***************************
APPLICATION FAILED TO START
***************************

Description:

The Tomcat connector configured to listen on port 8556 failed to start. The port may already be in use or the connector may be misconfigured.

　　猜測可能是因為端口被使用情況，可是查詢端口發現這個端口沒有被使用，而且更換端口也是無效，有一個印象比較深刻的是，我們查詢我們自己端口號和其他端口號時，總是有一個5開頭的端口被使用（比如說我們系統是8556端口，發現有一個58556端口被使用，查詢8561端口，也發現有一個58561端口被使用），后來發現有一個pid是12122的幾乎占據了所有5開頭的5位數端口號，這是一個java.exe，我們把它殺掉之后，重啟tomcat，發現ok了。

 

總結：

因為這個服務器部署了好多的項目，是測試服務器，同一時間有很多的連接和http請求，達到了window系統的上限，所以需要修改注冊表信息重啟，或是停掉消耗資源最多的那個應用。