Made with Remarkable!

解決tomcat 執行shutdown.sh 未能正常停止服務,釋放資源 出現如 * create a memory leak

近日,同事反饋說在搭建jenkins 部署,在打包完成,執行自動部署時,執行server tomcat 的shutdown.sh 后,tomcat 進程未關閉,資源未得到釋放,日志輸出如下:

警告: The web application [XXXX] appears to have started a thread named [commons-pool-EvictionTimer] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:552)
java.util.TimerThread.run(Timer.java:505)
十月 13, 2017 10:10:19 上午 org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
警告: The web application [XXXX appears to have started a thread named [Thread-6] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer $ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492) java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680) com.erp.cloudfi.report.util.LongTimeWorker.startWork(LongTimeWorker.java:38) com.erp.cloudfi.report.util.LongTimeWorker$ 1.run(LongTimeWorker.java:27)
java.lang.Thread.run(Thread.java:748)

經過日志反饋,查詢到 有兩種情況會造成線程存留.我們需要在spring 應用 shutdown時,destroy 相關資源並退出相關線程

1.LongTimeWorker(業務線程)引起的

代碼如下:

public class LongTimeWorker {
    /**
     * 任務隊列
     */
    BlockingQueue<Worker> works;
    public LongTimeWorker() {
        works = new LinkedBlockingDeque<Worker>();
        Thread thread = new Thread(new Runnable() {
            @Override
            public void run() {
                startWork();
            }
        });
        thread.setDaemon(true);
        thread.start();
    }
    public void startWork() {
        try {
            while(true) {
                Worker work = works.take();
                Worksheet sheet = work.getWorksheet();              
                work.run();
                if(sheet == null){
                    Thread.sleep(100);
                }else{
                    String id = sheet.getId();
                    existsKey.remove(id);
                }
            }
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }

    /**
     * 
     * @param work
     */
    public void setWork(Worker work) {
        try {
            works.put(work);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }
}

可以看到,在創建對象的時候,以demon 方式啟動了一個 死循環的 程序,不包含退出條件.

修改方式如下,添加 退出標志,放棄創建對象的時候啟動線程,增加啟動線程方法,與退出線程方法.
修改后代碼如下:

public class LongTimeWorker {
    /**
     * 任務隊列
     */
    BlockingQueue<Worker> works;
    private boolean workstatus=false;
    private Thread workthread = null;
    public LongTimeWorker() {
        works = new LinkedBlockingDeque<Worker>();
        workthread = new Thread(new Runnable() {
            @Override
            public void run() {
                startWork();
            }
        });
        workstatus=false;
        workthread.setDaemon(true);
    }
    public void startWork() {
        workstatus = true;
        workthread.start();
    }
    public void stopWork() {
        workstatus=false;
    }
    public void work() {
        try {
            while(workstatus) {
                Worker work = works.take();
                Worksheet sheet = work.getWorksheet();              
                work.run();
                if(sheet == null){
                    Thread.sleep(100);
                }
            }
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }

    /**
     * @param work
     */
    public void setWork(Worker work) {
        try {
            works.put(work);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }
}

2.commons-pool-EvictionTimer 未正常釋放資源引起的

實現類為 org.apache.commons.pool.impl.EvictionTimer 所屬於 commons-pool 包下,class 描述如下

Provides a shared idle object eviction timer for all pools. This class wraps the standard java.util.Timer and keeps track of how many pools are using it. If no pools are using the timer, it is canceled. This prevents a thread being left running which, in application server environments, can lead to memory leads and/or prevent applications from shutting down or reloading cleanly.

This class has package scope to prevent its inclusion in the pool public API. The class declaration below should *not* be changed to public.

提供了一個基於timer 機制的共享空閑 對象的通用池.

項目中涉及到 對象共享池的相關有 數據庫連接池,JedisPool,因為 未使用spring 提供的 redis 模塊 ,而是基於jedis 進行的封裝.所以針對JedisPool 連接池 .(數據庫連接池比較常用,以前未暴露此問題).經過檢查工程代碼,發現應用中 在應用啟動時,聲明了一個JedisPool,來進行業務緩存處理,但在應用退出時並未對jedispool 有任何處理.
參考 https://github.com/xetorthio/jedis/issues/936

Pool needs to be closed when it is no longer used - Failed to stop thread named [commons-pool-EvictionTimer]
pool在不被使用的時候需要釋放

guys from commons-pool you need to create a ServletContextListener and implement the contextDestroyed method. In thaUImethod you should get the reference to your JedisPool and call the close method.
需要在ServletContextListener 實現 contextDestroyed 方法 , 執行JedisPool 對象的destroy 方法,對資源進行釋放.

---

在應用啟動與關閉時,初始化與銷毀相關資源,步驟如下:

a.實現ServletContextListener 接口,分別在contextInitialized 與 contextDestroyed 中實現自己的資源初始化與銷毀邏輯(在 destroy 方法中 處理掉問題中未關閉的線程,與未關閉的redis pool )

    public class CloudfiApplicationContextListener implements ServletContextListener {
        private ServletContext servletContext;
        private LongTimeWorker longTimeWorker;
        private MybatisRedisCache redis;

        public void setRedis(MybatisRedisCache redis) {
            this.redis = redis;
        }

        public void setLongTimeWorker(LongTimeWorker longTimeWorker) {
            this.longTimeWorker = longTimeWorker;
        }

        @Override
        public void contextInitialized(ServletContextEvent sce) {
            servletContext = sce.getServletContext();
            SpringContextUtil.setApplicationContext(WebApplicationContextUtils.getWebApplicationContext(servletContext));

            longTimeWorker = SpringContextUtil.getBean( LongTimeWorker.class);
            redis = SpringContextUtil.getBean(MybatisRedisCache.class);
            longTimeWorker.startWork();
        }

        @Override
        public void contextDestroyed(ServletContextEvent sce) {
            longTimeWorker.stopWork();
            redis.getJedisPool().close();
        }
    }

b.在web.xml 中配置 a 中定義的listener

<servlet>
    <servlet-name>springmvc</servlet-name>
    <servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>
    <init-param>
        <param-name>contextConfigLocation</param-name>
        <param-value>classpath:springmvc.xml</param-value>
    </init-param>
    <load-on-startup>1</load-on-startup>
</servlet>
<listener>  
    <listener-class>org.springframework.web.context.ContextLoaderListener  
    </listener-class>  
</listener>
<listener>
    <listener-class>
        javacommon.init.CloudfiApplicationContextListener
    </listener-class>
</listener>

注意,如果需要在實現的listener 中,使用spring 中的bean ,由於listener 與 servlet 加載順序的限制,此處只能讀取 ContextLoaderListener
掃描到spring 的bean,無法獲取到 DispatcherServlet 掃描到spring 的bean (若需要獲取 ContextLoaderListener 的bean ,注意自定義listener 的配置需要在 ContextLoaderListener 配置之后)

至此,引起tomcat 執行 shutdown.sh 無法正常關閉的問題解決.

如果上述中出現錯誤,歡迎指正,若有問題請聯系[e-mail]
crazy~smn 星期五, 13. 十月 2017 02:55下午