根据进程找到高占用CPU的线程,查看线程的dump文件分析。 下面是在虚拟机上做的实验。
第一步:
在Linux上,使用ps -eLo pid,ppid,tid,pcpu,comm|grep <WL Spid> 获得CPU的线程。
root 2767 2730 0 05:12 pts/3 00:00:00 grep java
[root@slaver1 ~]# ps -eLo pid,ppid,tid,pcpu,comm|grep 2717
2717 2672 2717 0.0 java
2717 2672 2718 4.2 java
2717 2672 2719 2.0 java
2717 2672 2720 0.0 java
2717 2672 2721 0.0 java
2717 2672 2722 0.0 java
2717 2672 2723 3.4 java
2717 2672 2724 4.0 java
2717 2672 2725 0.0 java
2717 2672 2726 0.0 java
2717 2672 2727 0.0 java
2717 2672 2755 0.0 java
2717 2672 2756 17.9 java
2717 2672 2757 0.0 java
2717 2672 2758 0.0 java
2717 2672 2759 1.3 java
2717 2672 2760 0.0 java
2717 2672 2761 0.0 java
2717 2672 2762 0.0 java
2717 2672 2764 0.0 java
2717 2672 2765 0.0 java
2717 2672 2769 0.0 java
2717 2672 2770 0.0 java
2717 2672 2771 0.0 java
2717 2672 2772 0.0 java
2717 2672 2773 0.0 java
2717 2672 2774 0.0 java
2717 2672 2775 13.6 java
2717 2672 2776 0.0 java
第一列进程,第二列是父进程,第三列线程id,第四列cpu使用率。
找到对应的线程,转储线程dump文件的方法:weblogic不要后台执行,在另外一个窗口执行kill -3 线程号,就会把线程信息转储到weblogic的执行监控界面上了。
转储前weblogic日志停留界面:
<Jan 8, 2015 5:13:04 AM EST> <Notice> <Server> <BEA-002613> <Channel "Default" is now listening on 192.168.56.102:7001 for protocols iiop, t3, ldap, snmp, http.>
<Jan 8, 2015 5:13:04 AM EST> <Notice> <Server> <BEA-002613> <Channel "Default[3]" is now listening on 0:0:0:0:0:0:0:1:7001 for protocols iiop, t3, ldap, snmp, http.>
<Jan 8, 2015 5:13:04 AM EST> <Notice> <Server> <BEA-002613> <Channel "Default[2]" is now listening on 127.0.0.1:7001 for protocols iiop, t3, ldap, snmp, http.>
<Jan 8, 2015 5:13:04 AM EST> <Notice> <WebLogicServer> <BEA-000329> <Started WebLogic Admin Server "AdminServer" for domain "base_domain" running in Production Mode>
<Jan 8, 2015 5:13:05 AM EST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RUNNING>
<Jan 8, 2015 5:13:05 AM EST> <Notice> <WebLogicServer> <BEA-000360> <Server started in RUNNING mode>
执行[root@slaver1 ~]# kill -3 2775 后weblogic监控界面就会打印出线程转储信息:
<Jan 8, 2015 5:13:05 AM EST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RUNNING>
<Jan 8, 2015 5:13:05 AM EST> <Notice> <WebLogicServer> <BEA-000360> <Server started in RUNNING mode>
2015-01-08 05:15:55
Full thread dump Java HotSpot(TM) Server VM (24.45-b08 mixed mode):
"[STANDBY] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x09ed3800 nid=0xad8 in Object.wait() [0x00ac8000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x9dab2d38> (a weblogic.work.ExecuteThread)
at java.lang.Object.wait(Object.java:503)
at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:162)
- locked <0x9dab2d38> (a weblogic.work.ExecuteThread)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:183)
"[STANDBY] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x09fb1c00 nid=0xad7 in Object.wait() [0x0081d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x9daab238> (a weblogic.work.ExecuteThread)
at java.lang.Object.wait(Object.java:503)
at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:162)
- locked <0x9daab238> (a weblogic.work.ExecuteThread)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:183)
"DynamicListenThread[Default[1]]" daemon prio=10 tid=0x09fb5c00 nid=0xad6 runnable [0x007cc000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38)
at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535)
at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417)
at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173)
at java.lang.Thread.run(Thread.java:744)
"DynamicListenThread[Default[3]]" daemon prio=10 tid=0x096cc800 nid=0xad5 runnable [0x0071a000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38)
at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535)
at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417)
at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173)
at java.lang.Thread.run(Thread.java:744)
"DynamicListenThread[Default[2]]" daemon prio=10 tid=0x0a251000 nid=0xad4 runnable [0x006c9000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38)
at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535)
at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417)
at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173)
at java.lang.Thread.run(Thread.java:744)
"DynamicListenThread[Default]" daemon prio=10 tid=0x096cd400 nid=0xad3 runnable [0x00678000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38)
at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535)
at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417)
at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173)
at java.lang.Thread.run(Thread.java:744)
"weblogic.GCMonitor" daemon prio=10 tid=0x09b0b000 nid=0xad2 in Object.wait() [0x00627000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x9da543b8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
- locked <0x9da543b8> (a java.lang.ref.ReferenceQueue$Lock)
at weblogic.platform.GCMonitorThread.waitForNotification(GCMonitorThread.java:88)
at weblogic.platform.GCMonitorThread.run(GCMonitorThread.java:64)
"Thread-11" prio=10 tid=0x093f5800 nid=0xad1 waiting on condition [0x00541000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x9d95f948> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at weblogic.utils.concurrent.JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)
at weblogic.store.internal.PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:672)
at weblogic.store.internal.PersistentStoreImpl.run(PersistentStoreImpl.java:721)
at java.lang.Thread.run(Thread.java:744)
"Thread-10" prio=10 tid=0x083b7000 nid=0xacd waiting on condition [0x0049f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x9c66f560> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at weblogic.utils.concurrent.JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)
at weblogic.store.internal.PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:672)
at weblogic.store.internal.PersistentStoreImpl.run(PersistentStoreImpl.java:721)
at java.lang.Thread.run(Thread.java:744)
"DoSManager" daemon prio=10 tid=0x094f9400 nid=0xacc waiting on condition [0x004f0000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at com.octetstring.vde.DoSManager.run(DoSManager.java:433)
"VDE Transaction Processor Thread" daemon prio=10 tid=0x08f2f800 nid=0xaca in Object.wait() [0x003e8000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x9bb285f8> (a com.octetstring.vde.backend.standard.TransactionProcessor)
at java.lang.Object.wait(Object.java:503)
at com.octetstring.vde.backend.standard.TransactionProcessor.waitChange(TransactionProcessor.java:367)
- locked <0x9bb285f8> (a com.octetstring.vde.backend.standard.TransactionProcessor)
at com.octetstring.vde.backend.standard.TransactionProcessor.run(TransactionProcessor.java:212)
"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" daemon prio=10 tid=0x0887c400 nid=0xac9 in Object.wait() [0x00a77000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x9b9e8e98> (a [Lweblogic.socket.PosixSocketInfo$FdStruct;)
at java.lang.Object.wait(Object.java:503)
at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:97)
- locked <0x9b9e8e98> (a [Lweblogic.socket.PosixSocketInfo$FdStruct;)
- locked <0x9b9e2640> (a weblogic.socket.PosixSocketMuxer$1)
at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)
"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" daemon prio=10 tid=0x0887b400 nid=0xac8 waiting for monitor entry [0x002cd000]
java.lang.Thread.State: BLOCKED (on object monitor)
at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:95)
- waiting to lock <0x9b9e2640> (a weblogic.socket.PosixSocketMuxer$1)
at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)
"[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x08b99400 nid=0xac7 in Object.wait() [0x0025d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x9b90e780> (a weblogic.work.ExecuteThread)
at java.lang.Object.wait(Object.java:503)
at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:162)
- locked <0x9b90e780> (a weblogic.work.ExecuteThread)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:183)
"weblogic.timers.TimerThread" daemon prio=10 tid=0x08c4c000 nid=0xac6 in Object.wait() [0x0020c000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x9b8f1bf0> (a weblogic.timers.internal.TimerThread)
at weblogic.timers.internal.TimerThread$Thread.run(TimerThread.java:262)
- locked <0x9b8f1bf0> (a weblogic.timers.internal.TimerThread)
"weblogic.time.TimeEventGenerator" daemon prio=10 tid=0x08c79c00 nid=0xac5 in Object.wait() [0x001bb000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x9b8ed2d0> (a weblogic.time.common.internal.TimeTable)
at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:286)
- locked <0x9b8ed2d0> (a weblogic.time.common.internal.TimeTable)
at weblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java:117)
at java.lang.Thread.run(Thread.java:744)
"[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x08fb3400 nid=0xac4 in Object.wait() [0x00397000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x9b8e3498> (a weblogic.work.ExecuteThread)
at java.lang.Object.wait(Object.java:503)
at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:162)
- locked <0x9b8e3498> (a weblogic.work.ExecuteThread)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:183)
"Timer-1" daemon prio=10 tid=0x08b74800 nid=0xac3 in Object.wait() [0x0032b000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x9b8e2df0> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x9b8e2df0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"Timer-0" daemon prio=10 tid=0x08b7a000 nid=0xaa7 in Object.wait() [0x8f921000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x9bbfb0b8> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:503)
at java.util.TimerThread.mainLoop(Timer.java:526)
- locked <0x9bbfb0b8> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"Service Thread" daemon prio=10 tid=0x08127000 nid=0xaa5 runnable [0x00000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=10 tid=0x08125400 nid=0xaa4 waiting on condition [0x00000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=10 tid=0x0811ac00 nid=0xaa3 waiting on condition [0x00000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x08119400 nid=0xaa2 runnable [0x00000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x08105c00 nid=0xaa1 in Object.wait() [0x908bd000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x9bbfb2d0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
- locked <0x9bbfb2d0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:189)
"Reference Handler" daemon prio=10 tid=0x08103400 nid=0xaa0 in Object.wait() [0x9090e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x9bbfb358> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:503)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
- locked <0x9bbfb358> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0x080b1c00 nid=0xa9e in Object.wait() [0xb7efc000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x9bbfbf88> (a weblogic.t3.srvr.T3Srvr)
at java.lang.Object.wait(Object.java:503)
at weblogic.t3.srvr.T3Srvr.waitForDeath(T3Srvr.java:981)
- locked <0x9bbfbf88> (a weblogic.t3.srvr.T3Srvr)
at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:490)