Weblogic运行期间高CPU问题定位方法


根据进程找到高占用CPU的线程,查看线程的dump文件分析。 下面是在虚拟机上做的实验。 
第一步: 
在Linux上,使用ps -eLo pid,ppid,tid,pcpu,comm|grep <WL Spid> 获得CPU的线程。 

root 2767 2730 0 05:12 pts/3 00:00:00 grep java 
[root@slaver1 ~]# ps -eLo pid,ppid,tid,pcpu,comm|grep 2717 
2717 2672 2717 0.0 java 
2717 2672 2718 4.2 java 
2717 2672 2719 2.0 java 
2717 2672 2720 0.0 java 
2717 2672 2721 0.0 java 
2717 2672 2722 0.0 java 
2717 2672 2723 3.4 java 
2717 2672 2724 4.0 java 
2717 2672 2725 0.0 java 
2717 2672 2726 0.0 java 
2717 2672 2727 0.0 java 
2717 2672 2755 0.0 java 
2717 2672 2756 17.9 java 
2717 2672 2757 0.0 java 
2717 2672 2758 0.0 java 
2717 2672 2759 1.3 java 
2717 2672 2760 0.0 java 
2717 2672 2761 0.0 java 
2717 2672 2762 0.0 java 
2717 2672 2764 0.0 java 
2717 2672 2765 0.0 java 
2717 2672 2769 0.0 java 
2717 2672 2770 0.0 java 
2717 2672 2771 0.0 java 
2717 2672 2772 0.0 java 
2717 2672 2773 0.0 java 
2717 2672 2774 0.0 java 
2717 2672 2775 13.6 java 
2717 2672 2776 0.0 java 
第一列进程,第二列是父进程,第三列线程id,第四列cpu使用率。 

找到对应的线程,转储线程dump文件的方法:weblogic不要后台执行,在另外一个窗口执行kill -3 线程号,就会把线程信息转储到weblogic的执行监控界面上了。 
转储前weblogic日志停留界面: 
<Jan 8, 2015 5:13:04 AM EST> <Notice> <Server> <BEA-002613> <Channel "Default" is now listening on 192.168.56.102:7001 for protocols iiop, t3, ldap, snmp, http.> 
<Jan 8, 2015 5:13:04 AM EST> <Notice> <Server> <BEA-002613> <Channel "Default[3]" is now listening on 0:0:0:0:0:0:0:1:7001 for protocols iiop, t3, ldap, snmp, http.> 
<Jan 8, 2015 5:13:04 AM EST> <Notice> <Server> <BEA-002613> <Channel "Default[2]" is now listening on 127.0.0.1:7001 for protocols iiop, t3, ldap, snmp, http.> 
<Jan 8, 2015 5:13:04 AM EST> <Notice> <WebLogicServer> <BEA-000329> <Started WebLogic Admin Server "AdminServer" for domain "base_domain" running in Production Mode> 
<Jan 8, 2015 5:13:05 AM EST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RUNNING> 
<Jan 8, 2015 5:13:05 AM EST> <Notice> <WebLogicServer> <BEA-000360> <Server started in RUNNING mode> 
执行[root@slaver1 ~]# kill -3 2775 后weblogic监控界面就会打印出线程转储信息: 
<Jan 8, 2015 5:13:05 AM EST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RUNNING> 
<Jan 8, 2015 5:13:05 AM EST> <Notice> <WebLogicServer> <BEA-000360> <Server started in RUNNING mode> 
2015-01-08 05:15:55 
Full thread dump Java HotSpot(TM) Server VM (24.45-b08 mixed mode): 

"[STANDBY] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x09ed3800 nid=0xad8 in Object.wait() [0x00ac8000] 
java.lang.Thread.State: WAITING (on object monitor) 
at java.lang.Object.wait(Native Method) 
- waiting on <0x9dab2d38> (a weblogic.work.ExecuteThread) 
at java.lang.Object.wait(Object.java:503) 
at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:162) 
- locked <0x9dab2d38> (a weblogic.work.ExecuteThread) 
at weblogic.work.ExecuteThread.run(ExecuteThread.java:183) 

"[STANDBY] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x09fb1c00 nid=0xad7 in Object.wait() [0x0081d000] 
java.lang.Thread.State: WAITING (on object monitor) 
at java.lang.Object.wait(Native Method) 
- waiting on <0x9daab238> (a weblogic.work.ExecuteThread) 
at java.lang.Object.wait(Object.java:503) 
at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:162) 
- locked <0x9daab238> (a weblogic.work.ExecuteThread) 
at weblogic.work.ExecuteThread.run(ExecuteThread.java:183) 

"DynamicListenThread[Default[1]]" daemon prio=10 tid=0x09fb5c00 nid=0xad6 runnable [0x007cc000] 
java.lang.Thread.State: RUNNABLE 
at java.net.PlainSocketImpl.socketAccept(Native Method) 
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) 
at java.net.ServerSocket.implAccept(ServerSocket.java:530) 
at java.net.ServerSocket.accept(ServerSocket.java:498) 
at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38) 
at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535) 
at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417) 
at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173) 
at java.lang.Thread.run(Thread.java:744) 

"DynamicListenThread[Default[3]]" daemon prio=10 tid=0x096cc800 nid=0xad5 runnable [0x0071a000] 
java.lang.Thread.State: RUNNABLE 
at java.net.PlainSocketImpl.socketAccept(Native Method) 
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) 
at java.net.ServerSocket.implAccept(ServerSocket.java:530) 
at java.net.ServerSocket.accept(ServerSocket.java:498) 
at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38) 
at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535) 
at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417) 
at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173) 
at java.lang.Thread.run(Thread.java:744) 

"DynamicListenThread[Default[2]]" daemon prio=10 tid=0x0a251000 nid=0xad4 runnable [0x006c9000] 
java.lang.Thread.State: RUNNABLE 
at java.net.PlainSocketImpl.socketAccept(Native Method) 
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) 
at java.net.ServerSocket.implAccept(ServerSocket.java:530) 
at java.net.ServerSocket.accept(ServerSocket.java:498) 
at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38) 
at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535) 
at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417) 
at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173) 
at java.lang.Thread.run(Thread.java:744) 

"DynamicListenThread[Default]" daemon prio=10 tid=0x096cd400 nid=0xad3 runnable [0x00678000] 
java.lang.Thread.State: RUNNABLE 
at java.net.PlainSocketImpl.socketAccept(Native Method) 
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) 
at java.net.ServerSocket.implAccept(ServerSocket.java:530) 
at java.net.ServerSocket.accept(ServerSocket.java:498) 
at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38) 
at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535) 
at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417) 
at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173) 
at java.lang.Thread.run(Thread.java:744) 

"weblogic.GCMonitor" daemon prio=10 tid=0x09b0b000 nid=0xad2 in Object.wait() [0x00627000] 
java.lang.Thread.State: TIMED_WAITING (on object monitor) 
at java.lang.Object.wait(Native Method) 
- waiting on <0x9da543b8> (a java.lang.ref.ReferenceQueue$Lock) 
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) 
- locked <0x9da543b8> (a java.lang.ref.ReferenceQueue$Lock) 
at weblogic.platform.GCMonitorThread.waitForNotification(GCMonitorThread.java:88) 
at weblogic.platform.GCMonitorThread.run(GCMonitorThread.java:64) 

"Thread-11" prio=10 tid=0x093f5800 nid=0xad1 waiting on condition [0x00541000] 
java.lang.Thread.State: WAITING (parking) 
at sun.misc.Unsafe.park(Native Method) 
- parking to wait for <0x9d95f948> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) 
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) 
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) 
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) 
at weblogic.utils.concurrent.JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89) 
at weblogic.store.internal.PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:672) 
at weblogic.store.internal.PersistentStoreImpl.run(PersistentStoreImpl.java:721) 
at java.lang.Thread.run(Thread.java:744) 

"Thread-10" prio=10 tid=0x083b7000 nid=0xacd waiting on condition [0x0049f000] 
java.lang.Thread.State: WAITING (parking) 
at sun.misc.Unsafe.park(Native Method) 
- parking to wait for <0x9c66f560> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) 
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) 
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) 
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) 
at weblogic.utils.concurrent.JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89) 
at weblogic.store.internal.PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:672) 
at weblogic.store.internal.PersistentStoreImpl.run(PersistentStoreImpl.java:721) 
at java.lang.Thread.run(Thread.java:744) 

"DoSManager" daemon prio=10 tid=0x094f9400 nid=0xacc waiting on condition [0x004f0000] 
java.lang.Thread.State: TIMED_WAITING (sleeping) 
at java.lang.Thread.sleep(Native Method) 
at com.octetstring.vde.DoSManager.run(DoSManager.java:433) 

"VDE Transaction Processor Thread" daemon prio=10 tid=0x08f2f800 nid=0xaca in Object.wait() [0x003e8000] 
java.lang.Thread.State: WAITING (on object monitor) 
at java.lang.Object.wait(Native Method) 
- waiting on <0x9bb285f8> (a com.octetstring.vde.backend.standard.TransactionProcessor) 
at java.lang.Object.wait(Object.java:503) 
at com.octetstring.vde.backend.standard.TransactionProcessor.waitChange(TransactionProcessor.java:367) 
- locked <0x9bb285f8> (a com.octetstring.vde.backend.standard.TransactionProcessor) 
at com.octetstring.vde.backend.standard.TransactionProcessor.run(TransactionProcessor.java:212) 

"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" daemon prio=10 tid=0x0887c400 nid=0xac9 in Object.wait() [0x00a77000] 
java.lang.Thread.State: WAITING (on object monitor) 
at java.lang.Object.wait(Native Method) 
- waiting on <0x9b9e8e98> (a [Lweblogic.socket.PosixSocketInfo$FdStruct;) 
at java.lang.Object.wait(Object.java:503) 
at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:97) 
- locked <0x9b9e8e98> (a [Lweblogic.socket.PosixSocketInfo$FdStruct;) 
- locked <0x9b9e2640> (a weblogic.socket.PosixSocketMuxer$1) 
at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29) 
at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42) 
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145) 
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117) 

"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" daemon prio=10 tid=0x0887b400 nid=0xac8 waiting for monitor entry [0x002cd000] 
java.lang.Thread.State: BLOCKED (on object monitor) 
at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:95) 
- waiting to lock <0x9b9e2640> (a weblogic.socket.PosixSocketMuxer$1) 
at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29) 
at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42) 
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145) 
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117) 

"[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x08b99400 nid=0xac7 in Object.wait() [0x0025d000] 
java.lang.Thread.State: WAITING (on object monitor) 
at java.lang.Object.wait(Native Method) 
- waiting on <0x9b90e780> (a weblogic.work.ExecuteThread) 
at java.lang.Object.wait(Object.java:503) 
at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:162) 
- locked <0x9b90e780> (a weblogic.work.ExecuteThread) 
at weblogic.work.ExecuteThread.run(ExecuteThread.java:183) 

"weblogic.timers.TimerThread" daemon prio=10 tid=0x08c4c000 nid=0xac6 in Object.wait() [0x0020c000] 
java.lang.Thread.State: TIMED_WAITING (on object monitor) 
at java.lang.Object.wait(Native Method) 
- waiting on <0x9b8f1bf0> (a weblogic.timers.internal.TimerThread) 
at weblogic.timers.internal.TimerThread$Thread.run(TimerThread.java:262) 
- locked <0x9b8f1bf0> (a weblogic.timers.internal.TimerThread) 

"weblogic.time.TimeEventGenerator" daemon prio=10 tid=0x08c79c00 nid=0xac5 in Object.wait() [0x001bb000] 
java.lang.Thread.State: TIMED_WAITING (on object monitor) 
at java.lang.Object.wait(Native Method) 
- waiting on <0x9b8ed2d0> (a weblogic.time.common.internal.TimeTable) 
at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:286) 
- locked <0x9b8ed2d0> (a weblogic.time.common.internal.TimeTable) 
at weblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java:117) 
at java.lang.Thread.run(Thread.java:744) 

"[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x08fb3400 nid=0xac4 in Object.wait() [0x00397000] 
java.lang.Thread.State: WAITING (on object monitor) 
at java.lang.Object.wait(Native Method) 
- waiting on <0x9b8e3498> (a weblogic.work.ExecuteThread) 
at java.lang.Object.wait(Object.java:503) 
at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:162) 
- locked <0x9b8e3498> (a weblogic.work.ExecuteThread) 
at weblogic.work.ExecuteThread.run(ExecuteThread.java:183) 

"Timer-1" daemon prio=10 tid=0x08b74800 nid=0xac3 in Object.wait() [0x0032b000] 
java.lang.Thread.State: TIMED_WAITING (on object monitor) 
at java.lang.Object.wait(Native Method) 
- waiting on <0x9b8e2df0> (a java.util.TaskQueue) 
at java.util.TimerThread.mainLoop(Timer.java:552) 
- locked <0x9b8e2df0> (a java.util.TaskQueue) 
at java.util.TimerThread.run(Timer.java:505) 

"Timer-0" daemon prio=10 tid=0x08b7a000 nid=0xaa7 in Object.wait() [0x8f921000] 
java.lang.Thread.State: WAITING (on object monitor) 
at java.lang.Object.wait(Native Method) 
- waiting on <0x9bbfb0b8> (a java.util.TaskQueue) 
at java.lang.Object.wait(Object.java:503) 
at java.util.TimerThread.mainLoop(Timer.java:526) 
- locked <0x9bbfb0b8> (a java.util.TaskQueue) 
at java.util.TimerThread.run(Timer.java:505) 

"Service Thread" daemon prio=10 tid=0x08127000 nid=0xaa5 runnable [0x00000000] 
java.lang.Thread.State: RUNNABLE 

"C2 CompilerThread1" daemon prio=10 tid=0x08125400 nid=0xaa4 waiting on condition [0x00000000] 
java.lang.Thread.State: RUNNABLE 

"C2 CompilerThread0" daemon prio=10 tid=0x0811ac00 nid=0xaa3 waiting on condition [0x00000000] 
java.lang.Thread.State: RUNNABLE 

"Signal Dispatcher" daemon prio=10 tid=0x08119400 nid=0xaa2 runnable [0x00000000] 
java.lang.Thread.State: RUNNABLE 

"Finalizer" daemon prio=10 tid=0x08105c00 nid=0xaa1 in Object.wait() [0x908bd000] 
java.lang.Thread.State: WAITING (on object monitor) 
at java.lang.Object.wait(Native Method) 
- waiting on <0x9bbfb2d0> (a java.lang.ref.ReferenceQueue$Lock) 
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) 
- locked <0x9bbfb2d0> (a java.lang.ref.ReferenceQueue$Lock) 
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) 
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:189) 

"Reference Handler" daemon prio=10 tid=0x08103400 nid=0xaa0 in Object.wait() [0x9090e000] 
java.lang.Thread.State: WAITING (on object monitor) 
at java.lang.Object.wait(Native Method) 
- waiting on <0x9bbfb358> (a java.lang.ref.Reference$Lock) 
at java.lang.Object.wait(Object.java:503) 
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) 
- locked <0x9bbfb358> (a java.lang.ref.Reference$Lock) 

"main" prio=10 tid=0x080b1c00 nid=0xa9e in Object.wait() [0xb7efc000] 
java.lang.Thread.State: WAITING (on object monitor) 
at java.lang.Object.wait(Native Method) 
- waiting on <0x9bbfbf88> (a weblogic.t3.srvr.T3Srvr) 
at java.lang.Object.wait(Object.java:503) 
at weblogic.t3.srvr.T3Srvr.waitForDeath(T3Srvr.java:981) 
- locked <0x9bbfbf88> (a weblogic.t3.srvr.T3Srvr) 
at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:490) 
at weblogic.Server.main(Server.java:71) 

"VM Thread" prio=10 tid=0x08101c00 nid=0xa9f runnable 

"VM Periodic Task Thread" prio=10 tid=0x08129400 nid=0xaa6 waiting on condition 

JNI global references: 280 

Heap 
def new generation total 157248K, used 122434K [0x90de0000, 0x9b880000, 0x9b880000) 
eden space 139776K, 87% used [0x90de0000, 0x98570a88, 0x99660000) 
from space 17472K, 0% used [0x9a770000, 0x9a770000, 0x9b880000) 
to space 17472K, 0% used [0x99660000, 0x99660000, 0x9a770000) 
tenured generation total 349568K, used 45140K [0x9b880000, 0xb0de0000, 0xb0de0000) 
the space 349568K, 12% used [0x9b880000, 0x9e495168, 0x9e495200, 0xb0de0000) 
compacting perm gen total 61184K, used 61001K [0xb0de0000, 0xb49a0000, 0xb4de0000) 
the space 61184K, 99% used [0xb0de0000, 0xb49727e8, 0xb4972800, 0xb49a0000) 
No shared spaces configured. 


这里面的线程号是16进制的,把刚才执行kill -3 线程号的线程号在这里面找到,变可以查到高占用cpu的线程信息了。 

比如上面的2775线程号,对应的16进制是ad7,然后就可以在上面的转储信息里找到相关的线程占用高的信息。 


免责声明!

本站转载的文章为个人学习借鉴使用,本站对版权不负任何法律责任。如果侵犯了您的隐私权益,请联系本站邮箱yoyou2525@163.com删除。



 
粤ICP备18138465号  © 2018-2025 CODEPRJ.COM