Weblogic運行期間高CPU問題定位方法


根據進程找到高占用CPU的線程,查看線程的dump文件分析。 下面是在虛擬機上做的實驗。 
第一步: 
在Linux上,使用ps -eLo pid,ppid,tid,pcpu,comm|grep <WL Spid> 獲得CPU的線程。 

root 2767 2730 0 05:12 pts/3 00:00:00 grep java 
[root@slaver1 ~]# ps -eLo pid,ppid,tid,pcpu,comm|grep 2717 
2717 2672 2717 0.0 java 
2717 2672 2718 4.2 java 
2717 2672 2719 2.0 java 
2717 2672 2720 0.0 java 
2717 2672 2721 0.0 java 
2717 2672 2722 0.0 java 
2717 2672 2723 3.4 java 
2717 2672 2724 4.0 java 
2717 2672 2725 0.0 java 
2717 2672 2726 0.0 java 
2717 2672 2727 0.0 java 
2717 2672 2755 0.0 java 
2717 2672 2756 17.9 java 
2717 2672 2757 0.0 java 
2717 2672 2758 0.0 java 
2717 2672 2759 1.3 java 
2717 2672 2760 0.0 java 
2717 2672 2761 0.0 java 
2717 2672 2762 0.0 java 
2717 2672 2764 0.0 java 
2717 2672 2765 0.0 java 
2717 2672 2769 0.0 java 
2717 2672 2770 0.0 java 
2717 2672 2771 0.0 java 
2717 2672 2772 0.0 java 
2717 2672 2773 0.0 java 
2717 2672 2774 0.0 java 
2717 2672 2775 13.6 java 
2717 2672 2776 0.0 java 
第一列進程,第二列是父進程,第三列線程id,第四列cpu使用率。 

找到對應的線程,轉儲線程dump文件的方法:weblogic不要后台執行,在另外一個窗口執行kill -3 線程號,就會把線程信息轉儲到weblogic的執行監控界面上了。 
轉儲前weblogic日志停留界面: 
<Jan 8, 2015 5:13:04 AM EST> <Notice> <Server> <BEA-002613> <Channel "Default" is now listening on 192.168.56.102:7001 for protocols iiop, t3, ldap, snmp, http.> 
<Jan 8, 2015 5:13:04 AM EST> <Notice> <Server> <BEA-002613> <Channel "Default[3]" is now listening on 0:0:0:0:0:0:0:1:7001 for protocols iiop, t3, ldap, snmp, http.> 
<Jan 8, 2015 5:13:04 AM EST> <Notice> <Server> <BEA-002613> <Channel "Default[2]" is now listening on 127.0.0.1:7001 for protocols iiop, t3, ldap, snmp, http.> 
<Jan 8, 2015 5:13:04 AM EST> <Notice> <WebLogicServer> <BEA-000329> <Started WebLogic Admin Server "AdminServer" for domain "base_domain" running in Production Mode> 
<Jan 8, 2015 5:13:05 AM EST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RUNNING> 
<Jan 8, 2015 5:13:05 AM EST> <Notice> <WebLogicServer> <BEA-000360> <Server started in RUNNING mode> 
執行[root@slaver1 ~]# kill -3 2775 后weblogic監控界面就會打印出線程轉儲信息: 
<Jan 8, 2015 5:13:05 AM EST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RUNNING> 
<Jan 8, 2015 5:13:05 AM EST> <Notice> <WebLogicServer> <BEA-000360> <Server started in RUNNING mode> 
2015-01-08 05:15:55 
Full thread dump Java HotSpot(TM) Server VM (24.45-b08 mixed mode): 

"[STANDBY] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x09ed3800 nid=0xad8 in Object.wait() [0x00ac8000] 
java.lang.Thread.State: WAITING (on object monitor) 
at java.lang.Object.wait(Native Method) 
- waiting on <0x9dab2d38> (a weblogic.work.ExecuteThread) 
at java.lang.Object.wait(Object.java:503) 
at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:162) 
- locked <0x9dab2d38> (a weblogic.work.ExecuteThread) 
at weblogic.work.ExecuteThread.run(ExecuteThread.java:183) 

"[STANDBY] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x09fb1c00 nid=0xad7 in Object.wait() [0x0081d000] 
java.lang.Thread.State: WAITING (on object monitor) 
at java.lang.Object.wait(Native Method) 
- waiting on <0x9daab238> (a weblogic.work.ExecuteThread) 
at java.lang.Object.wait(Object.java:503) 
at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:162) 
- locked <0x9daab238> (a weblogic.work.ExecuteThread) 
at weblogic.work.ExecuteThread.run(ExecuteThread.java:183) 

"DynamicListenThread[Default[1]]" daemon prio=10 tid=0x09fb5c00 nid=0xad6 runnable [0x007cc000] 
java.lang.Thread.State: RUNNABLE 
at java.net.PlainSocketImpl.socketAccept(Native Method) 
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) 
at java.net.ServerSocket.implAccept(ServerSocket.java:530) 
at java.net.ServerSocket.accept(ServerSocket.java:498) 
at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38) 
at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535) 
at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417) 
at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173) 
at java.lang.Thread.run(Thread.java:744) 

"DynamicListenThread[Default[3]]" daemon prio=10 tid=0x096cc800 nid=0xad5 runnable [0x0071a000] 
java.lang.Thread.State: RUNNABLE 
at java.net.PlainSocketImpl.socketAccept(Native Method) 
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) 
at java.net.ServerSocket.implAccept(ServerSocket.java:530) 
at java.net.ServerSocket.accept(ServerSocket.java:498) 
at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38) 
at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535) 
at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417) 
at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173) 
at java.lang.Thread.run(Thread.java:744) 

"DynamicListenThread[Default[2]]" daemon prio=10 tid=0x0a251000 nid=0xad4 runnable [0x006c9000] 
java.lang.Thread.State: RUNNABLE 
at java.net.PlainSocketImpl.socketAccept(Native Method) 
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) 
at java.net.ServerSocket.implAccept(ServerSocket.java:530) 
at java.net.ServerSocket.accept(ServerSocket.java:498) 
at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38) 
at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535) 
at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417) 
at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173) 
at java.lang.Thread.run(Thread.java:744) 

"DynamicListenThread[Default]" daemon prio=10 tid=0x096cd400 nid=0xad3 runnable [0x00678000] 
java.lang.Thread.State: RUNNABLE 
at java.net.PlainSocketImpl.socketAccept(Native Method) 
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) 
at java.net.ServerSocket.implAccept(ServerSocket.java:530) 
at java.net.ServerSocket.accept(ServerSocket.java:498) 
at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38) 
at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535) 
at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417) 
at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173) 
at java.lang.Thread.run(Thread.java:744) 

"weblogic.GCMonitor" daemon prio=10 tid=0x09b0b000 nid=0xad2 in Object.wait() [0x00627000] 
java.lang.Thread.State: TIMED_WAITING (on object monitor) 
at java.lang.Object.wait(Native Method) 
- waiting on <0x9da543b8> (a java.lang.ref.ReferenceQueue$Lock) 
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) 
- locked <0x9da543b8> (a java.lang.ref.ReferenceQueue$Lock) 
at weblogic.platform.GCMonitorThread.waitForNotification(GCMonitorThread.java:88) 
at weblogic.platform.GCMonitorThread.run(GCMonitorThread.java:64) 

"Thread-11" prio=10 tid=0x093f5800 nid=0xad1 waiting on condition [0x00541000] 
java.lang.Thread.State: WAITING (parking) 
at sun.misc.Unsafe.park(Native Method) 
- parking to wait for <0x9d95f948> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) 
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) 
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) 
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) 
at weblogic.utils.concurrent.JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89) 
at weblogic.store.internal.PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:672) 
at weblogic.store.internal.PersistentStoreImpl.run(PersistentStoreImpl.java:721) 
at java.lang.Thread.run(Thread.java:744) 

"Thread-10" prio=10 tid=0x083b7000 nid=0xacd waiting on condition [0x0049f000] 
java.lang.Thread.State: WAITING (parking) 
at sun.misc.Unsafe.park(Native Method) 
- parking to wait for <0x9c66f560> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) 
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) 
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) 
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) 
at weblogic.utils.concurrent.JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89) 
at weblogic.store.internal.PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:672) 
at weblogic.store.internal.PersistentStoreImpl.run(PersistentStoreImpl.java:721) 
at java.lang.Thread.run(Thread.java:744) 

"DoSManager" daemon prio=10 tid=0x094f9400 nid=0xacc waiting on condition [0x004f0000] 
java.lang.Thread.State: TIMED_WAITING (sleeping) 
at java.lang.Thread.sleep(Native Method) 
at com.octetstring.vde.DoSManager.run(DoSManager.java:433) 

"VDE Transaction Processor Thread" daemon prio=10 tid=0x08f2f800 nid=0xaca in Object.wait() [0x003e8000] 
java.lang.Thread.State: WAITING (on object monitor) 
at java.lang.Object.wait(Native Method) 
- waiting on <0x9bb285f8> (a com.octetstring.vde.backend.standard.TransactionProcessor) 
at java.lang.Object.wait(Object.java:503) 
at com.octetstring.vde.backend.standard.TransactionProcessor.waitChange(TransactionProcessor.java:367) 
- locked <0x9bb285f8> (a com.octetstring.vde.backend.standard.TransactionProcessor) 
at com.octetstring.vde.backend.standard.TransactionProcessor.run(TransactionProcessor.java:212) 

"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" daemon prio=10 tid=0x0887c400 nid=0xac9 in Object.wait() [0x00a77000] 
java.lang.Thread.State: WAITING (on object monitor) 
at java.lang.Object.wait(Native Method) 
- waiting on <0x9b9e8e98> (a [Lweblogic.socket.PosixSocketInfo$FdStruct;) 
at java.lang.Object.wait(Object.java:503) 
at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:97) 
- locked <0x9b9e8e98> (a [Lweblogic.socket.PosixSocketInfo$FdStruct;) 
- locked <0x9b9e2640> (a weblogic.socket.PosixSocketMuxer$1) 
at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29) 
at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42) 
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145) 
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117) 

"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" daemon prio=10 tid=0x0887b400 nid=0xac8 waiting for monitor entry [0x002cd000] 
java.lang.Thread.State: BLOCKED (on object monitor) 
at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:95) 
- waiting to lock <0x9b9e2640> (a weblogic.socket.PosixSocketMuxer$1) 
at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29) 
at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42) 
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145) 
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117) 

"[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x08b99400 nid=0xac7 in Object.wait() [0x0025d000] 
java.lang.Thread.State: WAITING (on object monitor) 
at java.lang.Object.wait(Native Method) 
- waiting on <0x9b90e780> (a weblogic.work.ExecuteThread) 
at java.lang.Object.wait(Object.java:503) 
at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:162) 
- locked <0x9b90e780> (a weblogic.work.ExecuteThread) 
at weblogic.work.ExecuteThread.run(ExecuteThread.java:183) 

"weblogic.timers.TimerThread" daemon prio=10 tid=0x08c4c000 nid=0xac6 in Object.wait() [0x0020c000] 
java.lang.Thread.State: TIMED_WAITING (on object monitor) 
at java.lang.Object.wait(Native Method) 
- waiting on <0x9b8f1bf0> (a weblogic.timers.internal.TimerThread) 
at weblogic.timers.internal.TimerThread$Thread.run(TimerThread.java:262) 
- locked <0x9b8f1bf0> (a weblogic.timers.internal.TimerThread) 

"weblogic.time.TimeEventGenerator" daemon prio=10 tid=0x08c79c00 nid=0xac5 in Object.wait() [0x001bb000] 
java.lang.Thread.State: TIMED_WAITING (on object monitor) 
at java.lang.Object.wait(Native Method) 
- waiting on <0x9b8ed2d0> (a weblogic.time.common.internal.TimeTable) 
at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:286) 
- locked <0x9b8ed2d0> (a weblogic.time.common.internal.TimeTable) 
at weblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java:117) 
at java.lang.Thread.run(Thread.java:744) 

"[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x08fb3400 nid=0xac4 in Object.wait() [0x00397000] 
java.lang.Thread.State: WAITING (on object monitor) 
at java.lang.Object.wait(Native Method) 
- waiting on <0x9b8e3498> (a weblogic.work.ExecuteThread) 
at java.lang.Object.wait(Object.java:503) 
at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:162) 
- locked <0x9b8e3498> (a weblogic.work.ExecuteThread) 
at weblogic.work.ExecuteThread.run(ExecuteThread.java:183) 

"Timer-1" daemon prio=10 tid=0x08b74800 nid=0xac3 in Object.wait() [0x0032b000] 
java.lang.Thread.State: TIMED_WAITING (on object monitor) 
at java.lang.Object.wait(Native Method) 
- waiting on <0x9b8e2df0> (a java.util.TaskQueue) 
at java.util.TimerThread.mainLoop(Timer.java:552) 
- locked <0x9b8e2df0> (a java.util.TaskQueue) 
at java.util.TimerThread.run(Timer.java:505) 

"Timer-0" daemon prio=10 tid=0x08b7a000 nid=0xaa7 in Object.wait() [0x8f921000] 
java.lang.Thread.State: WAITING (on object monitor) 
at java.lang.Object.wait(Native Method) 
- waiting on <0x9bbfb0b8> (a java.util.TaskQueue) 
at java.lang.Object.wait(Object.java:503) 
at java.util.TimerThread.mainLoop(Timer.java:526) 
- locked <0x9bbfb0b8> (a java.util.TaskQueue) 
at java.util.TimerThread.run(Timer.java:505) 

"Service Thread" daemon prio=10 tid=0x08127000 nid=0xaa5 runnable [0x00000000] 
java.lang.Thread.State: RUNNABLE 

"C2 CompilerThread1" daemon prio=10 tid=0x08125400 nid=0xaa4 waiting on condition [0x00000000] 
java.lang.Thread.State: RUNNABLE 

"C2 CompilerThread0" daemon prio=10 tid=0x0811ac00 nid=0xaa3 waiting on condition [0x00000000] 
java.lang.Thread.State: RUNNABLE 

"Signal Dispatcher" daemon prio=10 tid=0x08119400 nid=0xaa2 runnable [0x00000000] 
java.lang.Thread.State: RUNNABLE 

"Finalizer" daemon prio=10 tid=0x08105c00 nid=0xaa1 in Object.wait() [0x908bd000] 
java.lang.Thread.State: WAITING (on object monitor) 
at java.lang.Object.wait(Native Method) 
- waiting on <0x9bbfb2d0> (a java.lang.ref.ReferenceQueue$Lock) 
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) 
- locked <0x9bbfb2d0> (a java.lang.ref.ReferenceQueue$Lock) 
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) 
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:189) 

"Reference Handler" daemon prio=10 tid=0x08103400 nid=0xaa0 in Object.wait() [0x9090e000] 
java.lang.Thread.State: WAITING (on object monitor) 
at java.lang.Object.wait(Native Method) 
- waiting on <0x9bbfb358> (a java.lang.ref.Reference$Lock) 
at java.lang.Object.wait(Object.java:503) 
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) 
- locked <0x9bbfb358> (a java.lang.ref.Reference$Lock) 

"main" prio=10 tid=0x080b1c00 nid=0xa9e in Object.wait() [0xb7efc000] 
java.lang.Thread.State: WAITING (on object monitor) 
at java.lang.Object.wait(Native Method) 
- waiting on <0x9bbfbf88> (a weblogic.t3.srvr.T3Srvr) 
at java.lang.Object.wait(Object.java:503) 
at weblogic.t3.srvr.T3Srvr.waitForDeath(T3Srvr.java:981) 
- locked <0x9bbfbf88> (a weblogic.t3.srvr.T3Srvr) 
at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:490) 
at weblogic.Server.main(Server.java:71) 

"VM Thread" prio=10 tid=0x08101c00 nid=0xa9f runnable 

"VM Periodic Task Thread" prio=10 tid=0x08129400 nid=0xaa6 waiting on condition 

JNI global references: 280 

Heap 
def new generation total 157248K, used 122434K [0x90de0000, 0x9b880000, 0x9b880000) 
eden space 139776K, 87% used [0x90de0000, 0x98570a88, 0x99660000) 
from space 17472K, 0% used [0x9a770000, 0x9a770000, 0x9b880000) 
to space 17472K, 0% used [0x99660000, 0x99660000, 0x9a770000) 
tenured generation total 349568K, used 45140K [0x9b880000, 0xb0de0000, 0xb0de0000) 
the space 349568K, 12% used [0x9b880000, 0x9e495168, 0x9e495200, 0xb0de0000) 
compacting perm gen total 61184K, used 61001K [0xb0de0000, 0xb49a0000, 0xb4de0000) 
the space 61184K, 99% used [0xb0de0000, 0xb49727e8, 0xb4972800, 0xb49a0000) 
No shared spaces configured. 


這里面的線程號是16進制的,把剛才執行kill -3 線程號的線程號在這里面找到,變可以查到高占用cpu的線程信息了。 

比如上面的2775線程號,對應的16進制是ad7,然后就可以在上面的轉儲信息里找到相關的線程占用高的信息。 


免責聲明!

本站轉載的文章為個人學習借鑒使用,本站對版權不負任何法律責任。如果侵犯了您的隱私權益,請聯系本站郵箱yoyou2525@163.com刪除。



 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM