問題描述
Java程序運行過程中拋出java.lang.OutOfMemoryError: unable to create new native thread，如下所示：
[java] view plain copy
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:691)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
[java] view plain copy
Caused by: java.lang.OutOfMemoryError
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.(ZipFile.java:214)
at java.util.zip.ZipFile.(ZipFile.java:144)
at java.util.jar.JarFile.(JarFile.java:153)
at java.util.jar.JarFile.(JarFile.java:117)
從JVM層面去解決
減小thread stack的大小
JVM默認thread stack的大小為1024，這樣當線程多時導致Native virtual memory被耗盡，實際上當thread stack的大小為128K 或 256K時是足夠的，所以我們如果明確指定thread stack為128K 或 256K即可，具體使用-Xss，例如在JVM啟動的JVM_OPT中添加如下配置
[java] view plain copy
-Xss128k
減小heap或permgen初始分配的大小
如果JVM啟動的JVM_OPT中有如下配置
[java] view plain copy
-Xms1303m -Xmx1303m -XX:PermSize=256m -XX:MaxPermSize=256m
我們可以刪除或減小初始化最小值的配置，如下
[java] view plain copy
-Xms256m -Xmx1303m -XX:PermSize=64m -XX:MaxPermSize=256m
[java] view plain copy
-Xmx1303m -XX:MaxPermSize=256m
升級JVM到最新的版本
最新版本的JVM一般在內存優化方面做的更好，升級JVM到最新的版本可能會緩解測問題
從操作系統層面去解決
使用64位操作系統
如果使用32位操作系統遇到unable to create new native thread，建議使用64位操作系統
增大OS對線程的限制
在Linux操作系統設定nofile和nproc，具體編輯/etc/security/limits.conf添加如下：
[html] view plain copy
soft nofile 2048
hard nofile 8192
[html] view plain copy
soft nproc 2048
hard nproc 8192
如果使用Red Hat Enterprise Linux 6，編輯/etc/security/limits.d/90-nproc.conf，添加如下配置：
[html] view plain copy
cat /etc/security/limits.d/90-nproc.conf
- soft nproc 1024
root soft nproc unlimited
user - nproc 2048
JVM Crash拋出如下信息：
[java] view plain copy
There is insufficient memory for the Java Runtime Environment to continue.
Native memory allocation (malloc) failed to allocate 813056 bytes for Chunk::new
An error report file with more information is saved as:
/home/kylin/work/brms/brms-standalone-5.3.1/jboss-as/bin/hs_err_pid26819.log
hs_err_pid26819.log文件內容如下：
[java] view plain copy
There is insufficient memory for the Java Runtime Environment to continue.
Native memory allocation (malloc) failed to allocate 813056 bytes for Chunk::new
Possible reasons:
The system is out of physical RAM or swap space
In 32 bit mode, the process size limit was hit
Possible solutions:
Reduce memory load on the system
Increase physical memory or swap space
Check if swap backing store is full
Use 64 bit Java on a 64 bit OS
Decrease Java heap size (-Xmx/-Xms)
Decrease number of Java threads
Decrease Java thread stack sizes (-Xss)
Set larger code cache with -XX:ReservedCodeCacheSize=
This output file may be truncated or incomplete.
Out of Memory Error (allocation.cpp:328), pid=26819, tid=1286601584
JRE version: 7.0_21-b11
Java VM: Java HotSpot(TM) Server VM (23.21-b01 mixed mode linux-x86 )
Failed to write core dump. Core dumps have been disabled. To enable core dumping, try “ulimit -c unlimited” before starting Java again
————— T H R E A D —————
Current thread (0x4cff4c00): JavaThread “C2 CompilerThread0” daemon [_thread_in_native, id=26829, stack(0x4ca7f000,0x4cb00000)]
Stack: [0x4ca7f000,0x4cb00000], sp=0x4cafd9a0, free space=506k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x722359] VMError::report_and_die()+0x199
V [libjvm.so+0x2e8ef2] report_vm_out_of_memory(char const*, int, unsigned int, char const*)+0x72
V [libjvm.so+0x14ddad] Chunk::operator new(unsigned int, unsigned int)+0x10d
V [libjvm.so+0x14dde9] Arena::grow(unsigned int)+0x29
V [libjvm.so+0x3d5fe2] PhaseIFG::init(unsigned int)+0x1c2
V [libjvm.so+0x23835c] PhaseChaitin::Register_Allocate()+0x75c
V [libjvm.so+0x2a2991] Compile::Code_Gen()+0x3b1
V [libjvm.so+0x2a5250] Compile::Compile(ciEnv*, C2Compiler*, ciMethod*, int, bool, bool)+0xbd0
V [libjvm.so+0x220f16] C2Compiler::compile_method(ciEnv*, ciMethod*, int)+0x176
V [libjvm.so+0x2a9f6a] CompileBroker::invoke_compiler_on_method(CompileTask*)+0x33a
V [libjvm.so+0x2ab037] CompileBroker::compiler_thread_loop()+0x417
V [libjvm.so+0x6ddd98] compiler_thread_entry(JavaThread*, Thread*)+0x18
V [libjvm.so+0x6e4944] JavaThread::thread_main_inner()+0xf4
V [libjvm.so+0x6e4ad1] JavaThread::run()+0x161
V [libjvm.so+0x5e1091] java_start(Thread*)+0x111
C [libpthread.so.0+0x6a2e] abort@@GLIBC_2.0+0x6a2e
阿里雲 Java環境 每小時在根目錄下生成 hs_err_pid*.log ，求教！
錯誤日志
OS環境：阿里雲 1核 1G內存；
安裝軟件：安裝了 JDK 1.7 TOMCAT 7.X；
部署內容：部署了 java web 工程，工程正常運行；
問題：每小時在跟目錄下生成一個 hs_err_pid.log
已經采取的措施：
1、配置服務器Swap
2、配置tomcat jvm 內存 （bin/catalina.sh）
未果，求牛人指點。
文件內容如下：
#
There is insufficient memory for the Java Runtime Environment to continue.
pthread_getattr_np
Possible reasons:
The system is out of physical RAM or swap space
In 32 bit mode, the process size limit was hit
Possible solutions:
Reduce memory load on the system
Increase physical memory or swap space
Check if swap backing store is full
Use 64 bit Java on a 64 bit OS
Decrease Java heap size (-Xmx/-Xms)
Decrease number of Java threads
Decrease Java thread stack sizes (-Xss)
Set larger code cache with -XX:ReservedCodeCacheSize=
This output file may be truncated or incomplete.
#
Out of Memory Error (os_linux_x86.cpp:718), pid=29229, tid=140505843455744
#
JRE version: Java(TM) SE Runtime Environment (7.0_67-b01) (build 1.7.0_67-b01)
Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 compressed oops)
Failed to write core dump. Core dumps have been disabled. To enable core dumping, try “ulimit -c unlimited” before starting Java again
#
————— T H R E A D —————
Current thread (0x00007fca0c08a800): JavaThread “Service Thread” daemon [_thread_new, id=29237, stack(0x0000000000000000,0x0000000000000000)]
Stack: [0x0000000000000000,0x0000000000000000], sp=0x00007fca10e34880, free space=137212737746k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x99eb8a]
V [libjvm.so+0x49721b]
V [libjvm.so+0x8237ca]
V [libjvm.so+0x823815]
V [libjvm.so+0x952614]
V [libjvm.so+0x958dd4]
V [libjvm.so+0x81f988]
————— P R O C E S S —————
Java Threads: ( => current thread )
=>0x00007fca0c08a800 JavaThread “Service Thread” daemon [_thread_new, id=29237, stack(0x0000000000000000,0x0000000000000000)]
0x00007fca0c088000 JavaThread “C2 CompilerThread1” daemon [_thread_blocked, id=29236, stack(0x00007fca10e36000,0x00007fca10f37000)]
0x00007fca0c085800 JavaThread “C2 CompilerThread0” daemon [_thread_blocked, id=29235, stack(0x00007fca10f37000,0x00007fca11038000)]
0x00007fca0c084000 JavaThread “Signal Dispatcher” daemon [_thread_blocked, id=29234, stack(0x00007fca11038000,0x00007fca11139000)]
0x00007fca0c064800 JavaThread “Finalizer” daemon [_thread_blocked, id=29233, stack(0x00007fca11139000,0x00007fca1123a000)]
0x00007fca0c062800 JavaThread “Reference Handler” daemon [_thread_blocked, id=29232, stack(0x00007fca1123a000,0x00007fca1133b000)]
0x00007fca0c008800 JavaThread “main” [_thread_in_vm, id=29230, stack(0x00007fca12195000,0x00007fca12296000)]
Other Threads:
0x00007fca0c05e000 VMThread [stack: 0x00007fca1133b000,0x00007fca1143c000] [id=29231]
VM state:not at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x00007fca0c006bc0] PeriodicTask_lock - owner thread: 0x00007fca0c008800
Heap
def new generation total 4800K, used 343K [0x00000000eb400000, 0x00000000eb930000, 0x00000000f0750000)
eden space 4288K, 8% used [0x00000000eb400000, 0x00000000eb455e40, 0x00000000eb830000)
from space 512K, 0% used [0x00000000eb830000, 0x00000000eb830000, 0x00000000eb8b0000)
to space 512K, 0% used [0x00000000eb8b0000, 0x00000000eb8b0000, 0x00000000eb930000)
tenured generation total 10624K, used 0K [0x00000000f0750000, 0x00000000f11b0000, 0x00000000fae00000)
the space 10624K, 0% used [0x00000000f0750000, 0x00000000f0750000, 0x00000000f0750200, 0x00000000f11b0000)
compacting perm gen total 21248K, used 2132K [0x00000000fae00000, 0x00000000fc2c0000, 0x0000000100000000)
the space 21248K, 10% used [0x00000000fae00000, 0x00000000fb015148, 0x00000000fb015200, 0x00000000fc2c0000)
No shared spaces configured.
Card table byte_map: [0x00007fca117bd000,0x00007fca11864000] byte_map_base: 0x00007fca11063000
Polling page: 0x00007fca1332b000
Code Cache [0x00007fca09000000, 0x00007fca09270000, 0x00007fca0c000000)
total_blobs=156 nmethods=0 adapters=126 free_code_cache=48779Kb largest_free_block=49950080
Compilation events (0 events):
No events
GC Heap History (0 events):
No events
Deoptimization events (0 events):
No events
Internal exceptions (1 events):
Event: 0.027 Thread 0x00007fca0c008800 Threw 0x00000000eb410380 at /HUDSON/workspace/7u-2-build-linux-amd64/jdk7u67/1368/hotspot/src/share/vm/prims/jni.cpp:3991
Events (10 events):
Event: 0.058 loading class 0x00007fca11701880
Event: 0.058 loading class 0x00007fca11701880 done
Event: 0.058 loading class 0x00007fca1170df80
Event: 0.058 loading class 0x00007fca1170df80 done
Event: 0.058 loading class 0x00007fca116c8ee0
Event: 0.058 loading class 0x00007fca116c8ee0 done
Event: 0.058 Thread 0x00007fca0c084000 Thread added: 0x00007fca0c084000
Event: 0.058 Thread 0x00007fca0c085800 Thread added: 0x00007fca0c085800
Event: 0.059 Thread 0x00007fca0c088000 Thread added: 0x00007fca0c088000
Event: 0.059 Thread 0x00007fca0c08a800 Thread added: 0x00007fca0c08a800
…..
資源暫時不可用錯誤(Out of memery)完美解決方案
首先說一下環境：
centos6.3
業務環境：weiboyi
情況說明一下：
為了適應公司業務發展需要，公司新購買了一批新機器，作為weiboyi的新環境，安裝了比較新的操作系統：通統：centos6.3
一開始硬件（Dell R410）主板壞了，更換后再進行各種測試。
測試前是用root用戶啟動業務，沒有什么問題。但是后來在用非root用戶啟動時出現種問題。java報如下錯誤：
There is insufficient memory for the Java Runtime Environment to continue.
Cannot create GC thread. Out of system resources.
Possible reasons:
The system is out of physical RAM or swap space
In 32 bit mode, the process size limit was hit
Possible solutions:
Reduce memory load on the system
Increase physical memory or swap space
Check if swap backing store is full
Use 64 bit Java on a 64 bit OS
Decrease Java heap size (-Xmx/-Xms)
Decrease number of Java threads
Decrease Java thread stack sizes (-Xss)
Set larger code cache with -XX:ReservedCodeCacheSize=
This output file may be truncated or incomplete.
#
Out of Memory Error (gcTaskThread.cpp:46), pid=50128, tid=140649625663232
同時運行java -version也報錯
執行一個ls也會報資源暫時不足
害得我又是查看系統的/etc/security/limit.conf 又是搞/etc/sysctl.conf 然后換JDK版本等等，搞得精疲力盡，到處找相關的資料！
搞了幾天沒有搞定，今天再接着進行這方面的研究，在以前的生產環境：centos5.5下可以正常使用的系統參數時突然發現：新系統的max user processes 只有1024 而舊系統默認為278528
你M呀，問題是出在這里了，再在/etc/security/下一看。centos6多出來一個limits.d目錄，下面有個文件： 90-nproc.config
此文件內容：
Default limit for number of user’s processes to prevent
accidental fork bombs.
See rhbz #432903 for reasoning.
- soft nproc 1024
這里限制了1024呀，果斷注釋。
再運行業務程序，問題解決！汗一下，centos6加這個搞什么呢，沒有搞清楚！