早上同事反映，mycat又假死了，估計還是內存溢出，查看了一下錯誤日志。
INFO | jvm 1 | 2016/03/25 07:09:06 | java.lang.OutOfMemoryError: Java heap space INFO | jvm 1 | 2016/03/25 07:09:06 | Exception in thread "Timer67" java.lang.OutOfMemoryError: Java heap space INFO | jvm 1 | 2016/03/25 07:09:06 | Exception in thread "Timer68" java.lang.OutOfMemoryError: Java heap space INFO | jvm 1 | 2016/03/25 07:09:06 | Exception in thread "Timer69" java.lang.OutOfMemoryError: Java heap space INFO | jvm 1 | 2016/03/25 07:09:06 | Exception in thread "Timer70" java.lang.OutOfMemoryError: Java heap space INFO | jvm 1 | 2016/03/25 07:09:06 | Exception in thread "Timer71" java.lang.OutOfMemoryError: Java heap space INFO | jvm 1 | 2016/03/25 07:09:06 | Exception in thread "Timer72" java.lang.OutOfMemoryError: Java heap space INFO | jvm 1 | 2016/03/25 07:09:07 | Exception in thread "Timer74" java.lang.OutOfMemoryError: Java heap space INFO | jvm 1 | 2016/03/25 07:09:07 | Exception in thread "Timer73" java.lang.OutOfMemoryError: Java heap space
堆內存溢出，以為是內存不足，但與前幾天的報錯又不一樣
INFO | jvm 1 | 2016/03/23 15:38:35 | Exception in thread "$_MyCatServer" java.lang.OutOfMemoryError: Direct buffer memory INFO | jvm 1 | 2016/03/23 15:38:35 | at java.nio.Bits.reserveMemory(Bits.java:658) INFO | jvm 1 | 2016/03/23 15:38:35 | at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) INFO | jvm 1 | 2016/03/23 15:38:35 | at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
而且，通過查看wrapper.conf，發現MaxDirectMemorySize已經設置為10G了，剛開始一直按默認2G來，偶爾會出現內存溢出。
wrapper.java.additional.5=-XX:MaxDirectMemorySize=10G
后來，一高人指點，通過jmap -heap pid命令查看，發現MaxHeapSize=16MB
Heap Configuration: MinHeapFreeRatio = 0 MaxHeapFreeRatio = 100 MaxHeapSize = 16777216 (16.0MB) NewSize = 1310720 (1.25MB) MaxNewSize = 17592186044415 MB OldSize = 5439488 (5.1875MB) NewRatio = 2 SurvivorRatio = 8 PermSize = 21757952 (20.75MB) MaxPermSize = 536870912 (512.0MB) G1HeapRegionSize = 0 (0.0MB)
不科學啊，wrapper.conf中有關堆內存的設置沒這么小啊
wrapper.java.additional.10=-Xmx4G wrapper.java.additional.11=-Xms1G # Initial Java Heap Size (in MB) wrapper.java.initmemory=4G # Maximum Java Heap Size (in MB) wrapper.java.maxmemory=15G
最后才發現，問題出在wrapper.java.initmemory=4G和wrapper.java.maxmemory=15G上了。
原來，這兩個參數直接設置數值就可以了，因為它的單位已經已經指明了，為MB。
如果設置為15G，jvm會直接屏蔽掉后面的G，使用15，然后MaxHeapSize將會被設置為16MB（后來測試了一下，如果設置為17，MaxHeapSize則會被設置為18MB，至於為什么每次都分配為偶數，這個就不太清楚了）。
后來測試了一下，得出以下結論：
1. wrapper.java.maxmemory參數會覆蓋wrapper.java.additional.10參數。
2. 即便wrapper.java.maxmemory的值超過了4096，MaxHeapSize最大為4096MB。
3. 如果堆大小要超過4096MB，可設置wrapper.java.additional.10，同時注釋掉wrapper.java.maxmemory。
建議，只使用wrapper.java.additional.10=-Xmx4G和wrapper.java.additional.11=-Xms1G配置堆內存的大小。
總結：
mycat的啟動方式有兩種
1. mycat/bin/mycat start
這種方式會使用mycat/conf/wrapper.conf作為配置文件
通過 sh -x mycat start即可發現
+ eval '"/usr/local/mycat/bin/./wrapper-linux-x86-64"' '"/usr/local/mycat/conf/wrapper.conf"' wrapper.syslog.ident=mycat 'wrapper.pidfile="/usr/local/mycat/logs/mycat.pid"' wrapper.daemonize=TRUE 'wrapper.lockfile="/var/lock/subsys/mycat"' ++ /usr/local/mycat/bin/./wrapper-linux-x86-64 /usr/local/mycat/conf/wrapper.conf wrapper.syslog.ident=mycat wrapper.pidfile=/usr/local/mycat/logs/mycat.pid wrapper.daemonize=TRUE wrapper.lockfile=/var/lock/subsys/myca
2. mycat/bin/startup_nowrap.sh
這個腳本里面同樣有關於jvm的配置
JAVA_OPTS="-server -Xms2G -Xmx2G -XX:MaxPermSize=64M -XX:+AggressiveOpts -XX:MaxDirectMemorySize=2G"
# ./startup_nowrap.sh "/usr/local/jdk1.7.0_79/bin/java" -DMYCAT_HOME="/usr/local/mycat" -classpath "/usr/local/mycat/conf:/usr/local/mycat/lib/classes:/usr/local/mycat/lib/curator-client-2.9.0.jar:/usr/local/mycat/lib/curator-framework-2.9.0.jar:/usr/local/mycat/lib/dom4j-1.6.1.jar:/usr/local/mycat/lib/druid-1.0.14.jar:/usr/local/mycat/lib/ehcache-core-2.6.11.jar:/usr/local/mycat/lib/fastjson-1.2.7.jar:/usr/local/mycat/lib/guava-18.0.jar:/usr/local/mycat/lib/jline-0.9.94.jar:/usr/local/mycat/lib/json-20151123.jar:/usr/local/mycat/lib/leveldb-0.7.jar:/usr/local/mycat/lib/leveldb-api-0.7.jar:/usr/local/mycat/lib/log4j-1.2.17.jar:/usr/local/mycat/lib/mapdb-1.0.7.jar:/usr/local/mycat/lib/mongo-java-driver-2.11.4.jar:/usr/local/mycat/lib/Mycat-server-1.5-GA.jar:/usr/local/mycat/lib/netty-3.7.0.Final.jar:/usr/local/mycat/lib/ojdbc14-10.2.0.3.0.jar:/usr/local/mycat/lib/sequoiadb-java-driver-1.0-20150615.070208-1.jar:/usr/local/mycat/lib/slf4j-api-1.7.12.jar:/usr/local/mycat/lib/slf4j-log4j12-1.7.12.jar:/usr/local/mycat/lib/snakeyaml-1.16.jar:/usr/local/mycat/lib/univocity-parsers-1.5.4.jar:/usr/local/mycat/lib/wrapper.jar:/usr/local/mycat/lib/xml-apis-1.0.b2.jar:/usr/local/mycat/lib/zookeeper-3.4.6.jar" -server -Xms2G -Xmx2G -XX:MaxPermSize=64M -XX:+AggressiveOpts -XX:MaxDirectMemorySize=2G org.opencloudb.MycatStartup >> "/usr/local/mycat/logs/console.log" 2>&1 &
后續-20160617
--------------------------------------------------------
通過mycat start啟動失敗，wrapper.log有如下報錯信息：
STATUS | wrapper | 2016/06/17 15:26:16 | Launching a JVM... ERROR | wrapper | 2016/06/17 15:26:46 | Startup failed: Timed out waiting for a signal from the JVM. ERROR | wrapper | 2016/06/17 15:26:46 | JVM did not exit on request, terminated INFO | wrapper | 2016/06/17 15:26:46 | JVM exited on its own while waiting to kill the application. STATUS | wrapper | 2016/06/17 15:26:46 | JVM exited in response to signal SIGKILL (9).
后來通過startup_nowrap.sh，發現可以啟動成功。
在網上找了一把，可修改conf/wrapper.conf中設置如下參數：
#設置超時時間為2小時wrapper.startup.timeout=7200
#設置心跳超時時間為1小時wrapper.ping.timeout=3600
重新通過mycat start啟動，啟動成功~