最近發現kunshan這家的生產環境進程mycat cpu 800%然后mycat基本上就假死了，無法查詢了。
后來回憶部署這家生產環境的時候，因為服務器內存大於等於64G 然后mycat 就啟動不了，然后配置了
<property name="useOffHeapForMerge">0</property> 然后就可以啟動了
這就是上面mycat cpu 800%的原因
這種方法能夠解決了筆者啟動報錯問題，但是禁用堆外內存犧牲性能，並沒有從根本上解決問題
后來找了網上一個文章，說的是將 wrapper.java.additional.5=-XX:MaxDirectMemorySize=2G改為8G加大MaxDirectMemorySize 內存，再次啟動，問題得到解決。
https://blog.csdn.net/u013716179/article/details/89886452
以前在進行Mycat調優的時候設置過Mycat的useOffHeapForMerge參數，發現在大數據量查詢聚合的時候啟用堆外內存對於查詢性能提升非常明顯，但是最近在新的生產環境部署Mycat啟動的時候總是報錯java.lang.NumberFormatException: Size must be specified as bytes (b), kibibytes (k), mebibytes (m), gibibytes (g), tebibytes (t), or pebibytes§. E.g. 50b, 100k, or 250m.
Failed to parse byte string: -1315333734B…類似這種，如下
2019-05-06 16:26:27.082 ERROR [WrapperSimpleAppMain] (io.mycat.MycatStartup.main(MycatStartup.java:63)) - 2019-05-06 16:26:27 startup error
java.lang.NumberFormatException: Size must be specified as bytes (b), kibibytes (k), mebibytes (m), gibibytes (g), tebibytes (t), or pebibytes(p). E.g. 50b, 100k, or 250m.
Failed to parse byte string: -1315333734B
at io.mycat.memory.unsafe.utils.JavaUtils.byteStringAs(JavaUtils.java:223) ~[Mycat-server-1.6.7.1-release.jar:?]
at io.mycat.memory.unsafe.utils.JavaUtils.byteStringAsBytes(JavaUtils.java:234) ~[Mycat-server-1.6.7.1-release.jar:?]
at io.mycat.memory.unsafe.utils.MycatPropertyConf.byteStringAsBytes(MycatPropertyConf.java:92) ~[Mycat-server-1.6.7.1-release.jar:?]
at io.mycat.memory.unsafe.utils.MycatPropertyConf.getSizeAsBytes(MycatPropertyConf.java:50) ~[Mycat-server-1.6.7.1-release.jar:?]
at io.mycat.memory.unsafe.memory.mm.MemoryManager.<init>(MemoryManager.java:30) ~[Mycat-server-1.6.7.1-release.jar:?]
at io.mycat.memory.unsafe.memory.mm.ResultMergeMemoryManager.<init>(ResultMergeMemoryManager.java:15) ~[Mycat-server-1.6.7.1-release.jar:?]
at io.mycat.memory.MyCatMemory.<init>(MyCatMemory.java:125) ~[Mycat-server-1.6.7.1-release.jar:?]
at io.mycat.MycatServer.startup(MycatServer.java:388) ~[Mycat-server-1.6.7.1-release.jar:?]
at io.mycat.MycatStartup.main(MycatStartup.java:58) ~[Mycat-server-1.6.7.1-release.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_161]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_161]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161]
at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:240) ~[wrapper.jar:3.2.3]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
對於此異常，筆者上網查詢過解決方案，幾乎全部是將useOffHeapForMerge置0
<property name="useOffHeapForMerge">1</property>
1
確實，這種方法能夠解決了筆者啟動報錯問題，但是禁用堆外內存犧牲性能，並沒有從根本上解決問題
至此，只能去github拉取mycat源碼看下問題究竟出在哪里，直接源碼里搜異常堆棧信息，找到包異常的地方在JavaUtils.java類的byteStringAs方法中
/**
* Convert a passed byte string (e.g. 50b, 100kb, or 250mb) to the given. If no suffix is
* provided, a direct conversion to the provided unit is attempted.
*/
public static long byteStringAs(String str, ByteUnit unit) {
String lower = str.toLowerCase().trim();
try {
Matcher m = Pattern.compile("([0-9]+)([a-z]+)?").matcher(lower);
Matcher fractionMatcher = Pattern.compile("([0-9]+\\.[0-9]+)([a-z]+)?").matcher(lower);
if (m.matches()) {
long val = Long.parseLong(m.group(1));
String suffix = m.group(2);
// Check for invalid suffixes
if (suffix != null && !byteSuffixes.containsKey(suffix)) {
throw new NumberFormatException("Invalid suffix: \"" + suffix + "\"");
}
// If suffix is valid use that, otherwise none was provided and use the default passed
return unit.convertFrom(val, suffix != null ? byteSuffixes.get(suffix) : unit);
} else if (fractionMatcher.matches()) {
throw new NumberFormatException("Fractional values are not supported. Input was: "
+ fractionMatcher.group(1));
} else {
throw new NumberFormatException("Failed to parse byte string: " + str);
}
} catch (NumberFormatException e) {
String byteError = "Size must be specified as bytes (b), " +
"kibibytes (k), mebibytes (m), gibibytes (g), tebibytes (t), or pebibytes(p). " +
"E.g. 50b, 100k, or 250m.";
throw new NumberFormatException(byteError + "\n" + e.getMessage());
}
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
首先我們可以看到，str解析異常會有三種情況，我們報的是最后一種即else里面的Failed to parse byte string，那么看看日志我們傳進來str是’-1315333734B’，首先會在方法的第一部分轉小寫，其次前兩個正則匹配整數或者小數+字母，前兩種均沒匹配上，出現最后一種情況原因只能是字符串開頭非數字，且是一個負數，那為什么會出現負數呢，我們接着看str如何生成的，進入MycatMemory類
public MyCatMemory(SystemConfig system,long totalNetWorkBufferSize) throws NoSuchFieldException, IllegalAccessException {
this.system = system;
LOGGER.info("useOffHeapForMerge = " + system.getUseOffHeapForMerge());
LOGGER.info("memoryPageSize = " + system.getMemoryPageSize());
LOGGER.info("spillsFileBufferSize = " + system.getSpillsFileBufferSize());
LOGGER.info("useStreamOutput = " + system.getUseStreamOutput());
LOGGER.info("systemReserveMemorySize = " + system.getSystemReserveMemorySize());
LOGGER.info("totalNetWorkBufferSize = " + JavaUtils.bytesToString2(totalNetWorkBufferSize));
LOGGER.info("dataNodeSortedTempDir = " + system.getDataNodeSortedTempDir());
this.conf = new MycatPropertyConf();
numCores = Runtime.getRuntime().availableProcessors();
this.systemReserveBufferSize = JavaUtils.
byteStringAsBytes(system.getSystemReserveMemorySize());
this.memoryPageSize = JavaUtils.
byteStringAsBytes(system.getMemoryPageSize());
this.spillsFileBufferSize = JavaUtils.
byteStringAsBytes(system.getSpillsFileBufferSize());
/**
* 目前merge，order by ，limit 沒有使用On Heap內存
*/
long maxOnHeapMemory = (Platform.getMaxHeapMemory()-systemReserveBufferSize);
assert maxOnHeapMemory > 0;
resultSetBufferSize =
(long)((Platform.getMaxDirectMemory()-2*totalNetWorkBufferSize)*DIRECT_SAFETY_FRACTION);
assert resultSetBufferSize > 0;
/**
* mycat.merge.memory.offHeap.enabled
* mycat.buffer.pageSize
* mycat.memory.offHeap.size
* mycat.merge.file.buffer
* mycat.direct.output.result
* mycat.local.dir
*/
if(system.getUseOffHeapForMerge()== 1){
conf.set("mycat.memory.offHeap.enabled","true");
}else{
conf.set("mycat.memory.offHeap.enabled","false");
}
if(system.getUseStreamOutput() == 1){
conf.set("mycat.stream.output.result","true");
}else{
conf.set("mycat.stream.output.result","false");
}
if(system.getMemoryPageSize() != null){
conf.set("mycat.buffer.pageSize",system.getMemoryPageSize());
}else{
conf.set("mycat.buffer.pageSize","32k");
}
if(system.getSpillsFileBufferSize() != null){
conf.set("mycat.merge.file.buffer",system.getSpillsFileBufferSize());
}else{
conf.set("mycat.merge.file.buffer","32k");
}
conf.set("mycat.pointer.array.len","1k")
.set("mycat.memory.offHeap.size", JavaUtils.bytesToString2(resultSetBufferSize));
LOGGER.info("mycat.memory.offHeap.size: " +
JavaUtils.bytesToString2(resultSetBufferSize));
resultMergeMemoryManager =
new ResultMergeMemoryManager(conf,numCores,maxOnHeapMemory);
serializerManager = new SerializerManager();
blockManager = new DataNodeDiskManager(conf,true,serializerManager);
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
可以看到，是根據Platform.getMaxDirectMemory()、totalNetWorkBufferSize、DIRECT_SAFETY_FRACTION三個參數程程的
resultSetBufferSize =
(long)((Platform.getMaxDirectMemory()-2*totalNetWorkBufferSize)*DIRECT_SAFETY_FRACTION);
1
2
而產生負數的原因無外乎前面值太小，或者后面值太大，而Platform.getMaxDirectMemory()取得值是MAX_DIRECT_MEMORY，這是wrapper.conf種的一個參數
#********************************************************************
# Wrapper Properties
#********************************************************************
# Java Application
wrapper.java.command=java
wrapper.working.dir=..
# Java Main class. This class must implement the WrapperListener interface
# or guarantee that the WrapperManager class is initialized. Helper
# classes are provided to do this for you. See the Integration section
# of the documentation for details.
wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperSimpleApp
set.default.REPO_DIR=lib
set.APP_BASE=.
# Java Classpath (include wrapper.jar) Add class path elements as
# needed starting from 1
wrapper.java.classpath.1=lib/wrapper.jar
wrapper.java.classpath.2=conf
wrapper.java.classpath.3=%REPO_DIR%/*
# Java Library Path (location of Wrapper.DLL or libwrapper.so)
wrapper.java.library.path.1=lib
# Java Additional Parameters
#wrapper.java.additional.1=
wrapper.java.additional.1=-DMYCAT_HOME=.
wrapper.java.additional.2=-server
wrapper.java.additional.3=-XX:MaxPermSize=64M
wrapper.java.additional.4=-XX:+AggressiveOpts
wrapper.java.additional.5=-XX:MaxDirectMemorySize=2G
wrapper.java.additional.6=-Dcom.sun.management.jmxremote
wrapper.java.additional.7=-Dcom.sun.management.jmxremote.port=1984
wrapper.java.additional.8=-Dcom.sun.management.jmxremote.authenticate=false
wrapper.java.additional.9=-Dcom.sun.management.jmxremote.ssl=false
wrapper.java.additional.10=-Xmx4G
wrapper.java.additional.11=-Xms1G
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
這里筆者將wrapper.java.additional.5=-XX:MaxDirectMemorySize=2G改為8G加大MaxDirectMemorySize內存，再次啟動，問題得到解決。
————————————————
版權聲明：本文為CSDN博主「張老七沒脾氣」的原創文章，遵循CC 4.0 BY-SA版權協議，轉載請附上原文出處鏈接及本聲明。
原文鏈接：https://blog.csdn.net/u013716179/article/details/89886452