使用-XX:+TraceClassPaths或者在服務器上執行jinfo時，都能得到classpath包含的jar包，例如：
java.class.path = local/aaa/lib/spring-data-redis-1.8.3.RELEASE.jar:/usr/local/aaa/lib/spring-tx-4.3.8.RELEASE.jar:/usr/local/aaa/lib/spring-jdbc-4.3.7.RELEASE.jar:/usr/local/aaa/lib/classmate-1.3.1.jar:/usr/local/aaa/lib/javax.servlet-api-3.1.0.jar:/usr/local/aaa/lib/mongodb-driver-3.4.2.jar:/usr/local/aaa/lib/xml-apis-2.0.2.jar:/usr/local/aaa/lib/ufc-api-utils-2.0.0.jar:/usr/local/aaa/lib/log4j-over-slf4j-1.7.25.jar:/usr/local/aaa/lib/tomcat-embed-websocket-8.5.14.jar:...
這些jar的順序不同的機器總是不一樣的，平時沒有問題，所以也沒有細想過，這些jar包的順序為什么會不一樣的。 在之前排查的一個問題 的結尾還留了一個問題，為什么有的機器會加載正確的類，有的就是錯的。因為這一段在上線一個項目，灰度公測階段，所以拖了些天，穿插着看了看加載相關的一些hotspot代碼，以前也沒看過，就邊猜測邊看了。因為是穿插着看，怕今天看的明天就忘了，所以博客結尾連接中流水賬記錄了我看的過程。
總之，經過一番查代碼，最終在rt.jar中找到了JarFile這個類，代碼在jdk的src.zip包里的。 用 https://github.com/saaavsaaa/warn-report/blob/master/src/main/java/report/btrace/JarBTrace.java 腳本跟蹤了一下：
正常的服務器腳本的輸出：
異常的服務器上腳本輸出：
這個問題在多台服務器上都出現了，所以搜集多台的輸出后可以確認這個規律，其實在結尾連接中代碼記錄里也可以看出，如果有多個同名的類只會加載其中第一個。在不出問題的服務器上commons-codec-1.10.jar是在http-1.1.0.jar之前加載的，而出問題的服務器正好相反。
/usr/lib/jvm/java-8-oracle/jre/lib/rt.jar!/sun/misc/URLClassPath.class中的private ArrayList path有所有加載的jar的URL。這個類里還有每個jar的loader：ArrayList<URLClassPath.Loader> loaders，jar路徑和jarLoader的映射：HashMap<String, URLClassPath.Loader> lmap。lmap通過private synchronized URLClassPath.Loader getLoader(int var1)方法從Stack urls中彈出的jar來構造loader並加入映射的。有兩個URLClassLoader加載不同的URLClassPath實例，一個只有jre/lib下的幾個jar，一個有所有的。不過這都不重要，這些jar都是從classpath加載的。於是我在幾個服務器上用jinfo輸出了java.class.path，並對比了一下發現正常服務器和服務器中，這兩個jar的順序果然是不一樣的。那么我估計問題是出現jvm對java.class.path賦值的前面了，或許是操作系統什么的。
於是決定看看java.class.path初始化的情況，代碼在/home/aaa/Github/hotspot/src/share/vm/runtime/arguments.cpp:_java_class_path = new SystemProperty("java.class.path", "", true)。SystemProperty的構造在/home/aaa/Github/hotspot/src/share/vm/runtime/arguments.hpp：
// Constructor
SystemProperty(const char* key, const char* value, bool writeable) {
if (key == NULL) {
_key = NULL;
} else {
_key = AllocateHeap(strlen(key)+1, mtInternal);
strcpy(_key, key);
}
if (value == NULL) {
_value = NULL;
} else {
_value = AllocateHeap(strlen(value)+1, mtInternal);
ba(_value, value);
}
_next = NULL;
_writeable = writeable;
}
};
構造里似乎沒什么關系，然后回到cpp仔細看了下，發現：
// following are JVMTI agent writeable properties.
// Properties values are set to NULL and they are
// os specific they are initialized in os::init_system_properties_values().
// Set OS specific system properties values
os::init_system_properties_values();
這個方法的定義在os.hpp中，不過實現是不同的系統不一樣，所以並沒有在對應的cpp中，我是linux系統，所以我去找了 /home/aaa/Github/hotspot/src/os/linux/vm/os_linux.cpp，不過也沒有我想要的。arguments.cpp中只找到了對endorsed，ext，bootclasspath的目錄文件讀取。
然后找到了-XX:+TraceClassPaths參數的輸出位置： [classpath: ...]的調用鏈：
/home/aaa/Github/hotspot/src/share/vm/runtime/thread.cpp:
// Parse arguments
jint parse_result = Arguments::parse(args);
// Parse JAVA_TOOL_OPTIONS environment variable (if present)
jint result = parse_java_tool_options_environment_variable(&scp, &scp_assembly_required);
if (result != JNI_OK) {
return result;
}
// Parse JavaVMInitArgs structure passed in
result = parse_each_vm_init_arg(args, &scp, &scp_assembly_required, Flag::COMMAND_LINE);
if (result != JNI_OK) {
return result;
}
// Parse _JAVA_OPTIONS environment variable (if present) (mimics classic VM)
result = parse_java_options_environment_variable(&scp, &scp_assembly_required); if (result != JNI_OK) { return result; }
parse_java_tool_options_environment_variable和parse_java_options_environment_variable都有調用parse_each_vm_init_arg，這里提一句_JAVA_OPTIONS會覆蓋JAVA_TOOL_OPTIONS的同key配置。
parse_each_vm_init_arg這方法一進來就看到這么一段略坑，不過也無所謂了響。。。
if (!match_option(option, "-Djava.class.path", &tail) &&
!match_option(option, "-Dsun.java.command", &tail) &&
!match_option(option, "-Dsun.java.launcher", &tail)) {
// add all jvm options to the jvm_args string. This string
// is used later to set the java.vm.args PerfData string constant.
// the -Djava.class.path and the -Dsun.java.command options are
// omitted from jvm_args string as each have their own PerfData
// string constant object.
build_jvm_args(option->optionString);
}
這就是打印上面那句的地方：
jint Arguments::parse_each_vm_init_arg
Arguments::fix_appclasspath():
if (!PrintSharedArchiveAndExit) {
ClassLoader::trace_class_path(tty, "[classpath: ", _java_class_path->value());
}
然后[Bootstrap loader class path=/usr/lib/jvm/java-8-oracle/jre/lib/resources.jar:/usr/lib/jvm/java-8-oracle/jre/lib/rt.jar:/usr/lib/jvm/java-8-oracle/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-8-oracle/jre/lib/jsse.jar:/usr/lib/jvm/java-8-oracle/jre/lib/jce.jar:/usr/lib/jvm/java-8-oracle/jre/lib/charsets.jar:/usr/lib/jvm/java-8-oracle/jre/lib/jfr.jar:/usr/lib/jvm/java-8-oracle/jre/classes]這一句輸出:
/home/aaa/Github/hotspot/src/share/vm/runtime/init.cpp
jint init_globals()
/home/aaa/Github/hotspot/src/share/vm/classfile/classLoader.cpp
void classLoader_init() {
ClassLoader::initialize();
}
void ClassLoader::initialize()
void ClassLoader::setup_bootstrap_search_path()
大體上代碼都在這一部分了，可以看出代碼中並沒有對加載順序有所改變，然后比較重要的線索就是加載方法用的是opendir和readdir函數，在ios同事的協助下，終於確定了，問題就是jar的加載順序問題，而這個順序實際上是由文件系統決定的，linux內部是用inode來指示文件的。
於是寫了個簡單的c方法來驗證，打印inode號：https://github.com/saaavsaaa/warn-report/blob/master/src/main/java/report/main.c
d_off:4066763678044974396 d_name: commons-codec-1.10.jar
d_off:4317940688349827723 d_name: commons-lang-2.5.jar
d_off:4319988583919237504 d_name: jul-to-slf4j-1.7.25.jar
d_off:4356646752930279233 d_name: spring-aop-4.3.8.RELEASE.jar
d_off:4579877846802590742 d_name: tomcat-embed-core-8.5.14.jar
d_off:4731608207059974753 d_name: groovy-2.4.3.jar
d_off:4771541818858258978 d_name: jpush-client-3.2.9.jar
d_off:4817159055427520488 d_name: httpcore-4.3.jar
d_off:5037058976412869958 d_name: rocketmq-common-3.2.6.jar
d_off:5112026581585883935 d_name: xmlParserAPIs-2.6.2.jar
d_off:5223887631628650300 d_name: commons-fileupload-1.2.2.jar
d_off:5270962929984509613 d_name: logback-core-1.1.11.jar
d_off:5295594039709144434 d_name: jackson-core-2.8.8.jar
d_off:5313324231349868064 d_name: .
d_off:5369671282559728007 d_name: spring-webmvc-4.3.8.RELEASE.jar
d_off:5480616600783493234 d_name: xalan-2.6.0.jar
d_off:5598132018198955916 d_name: unbescape-1.1.0.RELEASE.jar
d_off:5620136379821311472 d_name: spring-boot-starter-aop-1.5.3.RELEASE.jar
d_off:5639942392021544761 d_name: mybatis-3.4.4.jar
d_off:5688537857783605585 d_name: validation-api-1.1.0.Final.jar
d_off:5689351964973998306 d_name: json-lib-2.4-jdk15.jar
d_off:5708471019688158398 d_name: tagsoup-0.9.7.jar
d_off:5716046632217600256 d_name: xom-1.0b3.jar
d_off:5736731302988875630 d_name: p2p-repository-1.0-SNAPSHOT.jar
d_off:5770969695350360533 d_name: ognl-3.0.8.jar
d_off:5946256519116188501 d_name: jackson-annotations-2.8.0.jar
d_off:6011005565981751131 d_name: jxl-2.6.jar
d_off:6121401373763401899 d_name: spring-data-mongodb-1.10.3.RELEASE.jar
d_off:6155887434083949861 d_name: icu4j-2.6.1.jar
d_off:6203694621577639938 d_name: jboss-logging-3.3.0.Final.jar
d_off:6241670413890043636 d_name: jaxme-api-0.3.jar
d_off:6317802240634228683 d_name: thymeleaf-2.1.5.RELEASE.jar
d_off:6362118033392674189 d_name: logback-classic-1.1.11.jar
d_off:6391821784225315730 d_name: snakeyaml-1.17.jar
d_off:6447926989912518869 d_name: javassist-3.16.1-GA.jar
d_off:6586996539318953387 d_name: spring-boot-1.5.3.RELEASE.jar
d_off:6682174700565688505 d_name: mybatis-spring-1.3.1.jar
d_off:6858079168157560275 d_name: http-1.1.0.jar
對照前面正常服務器的截圖可以發現，順序和這是一樣的。而出錯的服務器：
d_off:7500897893766328572 d_name: http-1.1.0.jar
d_off:7630237192571233449 d_name: jaxen-1.1-beta-4.jar
d_off:7846439931967980783 d_name: xom-1.0b3.jar
d_off:7986273690820996399 d_name: jxl-2.6.jar
d_off:8013065173263952359 d_name: spring-boot-starter-thymeleaf-1.5.3.RELEASE.jar
d_off:8231450206996036007 d_name: spring-context-support-4.3.8.RELEASE.jar
d_off:8471297127500795042 d_name: jpush-client-3.2.9.jar
d_off:8635726305307688944 d_name: commons-codec-1.10.jar
同樣可以對照上面錯誤服務器的截圖，而且一般情況下，修改了文件名，再改回來，或者從新上傳一個，這個編號依然還是這個，所以在出問題的服務器上問題穩定復現。那么，這個問題終於是可以告一段落了，原因搞清楚了，心理就有底了。解決什么的就隨便了，我就隨便把名字改成zzzhttp然后問題就在兩個包都存在的情況下解決了。
d_off:8635726305307688944 d_name: commons-codec-1.10.jar
d_off:8710228832141418589 d_name: xercesImpl-2.6.2.jar
d_off:8722513994996448409 d_name: android-json-0.0.20131108.vaadin1.jar
d_off:8754081964290049159 d_name: ufc-api-utils-2.0.0.jar
d_off:8830796801498266528 d_name: httpcore-4.3.jar
d_off:8854152647200610772 d_name: tomcat-jdbc-8.5.11.jar
d_off:8971846312288780129 d_name: spring-tx-4.3.8.RELEASE.jar
d_off:8986236906371055996 d_name: bcprov-jdk15-1.45.jar
d_off:8988385379950226997 d_name: mysql-connector-java-5.1.30.jar
d_off:8994464230154278010 d_name: p2p-common-1.0-SNAPSHOT.jar
d_off:9012703293696799571 d_name: mybatis-spring-boot-starter-1.3.0.jar
d_off:9057592519440006836 d_name: zzzhttp-1.1.0.jar
流水賬記錄：https://saaavsaaa.github.io/aaa/Java_Class_Path.html
微信公眾號：
