Tomcat通過自帶的Cluster方式實現Session會話共享環境操作記錄

 

一般來說,在多個tomcat集群業務中,session會話共享是必須的需求,不然前端nginx轉發過來的請求不知道之前請求在哪台tomcat節點上，從而就找不到session以至於最終導致請求失敗。要實現tomcat session共享有多種方案,今天介紹下使用tomcat自帶的cluster方式，在多個tomcat節點間自動實時復制session信息，配置起來很簡單。tomcat自帶的這種session共享方案,對於規模較小的tomcat集群來說夠用了，但這個方案的效率比較低，在大並發下表現並不好。所以大規模的tomcat集群還是建議使用memcache或者redis來實現session共享。

1）基礎環境

ip                 主機名           應用                       端口
192.168.10.200     Nginx-node       nginx1.12.2                80
192.168.10.201     Tomcat-node1     java8.131、tomcat8.0.53    8080
192.168.10.202     Tomcat-node2     java8.131、tomcat8.0.53    8080

下面操作在三台機器上同樣執行：
[root@Nginx-node ~]# cat /etc/redhat-release
CentOS release 6.9 (Final)
   
為了方便測試，關閉iptables防火牆和selinux。如果是生產環境，開啟iptables后，需要開放對應的應用端口。
[root@Nginx-node ~]# setenforce 0
[root@Nginx-node ~]# getenforce
disabled
[root@Nginx-node ~]# cat /etc/sysconfig/selinux |grep "SELINUX=disabled"
SELINUX=disabled
[root@Nginx-node ~]# /etc/init.d/iptables stop

本案例環境部署中所需的軟件下載地址：https://pan.baidu.com/s/1732gmtRpVh1zY_i4H85Ecw
提取密碼:ze9d
下載到服務器上的/usr/local/src目錄下.另外:節點服務器的系統時間一定要保持一致!!

2）安裝Nginx（在192.168.10.200機器上操作）

[root@Nginx-node ~]# cd /usr/local/src/
[root@Nginx-node src]# ll
total 8920
-rw-rw-r--. 1 root root  981687 Oct 27  2017 nginx-1.12.2.tar.gz
-rw-rw-r--. 1 root root 5453234 Aug 23  2018 openssl-1.1.0i.tar.gz
-rw-rw-r--. 1 root root 2081413 Aug 23  2018 pcre-8.42.tar.gz
-rw-rw-r--. 1 root root  607698 Jan 16  2017 zlib-1.2.11.tar.gz
 
安裝依賴包
[root@Nginx-node src]# yum -y install gcc gcc-c++
 
安裝pcre庫
[root@Nginx-node src]# tar -zvxf pcre-8.42.tar.gz
[root@Nginx-node src]# cd pcre-8.42
[root@Nginx-node pcre-8.42]# ./configure && make && make install
 
安裝zlib庫
[root@Nginx-node pcre-8.42]# cd /usr/local/src/
[root@Nginx-node src]# tar -zvxf zlib-1.2.11.tar.gz
[root@Nginx-node src]# cd zlib-1.2.11
[root@Nginx-node zlib-1.2.11]# ./configure && make && make install
 
安裝openssl
[root@Nginx-node zlib-1.2.11]# cd /usr/local/src/
[root@Nginx-node src]# tar -zvxf openssl-1.1.0i.tar.gz
[root@Nginx-node src]# cd openssl-1.1.0i
[root@Nginx-node openssl-1.1.0i]# ./config && make && make install
 
安裝nginx，特別注意要指定prce zlib openssl原碼包位置
[root@Nginx-node openssl-1.1.0i]# cd /usr/local/src/
[root@Nginx-node src]# tar -zvxf nginx-1.12.2.tar.gz
[root@Nginx-node src]# cd nginx-1.12.2                
[root@Nginx-node nginx-1.12.2]# ./configure --prefix=/usr/local/nginx --with-http_ssl_module --with-pcre=/usr/local/src/pcre-8.42 --with-zlib=/usr/local/src/zlib-1.2.11  --with-openssl=/usr/local/src/openssl-1.1.0i
[root@Nginx-node nginx-1.12.2]# make && make install
 
安裝成功后配置nginx
[root@Nginx-node nginx-1.12.2]# cd /usr/local/nginx/conf/
[root@Nginx-node conf]# cp nginx.conf nginx.conf.bak
[root@Nginx-node conf]# cat nginx.conf
#user  nobody;
worker_processes  8;
  
#error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;
  
#pid        logs/nginx.pid;
  
worker_rlimit_nofile 65535;
events {
   use epoll;
    worker_connections  65535;
}
  
http {
    include       mime.types;
    default_type  application/octet-stream;
    charset utf-8;
          
    ######
    ## set access log format
    ######
    log_format  main  '$http_x_forwarded_for $remote_addr $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_cookie" $host $request_time';
  
    #######
    ## http setting
    #######
    sendfile       on;
    tcp_nopush     on;
    tcp_nodelay    on;
    keepalive_timeout  65;
    fastcgi_connect_timeout 30000;
    fastcgi_send_timeout 30000;
    fastcgi_read_timeout 30000;
    fastcgi_buffer_size 256k;
    fastcgi_buffers 8 256k;
    fastcgi_busy_buffers_size 256k;
    fastcgi_temp_file_write_size 256k;
    fastcgi_intercept_errors on;
  
    ##cache##
    client_header_timeout 60s;
    client_body_timeout 60s;
    client_max_body_size 10m;
    client_body_buffer_size 1m;
    proxy_connect_timeout 5;
    proxy_read_timeout 60;
    proxy_send_timeout 5;             
    proxy_buffer_size 64k;
    proxy_buffers 4 128k;
    proxy_busy_buffers_size 128k;
    proxy_temp_file_write_size 1m;
    proxy_temp_path /home/temp_dir;
    proxy_cache_path /home/cache levels=1:2 keys_zone=cache_one:200m inactive=1d max_size=30g;
    ##end##
  
    gzip  on;
    gzip_min_length  1k;
    gzip_buffers     4 16k;
    gzip_http_version 1.1;
    gzip_comp_level 9;
    gzip_types       text/plain application/x-javascript text/css application/xml text/javascript application/x-httpd-php;
    gzip_vary on;
  
    ## includes vhosts
    include vhosts/*.conf;
}
 
[root@Nginx-node conf]# mkdir vhosts
[root@Nginx-node conf]# cd vhosts/
[root@Nginx-node vhosts]# vim lb_tomcat.conf
  upstream tomcat-lb {
      server 192.168.10.201:8080;
      server 192.168.10.202:8080;
      }
                    
      server {
      listen  80;
      server_name www.kevin.com;
      location / {
          proxy_pass http://tomcat-lb;
          proxy_set_header  X-Real-IP $remote_addr;
          proxy_set_header  REMOTE-HOST $remote_addr;
          proxy_set_header  Host $host;
          proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
        }
        location ~ .*\.(gif|jpg|png|htm|html|css|ico|flv|swf)(.*) {
              proxy_pass http://tomcat-lb;
              proxy_redirect off;
              proxy_set_header Host $host;
              proxy_cache cache_one;
              proxy_cache_valid 200 302 1h;
              proxy_cache_valid 301 1d;
              proxy_cache_valid any 10m;
              expires 30d;
              proxy_cache_key $host$uri$is_args$args;
        }
}
 
[root@Nginx-node vhosts]# /usr/local/nginx/sbin/nginx -t
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
[root@Nginx-node conf]# /usr/local/nginx/sbin/nginx
[root@Nginx-node conf]# lsof -i:80
COMMAND   PID   USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
nginx   25292   root    6u  IPv4 19679665      0t0  TCP *:http (LISTEN)
nginx   25293 nobody    6u  IPv4 19679665      0t0  TCP *:http (LISTEN)
nginx   25294 nobody    6u  IPv4 19679665      0t0  TCP *:http (LISTEN)
nginx   25295 nobody    6u  IPv4 19679665      0t0  TCP *:http (LISTEN)
nginx   25296 nobody    6u  IPv4 19679665      0t0  TCP *:http (LISTEN)
nginx   25297 nobody    6u  IPv4 19679665      0t0  TCP *:http (LISTEN)
nginx   25298 nobody    6u  IPv4 19679665      0t0  TCP *:http (LISTEN)
nginx   25299 nobody    6u  IPv4 19679665      0t0  TCP *:http (LISTEN)
nginx   25300 nobody    6u  IPv4 19679665      0t0  TCP *:http (LISTEN)
 
將域名www.kevin.com解析到192.168.10.200上，訪問http://www.kevin.com，發現訪問請求結果會負載到192.168.10.201和192.168.10.202的tomcat上了。

3）安裝tomcat（在192.168.10.201和192.168.10.202兩台機器上操作）

安裝java8環境。先卸載掉系統自帶的java7，然后安裝java8
[root@Tomcat-node1 ~]# java -version
java version "1.7.0_131"
OpenJDK Runtime Environment (rhel-2.6.9.0.el6_8-x86_64 u131-b00)
OpenJDK 64-Bit Server VM (build 24.131-b00, mixed mode)
[root@Tomcat-node1 ~]# yum -y remove java-1.7.0-openjdk*
[root@Tomcat-node1 ~]# yum -y remove tzdata-java.noarch
[root@Tomcat-node1 ~]# java -version                 
-bash: /usr/bin/java: No such file or directory
   
[root@Tomcat-node1 ~]# ll /usr/local/src/jdk-8u131-linux-x64_.rpm
-rw-rw-r--. 1 root root 169983496 Nov 19  2017 /usr/local/src/jdk-8u131-linux-x64_.rpm
[root@Tomcat-node1 ~]# rpm -ivh /usr/local/src/jdk-8u131-linux-x64_.rpm --force
[root@Tomcat-node1 ~]# vim /etc/profile
......
JAVA_HOME=/usr/java/jdk1.8.0_131
JAVA_BIN=/usr/java/jdk1.8.0_131/bin
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/bin:/sbin/
CLASSPATH=.:/lib/dt.jar:/lib/tools.jar
export JAVA_HOME JAVA_BIN PATH CLASSPATH
   
[root@Tomcat-node1 ~]# source /etc/profile
[root@Tomcat-node1 ~]# java -version
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)
You have new mail in /var/spool/mail/root
   
安裝配置tomcat8
[root@Tomcat-node1 ~]# cd /usr/local/src/
[root@Tomcat-node1 src]# ll apache-tomcat-8.0.53.tar.gz
-rw-rw-r--. 1 root root 9472492 Nov  9  2017 apache-tomcat-8.0.53.tar.gz
[root@Tomcat-node1 src]# tar -zvxf apache-tomcat-8.0.53.tar.gz
[root@Tomcat-node1 src]# mv apache-tomcat-8.0.53 /usr/local/tomcat8
   

現在開始進行tomcat自帶cluster方式的session共享配置:
[root@Tomcat-node1 ~]# cd /usr/local/tomcat8/conf/
[root@Tomcat-node1 conf]# cp server.xml server.xml.bak
[root@Tomcat-node1 conf]# vim server.xml
.......
<Engine name="Catalina" defaultHost="localhost">

找到上面這一行,在此行的下面直接復制以下內容,不用修改,直接復制就可以
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
                 channelSendOptions="6">

          <Manager className="org.apache.catalina.ha.session.BackupManager"
                   expireSessionsOnShutdown="false"
                   notifyListenersOnReplication="true"
                   mapSendOptions="6"/>


          <Channel className="org.apache.catalina.tribes.group.GroupChannel">
            <Membership className="org.apache.catalina.tribes.membership.McastService"
                        address="228.0.0.4"
                        port="45564"
                        frequency="500"
                        dropTime="3000"/>
            <Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
                      address="192.168.10.201"
                      port="5000"
                      selectorTimeout="100"
                      maxThreads="6"/>

            <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
              <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
            </Sender>
            <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
            <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
            <Interceptor className="org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor"/>
          </Channel>

          <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
                 filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.*\.txt;"/>

          <Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer"
                    tempDir="/tmp/war-temp/"
                    deployDir="/tmp/war-deploy/"
                    watchDir="/tmp/war-listen/"
                    watchEnabled="false"/>

          <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/>
        </Cluster>


記得多個tomcat都需要配置，同一個集群里面配置信息保持一直就可以了。如果是其他集群，修改一下address就好了。
另一個tomcat節點直接拷貝這個server.xml文件,然后將address修改為192.168.10.202就可以了

==================================================================================================
配置好的server.xml可以直接使用,下載地址: https://pan.baidu.com/s/14Lgq97pyB_XLLu9j_CHtuw
提取密碼:mrz4
==================================================================================================

然后還有一個重要配置不要忘記了!否則session將不能實現共享!!
在tomcat項目的WEB-INF/web.xml文件中添加<distributable/>.
[root@Tomcat-node1 ~]# cd /usr/local/tomcat8/webapps/ROOT/WEB-INF/          #兩台tomcat節點都要修改
[root@Tomcat-node1 WEB-INF]# ls
web.xml
[root@Tomcat-node1 WEB-INF]# vim web.xml               #直接在底部</web-app>的前面添加既可以
.......
  <distributable/>


編寫一個測試頁面(直接修改index.jsp文件):
[root@Tomcat-node1 ~]# cat /usr/local/tomcat8/webapps/ROOT/index.jsp 
<html>
<head>
<title> session test</title>
</head>
<body>
SessionID is <%=session.getId()%>
<BR>
SessionIP is <%=request.getServerName()%>
<BR>
SessionPort is <%=request.getServerPort()%>
<%
out.println("Response from tomcat-node1");
%>
%</body>
%</html>

另一個tomcat2節點的測試內容:
[root@Tomcat-node2 ~]# cat /usr/local/tomcat8/webapps/ROOT/index.jsp 
<html>
<head>
<title> session test</title>
</head>
<body>
SessionID is <%=session.getId()%>
<BR>
SessionIP is <%=request.getServerName()%>
<BR>
SessionPort is <%=request.getServerPort()%>
<%
out.println("Response from tomcat-node2");
%>
%</body>
%</html>

然后重啟兩個節點的tomcat服務:

啟動tomcat
[root@Tomcat-node1 src]# /usr/local/tomcat8/bin/startup.sh
Using CATALINA_BASE:   /usr/local/tomcat8
Using CATALINA_HOME:   /usr/local/tomcat8
Using CATALINA_TMPDIR: /usr/local/tomcat8/temp
Using JRE_HOME:        /usr/java/jdk1.8.0_131
Using CLASSPATH:       /usr/local/tomcat8/bin/bootstrap.jar:/usr/local/tomcat8/bin/tomcat-juli.jar
Tomcat started.
You have new mail in /var/spool/mail/root
[root@Tomcat-node1 src]# ps -ef|grep tomcat
root      8477     1 87 03:11 pts/0    00:00:03 /usr/java/jdk1.8.0_131/bin/java -Djava.util.logging.config.file=/usr/local/tomcat8/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -classpath /usr/local/tomcat8/bin/bootstrap.jar:/usr/local/tomcat8/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/tomcat8 -Dcatalina.home=/usr/local/tomcat8 -Djava.io.tmpdir=/usr/local/tomcat8/temp org.apache.catalina.startup.Bootstrap start
root      8528  6829  0 03:11 pts/0    00:00:00 grep tomcat
[root@Tomcat-node1 src]# lsof -i:8080
COMMAND  PID USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
java    8477 root   49u  IPv6 12974768      0t0  TCP *:webcache (LISTEN)
 

4) Session共享測試
訪問http://www.kevin.com,不斷刷新頁面,發現頁面信息會改變,但是SessionID一直保持不變!!　
關閉192.168.10.201或192.168.10.202兩個中的任意一個tomcat服務,訪問頁面,不斷刷新,SessionID都會一直保持不變!!