由於我團隊開發的在線坐席系統，即將面對線上每周3000W的下行投放客戶，產品的咨詢量可能會很大，基於前期，200W的投放時，前10分鍾，大概800問題量，平均一個客戶大概8個問題，也就是說每分鍾10個客戶，折算3000W的話，就是每分鍾150客戶。系統容量應該問題不是太大，考慮一下極端情況（疊加效應），或者留下富余5倍，也就不到800吧，我們的生產環境，需要擴容消費者服務器，redis作為調度，需要擴容，由現在的一主從升級到一個集群（3對主從的cluster）。
這里大概記錄下我們開發環境的redis集群的配置和安裝過程，這些屬於運維的工作，但是也是系統架構的范疇，這些若沒有實戰的經驗，不要和我說什么架構設計。。。我就是搞這個的，我要對我的系統負責，但是又不會出現系統的過度設計。
有三台機器，准備搭建redis集群環境，redis的版本是3.2.8，源碼安裝，安裝過程略去。
10.90.7.2
10.90.7.10
10.90.2.102
每個上面部署兩個redis的實例，端口配置信息如下：
10.90.7.2 7000/7010
10.90.7.10 7001/7011
10.90.2.102 7002/7012
三台的redis的配置采用近乎一樣的配置，除了因為端口的不同造成的一點點差異。這里拿10.90.7.2的配置7000為例：
在/opt/redis-3.2.8下面創建目錄tkcluster。將/opt/redis-3.2.8下面的redis.conf文件copy到tkcluster目錄下，重命名為7000.conf，修改的內容如下，其他的采用默認值。
bind 10.90.7.2 protected-mode no port 7000 daemonize yes pidfile /var/run/redis_7000.pid appendonly yes appendfilename "appendonly7000.aof" cluster-enabled yes cluster-config-file nodes-7000.conf cluster-node-timeout 15000 notify-keyspace-events Ex
配置完后，copy一下7000.conf到7010.conf，將上述相應的7000改成7010的參數即可完成7010.conf的配置。為了方便啟動redis-server。寫一個簡單shell腳本:
#!/bin/bash /opt/redis-3.2.8/src/redis-server 7000.conf echo "status from 7000: $?" sleep 5 /opt/redis-3.2.8/src/redis-server 7010.conf echo "status from 7010: $?"
運行一下后，成功。得到文件列表：
[root@localhost tkcluster]# ll 總計 116 -rw-r--r-- 1 root root 46695 11-21 15:11 7000.conf -rw-r--r-- 1 root root 46684 11-21 15:11 7010.conf -rw-r--r-- 1 root root 54 11-22 19:10 appendonly7000.aof -rw-r--r-- 1 root root 0 11-22 18:43 appendonly7010.aof -rw-r--r-- 1 root root 90 11-22 19:10 dump.rdb -rw-r--r-- 1 root root 739 11-22 18:43 nodes-7000.conf -rw-r--r-- 1 root root 739 11-22 18:43 nodes-7010.conf -rwxr-xr-x 1 root root 166 11-21 14:59 stcluster.sh
檢查下，集群創建的指令是否能工作：
[root@bogon src]# ./redis-trib.rb --help
/usr/bin/env: ruby: 沒有那個文件或目錄
系統里面沒有ruby這個環境，需要安裝一下。
這個過程有點讓人頭疼，開始，在10.90.7.10的機器上進行安裝，通過yum，下面是我遇到的問題
[root@bogon src]# yum install -y ruby
源找到了，很快安裝成功。繼續嘗試：
[root@bogon src]# ./redis-trib.rb --help /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- redis (LoadError) from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require' from ./redis-trib.rb:25:in `<main>'
這個錯誤，呵呵，熟悉ruby的話，很容易看出來，是ruby的環境需要訪問redis，但是系統里面沒有ruby的redis客戶端程序，需要安裝一下ruby的redis客戶端插件。
[root@bogon src]# gem install redis Fetching: redis-4.0.1.gem (100%) ERROR: Error installing redis: redis requires Ruby version >= 2.2.2.
這個錯誤，是因為ruby版本太低，需要升級，采用下面的方式可以完成：
[root@bogon src]# curl -L get.rvm.io | bash -s stable % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 194 100 194 0 0 194 0 0:00:01 --:--:-- 0:00:01 249 100 24090 100 24090 0 0 24090 0 0:00:01 0:00:01 --:--:-- 25304 Downloading https://github.com/rvm/rvm/archive/1.29.3.tar.gz Downloading https://github.com/rvm/rvm/releases/download/1.29.3/1.29.3.tar.gz.asc curl: (22) The requested URL returned error: 503 Could not download 'https://github.com/rvm/rvm/releases/download/1.29.3/1.29.3.tar.gz.asc'. curl returned status '22'. Creating group 'rvm' Installing RVM to /usr/local/rvm/ Installation of RVM in /usr/local/rvm/ is almost complete: * First you need to add all users that will be using rvm to 'rvm' group, and logout - login again, anyone using rvm will be operating with `umask u=rwx,g=rwx,o=rx`. * To start using RVM you need to run `source /etc/profile.d/rvm.sh` in all your open shell windows, in rare cases you need to reopen all shell windows.
執行完成后，按照上面的提示，首先推出ssh，然后再登陸ssh。然后執行source /etc/profile.d/rvm.sh指令。接下來執行一下rvm list known指令，看看當前查看rvm庫中已知的ruby版本
[root@bogon redis-3.2.8]# rvm list known # MRI Rubies [ruby-]1.8.6[-p420] [ruby-]1.8.7[-head] # security released on head [ruby-]1.9.1[-p431] [ruby-]1.9.2[-p330] [ruby-]1.9.3[-p551] [ruby-]2.0.0[-p648] [ruby-]2.1[.10] [ruby-]2.2[.7] [ruby-]2.3[.4] [ruby-]2.4[.1] ruby-head # for forks use: rvm install ruby-head-<name> --url https://github.com/github/ruby.git --branch 2.2 # JRuby jruby-1.6[.8] jruby-1.7[.27] jruby[-9.1.13.0] jruby-head # Rubinius rbx-1[.4.3] rbx-2.3[.0] rbx-2.4[.1] rbx-2[.5.8] rbx-3[.84] rbx-head # Opal opal # Minimalistic ruby implementation - ISO 30170:2012 mruby-1.0.0 mruby-1.1.0 mruby-1.2.0 mruby-1[.3.0] mruby[-head] # Ruby Enterprise Edition ree-1.8.6 ree[-1.8.7][-2012.02] # Topaz topaz # MagLev maglev[-head] maglev-1.0.0 # Mac OS X Snow Leopard Or Newer macruby-0.10 macruby-0.11 macruby[-0.12] macruby-nightly macruby-head # IronRuby ironruby[-1.1.3] ironruby-head [root@bogon redis-3.2.8]#
查看后，看到前面有ruby的版本比較多，高於2.0.0的有4個，選一個即可。這里就選擇2.3.4吧。
[root@bogon redis-3.2.8]# rvm install 2.3.4 Searching for binary rubies, this might take some time. Found remote file https://rvm_io.global.ssl.fastly.net/binaries/centos/7/x86_64/ruby-2.3.4.tar.bz2 Checking requirements for centos. Installing requirements for centos. Installing required packages: bison, bzip2, libyaml-devel...... Requirements installation successful. ruby-2.3.4 - #configure ruby-2.3.4 - #download % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 25.2M 100 25.2M 0 0 149k 0 0:02:53 0:02:53 --:--:-- 165k No checksum for downloaded archive, recording checksum in user configuration. ruby-2.3.4 - #validate archive ruby-2.3.4 - #extract ruby-2.3.4 - #validate binary ruby-2.3.4 - #setup ruby-2.3.4 - #gemset created /usr/local/rvm/gems/ruby-2.3.4@global ruby-2.3.4 - #importing gemset /usr/local/rvm/gemsets/global.gems.............................. ruby-2.3.4 - #generating global wrappers........ Error running 'run_gem_wrappers_regenerate', please read /usr/local/rvm/log/1511253510_ruby-2.3.4/gemset.wrappers.global.log ruby-2.3.4 - #gemset created /usr/local/rvm/gems/ruby-2.3.4 ruby-2.3.4 - #importing gemsetfile /usr/local/rvm/gemsets/default.gems evaluated to empty gem list ruby-2.3.4 - #generating default wrappers......... Error running 'run_gem_wrappers_regenerate', please read /usr/local/rvm/log/1511253510_ruby-2.3.4/gemset.wrappers.default.log
在這個地方，我的10.90.7.10的機器總是報錯。。。
執行啟用新的ruby版本：
[root@bogon redis-3.2.8]# rvm use 2.3.4 Using /usr/local/rvm/gems/ruby-2.3.4 Error loading RubyGems plugin "/usr/local/rvm/gems/ruby-2.3.4@global/gems/executable-hooks-1.3.2/lib/rubygems_plugin.rb": /lib64/libcrypto.so.10: version `OPENSSL_1.0.2' not found (required by /usr/local/rvm/rubies/ruby-2.3.4/lib/ruby/2.3.0/x86_64-linux/openssl.so) - /usr/local/rvm/rubies/ruby-2.3.4/lib/ruby/2.3.0/x86_64-linux/openssl.so (LoadError) Error loading RubyGems plugin "/usr/local/rvm/gems/ruby-2.3.4@global/gems/gem-wrappers-1.3.2/lib/rubygems_plugin.rb": /lib64/libcrypto.so.10: version `OPENSSL_1.0.2' not found (required by /usr/local/rvm/rubies/ruby-2.3.4/lib/ruby/2.3.0/x86_64-linux/openssl.so) - /usr/local/rvm/rubies/ruby-2.3.4/lib/ruby/2.3.0/x86_64-linux/openssl.so (LoadError)
這個錯誤是找openssl 1.0.2的依賴，需要安裝。我以源碼來安裝這個吧。先從執行：
[root@bogon openssl-1.0.2l]# wget wget https://www.openssl.org/source/openssl-1.0.2.tar.gz
下載完畢后，解壓安裝：
[root@bogon openssl-1.0.2]# ./config -fPIC --prefix=/usr/local/openssl/ enable-shared Operating system: x86_64-whatever-linux2 Configuring for linux-x86_64 Configuring for linux-x86_64 no-ec_nistp_64_gcc_128 [default] OPENSSL_NO_EC_NISTP_64_GCC_128 (skip dir) no-gmp [default] OPENSSL_NO_GMP (skip dir) no-jpake [experimental] OPENSSL_NO_JPAKE (skip dir) no-krb5 [krb5-flavor not specified] OPENSSL_NO_KRB5 no-libunbound [experimental] OPENSSL_NO_LIBUNBOUND (skip dir) no-md2 [default] OPENSSL_NO_MD2 (skip dir) no-rc5 [default] OPENSSL_NO_RC5 (skip dir) no-rfc3779 [default] OPENSSL_NO_RFC3779 (skip dir) no-sctp [default] OPENSSL_NO_SCTP (skip dir) no-ssl-trace [default] OPENSSL_NO_SSL_TRACE (skip dir) no-store [experimental] OPENSSL_NO_STORE (skip dir) no-unit-test [default] OPENSSL_NO_UNIT_TEST (skip dir) no-zlib [default] no-zlib-dynamic [default] IsMK1MF=0 CC =gcc CFLAG =-fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -fPIC -Wa,--noexecstack -m64 -DL_ENDIAN -DTERMIO -O3 -Wall -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DECP_NISTZ256_ASM EX_LIBS =-ldl CPUID_OBJ =x86_64cpuid.o BN_ASM =x86_64-gcc.o x86_64-mont.o x86_64-mont5.o x86_64-gf2m.o rsaz_exp.o rsaz-x86_64.o rsaz-avx2.o EC_ASM =ecp_nistz256.o ecp_nistz256-x86_64.o DES_ENC =des_enc.o fcrypt_b.o AES_ENC =aes-x86_64.o vpaes-x86_64.o bsaes-x86_64.o aesni-x86_64.o aesni-sha1-x86_64.o aesni-sha256-x86_64.o aesni-mb-x86_64.o BF_ENC =bf_enc.o CAST_ENC =c_enc.o RC4_ENC =rc4-x86_64.o rc4-md5-x86_64.o RC5_ENC =rc5_enc.o MD5_OBJ_ASM =md5-x86_64.o SHA1_OBJ_ASM =sha1-x86_64.o sha256-x86_64.o sha512-x86_64.o sha1-mb-x86_64.o sha256-mb-x86_64.o RMD160_OBJ_ASM= CMLL_ENC =cmll-x86_64.o cmll_misc.o MODES_OBJ =ghash-x86_64.o aesni-gcm-x86_64.o ENGINES_OBJ = PROCESSOR = RANLIB =/usr/bin/ranlib ARFLAGS = PERL =/usr/bin/perl SIXTY_FOUR_BIT_LONG mode DES_UNROLL used DES_INT used RC4_CHUNK is unsigned long e_os2.h => include/openssl/e_os2.h making links in crypto... make[1]: 進入目錄“/opt/redis/openssl-1.0.2/crypto” crypto.h => ../include/openssl/crypto.h opensslv.h => ../include/openssl/opensslv.h opensslconf.h => ../include/openssl/opensslconf.h ebcdic.h => ../include/openssl/ebcdic.h symhacks.h => ../include/openssl/symhacks.h ossl_typ.h => ../include/openssl/ossl_typ.h constant_time_test.c => ../test/constant_time_test.c making links in crypto/objects... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/objects” objects.h => ../../include/openssl/objects.h obj_mac.h => ../../include/openssl/obj_mac.h make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/objects” making links in crypto/md4... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/md4” md4.h => ../../include/openssl/md4.h md4test.c => ../../test/md4test.c md4.c => ../../apps/md4.c make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/md4” making links in crypto/md5... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/md5” md5.h => ../../include/openssl/md5.h md5test.c => ../../test/md5test.c make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/md5” making links in crypto/sha... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/sha” sha.h => ../../include/openssl/sha.h shatest.c => ../../test/shatest.c sha1test.c => ../../test/sha1test.c sha256t.c => ../../test/sha256t.c sha512t.c => ../../test/sha512t.c make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/sha” making links in crypto/mdc2... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/mdc2” mdc2.h => ../../include/openssl/mdc2.h mdc2test.c => ../../test/mdc2test.c make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/mdc2” making links in crypto/hmac... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/hmac” hmac.h => ../../include/openssl/hmac.h hmactest.c => ../../test/hmactest.c make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/hmac” making links in crypto/ripemd... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/ripemd” ripemd.h => ../../include/openssl/ripemd.h rmdtest.c => ../../test/rmdtest.c make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/ripemd” making links in crypto/whrlpool... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/whrlpool” whrlpool.h => ../../include/openssl/whrlpool.h wp_test.c => ../../test/wp_test.c make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/whrlpool” making links in crypto/des... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/des” des.h => ../../include/openssl/des.h des_old.h => ../../include/openssl/des_old.h destest.c => ../../test/destest.c make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/des” making links in crypto/aes... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/aes” aes.h => ../../include/openssl/aes.h make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/aes” making links in crypto/rc2... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/rc2” rc2.h => ../../include/openssl/rc2.h rc2test.c => ../../test/rc2test.c make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/rc2” making links in crypto/rc4... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/rc4” rc4.h => ../../include/openssl/rc4.h rc4test.c => ../../test/rc4test.c make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/rc4” making links in crypto/idea... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/idea” idea.h => ../../include/openssl/idea.h ideatest.c => ../../test/ideatest.c make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/idea” making links in crypto/bf... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/bf” blowfish.h => ../../include/openssl/blowfish.h bftest.c => ../../test/bftest.c make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/bf” making links in crypto/cast... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/cast” cast.h => ../../include/openssl/cast.h casttest.c => ../../test/casttest.c make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/cast” making links in crypto/camellia... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/camellia” camellia.h => ../../include/openssl/camellia.h make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/camellia” making links in crypto/seed... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/seed” seed.h => ../../include/openssl/seed.h make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/seed” making links in crypto/modes... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/modes” modes.h => ../../include/openssl/modes.h make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/modes” making links in crypto/bn... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/bn” bn.h => ../../include/openssl/bn.h bntest.c => ../../test/bntest.c exptest.c => ../../test/exptest.c make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/bn” making links in crypto/ec... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/ec” ec.h => ../../include/openssl/ec.h ectest.c => ../../test/ectest.c make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/ec” making links in crypto/rsa... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/rsa” rsa.h => ../../include/openssl/rsa.h rsa_test.c => ../../test/rsa_test.c make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/rsa” making links in crypto/dsa... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/dsa” dsa.h => ../../include/openssl/dsa.h dsatest.c => ../../test/dsatest.c make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/dsa” making links in crypto/ecdsa... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/ecdsa” ecdsa.h => ../../include/openssl/ecdsa.h ecdsatest.c => ../../test/ecdsatest.c make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/ecdsa” making links in crypto/dh... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/dh” dh.h => ../../include/openssl/dh.h dhtest.c => ../../test/dhtest.c make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/dh” making links in crypto/ecdh... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/ecdh” ecdh.h => ../../include/openssl/ecdh.h ecdhtest.c => ../../test/ecdhtest.c make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/ecdh” making links in crypto/dso... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/dso” dso.h => ../../include/openssl/dso.h make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/dso” making links in crypto/engine... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/engine” engine.h => ../../include/openssl/engine.h enginetest.c => ../../test/enginetest.c make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/engine” making links in crypto/buffer... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/buffer” buffer.h => ../../include/openssl/buffer.h make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/buffer” making links in crypto/bio... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/bio” bio.h => ../../include/openssl/bio.h make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/bio” making links in crypto/stack... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/stack” stack.h => ../../include/openssl/stack.h safestack.h => ../../include/openssl/safestack.h make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/stack” making links in crypto/lhash... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/lhash” lhash.h => ../../include/openssl/lhash.h make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/lhash” making links in crypto/rand... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/rand” rand.h => ../../include/openssl/rand.h randtest.c => ../../test/randtest.c make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/rand” making links in crypto/err... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/err” err.h => ../../include/openssl/err.h make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/err” making links in crypto/evp... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/evp” evp.h => ../../include/openssl/evp.h evp_test.c => ../../test/evp_test.c evptests.txt -> ../../test/evptests.txt make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/evp” making links in crypto/asn1... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/asn1” asn1.h => ../../include/openssl/asn1.h asn1_mac.h => ../../include/openssl/asn1_mac.h asn1t.h => ../../include/openssl/asn1t.h make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/asn1” making links in crypto/pem... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/pem” pem.h => ../../include/openssl/pem.h pem2.h => ../../include/openssl/pem2.h make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/pem” making links in crypto/x509... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/x509” x509.h => ../../include/openssl/x509.h x509_vfy.h => ../../include/openssl/x509_vfy.h make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/x509” making links in crypto/x509v3... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/x509v3” x509v3.h => ../../include/openssl/x509v3.h v3nametest.c => ../../test/v3nametest.c make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/x509v3” making links in crypto/conf... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/conf” conf.h => ../../include/openssl/conf.h conf_api.h => ../../include/openssl/conf_api.h make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/conf” making links in crypto/txt_db... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/txt_db” txt_db.h => ../../include/openssl/txt_db.h make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/txt_db” making links in crypto/pkcs7... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/pkcs7” pkcs7.h => ../../include/openssl/pkcs7.h make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/pkcs7” making links in crypto/pkcs12... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/pkcs12” pkcs12.h => ../../include/openssl/pkcs12.h make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/pkcs12” making links in crypto/comp... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/comp” comp.h => ../../include/openssl/comp.h make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/comp” making links in crypto/ocsp... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/ocsp” ocsp.h => ../../include/openssl/ocsp.h make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/ocsp” making links in crypto/ui... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/ui” ui.h => ../../include/openssl/ui.h ui_compat.h => ../../include/openssl/ui_compat.h make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/ui” making links in crypto/krb5... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/krb5” krb5_asn.h => ../../include/openssl/krb5_asn.h make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/krb5” making links in crypto/cms... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/cms” cms.h => ../../include/openssl/cms.h make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/cms” making links in crypto/pqueue... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/pqueue” pqueue.h => ../../include/openssl/pqueue.h make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/pqueue” making links in crypto/ts... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/ts” ts.h => ../../include/openssl/ts.h make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/ts” making links in crypto/srp... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/srp” srp.h => ../../include/openssl/srp.h srptest.c => ../../test/srptest.c make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/srp” making links in crypto/cmac... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/crypto/cmac” cmac.h => ../../include/openssl/cmac.h make[2]: 離開目錄“/opt/redis/openssl-1.0.2/crypto/cmac” make[1]: 離開目錄“/opt/redis/openssl-1.0.2/crypto” making links in ssl... make[1]: 進入目錄“/opt/redis/openssl-1.0.2/ssl” ssl.h => ../include/openssl/ssl.h ssl2.h => ../include/openssl/ssl2.h ssl3.h => ../include/openssl/ssl3.h ssl23.h => ../include/openssl/ssl23.h tls1.h => ../include/openssl/tls1.h dtls1.h => ../include/openssl/dtls1.h kssl.h => ../include/openssl/kssl.h srtp.h => ../include/openssl/srtp.h ssltest.c => ../test/ssltest.c heartbeat_test.c => ../test/heartbeat_test.c make[1]: 離開目錄“/opt/redis/openssl-1.0.2/ssl” making links in engines... make[1]: 進入目錄“/opt/redis/openssl-1.0.2/engines” making links in engines/ccgost... make[2]: 進入目錄“/opt/redis/openssl-1.0.2/engines/ccgost” make[2]: 對“links”無需做任何事。 make[2]: 離開目錄“/opt/redis/openssl-1.0.2/engines/ccgost” make[1]: 離開目錄“/opt/redis/openssl-1.0.2/engines” making links in apps... make[1]: 進入目錄“/opt/redis/openssl-1.0.2/apps” make[1]: 對“links”無需做任何事。 make[1]: 離開目錄“/opt/redis/openssl-1.0.2/apps” making links in test... make[1]: 進入目錄“/opt/redis/openssl-1.0.2/test” make[1]: 對“links”無需做任何事。 make[1]: 離開目錄“/opt/redis/openssl-1.0.2/test” making links in tools... make[1]: 進入目錄“/opt/redis/openssl-1.0.2/tools” make[1]: 對“links”無需做任何事。 make[1]: 離開目錄“/opt/redis/openssl-1.0.2/tools” generating dummy tests (if needed)... make[1]: 進入目錄“/opt/redis/openssl-1.0.2/test” make[1]: 對“generate”無需做任何事。 make[1]: 離開目錄“/opt/redis/openssl-1.0.2/test” Configured for linux-x86_64. [root@bogon openssl-1.0.2]
然后進行make操作，生成可執行文件。 完成后，看看當前的版本。openssl安裝后，執行redis-trib還是報錯。。。
到此為止，因為這個過程我反復嘗試各種修正openssl的操作，都報錯，應該是我的這個機器環境什么地方出問題了，不折騰這個了，既然redis配置的是集群的環境，那么，我在另外的一台機器上處理，應該也沒有什么問題。
接下來的，配置操作是在10.90.7.2的服務器上進行的，一切換一種方式進行操作，這次主要是基於源碼的安裝。
源碼安裝ruby 2.4.2。從 http://www.ruby-lang.org/en/downloads/下載需要的版本。 解壓后，進入ruby目錄，分別進行./configure，./make， ./make install
查看版本信息，得到下面的內容，說明安裝成功。
[root@localhost ~]# ruby -v ruby 2.4.2p198 (2017-09-14 revision 59899) [x86_64-linux]
繼續執行redis-trib.rb指令。
[root@localhost src]# ./redis-trib.rb --help /usr/local/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- redis (LoadError) from /usr/local/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require' from ./redis-trib.rb:25:in `<main>'
這個錯誤和上面的基於yum安裝的一個錯誤，缺少ruby版本的redis。依然下載源碼安裝。https://rubygems.org/downloads/redis-4.0.1.gem下載完畢后，用gem進行安裝。
[root@localhost opt]# gem install redis-4.0.1.gem Successfully installed redis-4.0.1 Parsing documentation for redis-4.0.1 Installing ri documentation for redis-4.0.1 Done installing documentation for redis after 0 seconds 1 gem installed [root@localhost opt]#
再次執行redis-trib指令，檢查redis-trib是否可以操作集群。
[root@localhost src]# ./redis-trib.rb Usage: redis-trib <command> <options> <arguments ...> create host1:port1 ... hostN:portN --replicas <arg> check host:port info host:port fix host:port --timeout <arg> reshard host:port --from <arg> --to <arg> --slots <arg> --yes --timeout <arg> --pipeline <arg> rebalance host:port --weight <arg> --auto-weights --use-empty-masters --timeout <arg> --simulate --pipeline <arg> --threshold <arg> add-node new_host:new_port existing_host:existing_port --slave --master-id <arg> del-node host:port node_id set-timeout host:port milliseconds call host:port command arg arg .. arg import host:port --from <arg> --copy --replace help (show this help) For check, fix, reshard, del-node, set-timeout you can specify the host and port of any working node in the cluster. [root@localhost src]#
這次，說明redis的起集群的操作成功，下一步准備集群創建。
查看目前啟動的幾台redis機器信息（info）
[root@localhost src]# ./redis-trib.rb info 10.90.7.2:7001 [ERR] Sorry, can't connect to node 10.90.7.2:7001 [root@localhost src]# ./redis-trib.rb info 10.90.7.2:7000 10.90.7.2:7000 (54941737...) -> 0 keys | 0 slots | 0 slaves. [OK] 0 keys in 1 masters. 0.00 keys per slot on average. [root@localhost src]# [root@localhost src]# ./redis-trib.rb info 10.90.7.2:7010 10.90.7.2:7010 (79ebc53a...) -> 0 keys | 0 slots | 0 slaves. [OK] 0 keys in 1 masters. 0.00 keys per slot on average. [root@localhost src]# [root@localhost src]# ./redis-trib.rb info 10.90.7.10:7001 10.90.7.10:7001 (7fd72f31...) -> 0 keys | 0 slots | 0 slaves. [OK] 0 keys in 1 masters. 0.00 keys per slot on average. [root@localhost src]# ./redis-trib.rb info 10.90.7.10:7011 10.90.7.10:7011 (d5a0fa35...) -> 0 keys | 0 slots | 0 slaves. [OK] 0 keys in 1 masters. 0.00 keys per slot on average. [root@localhost src]# ./redis-trib.rb info 10.90.2.102:7002 10.90.2.102:7002 (8d539663...) -> 0 keys | 0 slots | 0 slaves. [OK] 0 keys in 1 masters. 0.00 keys per slot on average. [root@localhost src]# ./redis-trib.rb info 10.90.2.102:7012 10.90.2.102:7012 (175061fa...) -> 0 keys | 0 slots | 0 slaves. [OK] 0 keys in 1 masters. 0.00 keys per slot on average. [root@localhost src]#
創建redis集群，通過redis-trib命令實現：
[root@localhost src]# ./redis-trib.rb create --replicas 1 10.90.7.2:7000 10.90.7.2:7010 10.90.7.10:7001 10.90.7.10:7011 10.90.2.102:7002 10.90.2.102:7012 >>> Creating cluster >>> Performing hash slots allocation on 6 nodes... Using 3 masters: 10.90.7.2:7000 10.90.7.10:7001 10.90.2.102:7002 Adding replica 10.90.7.10:7011 to 10.90.7.2:7000 Adding replica 10.90.7.2:7010 to 10.90.7.10:7001 Adding replica 10.90.2.102:7012 to 10.90.2.102:7002 M: 549417379f072c2f9551c1e662fbf0ef3ecc73eb 10.90.7.2:7000 slots:0-5460 (5461 slots) master S: 79ebc53a5193dac898ec01fe2c5b0023990a379b 10.90.7.2:7010 replicates 7fd72f31bf4faaae4f30cec6ac8ae1ab9cdaa597 M: 7fd72f31bf4faaae4f30cec6ac8ae1ab9cdaa597 10.90.7.10:7001 slots:5461-10922 (5462 slots) master S: d5a0fa35859c40e8588bf8648e036ab152bcece7 10.90.7.10:7011 replicates 549417379f072c2f9551c1e662fbf0ef3ecc73eb M: 8d53966305162f6f0e53714f22e554a38f74d67b 10.90.2.102:7002 slots:10923-16383 (5461 slots) master S: 175061fa6addecc8eaaf9b999372a9d5a8964d44 10.90.2.102:7012 replicates 8d53966305162f6f0e53714f22e554a38f74d67b Can I set the above configuration? (type 'yes' to accept): yes >>> Nodes configuration updated >>> Assign a different config epoch to each node >>> Sending CLUSTER MEET messages to join the cluster Waiting for the cluster to join..... >>> Performing Cluster Check (using node 10.90.7.2:7000) M: 549417379f072c2f9551c1e662fbf0ef3ecc73eb 10.90.7.2:7000 slots:0-5460 (5461 slots) master 1 additional replica(s) S: 79ebc53a5193dac898ec01fe2c5b0023990a379b 10.90.7.2:7010 slots: (0 slots) slave replicates 7fd72f31bf4faaae4f30cec6ac8ae1ab9cdaa597 S: d5a0fa35859c40e8588bf8648e036ab152bcece7 10.90.7.10:7011 slots: (0 slots) slave replicates 549417379f072c2f9551c1e662fbf0ef3ecc73eb M: 7fd72f31bf4faaae4f30cec6ac8ae1ab9cdaa597 10.90.7.10:7001 slots:5461-10922 (5462 slots) master 1 additional replica(s) M: 8d53966305162f6f0e53714f22e554a38f74d67b 10.90.2.102:7002 slots:10923-16383 (5461 slots) master 1 additional replica(s) S: 175061fa6addecc8eaaf9b999372a9d5a8964d44 10.90.2.102:7012 slots: (0 slots) slave replicates 8d53966305162f6f0e53714f22e554a38f74d67b [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered.
指令查看redis集群信息：
[root@localhost src]# ./redis-cli -c -h 10.90.7.2 -p 7000 10.90.7.2:7000> info # Server redis_version:3.2.8 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:6319bd4d85613682 redis_mode:cluster os:Linux 2.6.18-274.el5 x86_64 arch_bits:64 multiplexing_api:epoll gcc_version:4.1.2 process_id:29047 run_id:428293cc0d9842e0ba2f3917322a0deb136196ee tcp_port:7000 uptime_in_seconds:100413 uptime_in_days:1 hz:10 lru_clock:1399317 executable:/opt/redis-3.2.8/src/redis-server config_file:/opt/redis-3.2.8/tkcluster/7000.conf # Clients connected_clients:1 client_longest_output_list:0 client_biggest_input_buf:0 blocked_clients:0 # Memory used_memory:2418464 used_memory_human:2.31M used_memory_rss:3018752 used_memory_rss_human:2.88M used_memory_peak:2418464 used_memory_peak_human:2.31M total_system_memory:8263512064 total_system_memory_human:7.70G used_memory_lua:37888 used_memory_lua_human:37.00K maxmemory:0 maxmemory_human:0B maxmemory_policy:noeviction mem_fragmentation_ratio:1.25 mem_allocator:jemalloc-4.0.3 # Persistence loading:0 rdb_changes_since_last_save:0 rdb_bgsave_in_progress:0 rdb_last_save_time:1511347436 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:0 rdb_current_bgsave_time_sec:-1 aof_enabled:1 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_last_write_status:ok aof_current_size:0 aof_base_size:0 aof_pending_rewrite:0 aof_buffer_length:0 aof_rewrite_buffer_length:0 aof_pending_bio_fsync:0 aof_delayed_fsync:0 # Stats total_connections_received:5 total_commands_processed:1349 instantaneous_ops_per_sec:1 total_net_input_bytes:102199 total_net_output_bytes:6045767 instantaneous_input_kbps:0.04 instantaneous_output_kbps:0.00 rejected_connections:0 sync_full:1 sync_partial_ok:0 sync_partial_err:0 expired_keys:0 evicted_keys:0 keyspace_hits:0 keyspace_misses:0 pubsub_channels:0 pubsub_patterns:0 latest_fork_usec:4132 migrate_cached_sockets:0 # Replication role:master connected_slaves:1 slave0:ip=10.90.7.10,port=7011,state=online,offset=1849,lag=1 master_repl_offset:1849 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:2 repl_backlog_histlen:1848 # CPU used_cpu_sys:0.09 used_cpu_user:0.11 used_cpu_sys_children:0.01 used_cpu_user_children:0.00 # Cluster cluster_enabled:1 # Keyspace 10.90.7.2:7000>
在10.90.7.2上設置一個數據：
10.90.7.2:7000> set hello 1 OK 10.90.7.2:7000> get hello "1"
查看一下集群的信息點：
10.90.7.2:7000> cluster info cluster_state:ok cluster_slots_assigned:16384 cluster_slots_ok:16384 cluster_slots_pfail:0 cluster_slots_fail:0 cluster_known_nodes:6 cluster_size:3 cluster_current_epoch:6 cluster_my_epoch:1 cluster_stats_messages_sent:15335 cluster_stats_messages_received:15335
從另外一個主機上，測試看看：
[root@bogon src]# ./redis-cli -c -h 10.90.7.10 -p 7001 10.90.7.10:7001> info # Server redis_version:3.2.8 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:b50c19b6d7e8c0c8 redis_mode:cluster os:Linux 3.10.0-229.el7.x86_64 x86_64 arch_bits:64 multiplexing_api:epoll gcc_version:4.8.3 process_id:40885 run_id:5b03ae96c0af7156d9ee0fae9ab96f9c56c7f6b7 tcp_port:7001 uptime_in_seconds:98920 uptime_in_days:1 hz:10 lru_clock:1399401 executable:/opt/redis/redis-3.2.8/src/redis-server config_file:/opt/redis/redis-3.2.8/tkcluster/7001.conf # Clients connected_clients:1 client_longest_output_list:0 client_biggest_input_buf:0 blocked_clients:0 # Memory used_memory:2418496 used_memory_human:2.31M used_memory_rss:10039296 used_memory_rss_human:9.57M used_memory_peak:2418496 used_memory_peak_human:2.31M total_system_memory:33450364928 total_system_memory_human:31.15G used_memory_lua:37888 used_memory_lua_human:37.00K maxmemory:0 maxmemory_human:0B maxmemory_policy:noeviction mem_fragmentation_ratio:4.15 mem_allocator:jemalloc-4.0.3 # Persistence loading:0 rdb_changes_since_last_save:0 rdb_bgsave_in_progress:0 rdb_last_save_time:1511347221 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:0 rdb_current_bgsave_time_sec:-1 aof_enabled:1 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_last_write_status:ok aof_current_size:0 aof_base_size:0 aof_pending_rewrite:0 aof_buffer_length:0 aof_rewrite_buffer_length:0 aof_pending_bio_fsync:0 aof_delayed_fsync:0 # Stats total_connections_received:5 total_commands_processed:1647 instantaneous_ops_per_sec:0 total_net_input_bytes:115303 total_net_output_bytes:6128379 instantaneous_input_kbps:0.02 instantaneous_output_kbps:0.00 rejected_connections:0 sync_full:1 sync_partial_ok:0 sync_partial_err:0 expired_keys:0 evicted_keys:0 keyspace_hits:0 keyspace_misses:0 pubsub_channels:0 pubsub_patterns:0 latest_fork_usec:789 migrate_cached_sockets:0 # Replication role:master connected_slaves:1 slave0:ip=10.90.7.2,port=7010,state=online,offset=2255,lag=1 master_repl_offset:2255 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:2 repl_backlog_histlen:2254 # CPU used_cpu_sys:35.23 used_cpu_user:19.00 used_cpu_sys_children:0.00 used_cpu_user_children:0.00 # Cluster cluster_enabled:1 # Keyspace 10.90.7.10:7001> info # Server redis_version:3.2.8 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:b50c19b6d7e8c0c8 redis_mode:cluster os:Linux 3.10.0-229.el7.x86_64 x86_64 arch_bits:64 multiplexing_api:epoll gcc_version:4.8.3 process_id:40885 run_id:5b03ae96c0af7156d9ee0fae9ab96f9c56c7f6b7 tcp_port:7001 uptime_in_seconds:98926 uptime_in_days:1 hz:10 lru_clock:1399407 executable:/opt/redis/redis-3.2.8/src/redis-server config_file:/opt/redis/redis-3.2.8/tkcluster/7001.conf # Clients connected_clients:1 client_longest_output_list:0 client_biggest_input_buf:0 blocked_clients:0 # Memory used_memory:2418496 used_memory_human:2.31M used_memory_rss:10039296 used_memory_rss_human:9.57M used_memory_peak:2418496 used_memory_peak_human:2.31M total_system_memory:33450364928 total_system_memory_human:31.15G used_memory_lua:37888 used_memory_lua_human:37.00K maxmemory:0 maxmemory_human:0B maxmemory_policy:noeviction mem_fragmentation_ratio:4.15 mem_allocator:jemalloc-4.0.3 # Persistence loading:0 rdb_changes_since_last_save:0 rdb_bgsave_in_progress:0 rdb_last_save_time:1511347221 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:0 rdb_current_bgsave_time_sec:-1 aof_enabled:1 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_last_write_status:ok aof_current_size:0 aof_base_size:0 aof_pending_rewrite:0 aof_buffer_length:0 aof_rewrite_buffer_length:0 aof_pending_bio_fsync:0 aof_delayed_fsync:0 # Stats total_connections_received:5 total_commands_processed:1655 instantaneous_ops_per_sec:1 total_net_input_bytes:115576 total_net_output_bytes:6130809 instantaneous_input_kbps:0.04 instantaneous_output_kbps:0.00 rejected_connections:0 sync_full:1 sync_partial_ok:0 sync_partial_err:0 expired_keys:0 evicted_keys:0 keyspace_hits:0 keyspace_misses:0 pubsub_channels:0 pubsub_patterns:0 latest_fork_usec:789 migrate_cached_sockets:0 # Replication role:master connected_slaves:1 slave0:ip=10.90.7.2,port=7010,state=online,offset=2269,lag=0 master_repl_offset:2269 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:2 repl_backlog_histlen:2268 # CPU used_cpu_sys:35.23 used_cpu_user:19.00 used_cpu_sys_children:0.00 used_cpu_user_children:0.00 # Cluster cluster_enabled:1 # Keyspace 10.90.7.10:7001> get hello -> Redirected to slot [866] located at 10.90.7.2:7000 "1" 10.90.7.2:7000>
選擇database的操作：
10.90.7.2:7000> select 0 OK 10.90.7.2:7000> select 1 (error) ERR SELECT is not allowed in cluster mode 10.90.7.2:7000>
特別說明，redis集群下，不支持database的非0的選擇，其實是不支持select指令了，但是對於默認的database 0又默認支持。有點設計的不美妙！
另外，備注：集群環境下，不支持slaveof的配置，即配置文件里面，不能啟動slaveof的配置。否則如我開始沒有注意到這個，將conf文件中的slaveof指定了。就出現下面的錯誤。
[root@localhost tkcluster]# ./stcluster.sh status from 7010: 0 *** FATAL CONFIG FILE ERROR *** Reading the configuration file, at line 265 >>> 'slaveof 10.90.7.2 7010' slaveof directive not allowed in cluster mode status from 7011: 1 [root@localhost tkcluster]#
OK，到此，redis的集群環境搭建到此結束，成功完成！