環境部署
1.服務器准備:centos7.4系統,Jdk1.8 cat /etc/redhat-release。Es6.x
如果是自帶的openjdk需要先卸載:
rpm -qa|grep java
rpm -e –nodeps *
2.ElasticSearch安裝:我采用的版本為6.6.0 https://elasticsearch.cn/download/
(1)地址:https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.6.0.tar.gz
(2)解壓:tar -zxvf elasticsearch/elasticsearch-6.6.0.tar.gz
(3)修改系統配置:
- 設置內核參數 :vim /etc/sysctl.conf
添加如下內容:
fs.file-max=65536
vm.max_map_count=262144
sysctl -p 刷新下配置,
sysctl -a查看是否生效 ,
如果不成功的(啟動es還是失敗,不是所有人都碰得到,好像是在7.6碰到了):
rm -f /sbin/modprobe
ln -s /bin/true /sbin/modprobe
rm -f /sbin/sysctl
ln -s /bin/true /sbin/sysctl
-
設置資源參數
vi /etc/security/limits.conf
# 添加一下內容:
* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096 -
修改進程數
vi /etc/security/limits.d/20-nproc.conf
* soft nproc 4096
配置完成后 要關掉鏈接窗口,重新打開一個
不能用root用戶啟動 -
修改es配置文件: vim elasticsearch安裝路徑/config/elasticsearch.yml
# 配置es的集群名稱, es會自動發現在同一網段下的es,如果在同一網段下有多個集群,就可以用這個屬性來區分不同的集群。
cluster.name: bi-cluster
# 節點名稱
node.name: node-master
# 存儲數據的目錄
path.data: /home/elasticsearch/data
# 存儲日志的目錄
path.logs: /home/elasticsearch/logs
# 設置綁定的ip地址還有其它節點和該節點交互的ip地址
network.host: 0.0.0.0
# 指定http端口,你使用head、kopf等相關插件使用的端口
http.port: 9200
# 設置節點間交互的tcp端口,默認是9300
transport.tcp.port: 9300
#設置集群中master節點的初始列表,可以通過這些節點來自動發現新加入集群的節點。
discovery.zen.ping.unicast.hosts: ["10.108.4.203:9300", "10.108.4.204:9300", "10.108.4.205:9300"]
#如果沒有這種設置,遭受網絡故障的集群就有可能將集群分成兩個獨立的集群 - 分裂的大腦 - 這將導致數據丟失
discovery.zen.minimum_master_nodes: 3
http.cors.enabled: true #es5.x版本以上需要,head訪問
http.cors.allow-origin: "*"
bootstrap.memory_lock: false #某些系統需要 是因為centos6.x操作系統不支持SecComp,而elasticsearch 5.5.2默認 bootstrap.system_call_filter為true進行檢測,所以導致檢測失敗,失敗后直接導致ES不能啟動。
我的配置如下:# ======================== Elasticsearch Configuration ==================== ===== # # NOTE: Elasticsearch comes with reasonable defaults for most settings. # Before you set out to tweak and tune the configuration, make sure y ou # understand what are you trying to accomplish and the consequences. # # The primary way of configuring a node is via this file. This template lis ts # the most important settings you may want to configure for a production cl uster. # # Please consult the documentation for further information on configuration options: # https://www.elastic.co/guide/en/elasticsearch/reference/index.html # # ---------------------------------- Cluster ------------------------------ ----- # # Use a descriptive name for your cluster: # #cluster.name: my-application # # ------------------------------------ Node ------------------------------- ----- # # Use a descriptive name for the node: # #node.name: node-1 # # Add custom attributes to the node: # #node.attr.rack: r1 # # ----------------------------------- Paths ------------------------------- ----- # # Path to directory where to store the data (separate multiple locations by comma): # #path.data: /path/to/data # # Path to log files: # #path.logs: /path/to/logs # # ----------------------------------- Memory ------------------------------ ----- # # Lock the memory on startup: # #bootstrap.memory_lock: true # # Make sure that the heap size is set to about half the memory available # on the system and that the owner of the process is allowed to use this # limit. # # Elasticsearch performs poorly when the system is swapping the memory. # # ---------------------------------- Network ------------------------------ ----- # # Set the bind address to a specific IP (IPv4 or IPv6): # #network.host: 192.168.0.1 # # Set a custom port for HTTP: # #http.port: 9200 # # For more information, consult the network module documentation. # # --------------------------------- Discovery ----------------------------- ----- # # Pass an initial list of hosts to perform discovery when new node is start ed: # The default list of hosts is ["127.0.0.1", "[::1]"] # #discovery.zen.ping.unicast.hosts: ["host1", "host2"] # # Prevent the "split brain" by configuring the majority of nodes (total num ber of master-eligible nodes / 2 + 1): # #discovery.zen.minimum_master_nodes: # # For more information, consult the zen discovery module documentation. # # ---------------------------------- Gateway ------------------------------ ----- # # Block initial recovery after a full cluster restart until N nodes are sta rted: # #gateway.recover_after_nodes: 3 # # For more information, consult the gateway module documentation. # # ---------------------------------- Various ------------------------------ ----- # # Require explicit names when deleting indices: # #action.destructive_requires_name: true cluster.name: my-es node.name: node-128 network.host: 0.0.0.0 http.port: 19200 transport.tcp.port: 19300 #discovery.zen.ping.unicast.hosts: ["127.0.0.1:9300"] #discovery.zen.minimum_master_nodes: 3 http.cors.enabled: true http.cors.allow-origin: "*" bootstrap.memory_lock: false bootstrap.system_call_filter: false
-
啟動es:./bin/elasticsearch 后台啟動加 –d參數
-
打開瀏覽器訪問 http://你的ip:19200/查看是否能夠正常訪問,看到以下界面表示啟動ok
如果在外網訪問失敗的話,請按照以下步驟執行:
- 檢查你的elasticsearch.yml配置文件的以下三處是否配置正確
network.host: 0.0.0.0
bootstrap.memory_lock: false bootstrap.system_call_filter: false
- 檢查防火牆是否關閉(需要關閉防火牆外網才可以訪問)
開啟、重啟、關閉、firewalld.service服務
# 開啟
service firewalld start
# 重啟
service firewalld restart
# 關閉
service firewalld stop
查看防火牆規則:firewall-cmd --list-all
查詢、開放、關閉端口:# 查詢端口是否開放
執行以上步驟之后就可以在外網訪問了。以下是我遇到的啟動錯誤的問題以及解決辦法:
firewall-cmd --query-port=8080/tcp
# 開放80端口
firewall-cmd --permanent --add-port=80/tcp
# 移除端口
firewall-cmd --permanent --remove-port=8080/tcp
#重啟防火牆(修改配置后要重啟防火牆)
firewall-cmd --reload - (1)max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
每個進程最大同時打開文件數太小,可通過下面2個命令查看當前數量 -
ulimit -Hn ulimit -Sn
修改/etc/security/limits.conf文件,增加配置,用戶退出后重新登錄生效
* soft nofile 65536 * hard nofile 65536
(2)max number of threads [3818] for user [es] is too low, increase to at least [4096]
問題同上,最大線程個數太低。修改配置文件/etc/security/limits.conf(和問題1是一個文件),增加配置
* soft nproc 4096 * hard nproc 4096
可通過命令查看
ulimit -Hu ulimit -Su
修改后的文件:
如果更改后發現ulimit -Su 的結果沒有變化的話,此時需要在這個文件中添加如下信息:(admin是我自己的用戶名稱,可以根據自己的實際用戶更改)
admin - nproc 4096
-
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
修改/etc/sysctl.conf文件,增加配置vm.max_map_count=262144
vi /etc/sysctl.conf
sysctl -p執行命令sysctl -p生效
4、Exception in thread "main" java.nio.file.AccessDeniedException: /usr/local/elasticsearch/elasticsearch-6.2.2-1/config/jvm.options
elasticsearch用戶沒有該文件夾的權限,執行命令
chown -R es:es /usr/local/elasticsearch/
安裝kibana:
1、將下載的kibana安裝包解壓: tar -zxvf kibana-6.6.0-linux-x86_64.tar.gz
2、更改kibana的配置
# Kibana is served by a back end server. This setting specifies the port to use. server.port: 15601 # Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values. # The default is 'localhost', which usually means remote machines will not be able to connect. # To allow connections from remote users, set this parameter to a non-loopb ack address. #server.host: "localhost" server.host: "0.0.0.0" # Enables you to specify a path to mount Kibana at if you are running behin d a proxy. # Use the `server.rewriteBasePath` setting to tell Kibana if it should remo ve the basePath # from requests it receives, and to prevent a deprecation warning at startu p. # This setting cannot end in a slash. #server.basePath: "" # Specifies whether Kibana should rewrite requests that are prefixed with # `server.basePath` or require that they are rewritten by your reverse prox y. # This setting was effectively always `false` before Kibana 6.3 and will # default to `true` starting in Kibana 7.0. #server.rewriteBasePath: false # The maximum payload size in bytes for incoming server requests. #server.maxPayloadBytes: 1048576 # The Kibana server's name. This is used for display purposes. #server.name: "your-hostname" # The URLs of the Elasticsearch instances to use for all your queries. #elasticsearch.hosts: ["http://localhost:9200"] elasticsearch.hosts: ["http://localhost:19200"] # When this setting's value is true Kibana uses the hostname specified in t he server.host # setting. When the value of this setting is false, Kibana uses the hostnam e of the host # that connects to this Kibana instance. #elasticsearch.preserveHost: true # Kibana uses an index in Elasticsearch to store saved searches, visualizat ions and # dashboards. Kibana creates a new index if the index doesn't already exist . #kibana.index: ".kibana" # The default application to load. #kibana.defaultAppId: "home" # If your Elasticsearch is protected with basic authentication, these setti ngs provide # the username and password that the Kibana server uses to perform maintena nce on the Kibana # index at startup. Your Kibana users still need to authenticate with Elast icsearch, which # is proxied through the Kibana server. #elasticsearch.username: "user" #elasticsearch.password: "pass" # Enables SSL and paths to the PEM-format SSL certificate and SSL key files , respectively. # These settings enable SSL for outgoing requests from the Kibana server to the browser. #server.ssl.enabled: false #server.ssl.certificate: /path/to/your/server.crt #server.ssl.key: /path/to/your/server.key # Optional settings that provide the paths to the PEM-format SSL certificat e and key files. # These files validate that your Elasticsearch backend uses the same key fi les. #elasticsearch.ssl.certificate: /path/to/your/client.crt #elasticsearch.ssl.key: /path/to/your/client.key # Optional setting that enables you to specify a path to the PEM file for t he certificate # authority for your Elasticsearch instance. #elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ] # To disregard the validity of SSL certificates, change this setting's valu e to 'none'. #elasticsearch.ssl.verificationMode: full # Time in milliseconds to wait for Elasticsearch to respond to pings. Defau lts to the value of # the elasticsearch.requestTimeout setting. #elasticsearch.pingTimeout: 1500 # Time in milliseconds to wait for responses from the back end or Elasticse arch. This value # must be a positive integer. #elasticsearch.requestTimeout: 30000 # List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side # headers, set this value to [] (an empty list). #elasticsearch.requestHeadersWhitelist: [ authorization ] # Header names and values that are sent to Elasticsearch. Any custom header s cannot be overwritten # by client-side headers, regardless of the elasticsearch.requestHeadersWhi telist configuration. #elasticsearch.customHeaders: {} # Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable. #elasticsearch.shardTimeout: 30000 # Time in milliseconds to wait for Elasticsearch at Kibana startup before r etrying. #elasticsearch.startupTimeout: 5000 [admin@localhost kibana-6.6.0-linux-x86_64]$ cat config/kibana.yml # Kibana is served by a back end server. This setting specifies the port to use. server.port: 15601 # Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values. # The default is 'localhost', which usually means remote machines will not be able to connect. # To allow connections from remote users, set this parameter to a non-loopb ack address. #server.host: "localhost" server.host: "0.0.0.0" # Enables you to specify a path to mount Kibana at if you are running behin d a proxy. # Use the `server.rewriteBasePath` setting to tell Kibana if it should remo ve the basePath # from requests it receives, and to prevent a deprecation warning at startu p. # This setting cannot end in a slash. #server.basePath: "" # Specifies whether Kibana should rewrite requests that are prefixed with # `server.basePath` or require that they are rewritten by your reverse prox y. # This setting was effectively always `false` before Kibana 6.3 and will # default to `true` starting in Kibana 7.0. #server.rewriteBasePath: false # The maximum payload size in bytes for incoming server requests. #server.maxPayloadBytes: 1048576 # The Kibana server's name. This is used for display purposes. #server.name: "your-hostname" # The URLs of the Elasticsearch instances to use for all your queries. #elasticsearch.hosts: ["http://localhost:9200"] elasticsearch.hosts: ["http://localhost:19200"] # When this setting's value is true Kibana uses the hostname specified in t he server.host # setting. When the value of this setting is false, Kibana uses the hostnam e of the host # that connects to this Kibana instance. #elasticsearch.preserveHost: true # Kibana uses an index in Elasticsearch to store saved searches, visualizat ions and # dashboards. Kibana creates a new index if the index doesn't already exist . #kibana.index: ".kibana" # The default application to load. #kibana.defaultAppId: "home" # If your Elasticsearch is protected with basic authentication, these setti ngs provide # the username and password that the Kibana server uses to perform maintena nce on the Kibana # index at startup. Your Kibana users still need to authenticate with Elast icsearch, which # is proxied through the Kibana server. #elasticsearch.username: "user" #elasticsearch.password: "pass" # Enables SSL and paths to the PEM-format SSL certificate and SSL key files , respectively. # These settings enable SSL for outgoing requests from the Kibana server to the browser. #server.ssl.enabled: false #server.ssl.certificate: /path/to/your/server.crt #server.ssl.key: /path/to/your/server.key # Optional settings that provide the paths to the PEM-format SSL certificat e and key files. # These files validate that your Elasticsearch backend uses the same key fi les. #elasticsearch.ssl.certificate: /path/to/your/client.crt #elasticsearch.ssl.key: /path/to/your/client.key # Optional setting that enables you to specify a path to the PEM file for t he certificate # authority for your Elasticsearch instance. #elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ] # To disregard the validity of SSL certificates, change this setting's valu e to 'none'. #elasticsearch.ssl.verificationMode: full # Time in milliseconds to wait for Elasticsearch to respond to pings. Defau lts to the value of # the elasticsearch.requestTimeout setting. #elasticsearch.pingTimeout: 1500 # Time in milliseconds to wait for responses from the back end or Elasticse arch. This value # must be a positive integer. #elasticsearch.requestTimeout: 30000 # List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side # headers, set this value to [] (an empty list). #elasticsearch.requestHeadersWhitelist: [ authorization ] # Header names and values that are sent to Elasticsearch. Any custom header s cannot be overwritten # by client-side headers, regardless of the elasticsearch.requestHeadersWhi telist configuration. #elasticsearch.customHeaders: {} # Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable. #elasticsearch.shardTimeout: 30000 # Time in milliseconds to wait for Elasticsearch at Kibana startup before r etrying. #elasticsearch.startupTimeout: 5000 # Logs queries sent to Elasticsearch. Requires logging.verbose set to true. #elasticsearch.logQueries: false # Specifies the path where Kibana creates the process ID file. #pid.file: /var/run/kibana.pid # Enables you specify a file where Kibana stores log output. #logging.dest: stdout # Set the value of this setting to true to suppress all logging output. #logging.silent: false # Set the value of this setting to true to suppress all logging output othe r than error messages. #logging.quiet: false # Set the value of this setting to true to log all events, including system usage information # and all requests. #logging.verbose: false # Set the interval in milliseconds to sample system and process performance # metrics. Minimum is 100ms. Defaults to 5000. #ops.interval: 5000 # Specifies locale to be used for all localizable strings, dates and number formats. #i18n.locale: "en"
3、啟動kibana: ./bin/kibana
4、外網通過ip地址訪問:
注意兩個點:
(1)修改啟動的es鏈接地址
(2)修改地址綁定不然只有本地才能訪問
(3)kibana的版本和elasticsearch的版本必須一致,比如都是6.6
elasticsearch集群搭建:
1、將以上安裝成功的elastic解壓包拷貝一份,重新命名為XXXX-node;
2、將elasticsearch-6.6.0里面的配置文件打開,elasticsearch.yml,修改以下標記內容
# ======================== Elasticsearch Configuration ================== ======= # # NOTE: Elasticsearch comes with reasonable defaults for most settings. # Before you set out to tweak and tune the configuration, make sure you # understand what are you trying to accomplish and the consequences . # # The primary way of configuring a node is via this file. This template l ists # the most important settings you may want to configure for a production cluster. # # Please consult the documentation for further information on configurati on options: # https://www.elastic.co/guide/en/elasticsearch/reference/index.html # # ---------------------------------- Cluster ---------------------------- ------- # # Use a descriptive name for your cluster: # #cluster.name: my-application # # ------------------------------------ Node ----------------------------- ------- # # Use a descriptive name for the node: # #node.name: node-1 # # Add custom attributes to the node: # #node.attr.rack: r1 # # ----------------------------------- Paths ----------------------------- ------- # # Path to directory where to store the data (separate multiple locations by comma): # #path.data: /path/to/data # # Path to log files: # #path.logs: /path/to/logs # # ----------------------------------- Memory ---------------------------- ------- # # Lock the memory on startup: # #bootstrap.memory_lock: true # # Make sure that the heap size is set to about half the memory available # on the system and that the owner of the process is allowed to use this # limit. # # Elasticsearch performs poorly when the system is swapping the memory. # # ---------------------------------- Network ---------------------------- ------- # # Set the bind address to a specific IP (IPv4 or IPv6): # #network.host: 192.168.0.1 # # Set a custom port for HTTP: # #http.port: 9200 # # For more information, consult the network module documentation. # # --------------------------------- Discovery --------------------------- ------- # # Pass an initial list of hosts to perform discovery when new node is sta rted: # The default list of hosts is ["127.0.0.1", "[::1]"] # #discovery.zen.ping.unicast.hosts: ["host1", "host2"] # # Prevent the "split brain" by configuring the majority of nodes (total n umber of master-eligible nodes / 2 + 1): # #discovery.zen.minimum_master_nodes: # # For more information, consult the zen discovery module documentation. # # ---------------------------------- Gateway ---------------------------- ------- # # Block initial recovery after a full cluster restart until N nodes are s tarted: # #gateway.recover_after_nodes: 3 # # For more information, consult the gateway module documentation. # # ---------------------------------- Various ---------------------------- ------- # # Require explicit names when deleting indices: # #action.destructive_requires_name: true cluster.name: my-es node.name: node-128 network.host: 0.0.0.0 http.port: 19200 transport.tcp.port: 19300 discovery.zen.ping.unicast.hosts: ["127.0.0.1:19300","127.0.0.1:29300"] #discovery.zen.minimum_master_nodes: 3 http.cors.enabled: true http.cors.allow-origin: "*" bootstrap.memory_lock: false bootstrap.system_call_filter: false
3、修改好以后打開拷貝的那個elasticsearch_node1,也打開該文件,修改:
# ======================== Elasticsearch Configuration ================== ======= # # NOTE: Elasticsearch comes with reasonable defaults for most settings. # Before you set out to tweak and tune the configuration, make sure you # understand what are you trying to accomplish and the consequences . # # The primary way of configuring a node is via this file. This template l ists # the most important settings you may want to configure for a production cluster. # # Please consult the documentation for further information on configurati on options: # https://www.elastic.co/guide/en/elasticsearch/reference/index.html # # ---------------------------------- Cluster ---------------------------- ------- # # Use a descriptive name for your cluster: # #cluster.name: my-application # # ------------------------------------ Node ----------------------------- ------- # # Use a descriptive name for the node: # #node.name: node-1 # # Add custom attributes to the node: # #node.attr.rack: r1 # # ----------------------------------- Paths ----------------------------- ------- # # Path to directory where to store the data (separate multiple locations by comma): # #path.data: /path/to/data # # Path to log files: # #path.logs: /path/to/logs # # ----------------------------------- Memory ---------------------------- ------- # # Lock the memory on startup: # #bootstrap.memory_lock: true # # Make sure that the heap size is set to about half the memory available # on the system and that the owner of the process is allowed to use this # limit. # # Elasticsearch performs poorly when the system is swapping the memory. # # ---------------------------------- Network ---------------------------- ------- # # Set the bind address to a specific IP (IPv4 or IPv6): # #network.host: 192.168.0.1 # # Set a custom port for HTTP: # #http.port: 9200 # # For more information, consult the network module documentation. # # --------------------------------- Discovery --------------------------- ------- # # Pass an initial list of hosts to perform discovery when new node is sta rted: # The default list of hosts is ["127.0.0.1", "[::1]"] # #discovery.zen.ping.unicast.hosts: ["host1", "host2"] # # Prevent the "split brain" by configuring the majority of nodes (total n umber of master-eligible nodes / 2 + 1): # #discovery.zen.minimum_master_nodes: # # For more information, consult the zen discovery module documentation. # # ---------------------------------- Gateway ---------------------------- ------- # # Block initial recovery after a full cluster restart until N nodes are s tarted: # #gateway.recover_after_nodes: 3 # # For more information, consult the gateway module documentation. # # ---------------------------------- Various ---------------------------- ------- # # Require explicit names when deleting indices: # #action.destructive_requires_name: true cluster.name: my-es node.name: node-128-1 network.host: 0.0.0.0 http.port: 29200 transport.tcp.port: 29300 discovery.zen.ping.unicast.hosts: ["127.0.0.1:19300","127.0.0.1:29300"] #discovery.zen.minimum_master_nodes: 3 http.cors.enabled: true http.cors.allow-origin: "*" bootstrap.memory_lock: false bootstrap.system_call_filter: false
4、刪除node1文件路徑下data文件夾下的所有東西,因為這些是從原來的那里拷貝過來的,有問題:
cd elasticsearch_node1/data(你自己節點的路徑)
rm -rf *
5、然后啟動兩個節點就可以了,集群就搭建好了。
注意:我是在一台虛擬機上搭建的,所以ip地址一樣,端口不一樣,如果是不同的虛擬機的話ip地址不一樣,端口可以一樣
指南:https://www.elastic.co/guide/cn/elasticsearch/guide/cn/index.html