elasticsearch-7.2.0 在windows環境的部署應用

1、下載解壓版"elasticsearch-7.2.0-windows-x86_64.zip",解壓至"D:\server\elasticsearch-7.2.0"目錄；

2、下載“elasticsearch-analysis-ik-7.2.0.zip”，解壓至“D:\server\elasticsearch-7.2.0\plugins\elasticsearch-analysis-ik-7.2.0”；

3、配置“D:\server\elasticsearch-7.2.0\config\elasticsearch.yml”文件：

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["host1"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["node-1"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true

4、啟動服務“D:\server\elasticsearch-7.2.0\bin\elasticsearch.bat”

5、下載解壓curl-7.65.1-win64-mingw.zip至“D:\curl-7.65.1-win64-mingw”，配置系統環境path增加路徑“D:\curl-7.65.1-win64-mingw\bin”；

6、運行dos命令， 執行“curl -help”驗證是否配置成功；

7、在DOS窗口中，設置用戶密碼：

D:\server\elasticsearch-7.2.0\bin>elasticsearch-setup-passwords auto

Changed password for user apm_system
PASSWORD apm_system = wXP7CnMmpulr70PrhOy9

Changed password for user kibana
PASSWORD kibana = 16P4FhGKQjRxTkI3c8KR

Changed password for user logstash_system
PASSWORD logstash_system = 1lyczLndTf2xACdJEt7U

Changed password for user beats_system
PASSWORD beats_system = jZRMVw05cQD8xxuCEleY

Changed password for user remote_monitoring_user
PASSWORD remote_monitoring_user = MR44yUAc046FihQDhO43

Changed password for user elastic
PASSWORD elastic = N0AE4qdTZp0l6zJzGNZ1

記住以上自動生成的密碼；

8、在DOS窗口中，運行以下命令，創建索引並設置分詞器，過濾器：

curl -u elastic:9lHDbltrSSIPmcjexseY -XPUT "http://192.168.1.230:9200/xuetz" -H "Content-Type:application/json" -d "{\"settings\":{\"analysis\":{\"analyzer\":{\"myAnalyzer\":{\"tokenizer\":\"ik_max_word\",\"char_filter\":[\"html_strip\"]}}}}}"

9、在DOS窗口中，運行以下命令，設置各個field屬性對應的解析器：

curl -u elastic:9lHDbltrSSIPmcjexseY -XPUT "http://192.168.1.230:9200/xuetz/_mapping" -H "Content-Type:application/json" -d "{\"properties\":{\"opName\":{\"type\":\"text\",\"analyzer\":\"myAnalyzer\",\"search_analyzer\":\"ik_smart\"},\"brief\":{\"type\":\"text\",\"analyzer\":\"myAnalyzer\",\"search_analyzer\":\"ik_smart\"},\"editor\":{\"type\":\"text\",\"analyzer\":\"myAnalyzer\",\"search_analyzer\":\"ik_smart\"},\"keywords\":{\"type\":\"text\",\"analyzer\":\"myAnalyzer\",\"search_analyzer\":\"ik_smart\"}}}"

如一次性返回buckets數量大於10000,需設置search.max_buckets參數：

curl -u elastic:9lHDbltrSSIPmcjexseY -XPUT "http://192.168.1.230:9200/_cluster/settings" -H "Content-Type:application/json" -d   "{\"persistent\":{\"search.max_buckets\":50000}}"

10、JAVA采用Java High Level REST Client 調用

10.1  ESBean.java

package com.xrh.extend.elasticsearch;

public class ESBean implements java.io.Serializable {

    /**
     * 
     */
    private static final long serialVersionUID = -8973010685088776452L;

    private String id;       //ID
    private String opName;   //標題
    private String opNameHL; //標題-高亮
    private String brief;    //簡介
    private String briefHL;  //簡介-高亮
    private String keywords; //關鍵字
    private String keyWordsHL;//關鍵字-高亮
    private String editor;   //內容
    private String editorHL; //內容-高亮
    private String picUrl;   //封面圖
    private String opSite;   //站點
    private String parentId; //所屬欄目ID或父級ID
    private String publishDate; //發布時間
    private String tableName; //表名
    
    private float score; //文檔評分
    
    public String getId() {
        return id;
    }
    public void setId(String id) {
        this.id = id;
    }
    public String getOpName() {
        return opName;
    }
    public void setOpName(String opName) {
        this.opName = opName;
    }
    public String getBrief() {
        return brief;
    }
    public void setBrief(String brief) {
        this.brief = brief;
    }
    public String getKeywords() {
        return keywords;
    }
    public void setKeywords(String keywords) {
        this.keywords = keywords;
    }
    public String getEditor() {
        return editor;
    }
    public void setEditor(String editor) {
        this.editor = editor;
    }
    public String getOpSite() {
        return opSite;
    }
    public void setOpSite(String opSite) {
        this.opSite = opSite;
    }
    public String getParentId() {
        return parentId;
    }
    public void setParentId(String parentId) {
        this.parentId = parentId;
    }
    public String getPublishDate() {
        return publishDate;
    }
    public void setPublishDate(String publishDate) {
        this.publishDate = publishDate;
    }
    public String getTableName() {
        return tableName;
    }
    public void setTableName(String tableName) {
        this.tableName = tableName;
    }
    public String getOpNameHL() {
        if (opNameHL == null){
            return opName;
        }
        return opNameHL;
    }
    public void setOpNameHL(String opNameHL) {
        this.opNameHL = opNameHL;
    }
    public String getBriefHL() {
        if (briefHL == null){
            return brief;
        }
        return briefHL;
    }
    public void setBriefHL(String briefHL) {
        this.briefHL = briefHL;
    }
    public String getKeyWordsHL() {
        if (keyWordsHL == null){
            return keywords;
        }
        return keyWordsHL;
    }
    public void setKeyWordsHL(String keyWordsHL) {
        this.keyWordsHL = keyWordsHL;
    }
    public String getEditorHL() {
        if (editorHL == null){
            return editor;
        }
        return editorHL;
    }
    public void setEditorHL(String editorHL) {
        this.editorHL = editorHL;
    }
    public float getScore() {
        return score;
    }
    public void setScore(float score) {
        this.score = score;
    }
    public String getPicUrl() {
        return picUrl;
    }
    public void setPicUrl(String picUrl) {
        this.picUrl = picUrl;
    }
    
    
}

10.2  ESResult.java

package com.xrh.extend.elasticsearch;

import java.util.List;

public class ESResult {

    private List<ESBean> datas;
    private long totalHits;
    
    public List<ESBean> getDatas() {
        return datas;
    }
    public void setDatas(List<ESBean> datas) {
        this.datas = datas;
    }
    public long getTotalHits() {
        return totalHits;
    }
    public void setTotalHits(long totalHits) {
        this.totalHits =totalHits;
    }
    
    
}

10.3 ESTool.java

package com.xrh.extend.elasticsearch;

import java.io.IOException;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.logging.Logger;

import org.apache.http.HttpHost;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.CredentialsProvider;
import org.apache.http.client.config.RequestConfig.Builder;
import org.apache.http.impl.client.BasicCredentialsProvider;
import org.apache.http.impl.nio.client.HttpAsyncClientBuilder;
import org.elasticsearch.action.DocWriteResponse;
import org.elasticsearch.action.delete.DeleteRequest;
import org.elasticsearch.action.delete.DeleteResponse;
import org.elasticsearch.action.search.SearchRequest;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.action.update.UpdateRequest;
import org.elasticsearch.action.update.UpdateResponse;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestClientBuilder;
import org.elasticsearch.client.RestClientBuilder.HttpClientConfigCallback;
import org.elasticsearch.client.RestClientBuilder.RequestConfigCallback;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.common.text.Text;
import org.elasticsearch.common.unit.TimeValue;
import org.elasticsearch.common.xcontent.XContentBuilder;
import org.elasticsearch.common.xcontent.XContentFactory;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.rest.RestStatus;
import org.elasticsearch.search.SearchHit;
import org.elasticsearch.search.SearchHits;
import org.elasticsearch.search.builder.SearchSourceBuilder;
import org.elasticsearch.search.fetch.subphase.highlight.HighlightBuilder;
import org.elasticsearch.search.fetch.subphase.highlight.HighlightField;

import com.xrh.base.config.DAO_Config;
import com.xrh.core.util.JsonUtil;
import com.xrh.core.util.ObjectUtil;

public class ESTool {

    private static Logger logger = Logger.getLogger(ESTool.class.getName());
    
    private RestHighLevelClient restClient = null;

    public ESTool(){
        
        String hosts = DAO_Config.getConfigValue("es-hosts"); // 集群地址
        String port = DAO_Config.getConfigValue("es-port"); // 使用的端口號
        String schema = DAO_Config.getConfigValue("es-schema"); // 使用的協議
        
        if (ObjectUtil.isNull(hosts)){
            logger.info("ES集群地址不能為空！無法完成初始化.");
            return ;
        }
        
        if (ObjectUtil.isNull(port)){
            logger.info("ES端口后不能為空！無法完成初始化.");
            return ;
        }
        
        if (ObjectUtil.isNull(schema)){
            logger.info("ES協議不能為空！無法完成初始化.");
            return ;
        }
        
        ArrayList<HttpHost> hostList = new ArrayList<>();
        String[] hostStrs = hosts.split(",");
        for (String host : hostStrs) {
            hostList.add(new HttpHost(host, Integer.parseInt(port), schema));
        }
        
        RestClientBuilder builder = RestClient.builder(hostList.toArray(new HttpHost[0]))
                .setFailureListener(new RestClient.FailureListener() { // 連接失敗策略
                    //
                }).setRequestConfigCallback(new RequestConfigCallback() {
                    @Override
                    public Builder customizeRequestConfig(Builder requestConfigBuilder) {
                        String connectTimeOut = DAO_Config.getConfigValue("es-connectTimeOut");
                        if (!ObjectUtil.isNull(connectTimeOut)){
                            requestConfigBuilder.setConnectTimeout(Integer.parseInt(connectTimeOut));
                        }
                        String socketTimeout = DAO_Config.getConfigValue("es-socketTimeout");
                        if (!ObjectUtil.isNull(socketTimeout)){
                            requestConfigBuilder.setSocketTimeout(Integer.parseInt(socketTimeout));
                        }
                        String connectionRequestTimeout = DAO_Config.getConfigValue("es-connectionRequestTimeout");
                        if (!ObjectUtil.isNull(connectionRequestTimeout)){
                            requestConfigBuilder.setConnectionRequestTimeout(Integer.parseInt(connectionRequestTimeout));
                        }
                        return requestConfigBuilder;
                    }
                })
                .setHttpClientConfigCallback(new HttpClientConfigCallback() { // 認證
                    @Override
                    public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {
                        
                        String maxConnTotal = DAO_Config.getConfigValue("es-maxConnTotal");
                        if (!ObjectUtil.isNull(maxConnTotal)){
                            httpClientBuilder.setMaxConnTotal(Integer.parseInt(maxConnTotal));
                        }
                        
                        String maxConnPerRoute = DAO_Config.getConfigValue("es-maxConnPerRoute");
                        if (!ObjectUtil.isNull(maxConnPerRoute)){
                            httpClientBuilder.setMaxConnTotal(Integer.parseInt(maxConnPerRoute));
                        }

                        String userName = DAO_Config.getConfigValue("es-userName");
                        String password = DAO_Config.getConfigValue("es-password");
                        if (ObjectUtil.isNull(userName) || ObjectUtil.isNull(password)){
                            return httpClientBuilder;
                        }else{
                            CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
                            credentialsProvider.setCredentials(AuthScope.ANY,
                                    new UsernamePasswordCredentials(userName, password));
                            return httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
                        }
                    }
                });
        
        restClient = new RestHighLevelClient(builder);
    }
    
    /**
     * 生成索引文檔（重復則自動更新）
     * @param bean
     * @return
     */
    public boolean doSave(ESBean bean){
        
        if (restClient == null){
            logger.info("系統未配置搜索服務參數，無法完成索引保存操作！");
            return false;
        }
        
        if(bean == null){
            logger.info("bean 對象為空，無法創建索引！");
            return false;
        }
        
        if (ObjectUtil.isNull(bean.getOpSite())){
            logger.info("對象opSite為空，無法創建索引！");
            return false;
        }
        
        if (ObjectUtil.isNull(bean.getTableName())){
            logger.info("對象tableName為空，無法生成文檔唯一ID！");
            return false;
        }
        
        if (ObjectUtil.isNull(bean.getId())){
            logger.info("對象id為空，無法生成文檔唯一ID！");
            return false;
        }
        
        XContentBuilder builder = getXContentBuilder(bean);
        UpdateRequest request = new UpdateRequest(bean.getOpSite(), 
                    generateId(bean.getTableName(), bean.getId()))
                    .doc(builder)
                    .upsert(builder);//upsert--id不存在時就插入
        try {
            UpdateResponse response = restClient.update(request, RequestOptions.DEFAULT);
            response.forcedRefresh();
            if (response.status().equals(RestStatus.OK) 
                    || response.status().equals(RestStatus.CREATED)){
                logger.info("索引文檔【id="+bean.getId()+" opName="+bean.getOpName()+"】保存成功!");
                return true;
            }else{
                logger.info("保存失敗，狀態="+response.status()+" "+response.toString());
                return false;
            }
        } catch (IOException e1) {
            // TODO Auto-generated catch block
            e1.printStackTrace();
            return false;
        }
    }
    
    /**
     * 刪除索引文檔
     * @param site       站點ID
     * @param tableName  表名
     * @param id         主鍵
     * @return
     */
    public boolean doDelete(String site, String tableName, String id){
        
        if (restClient == null){
            logger.info("系統未配置搜索服務參數，無法完成索引刪除操作！");
            return false;
        }
        
        if (ObjectUtil.isNull(site)){
            logger.info("site為空，定位索引！");
            return false;
        }
        
        DeleteRequest request = new DeleteRequest()
                .index(site)
                .id(generateId(tableName, id));
        try {
            DeleteResponse deleteResponse = restClient.delete(
                    request, RequestOptions.DEFAULT);
            if (deleteResponse.getResult() == DocWriteResponse.Result.NOT_FOUND) {
                logger.info("未找到要刪除的文檔【"+generateId(tableName, id)+"】");
                return true;
            }
            if (deleteResponse.status().equals(RestStatus.OK)){
                logger.info("文檔【"+generateId(tableName, id)+"】刪除成功！");
                return true;
            }else{
                logger.info("刪除狀態="+deleteResponse.status() + " "+deleteResponse.toString());
                return false;
            }
            
        } catch (IOException e1) {
            // TODO Auto-generated catch block
            e1.printStackTrace();
            return false;
        }
    }
    
    /**
     * 搜索文檔
     * @param site            站點ID
     * @param sourceBuilder   searchSourceBuilder
     * @param from            起始位置
     * @param size            獲取條數
     * @param highlightFields 高亮屬性
     * @return
     */
    public ESResult doSearch(String site, 
            SearchSourceBuilder sourceBuilder,
            int from,
            int size,
            String[] highlightFields){
        
        if (restClient == null){
            logger.info("系統未配置搜索服務參數，無法完成檢索操作！");
            return null;
        }
        
        if (ObjectUtil.isNull(site)){
            logger.info("site為空，定位索引！");
            return null;
        }
        
        SearchRequest searchRequest = new SearchRequest();
        searchRequest.indices(site);
        
        sourceBuilder.from(from); //設置from選項，確定要開始搜索的結果索引。 默認為0。
        sourceBuilder.size(size); //設置大小選項，確定要返回的搜索匹配數。 默認為10。
        String timeout = DAO_Config.getConfigValue("es-timeout"); //搜索超時時間
        if (!ObjectUtil.isNull(timeout)){
            sourceBuilder.timeout(new TimeValue(Integer.parseInt(timeout)));
        }
        
        if (highlightFields != null){
            String preTags = DAO_Config.getConfigValue("es-preTags"); //高亮前綴
            String postTags = DAO_Config.getConfigValue("es-postTags"); //高亮后綴
            HighlightBuilder highlightBuilder = new HighlightBuilder();
            for (String highlightField : highlightFields){
                HighlightBuilder.Field field = new HighlightBuilder.Field(highlightField);
                highlightBuilder.field(field);
            }
            if (!ObjectUtil.isNull(preTags) 
                    && !ObjectUtil.isNull(postTags)){
                highlightBuilder.preTags(preTags);
                highlightBuilder.postTags(postTags); 
            }
            sourceBuilder.highlighter(highlightBuilder);
        }
        
        searchRequest.source(sourceBuilder);
        
        try {
            SearchResponse response = restClient.search(searchRequest, RequestOptions.DEFAULT);
            logger.info("文檔搜索結果 status="+response.status());
            ESResult esResult = new ESResult();
            
            SearchHits searchHits = response.getHits();
            esResult.setTotalHits(searchHits.getTotalHits().value);
            
            List<ESBean> datas = new ArrayList();
            Iterator<SearchHit> iterator = searchHits.iterator();
            while (iterator.hasNext()){
                SearchHit searchHit = iterator.next();
                ESBean data = JsonUtil.json2Obj(searchHit.getSourceAsString(), ESBean.class);
                data.setScore(searchHit.getScore());
                
                if (highlightFields != null){
                    Map<String, HighlightField> hlFieldsMap = searchHit.getHighlightFields();
                    
                    for (String highlightField : highlightFields){
                        if (hlFieldsMap.containsKey(highlightField)){
                            HighlightField highlight = hlFieldsMap.get(highlightField);
                            Text[] fragments = highlight.fragments();
                            if (fragments.length > 0){
                                StringBuffer sb = new StringBuffer();;
                                for (Text fragment : fragments){
                                    sb.append(fragment.string());
                                }
                                if (highlightField.toLowerCase().equals("opname")){
                                    data.setOpNameHL(sb.toString());
                                } else if (highlightField.toLowerCase().equals("brief")){
                                    data.setBriefHL(sb.toString());
                                } else if (highlightField.toLowerCase().equals("keywords")){
                                    data.setKeyWordsHL(sb.toString());
                                } else if (highlightField.toLowerCase().equals("editor")){
                                    data.setEditorHL(fiterHtmlTag(sb.toString()));
                                } else {
                                    logger.info("highlightField【"+highlightField+"】不在ESBean屬性范圍！");
                                }
                            }
                        }
                    }
                }
                datas.add(data);
            }
            esResult.setDatas(datas);
            return esResult;
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
        return null;
    }
    
    /**
     * 關閉restClient對象
     */
    public void close(){
        try {
            if (restClient != null){
                restClient.close();
                restClient = null;
            }
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
    }
    
    private static String fiterHtmlTag(String html){
        String filterTags = "(?!<(font|/font|em|/em).*?>)<.*?>"; //html不過濾的標簽
        if (!ObjectUtil.isNull(filterTags) && !ObjectUtil.isNull(html)){
            return html.replaceAll(filterTags, "");
        }
        return html;
    }
 
    /**
     * 生成索引文檔對象
     * @param bean
     * @return
     */
    private XContentBuilder getXContentBuilder(ESBean bean){
        XContentBuilder builder = null;
        try {
            builder = XContentFactory.jsonBuilder().startObject()
                    .field("id", bean.getId())
                    .field("opName", bean.getOpName())
                    .field("brief", bean.getBrief())
                    .field("keywords", bean.getKeywords())
                    .field("editor", bean.getEditor())
                    .field("opSite", bean.getOpSite())
                    .field("parentId", bean.getParentId())
                    .field("picUrl", bean.getPicUrl())
                    .field("publishDate", bean.getPublishDate())
                    .field("tableName", bean.getTableName())
                    .endObject();
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
        return builder;
        
    }
    
    /**
     * 生成文檔ID
     * @param tableName  表名
     * @param id         主鍵
     * @return
     */
    private String generateId(String tableName, String id){
        return tableName + "_" + id;
    }
    
    public static void main(String[] args){
         ESTool esTool = new ESTool();
         
         SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
         sourceBuilder.query(QueryBuilders.boolQuery()
               .must(QueryBuilders.multiMatchQuery("住房公積金管理中心服務大廳", 
                       "opName", "editor", "brief", "keywords"))     // 自動分詞匹配
//               .must(QueryBuilders.termsQuery("parentId","21","46")) // 全匹配(文本不會進行分詞)，可一次匹配多個值
//               .must(QueryBuilders.rangeQuery("publishDate")
//                       .from("2019-01-01 00:00:00").to("2019-01-31 23:59:59")) //區間搜索
                 );
         
         ESResult result = esTool.doSearch("xuetz", sourceBuilder, 0, 2, 
               new String[]{"opName", "brief"});

         esTool.close();
 
    }
}