目錄
ElasticSearch
安裝
ElasticSearch
docker pull elasticsearch:7.4.2
Kibana用於可視化
docker pull kibana:
運行
准備es
mkdir -p /mydata/elasticsearch/config #存放ES配置
mkdir -p /mydata/elasticsearch/data #存放ES數據
chmod -R 777 /mydata/elasticsearch #修改權限
#進入config目錄
echo "http.host: 0.0.0.0">>/mydata/elasticsearch/config/elasticsearch.yml #將可遠程訪問寫入配置文件
運行es
# 9200是我們向es發請求的http端口，9300是es集群節點之間的通信端口
docker run --name es -p 9200:9200 -p 9300:9300 \
-e "discovery.type=single-node" \ #單節點設置
-e ES_JAVA_OPTS="-Xms64m -Xmx512m" \ #內存占用，此配置為開發測試
#### 掛載###
-v /mydata/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \ #配置
-v /mydata/elasticsearch/data:/usr/share/elasticsearch/data \ #數據
-v /mydata/elasticsearch/plugins:/usr/share/elasticsearch/plugins \ #插件
-d elasticsearch:7.4.2
運行kibana
docker run --name kibana -e ELASTICSEARCH_HOSTS=http://123.56.16.54:9200 -p 5601:5601 -d kibana:7.4.2
初步檢索
- GET請求：_cat/nodes
- GET請求：_cat/health
- GET請求：_cat/master 查看主節點
- GET請求：_cat/indices 查看索引
新增文檔
-
PUT請求：index/type/id （不存在該id）新增數據，（存在該id）更新數據
-
POST請求：index/type/id （不帶id，隨機指定）新增數據，（帶且存在該id）更新數據
查詢文檔
-
GET請求：index/type/id 查找
樂觀鎖，插敘的時候帶上
?if_sq_no=1&if_primary_term=1
更新文檔
-
POST請求：index/type/id/_update
方法1：若是此次更新數據與上次一樣，則不會增加版本號，提示noop無操作
方法2：不帶_update，同樣的更新也能增加版本號，注意json數據寫法不一樣
- PUT請求：index/type/id
- _update只能是post請求
- 不會檢查是否與上次更新操作一樣
刪除文檔/索引
- DELETE請求：index/type/id
- DELETE請求：index
批量操作
兩行一個數據，刪除沒有請求體，只有一行
進階檢索
Search API
#方法1 uri+檢索參數
GET index01/_search?q=*&sort=_id:asc
#方法2 uri+請求體
GET index01/_search
{
"query": {
"match_all": {}
},
"sort": [
{
"_id": {
"order": "asc" #按id升序 desc降序
}
},
...
]
}
#默認只顯示10個信息
Query DSL
match
# 1. match_all 查詢所有
GET /bank/_search
{
"query": {
"match_all": {}
},
"sort": [
{
"balance": {
"order": "asc" #按id升序 desc降序
}
}
],
"from": 5, # 指定從哪里開始
"size": 5, # 指定一次查詢多少個
"_source": ["balance","firstname"] # 指定返回哪些字段
}
# 2. match用於查找字段xxx,值為xxx的數據，
GET /bank/_search
{
"query": {
"match": {
"account_number": 20
}
}
}
# 也可以是模糊匹配，單詞用空格隔開,按匹配度（匹配單詞數量）降序 ”max_score“匹配最高的得分
GET /bank/_search
{
"query": {
"match": {
"address": "mill lane"
}
}
}
# 3.match_phrase 短語完全匹配
GET /bank/_search
{
"query": {
"match_phrase": {
"address": "Lane mill"
}
}
}
# 4.multi_match多字段匹配 指定字段內包含query值越多越匹配
GET /bank/_search
{
"query": {
"multi_match": {
"query": "Lane Brogan",
"fields": ["address","city"]
}
}
}
bool
- must
- must_not
- should：並非必要，只是匹配的話得分更高
# 以下條件篩選"gender"為"M"，"address"包含 "mill"，但是age不為28，且最好last有Holland的數據
GET /bank/_search
{
"query":{
"bool": {
"must": [
{"match": {
"gender": "M"
}},
{
"match": {
"address": "mill"
}
}
],
"must_not": [
{"match": {
"age": "28"
}}
],
"should": [
{"match": {
"lastname": "Holland"
}}
]
}
}
}
filter
GET /bank/_search
{
"query":{
"bool": {
"filter": { #不影響得分
"range": {
"age": {
"gte": 10,
"lte": 40
}
}
}
}
}
}
term
非text字段檢索，match適合全文檢索字段
GET /bank/_search
{
"query": {
"term": {
"balance": "32838"
}
}
}
字段.keyword以及match區分
必須完全匹配，空格也要，大小寫也要
GET /bank/_search
{
"query": {
"match": {
"address.keyword": "789 Madison Street"
}
}
}
#下例中 match_phrase，只要存在該完整短語(完整單詞A 完整單詞B)即可（大小寫無關，但是單詞順序不能亂）
GET /bank/_search
{
"query": {
"match_phrase": {
"address": "789 madison"
}
}
}
#下例中 match 只要存在當中任意一個單詞即可，與單詞順序、拼寫大小寫無關，匹配越多單詞得分越高
GET /bank/_search
{
"query": {
"match": {
"address": "789 madison"
}
}
}
# 注意：上述匹配均是需要匹配完整單詞，每個單詞長度一樣才可匹配
Aggregations
GET bank/_search
{
"query": {
"match_all": {}
},
"aggs": {
"myagg1": {
"terms": {
"field": "age" #查看年齡分布
}
},
"myaggAVG":{
"avg": {
"field": "balance" #查看平均余額
}
}
},
"size": 0
}
GET bank/_search
{
"query": {
"match_all": {}
},
"aggs": {
"myagg1": {
"terms": {
"field": "age"
},
"aggs": {
"myaggAVG": { # 子聚合 即查看每個年齡的分布清空以及每個年齡的平均余額
"avg": {
"field": "balance"
}
}
}
}
},
"size": 0 #不顯示查出的結果，只顯示聚合結果
}
GET bank/_search
{
"query": {
"match_all": {}
},
"aggs": {
"myagg": {
"terms": {
"field": "age",
"size": 10
},
"aggs": {
"myaggGender": {
"terms": {
"field": "gender.keyword",
"size": 10
},
"aggs": {
"myaggGenderAVG": {
"avg": {
"field": "balance"
}
}
}
},
"myaggAllAVG":{
"avg": {
"field": "balance"
}
}
}
}
},
"size": 0
}
# 查看各個年齡段的男和女分別的平均余額以及這個年齡段的平均余額
Mapping
創建映射關系
PUT /my_index
{
"mappings": {
"properties": {
"age": {"type": "integer"},
"email":{"type": "keyword"},
"name": {"type": "text"，"index":true} #index默認true，默認可以被索引
}
}
}
查看映射信息
GET請求：index/_mapping
修改映射信息
添加新字段
PUT /my_index/_mapping
{
"properties":{
"address":{
"type":"keyword",
"index":false
}
}
}
注意：以及存在的映射信息不可修改
如果要修改，只能指定正確的映射關系，然后數據遷移
PUT /new_bank #創建新的映射
{
"mappings": {
"properties": {
"account_number": {
"type": "long"
},
"address": {
"type": "text"
},
"age": {
"type": "integer"
},
"balance": {
"type": "long"
},
"city": {
"type": "keyword"
},
"email": {
"type": "keyword"
},
"employer": {
"type": "keyword"
},
"firstname": {
"type": "text"
},
"gender": {
"type": "keyword"
},
"lastname": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"state": {
"type": "keyword"
}
}
}
}
POST _reindex #數據遷移，不再用type
{
"source": {
"index": "bank",
"type": "account" #原始數據擁有類型，6.0以后不用type
},
"dest": {
"index": "new_bank" #不用type
}
}
原始type是自己創建的適合定義的
不用type之后所有的type為_doc
分詞
安裝插件
下載elasticsearch-analysis-ik-7.4.2
解壓后放到掛載的plugins目錄，或者進入容器放進去
檢查插件是否安裝成功
docker exec -it es /bin/bash
cd bin
elasticsearch-plugin list # 列出所有插件
POST _analyze
{
"analyzer": "ik_max_word",
"text": ["我是中國人"]
}
POST _analyze
{
"analyzer": "ik_smart",
"text": ["我是中國人"]
}
自定義詞庫
安裝nginx
# 隨便啟動一個nginx實例
docker run -p 80:80 --name nginx -d nginx:1.10
# 把容器內配置文件復制出來,(后面有個 空格+. )
docker container cp nginx:/etc/nginx .
# 停止、刪除這個容器
# 把配置放到conf目錄再轉移到/mydata/nginx目錄下
mv nginx conf
mkdir nginx
mv conf /mydata/nginx/
# 運行
docker run -p 80:80 --name nginx \
-v /mydata/nginx/html:/usr/share/nginx/html \
-v /mydata/nginx/logs:/var/log/nginx \
-v /mydata/nginx/conf:/etc/nginx \
-d nginx:1.10
#測試是否啟動成功
cd html
vi index.html
#寫入
<h1>hahahha~~~~~</h1>
#保存退出，訪問80端口
創建自定義詞庫
-
在nginx的html目錄下新建es文件夾新建txt文件，錄入單詞，保存訪問ip:80/es/xxx.txt
-
進入plugins/elasticsearch-analysis-ik-7.4.2/config目錄修改
IKAnalyzer.cfg.xml文件在擴展字典那里寫上各個的訪問路徑
<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd"> <properties> <comment>IK Analyzer 擴展配置</comment> <!--用戶可以在這里配置自己的擴展字典 --> <entry key="ext_dict"></entry> <!--用戶可以在這里配置自己的擴展停止詞字典--> <entry key="ext_stopwords"></entry> <!--用戶可以在這里配置遠程擴展字典 --> <entry key="remote_ext_dict">http://你的ip/es/participle.txt</entry> <!--用戶可以在這里配置遠程擴展停止詞字典--> <!-- <entry key="remote_ext_stopwords">words_location</entry> --> </properties> -
重啟
整合SpringBoot
- 創建springboot項目，選擇web不選擇springboot整合的elasticsearch，因其版本沒有更新到7.0之后
- 導入依賴
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>elasticsearch-rest-high-level-client</artifactId>
<version>7.4.2</version>
</dependency>
- 調整版本，springboot 2.2.7內置6.8.8，指定版本即可
<properties>
<java.version>1.8</java.version>
<elasticsearch.version>7.4.2</elasticsearch.version>
</properties>
- 配置
@Configuration
public class ElasticSearchConfig {
public static final RequestOptions COMMON_OPTIONS;
static {
RequestOptions.Builder builder = RequestOptions.DEFAULT.toBuilder();
/*builder.addHeader("Authorization", "Bearer " + TOKEN);
builder.setHttpAsyncResponseConsumerFactory(
new HttpAsyncResponseConsumerFactory
.HeapBufferedResponseConsumerFactory(30 * 1024 * 1024 * 1024));*/
COMMON_OPTIONS = builder.build();
}
@Bean
public RestHighLevelClient esRestCilent(){
RestHighLevelClient client = new RestHighLevelClient(
RestClient.builder(
new HttpHost("ip...", 9200, "http")));
return client;
}
}
創建索引
@Test
void index() throws IOException{
//新建一個索引
IndexRequest indexRequest = new IndexRequest("users");
//指定id
indexRequest.id("1");
User user = new User();
user.setUserName("小明");
user.setGender("M");
user.setAge(18);
// 轉為json字符
String u = JSON.toJSONString(user);
// 保存json
indexRequest.source(u, XContentType.JSON);
// 執行操作
IndexResponse index = client.index(indexRequest, ElasticSearchConfig.COMMON_OPTIONS);
System.out.println(index);
}
輸出結果
IndexResponse[index=users,type=_doc,id=1,version=1,result=created,seqNo=0,primaryTerm=1,shards={"total":2,"successful":1,"failed":0}]
獲取
@Test
public void get() throws IOException{
GetRequest getRequest = new GetRequest(
"users",
"1");
GetResponse fields = client.get(getRequest, ElasticSearchConfig.COMMON_OPTIONS);
System.out.println(fields);
}
輸出
{"_index":"users","_type":"_doc","_id":"1","_version":1,"_seq_no":0,"_primary_term":1,"found":true,"_source":{"age":18,"gender":"M","userName":"小明"}}
刪除
@Test
public void delete() throws IOException{
DeleteRequest request = new DeleteRequest(
"users",
"1");
DeleteResponse delete = client.delete(request, ElasticSearchConfig.COMMON_OPTIONS);
System.out.println(delete);
}
檢索
json工具網站
@Test
public void search() throws IOException{
//創建請求
SearchRequest searchRequest = new SearchRequest("new_bank");
//封裝條件
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
//篩選address有mill單詞的數據
searchSourceBuilder.query(QueryBuilders.matchQuery("address","mill"));
//按年齡分布聚合，
TermsAggregationBuilder ageTerm = AggregationBuilders.terms("ageTerm").field("age");
//求余額平均值
AvgAggregationBuilder balanceAvg = AggregationBuilders.avg("balanceAvg").field("balance");
//添加聚合
searchSourceBuilder.aggregation(ageTerm);
searchSourceBuilder.aggregation(balanceAvg);
System.out.println("請求"+searchSourceBuilder);
//保存
searchRequest.source(searchSourceBuilder);
//執行
SearchResponse response = client.search(searchRequest, ElasticSearchConfig.COMMON_OPTIONS);
System.out.println("檢索返回信息"+response);
//分析返回信息
SearchHits hits = response.getHits();
System.out.println("檢索結果有幾個："+hits.getTotalHits());
SearchHit[] searchHits = hits.getHits();
for(SearchHit hit :searchHits){
String string = hit.getSourceAsString();
Account account = JSON.parseObject(string,Account.class);
System.out.println("檢索結果"+account);
}
//分析聚合結果
Aggregations aggregations = response.getAggregations();
Terms term = aggregations.get("ageTerm");
for (Terms.Bucket bucket : term.getBuckets()) {
String keyAsString = bucket.getKeyAsString();
System.out.println("年齡為："+keyAsString+"有："+bucket.getDocCount()+"個");
}
Avg avg = aggregations.get("balanceAvg");
System.out.println("求得余額平均值為："+avg.getValue());
}
輸出的結果
請求
{
"query": {
"match": {
"address": {
"query": "mill",
"operator": "OR",
"prefix_length": 0,
"max_expansions": 50,
"fuzzy_transpositions": true,
"lenient": false,
"zero_terms_query": "NONE",
"auto_generate_synonyms_phrase_query": true,
"boost": 1.0
}
}
},
"aggregations": {
"ageTerm": {
"terms": {
"field": "age",
"size": 10,
"min_doc_count": 1,
"shard_min_doc_count": 0,
"show_term_doc_count_error": false,
"order": [{
"_count": "desc"
}, {
"_key": "asc"
}]
}
},
"balanceAvg": {
"avg": {
"field": "balance"
}
}
}
}
檢索返回信息
{
"took": 2,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 4,
"relation": "eq"
},
"max_score": 5.4032025,
"hits": [{
"_index": "new_bank",
"_type": "_doc",
"_id": "970",
"_score": 5.4032025,
"_source": {
"account_number": 970,
"balance": 19648,
"firstname": "Forbes",
"lastname": "Wallace",
"age": 28,
"gender": "M",
"address": "990 Mill Road",
"employer": "Pheast",
"email": "forbeswallace@pheast.com",
"city": "Lopezo",
"state": "AK"
}
}, ...]
},
"aggregations": {
"avg#balanceAvg": {
"value": 25208.0
},
"lterms#ageTerm": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [{
"key": 38,
"doc_count": 2
}, {
"key": 28,
"doc_count": 1
}, {
"key": 32,
"doc_count": 1
}]
}
}
}
檢索結果有幾個：4 hits
檢索結果Account(account_number=970, balance=19648, firstname=Forbes, lastname=Wallace, age=28, gender=M, address=990 Mill Road, employer=Pheast, email=forbeswallace@pheast.com, city=Lopezo, state=AK)
檢索結果Account(account_number=136, balance=45801, firstname=Winnie, lastname=Holland, age=38, gender=M, address=198 Mill Lane, employer=Neteria, email=winnieholland@neteria.com, city=Urie, state=IL)
檢索結果Account(account_number=345, balance=9812, firstname=Parker, lastname=Hines, age=38, gender=M, address=715 Mill Avenue, employer=Baluba, email=parkerhines@baluba.com, city=Blackgum, state=KY)
檢索結果Account(account_number=472, balance=25571, firstname=Lee, lastname=Long, age=32, gender=F, address=288 Mill Street, employer=Comverges, email=leelong@comverges.com, city=Movico, state=MT)
年齡為：38有：2個
年齡為：28有：1個
年齡為：32有：1個
求得余額平均值為：25208.0
用kibana測試
