前言
-
es是什么?
es是基於Apache Lucene的開源分布式(全文)搜索引擎,,提供簡單的RESTful API來隱藏Lucene的復雜性。
es除了全文搜索引擎之外,還可以這樣描述它:
1、分布式的實時文件存儲,每個字段都被索引並可被搜索
2、分布式的實時分析搜索引擎
3、可以擴展到成百上千台服務器,處理PB級結構化或非結構化數據。 -
ES的數據組織類比
Relational DB | Elasticsearch |
---|---|
數據庫(database) | 索引(indices) |
表(tables) | types |
行(rows) | documents |
字段(columns) | fields |
- mac安裝ES
- 1、更新brew ```brew update``` - 2、安裝java1.8版本 ```brew cask install homebrew/cask-versions/java8``` - 3、安裝ES ```brew install elasticsearch``` - 4、啟動本地ES ```brew services start elasticsearch``` - 5、本地訪問9200端口查看ES安裝 ```http://localhost:9200``` - 6、安裝kibana ```Kibana是ES的一個配套工具,可以讓用戶在網頁中與ES進行交互``` ```brew install kibana``` - 7、本地啟動kibana ```brew services start kibana``` - 8、本地訪問5601端口進入kibana交互界面 ```http://localhost:5601```
一、 ES簡單的增刪改查
1、創建一篇文檔(有則修改,無則創建)
PUT test/doc/2
{
"name":"wangfei",
"age":27,
"desc":"熱天還不讓后人不認同"
}
PUT test/doc/1
{
"name":"wangjifei",
"age":27,
"desc":"薩芬我反胃為范圍額"
}
PUT test/doc/3
{
"name":"wangyang",
"age":30,
"desc":"點在我心內的幾首歌"
}
2、查詢指定索引信息
GET test
3、 查詢指定文檔信息
GET test/doc/1
GET test/doc/2
4、查詢對應索引下所有數據
GET test/doc/_search
或
GET test/doc/_search
{
"query": {
"match_all": {}
}
}
5、刪除指定文檔
DELETE test/doc/3
6、刪除索引
DELETE test
7、修改指定文檔方式
- 修改時,不指定的屬性會自動覆蓋,只保留指定的屬性(不正確的修改指定文檔方式)
PUT test/doc/1
{
"name":"王計飛"
}
- 使用POST命令,在id后面跟_update,要修改的內容放到doc文檔(屬性)中(正確的修改指定文檔方式)
POST test/doc/1/_update
{
"doc":{
"desc":"生活就像 茫茫海上"
}
}
二、ES查詢的兩種方式
1、查詢字符串搜索
GET test/doc/_search?q=name:wangfei
2、結構化查詢(單字段查詢,不能多字段組合查詢)
GET test/doc/_search
{
"query":{
"match":{
"name":"wang"
}
}
}
三、match系列之操作
1、match系列之match_all (查詢全部)
GET test/doc/_search
{
"query":{
"match_all": {
}
}
}
2、match系列之match_phrase(短語查詢)
准備數據
PUT test1/doc/1
{
"title": "中國是世界上人口最多的國家"
}
PUT test1/doc/2
{
"title": "美國是世界上軍事實力最強大的國家"
}
PUT test1/doc/3
{
"title": "北京是中國的首都"
}
查詢語句
GET test1/doc/_search { "query":{ "match":{ "title":"中國" } } } >>>輸出結果 { "took" : 241, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 3, "max_score" : 0.68324494, "hits" : [ { "_index" : "test1", "_type" : "doc", "_id" : "1", "_score" : 0.68324494, "_source" : { "title" : "中國是世界上人口最多的國家" } }, { "_index" : "test1", "_type" : "doc", "_id" : "3", "_score" : 0.5753642, "_source" : { "title" : "北京是中國的首都" } }, { "_index" : "test1", "_type" : "doc", "_id" : "2", "_score" : 0.39556286, "_source" : { "title" : "美國是世界上軍事實力最強大的國家" } } ] } }
通過觀察結果可以發現,雖然如期的返回了中國的文檔。但是卻把和美國的文檔也返回了,這並不是我們想要的。是怎么回事呢?因為這是elasticsearch在內部對文檔做分詞的時候,對於中文來說,就是一個字一個字分的,所以,我們搜中國,中和國都符合條件,返回,而美國的國也符合。而我們認為中國是個短語,是一個有具體含義的詞。所以elasticsearch在處理中文分詞方面比較弱勢。后面會講針對中文的插件。但目前我們還有辦法解決,那就是使用短語查詢 用match_phrase
GET test1/doc/_search { "query":{ "match_phrase": { "title": "中國" } } } >>>查詢結果 { "took" : 10, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 2, "max_score" : 0.5753642, "hits" : [ { "_index" : "test1", "_type" : "doc", "_id" : "1", "_score" : 0.5753642, "_source" : { "title" : "中國是世界上人口最多的國家" } }, { "_index" : "test1", "_type" : "doc", "_id" : "3", "_score" : 0.5753642, "_source" : { "title" : "北京是中國的首都" } } ] } }
我們搜索中國和世界這兩個指定詞組時,但又不清楚兩個詞組之間有多少別的詞間隔。那么在搜的時候就要留有一些余地。這時就要用到了slop了。相當於正則中的中國.*?世界。這個間隔默認為0
GET test1/doc/_search { "query":{ "match_phrase": { "title": { "query": "中國世界", "slop":2 } } } } >>>查詢結果 { "took" : 23, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.7445889, "hits" : [ { "_index" : "test1", "_type" : "doc", "_id" : "1", "_score" : 0.7445889, "_source" : { "title" : "中國是世界上人口最多的國家" } } ] } }
3、match系列之match_phrase_prefix(最左前綴查詢)智能搜索--以什么開頭
數據准備
PUT test2/doc/1
{
"title": "prefix1",
"desc": "beautiful girl you are beautiful so"
}
PUT test2/doc/2
{
"title": "beautiful",
"desc": "I like basking on the beach"
}
搜索特定英文開頭的數據
查詢語句
GET test2/doc/_search { "query": { "match_phrase_prefix": { "desc": "bea" } } } >>>查詢結果() { "took" : 5, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 2, "max_score" : 0.39556286, "hits" : [ { "_index" : "test2", "_type" : "doc", "_id" : "1", "_score" : 0.39556286, "_source" : { "title" : "prefix1", "desc" : "beautiful girl you are beautiful so" } }, { "_index" : "test2", "_type" : "doc", "_id" : "2", "_score" : 0.2876821, "_source" : { "title" : "beautiful", "desc" : "I like basking on the beach" } } ] } } 查詢短語 GET test2/doc/_search { "query": { "match_phrase_prefix": { "desc": "you are bea" } } } >>>查詢結果 { "took" : 28, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.8630463, "hits" : [ { "_index" : "test2", "_type" : "doc", "_id" : "1", "_score" : 0.8630463, "_source" : { "title" : "prefix1", "desc" : "beautiful girl you are beautiful so" } } ] } }
max_expansions 參數理解 前綴查詢會非常的影響性能,要對結果集進行限制,就加上這個參數。
GET test2/doc/_search
{
"query": {
"match_phrase_prefix": {
"desc": {
"query": "bea",
"max_expansions":1
}
}
}
}
4、match系列之multi_match(多字段查詢)
- multi_match是要在多個字段中查詢同一個關鍵字 除此之外,mulit_match甚至可以當做match_phrase和match_phrase_prefix使用,只需要指定type類型即可
GET test2/doc/_search { "query": { "multi_match": { "query": "beautiful", "fields": ["title","desc"] } } } >>查詢結果 { "took" : 43, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 2, "max_score" : 0.39556286, "hits" : [ { "_index" : "test2", "_type" : "doc", "_id" : "1", "_score" : 0.39556286, "_source" : { "title" : "prefix1", "desc" : "beautiful girl you are beautiful so" } }, { "_index" : "test2", "_type" : "doc", "_id" : "2", "_score" : 0.2876821, "_source" : { "title" : "beautiful", "desc" : "I like basking on the beach" } } ] } }
- 當設置屬性 type:phrase 時 等同於 短語查詢
GET test1/doc/_search { "query": { "multi_match": { "query": "中國", "fields": ["title"], "type": "phrase" } } } >>>查詢結果 { "took" : 47, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 2, "max_score" : 0.5753642, "hits" : [ { "_index" : "test1", "_type" : "doc", "_id" : "1", "_score" : 0.5753642, "_source" : { "title" : "中國是世界上人口最多的國家" } }, { "_index" : "test1", "_type" : "doc", "_id" : "3", "_score" : 0.5753642, "_source" : { "title" : "北京是中國的首都" } } ] } }
- 當設置屬性 type:phrase_prefix時 等同於 最左前綴查詢
GET test2/doc/_search { "query": { "multi_match": { "query": "bea", "fields": ["desc"], "type": "phrase_prefix" } } } >>查詢結果 { "took" : 5, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 2, "max_score" : 0.5753642, "hits" : [ { "_index" : "test1", "_type" : "doc", "_id" : "1", "_score" : 0.5753642, "_source" : { "title" : "中國是世界上人口最多的國家" } }, { "_index" : "test1", "_type" : "doc", "_id" : "3", "_score" : 0.5753642, "_source" : { "title" : "北京是中國的首都" } } ] } }
match 查詢相關總結
1、match:返回所有匹配的分詞。
2、match_all:查詢全部。
3、match_phrase:短語查詢,在match的基礎上進一步查詢詞組,可以指定slop分詞間隔。
4、match_phrase_prefix:前綴查詢,根據短語中最后一個詞組做前綴匹配,可以應用於搜索提示,但注意和max_expanions搭配。其實默認是50.......
5、multi_match:多字段查詢,使用相當的靈活,可以完成match_phrase和match_phrase_prefix的工作。
四、ES的排序查詢
es 6.8.4版本中,需要分詞的字段不可以直接排序,比如:text類型,如果想要對這類字段進行排序,需要特別設置:對字段索引兩次,一次索引分詞(用於搜索)一次索引不分詞(用於排序),es默認生成的text類型字段就是通過這樣的方法實現可排序的。
-
倒敘排序
GET test/doc/_search { "query": { "match_all": {} }, "sort": [ { "age": { "order": "desc" } } ] } >>排序結果 { "took" : 152, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 3, "max_score" : null, "hits" : [ { "_index" : "test", "_type" : "doc", "_id" : "3", "_score" : null, "_source" : { "name" : "wangyang", "age" : 30, "desc" : "點在我心內的幾首歌" }, "sort" : [ 30 ] }, { "_index" : "test", "_type" : "doc", "_id" : "2", "_score" : null, "_source" : { "name" : "wangfei", "age" : 27, "desc" : "熱天還不讓后人不認同" }, "sort" : [ 27 ] }, { "_index" : "test", "_type" : "doc", "_id" : "1", "_score" : null, "_source" : { "name" : "wangjifei", "age" : 27, "desc" : "生活就像 茫茫海上" }, "sort" : [ 27 ] } ] } }
- 升序排序
GET test/doc/_search
{
"query": {
"match_all": {}
},
"sort": [
{
"age": {
"order": "asc"
}
}
]
}
五、ES的分頁查詢
- from:從哪開始查 size:返回幾條結果
GET test/doc/_search { "query": { "match_phrase_prefix": { "name": "wang" } }, "from": 0, "size": 1 } >>查詢結果 { "took" : 3, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 3, "max_score" : 0.2876821, "hits" : [ { "_index" : "test", "_type" : "doc", "_id" : "2", "_score" : 0.2876821, "_source" : { "name" : "wangfei", "age" : 27, "desc" : "熱天還不讓后人不認同" } } ] } }
六、ES的bool查詢 (must、should)
- must (must字段對應的是個列表,也就是說可以有多個並列的查詢條件,一個文檔滿足各個子條件后才最終返回)
#### 單條件查詢 GET test/doc/_search { "query": { "bool": { "must": [ { "match": { "name": "wangfei" } } ] } } } >>查詢結果 { "took" : 4, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.2876821, "hits" : [ { "_index" : "test", "_type" : "doc", "_id" : "2", "_score" : 0.2876821, "_source" : { "name" : "wangfei", "age" : 27, "desc" : "熱天還不讓后人不認同" } } ] } }
#### 多條件組合查詢 GET test/doc/_search { "query": { "bool": { "must": [ { "match": { "name": "wanggfei" } },{ "match": { "age": 25 } } ] } } } >>查詢結果 { "took" : 21, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 0, "max_score" : null, "hits" : [ ] } }
- should (只要符合其中一個條件就返回)
GET test/doc/_search { "query": { "bool": { "should": [ { "match": { "name": "wangjifei" } },{ "match": { "age": 27 } } ] } } } >>查詢結果 { "took" : 34, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 2, "max_score" : 1.287682, "hits" : [ { "_index" : "test", "_type" : "doc", "_id" : "1", "_score" : 1.287682, "_source" : { "name" : "wangjifei", "age" : 27, "desc" : "生活就像 茫茫海上" } }, { "_index" : "test", "_type" : "doc", "_id" : "2", "_score" : 1.0, "_source" : { "name" : "wangfei", "age" : 27, "desc" : "熱天還不讓后人不認同" } } ] } }
- must_not 顧名思義
GET test/doc/_search { "query": { "bool": { "must_not": [ { "match": { "name": "wangjifei" } },{ "match": { "age": 27 } } ] } } } >>查詢結果 { "took" : 13, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 1.0, "hits" : [ { "_index" : "test", "_type" : "doc", "_id" : "3", "_score" : 1.0, "_source" : { "name" : "wangyang", "age" : 30, "desc" : "點在我心內的幾首歌" } } ] } }
- filter(條件過濾查詢,過濾條件的范圍用range表示gt表示大於、lt表示小於、gte表示大於等於、lte表示小於等於)
GET test/doc/_search { "query": { "bool": { "must": [ { "match": { "name": "wangjifei" } } ], "filter": { "range": { "age": { "gte": 10, "lt": 27 } } } } } } >>查詢結果 { "took" : 33, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 0, "max_score" : null, "hits" : [ ] } }
bool查詢總結
must:與關系,相當於關系型數據庫中的 and。
should:或關系,相當於關系型數據庫中的 or。
must_not:非關系,相當於關系型數據庫中的 not。
filter:過濾條件。
range:條件篩選范圍。
gt:大於,相當於關系型數據庫中的 >。
gte:大於等於,相當於關系型數據庫中的 >=。
lt:小於,相當於關系型數據庫中的 <。
lte:小於等於,相當於關系型數據庫中的 <=。
七、ES之查詢結果過濾
####准備數據
PUT test3/doc/1
{
"name":"顧老二",
"age":30,
"from": "gu",
"desc": "皮膚黑、武器長、性格直",
"tags": ["黑", "長", "直"]
}
- 現在,在所有的結果中,我只需要查看name和age兩個屬性,提高查詢效率
GET test3/doc/_search { "query": { "match": { "name": "顧" } }, "_source": ["name","age"] } >>查詢結果 { "took" : 58, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.2876821, "hits" : [ { "_index" : "test3", "_type" : "doc", "_id" : "1", "_score" : 0.2876821, "_source" : { "name" : "顧老二", "age" : 30 } } ] } }
八、ES之查詢結果高亮顯示
- ES的默認高亮顯示
GET test3/doc/_search { "query": { "match": { "name": "顧老二" } }, "highlight": { "fields": { "name": {} } } } >>查詢結果 { "took" : 216, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.8630463, "hits" : [ { "_index" : "test3", "_type" : "doc", "_id" : "1", "_score" : 0.8630463, "_source" : { "name" : "顧老二", "age" : 30, "from" : "gu", "desc" : "皮膚黑、武器長、性格直", "tags" : [ "黑", "長", "直" ] }, "highlight" : { "name" : [ "<em>顧</em><em>老</em><em>二</em>" ] } } ] } }
ES自定義高亮顯示(在highlight中,pre_tags用來實現我們的自定義標簽的前半部分,在這里,我們也可以為自定義的 標簽添加屬性和樣式。post_tags實現標簽的后半部分,組成一個完整的標簽。至於標簽中的內容,則還是交給fields來完成)
GET test3/doc/_search { "query": { "match": { "desc": "性格直" } }, "highlight": { "pre_tags": "<b class='key' style='color:red'>", "post_tags": "</b>", "fields": { "desc": {} } } } >>查詢結果 { "took" : 6, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.8630463, "hits" : [ { "_index" : "test3", "_type" : "doc", "_id" : "1", "_score" : 0.8630463, "_source" : { "name" : "顧老二", "age" : 30, "from" : "gu", "desc" : "皮膚黑、武器長、性格直", "tags" : [ "黑", "長", "直" ] }, "highlight" : { "desc" : [ "皮膚黑、武器長、<b class='key' style='color:red'>性</b><b class='key' style='color:red'>格</b><b class='key' style='color:red'>直</b>" ] } } ] } }
十、ES之精確查詢與模糊查詢
- term查詢查找包含文檔精確的倒排索引指定的詞條。也就是精確查找。
term和match的區別是:match是經過analyer的,也就是說,文檔首先被分析器給處理了。根據不同的分析器,分析的結果也稍顯不同,然后再根據分詞結果進行匹配。term則不經過分詞,它是直接去倒排索引中查找了精確的值了。
#### 准備數據
PUT w1
{
"mappings": {
"doc": {
"properties":{
"t1":{
"type": "text"
},
"t2": {
"type": "keyword"
}
}
}
}
}
PUT w1/doc/1
{
"t1": "hi single dog",
"t2": "hi single dog"
}
- 對比兩者的不同 (結果就不展示出來了,只展示結果的文字敘述)
# t1類型為text,會經過分詞,match查詢時條件也會經過分詞,所以下面兩種查詢都能查到結果 GET w1/doc/_search { "query": { "match": { "t1": "hi single dog" } } } GET w1/doc/_search { "query": { "match": { "t1": "hi" } } } # t2類型為keyword類型,不會經過分詞,match查詢時條件會經過分詞,所以只能當值為"hi single dog"時能查詢到 GET w1/doc/_search { "query": { "match": { "t2": "hi" } } } GET w1/doc/_search { "query": { "match": { "t2": "hi single dog" } } } # t1類型為text,會經過分詞,term查詢時條件不會經過分詞,所以只有當值為"hi"時能查詢到 GET w1/doc/_search { "query": { "term": { "t1": "hi single dog" } } } GET w1/doc/_search { "query": { "term": { "t1": "hi" } } } # t2類型為keyword類型,不會經過分詞,term查詢時條件不會經過分詞,所以只能當值為"hi single dog"時能查詢到 GET w1/doc/_search { "query": { "term": { "t2": "hi single dog" } } } GET w1/doc/_search { "query": { "term": { "t2": "hi" } } }
- 查找多個精確值(terms)
#### 第一個查詢方式
GET test/doc/_search
{
"query": {
"bool": {
"should": [
{
"term": {
"age":27
}
},{
"term":{
"age":28
}
}
]
}
}
}
# 第二個查詢方式
GET test/doc/_search
{
"query": {
"terms": {
"age": [
"27",
"28"
]
}
}
}
>>>兩種方式的查詢結果都是一下結果
{
"took" : 10,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 2,
"max_score" : 1.0,
"hits" : [
{
"_index" : "test",
"_type" : "doc",
"_id" : "2",
"_score" : 1.0,
"_source" : {
"name" : "wangfei",
"age" : 27,
"desc" : "熱天還不讓后人不認同"
}
},
{
"_index" : "test",
"_type" : "doc",
"_id" : "1",
"_score" : 1.0,
"_source" : {
"name" : "wangjifei",
"age" : 27,
"desc" : "生活就像 茫茫海上"
}
}
]
}
}
十一、ES的聚合查詢avg、max、min、sum
#### 數據准備
PUT zhifou/doc/1
{
"name":"顧老二",
"age":30,
"from": "gu",
"desc": "皮膚黑、武器長、性格直",
"tags": ["黑", "長", "直"]
}
PUT zhifou/doc/2
{
"name":"大娘子",
"age":18,
"from":"sheng",
"desc":"膚白貌美,嬌憨可愛",
"tags":["白", "富","美"]
}
PUT zhifou/doc/3
{
"name":"龍套偏房",
"age":22,
"from":"gu",
"desc":"mmp,沒怎么看,不知道怎么形容",
"tags":["造數據", "真","難"]
}
PUT zhifou/doc/4
{
"name":"石頭",
"age":29,
"from":"gu",
"desc":"粗中有細,狐假虎威",
"tags":["粗", "大","猛"]
}
PUT zhifou/doc/5
{
"name":"魏行首",
"age":25,
"from":"廣雲台",
"desc":"仿佛兮若輕雲之蔽月,飄飄兮若流風之回雪,mmp,最后竟然沒有嫁給顧老二!",
"tags":["閉月","羞花"]
}
GET zhifou/doc/_search
{
"query": {
"match_all": {}
}
}
- 需求1、查詢from是gu的人的平均年齡。
GET zhifou/doc/_search { "query": { "match": { "from": "gu" } }, "aggs": { "my_avg": { "avg": { "field": "age" } } }, "_source": ["name", "age"] } >>>查詢結果 { "took" : 83, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 3, "max_score" : 0.6931472, "hits" : [ { "_index" : "zhifou", "_type" : "doc", "_id" : "4", "_score" : 0.6931472, "_source" : { "name" : "石頭", "age" : 29 } }, { "_index" : "zhifou", "_type" : "doc", "_id" : "1", "_score" : 0.2876821, "_source" : { "name" : "顧老二", "age" : 30 } }, { "_index" : "zhifou", "_type" : "doc", "_id" : "3", "_score" : 0.2876821, "_source" : { "name" : "龍套偏房", "age" : 22 } } ] }, "aggregations" : { "my_avg" : { "value" : 27.0 } } }
上例中,首先匹配查詢from是gu的數據。在此基礎上做查詢平均值的操作,這里就用到了聚合函數,其語法被封裝在aggs中,而my_avg則是為查詢結果起個別名,封裝了計算出的平均值。那么,要以什么屬性作為條件呢?是age年齡,查年齡的什么呢?是avg,查平均年齡。
如果只想看輸出的值,而不關心輸出的文檔的話可以通過size=0來控制
GET zhifou/doc/_search { "query": { "match": { "from": "gu" } }, "aggs":{ "my_avg":{ "avg": { "field": "age" } } }, "size":0, "_source":["name","age"] } >>>查詢結果 { "took" : 35, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 3, "max_score" : 0.0, "hits" : [ ] }, "aggregations" : { "my_avg" : { "value" : 27.0 } } }
- 需求2、查詢年齡的最大值
GET zhifou/doc/_search { "query": { "match_all": {} }, "aggs": { "my_max": { "max": { "field": "age" } } }, "size": 0, "_source": ["name","age","from"] } >>>查詢結果 { "took" : 10, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 5, "max_score" : 0.0, "hits" : [ ] }, "aggregations" : { "my_max" : { "value" : 30.0 } } }
- 需求3、查詢年齡的最小值
GET zhifou/doc/_search { "query": { "match_all": {} }, "aggs": { "my_min": { "min": { "field": "age" } } }, "size": 0, "_source": ["name","age","from"] } >>>查詢結果 { "took" : 2, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 5, "max_score" : 0.0, "hits" : [ ] }, "aggregations" : { "my_min" : { "value" : 18.0 } } }
- 需求4、查詢符合條件的年齡之和
GET zhifou/doc/_search { "query": { "match": { "from": "gu" } }, "aggs": { "my_sum": { "sum": { "field": "age" } } }, "size": 0, "_source": ["name","age","from"] } >>>查詢結果 { "took" : 4, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 3, "max_score" : 0.0, "hits" : [ ] }, "aggregations" : { "my_sum" : { "value" : 81.0 } } }
十二、ES的分組查詢
- 需求: 要查詢所有人的年齡段,並且按照1520,2025,25~30分組,並且算出每組的平均年齡。
GET zhifou/doc/_search { "size": 0, "query": { "match_all": {} }, "aggs": { "age_group": { "range": { "field": "age", "ranges": [ { "from": 15, "to": 20 }, { "from": 20, "to": 25 }, { "from": 25, "to": 30 } ] } } } } >>>查詢結果 { "took" : 9, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 5, "max_score" : 0.0, "hits" : [ ] }, "aggregations" : { "age_group" : { "buckets" : [ { "key" : "15.0-20.0", "from" : 15.0, "to" : 20.0, "doc_count" : 1 }, { "key" : "20.0-25.0", "from" : 20.0, "to" : 25.0, "doc_count" : 1 }, { "key" : "25.0-30.0", "from" : 25.0, "to" : 30.0, "doc_count" : 2 } ] } } }
上例中,在aggs的自定義別名age_group中,使用range來做分組,field是以age為分組,分組使用ranges來做,from和to是范圍
- 接下來,我們就要對每個小組內的數據做平均年齡處理。
GET zhifou/doc/_search { "size": 0, "query": { "match_all": {} }, "aggs": { "age_group": { "range": { "field": "age", "ranges": [ { "from": 15, "to": 20 }, { "from": 20, "to": 25 }, { "from": 25, "to": 30 } ] }, "aggs": { "my_avg": { "avg": { "field": "age" } } } } } } >>>查詢結果 { "took" : 1, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 5, "max_score" : 0.0, "hits" : [ ] }, "aggregations" : { "age_group" : { "buckets" : [ { "key" : "15.0-20.0", "from" : 15.0, "to" : 20.0, "doc_count" : 1, "my_avg" : { "value" : 18.0 } }, { "key" : "20.0-25.0", "from" : 20.0, "to" : 25.0, "doc_count" : 1, "my_avg" : { "value" : 22.0 } }, { "key" : "25.0-30.0", "from" : 25.0, "to" : 30.0, "doc_count" : 2, "my_avg" : { "value" : 27.0 } } ] } } }
ES的聚合查詢的總結:聚合函數的使用,一定是先查出結果,然后對結果使用聚合函數做處理
avg:求平均
max:最大值
min:最小值
sum:求和
十三、ES之Mappings
GET test >>>查詢結果 { "test" : { "aliases" : { }, "mappings" : { "doc" : { "properties" : { "age" : { "type" : "long" }, "desc" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "name" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } } }, "settings" : { "index" : { "creation_date" : "1569133097594", "number_of_shards" : "5", "number_of_replicas" : "1", "uuid" : "AztO9waYQiyHvzP6dlk4tA", "version" : { "created" : "6080299" }, "provided_name" : "test" } } } }
由返回結果可以看到,分為兩大部分:
第一部分關於t1索引類型相關的,包括該索引是否有別名aliases,然后就是mappings信息,
包括索引類型doc,各字段的詳細映射關系都收集在properties中。
另一部分是關於索引t1的settings設置。包括該索引的創建時間,主副分片的信息,UUID等等。
1. mappings 是什么?
映射就是在創建索引的時候,有更多定制的內容,更加的貼合業務場景。
用來定義一個文檔及其包含的字段如何存儲和索引的過程。
2. 字段的數據類型
簡單類型如文本(text)、關鍵字(keyword)、日期(data)、整形(long)、雙精度 (double)、布爾(boolean)或ip。 可以是支持JSON的層次結構性質的類型,如對象或嵌套。 或者一種特殊類型,如geo_point、geo_shape或completion。為了不同的目的, 以不同的方式索引相同的字段通常是有用的。例如,字符串字段可以作為全文搜索的文本字段進行索引, 也可以作為排序或聚合的關鍵字字段進行索引。或者,可以使用標准分析器、英語分析器和 法語分析器索引字符串字段。這就是多字段的目的。大多數數據類型通過fields參數支持多字段。
- 一個簡單的映射示例
PUT mapping_test
{
"mappings": {
"test1":{
"properties":{
"name":{"type": "text"},
"age":{"type":"long"}
}
}
}
}
我們在創建索引PUT mapping_test1的過程中,為該索引定制化類型(設計表結構),添加一個映射類型test1;指定字段或者屬性都在properties內完成。
GET mapping_test >>>查詢結果 { "mapping_test" : { "aliases" : { }, "mappings" : { "test1" : { "properties" : { "age" : { "type" : "long" }, "name" : { "type" : "text" } } } }, "settings" : { "index" : { "creation_date" : "1570794586526", "number_of_shards" : "5", "number_of_replicas" : "1", "uuid" : "P4-trriPTxq-nJj89iYXZA", "version" : { "created" : "6080299" }, "provided_name" : "mapping_test" } } } }
返回的結果中你肯定很熟悉!映射類型是test1,具體的屬性都被封裝在properties中。
3. ES mappings之dynamic的三種狀態
- 一般的,mapping則又可以分為動態映射(dynamic mapping)和靜態(顯示)映射(explicit mapping)和精確(嚴格)映射(strict mappings),具體由dynamic屬性控制。默認為動態映射
##### 默認為動態映射 PUT test4 { "mappings": { "doc":{ "properties": { "name": { "type": "text" }, "age": { "type": "long" } } } } } GET test4/_mapping >>>查詢結果 { "test4" : { "mappings" : { "doc" : { "properties" : { "age" : { "type" : "long" }, "name" : { "type" : "text" }, "sex" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } } } } } #####添加數據 PUT test4/doc/1 { "name":"wangjifei", "age":"18", "sex":"不詳" } #####查看數據 GET test4/doc/_search { "query": { "match_all": {} } } >>>查詢結果 { "took" : 8, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 1.0, "hits" : [ { "_index" : "test4", "_type" : "doc", "_id" : "1", "_score" : 1.0, "_source" : { "name" : "wangjifei", "age" : "18", "sex" : "不詳" } } ] } }
- 測試靜態映射:當elasticsearch察覺到有新增字段時,因為dynamic:false的關系,會忽略該字段,但是仍會存儲該字段。
#####創建靜態mapping PUT test5 { "mappings": { "doc":{ "dynamic":false, "properties": { "name": { "type": "text" }, "age": { "type": "long" } } } } } #####插入數據 PUT test5/doc/1 { "name":"wangjifei", "age":"18", "sex":"不詳" } ####條件查詢 GET test5/doc/_search { "query": { "match": { "sex": "不詳" } } } >>>查詢結果 { "took" : 9, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 0, "max_score" : null, "hits" : [ ] } } #####查看所有數據 GET /test5/doc/_search { "query": { "match_all": {} } } >>>查詢結果 { "took" : 1, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 1.0, "hits" : [ { "_index" : "test5", "_type" : "doc", "_id" : "1", "_score" : 1.0, "_source" : { "name" : "wangjifei", "age" : "18", "sex" : "不詳" } } ] } }
- 測試嚴格映射:當elasticsearch察覺到有新增字段時,因為dynamic:strict 的關系,就會報錯,不能插入成功。
#####創建嚴格mapping PUT test6 { "mappings": { "doc":{ "dynamic":"strict", "properties": { "name": { "type": "text" }, "age": { "type": "long" } } } } } #####插入數據 PUT test6/doc/1 { "name":"wangjifei", "age":"18", "sex":"不詳" } >>>插入結果 { "error": { "root_cause": [ { "type": "strict_dynamic_mapping_exception", "reason": "mapping set to strict, dynamic introduction of [sex] within [doc] is not allowed" } ], "type": "strict_dynamic_mapping_exception", "reason": "mapping set to strict, dynamic introduction of [sex] within [doc] is not allowed" }, "status": 400 }
小結: 動態映射(dynamic:true):動態添加新的字段(或缺省)。 靜態映射(dynamic:false):忽略新的字段。在原有的映射基礎上,當有新的字段時,不會主動的添加新的映射關系,只作為查詢結果出現在查詢中。 嚴格模式(dynamic:strict):如果遇到新的字段,就拋出異常。一般靜態映射用的較多。就像HTML的img標簽一樣,src為自帶的屬性,你可以在需要的時候添加id或者class屬性。當然,如果你非常非常了解你的數據,並且未來很長一段時間不會改變,strict不失為一個好選擇。
4. ES之mappings的 index 屬性
- index屬性默認為true,如果該屬性設置為false,那么,elasticsearch不會為該屬性創建索引,也就是說無法當做主查詢條件。
PUT test7 { "mappings": { "doc": { "properties": { "name": { "type": "text", "index": true }, "age": { "type": "long", "index": false } } } } } ####插入數據 PUT test7/doc/1 { "name":"wangjifei", "age":18 } ####條件查詢數據 GET test7/doc/_search { "query": { "match": { "name": "wangjifei" } } } >>>查詢結果 { "took" : 18, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.2876821, "hits" : [ { "_index" : "test7", "_type" : "doc", "_id" : "1", "_score" : 0.2876821, "_source" : { "name" : "wangjifei", "age" : 18 } } ] } } #####條件查詢 GET test7/doc/_search { "query": { "match": { "age": 18 } } } >>>查詢結果 { "error": { "root_cause": [ { "type": "query_shard_exception", "reason": "failed to create query: {\n \"match\" : {\n \"age\" : {\n \"query\" : 18,\n \"operator\" : \"OR\",\n \"prefix_length\" : 0,\n \"max_expansions\" : 50,\n \"fuzzy_transpositions\" : true,\n \"lenient\" : false,\n \"zero_terms_query\" : \"NONE\",\n \"auto_generate_synonyms_phrase_query\" : true,\n \"boost\" : 1.0\n }\n }\n}", "index_uuid": "fzN9frSZRy2OzinRjeMKGA", "index": "test7" } ], "type": "search_phase_execution_exception", "reason": "all shards failed", "phase": "query", "grouped": true, "failed_shards": [ { "shard": 0, "index": "test7", "node": "INueKtviRpO1dbNWngcjJA", "reason": { "type": "query_shard_exception", "reason": "failed to create query: {\n \"match\" : {\n \"age\" : {\n \"query\" : 18,\n \"operator\" : \"OR\",\n \"prefix_length\" : 0,\n \"max_expansions\" : 50,\n \"fuzzy_transpositions\" : true,\n \"lenient\" : false,\n \"zero_terms_query\" : \"NONE\",\n \"auto_generate_synonyms_phrase_query\" : true,\n \"boost\" : 1.0\n }\n }\n}", "index_uuid": "fzN9frSZRy2OzinRjeMKGA", "index": "test7", "caused_by"