Elasticsearch .net client NEST使用說明 2.x
Elasticsearch.Net與NEST是Elasticsearch為C#提供的一套客戶端驅動,方便C#調用Elasticsearch服務接口。Elasticsearch.Net是較基層的對Elasticsearch服務接口請求響應的實現,NEST是在前者基礎之上進行的封裝。本文是針對NEST的使用的總結。
目錄:
Elasticsearch.Net、NEST 交流群:523061899
demo源碼 https://github.com/huhangfei/NestDemos
引用
Install-Package NEST
包含以下dll
NEST.dll
Elasticsearch.Net.dll
Newtonsoft.Json.dll
概念
存儲結構:
在Elasticsearch中,文檔(Document
)歸屬於一種類型(type),而這些類型存在於索引(index)中.
類比傳統關系型數據庫:
Relational DB -> Databases -> Tables -> Rows -> Columns
Elasticsearch -> Indices -> Types -> Documents -> Fields
距離單位:
-
mm
(毫米) -
cm
(厘米) -
m
(米) -
km
(千米) -
in
(英寸) -
ft
(英尺) -
yd
(碼) -
mi
(英里) -
nmi
orNM
(海里)
日期單位:
- y (year)
- M (month)
- w (week)
- d (day)
- h (hour)
- m (minute)
- s (second)
- ms (milliseconds)
客戶端語法
鏈式lambda 表達式( powerful query DSL)
s => s .Query(q => q .Term(p => p.Name, "elasticsearch") )
對象初始化語法
var searchRequest = new SearchRequest<VendorPriceInfo> { Query = new TermQuery { Field = "name", Value = "elasticsearch" } };
Connection鏈接
//單node Var node = new Uri(“……”); var settings = new ConnectionSettings(node); //多uri Var uris = new Uri [] { new Uri(“……”), new Uri(“……”) }; //多node Var nodes = new Node [] { new Node (new Uri(“……”)), new Node (new Uri(“……”)) }; var pool = new StaticConnectionPool(nodes);
var pool = new StaticConnectionPool(uris);
var settings = new ConnectionSettings(pool);
var client = new ElasticClient(settings);
連接池類型:
//對單節點請求 IConnectionPool pool = new SingleNodeConnectionPool(urls.FirstOrDefault()); //請求時隨機請求各個正常節點,不請求異常節點,異常節點恢復后會重新被請求 IConnectionPool pool = new StaticConnectionPool(urls); IConnectionPool pool = new SniffingConnectionPool(urls); //false.創建客戶端時,隨機選擇一個節點作為客戶端的請求對象,該節點異常后不會切換其它節點 //true,請求時隨機請求各個正常節點,不請求異常節點,但異常節點恢復后不會重新被請求 pool.SniffedOnStartup = true; //創建客戶端時,選擇第一個節點作為請求主節點,該節點異常后會切換其它節點,待主節點恢復后會自動切換回來 IConnectionPool pool = new StickyConnectionPool(urls);
索引選擇
方式1:
var settings = new ConnectionSettings().DefaultIndex("defaultindex");
方式2:
var settings = new ConnectionSettings().MapDefaultTypeIndices(m => m.Add(typeof(Project), "projects") );
方式3:
client.Search<VendorPriceInfo>(s => s.Index("test-index")); client.Index(data,o=>o.Index("test-index"));
優先級:方式3 > 方式2 > 方式1
索引
索引唯一Id:
1) 默認以“Id”字段值作為索引唯一Id值,無“Id”屬性,Es自動生成唯一Id值,添加數據時統一類型數據唯一ID已存在相等值,將只做更新處理。
注:自動生成的ID有22個字符長,URL-safe, Base64-encoded string universally unique identifiers, 或者叫UUIDs。
2) 標記唯一Id值
[ElasticsearchType(IdProperty = "priceID")] public class VendorPriceInfo { public Int64 priceID { get; set; } public int oldID { get; set; } public int source { get; set; } }
3) 索引時指定
client.Index(data, o => o.Id(data.vendorName));
優先級: 3) > 2) > 1)
索引類型:
1) 默認類型為索引數據的類名(自動轉換為全小寫)
2) 標記類型
[ElasticsearchType(Name = "datatype")] public class VendorPriceInfo { public Int64 priceID { get; set; } public int oldID { get; set; } public int source { get; set; } }
3) 索引時指定
client.Index(data, o => o.Type(new TypeName() { Name = "datatype", Type = typeof(VendorPriceInfo) })); 或 client.Index(data, o => o.Type<MyClass>());//使用 2)標記的類型
優先級:3)> 2) > 1)
創建:
client.CreateIndex("test2"); //基本配置 IIndexState indexState=new IndexState() { Settings = new IndexSettings() { NumberOfReplicas = 1,//副本數 NumberOfShards = 5//分片數 } }; client.CreateIndex("test2", p => p.InitializeUsing(indexState)); //創建並Mapping client.CreateIndex("test-index3", p => p.InitializeUsing(indexState).Mappings(m => m.Map<VendorPriceInfo>(mp => mp.AutoMap())));
注:索引名稱必須小寫
判斷:
client.IndexExists("test2");
刪除:
client.DeleteIndex("test2");
Open/Close:
client.OpenIndex("index"); client.CloseIndex("index");
映射
概念:
每個類型擁有自己的映射(mapping)或者模式定義(schema definition)。一個映射定義了字段類型,每個字段的數據類型,以及字段被Elasticsearch處理的方式。映射還用於設置關聯到類型上的元數據。
獲取映射
var resule = client.GetMapping<VendorPriceInfo>();
特性
/// <summary> /// VendorPrice 實體 /// </summary> [ElasticsearchType(IdProperty = "priceID", Name = "VendorPriceInfo")] public class VendorPriceInfo { [Number(NumberType.Long)] public Int64 priceID { get; set; } [Date(Format = "mmddyyyy")] public DateTime modifyTime { get; set; } /// <summary> /// 如果string 類型的字段不需要被分析器拆分,要作為一個正體進行查詢,需標記此聲明,否則索引的值將被分析器拆分 /// </summary> [String(Index = FieldIndexOption.NotAnalyzed)] public string pvc_Name { get; set; } /// <summary> /// 設置索引時字段的名稱 /// </summary> [String(Name = "PvcDesc")] public string pvc_Desc { get; set; } /// <summary> /// 如需使用坐標點類型需添加坐標點特性,在maping時會自動映射類型 /// </summary> [GeoPoint(Name = "ZuoBiao",LatLon = true)] public GeoLocation Location { get; set; } }
映射
//根據對象類型自動映射 var result= client.Map<VendorPriceInfo>(m => m.AutoMap()); //手動指定 var result1 = client.Map<VendorPriceInfo>(m => m.Properties(p => p .GeoPoint(gp => gp.Name(n => n.Location)// 坐標點類型 .Fielddata(fd => fd .Format(GeoPointFielddataFormat.Compressed)//格式 array doc_values compressed disabled .Precision(new Distance(2, DistanceUnit.Meters)) //精確度 )) .String(s => s.Name(n => n.b_id))//string 類型 )); //在原有字段下新增字段(用於存儲不同格式的數據,查詢方法查看SearchBaseDemo) //eg:在 vendorName 下添加無需分析器分析的值 temp var result2 = client.Map<VendorPriceInfo>( m => m .Properties(p => p.String(s => s.Name(n => n.vendorName).Fields(fd => fd.String(ss => ss.Name("temp").Index(FieldIndexOption.NotAnalyzed))))));
注:映射時已存在的字段將無法重新映射,只有新加的字段能映射成功。
注:映射時同一索引中,多個類型中如果有相同字段名,那么在索引時可能會出現問題(會使用第一個映射類型)。
數據
數據操作
提示:
- 添加數據時,如果文檔的唯一id在索引里已存在,那么會替換掉原數據;
- 添加數據時,如果索引不存在,服務會自動創建索引;
- 如果服務自動創建索引,並索引了數據,那么索引的映射關系就是服務器自動設置的;
- 通常正確的使用方法是在緊接着創建索引操作之后進行映射關系的操作,以保證索引數據的映射是正確的。然后才是索引數據;
- 文檔在Elasticsearch中是不可變的,執行Update事實上Elasticsearch的處理過程如下:
-
- 從舊文檔中檢索JSON
- 修改它
- 刪除舊文檔
- 索引新文檔
所以我們也可以使用Index來更新已存在文檔,只需對應文檔的唯一id。
添加索引數據:
添加單條數據
var data = new VendorPriceInfo() { vendorName = "測試"}; client.Index(data);
添加多條數據
var datas = new List<VendorPriceInfo> { new VendorPriceInfo(){priceID = 1,vendorName = "test1"}, new VendorPriceInfo(){priceID = 2,vendorName = "test2"}}; client.IndexMany(datas);
刪除數據:
單條數據
DocumentPath<VendorPriceInfo> deletePath=new DocumentPath<VendorPriceInfo>(7); client.Delete(deletePath); 或 IDeleteRequest request = new DeleteRequest("test3", "vendorpriceinfo", 0); client.Delete(request);
注:刪除時根據唯一id刪除
集合數據
Indices indices = "test-1"; Types types = "vendorpriceinfo"; //批量刪除 需要es安裝 delete-by-query插件 var result = client.DeleteByQuery<VendorPriceInfo>(indices, types, dq => dq.Query( q => q.TermRange(tr => tr.Field(fd => fd.priceID).GreaterThanOrEquals("5").LessThanOrEquals("10"))) );
更新數據:
更新所有字段
DocumentPath<VendorPriceInfo> deletePath=new DocumentPath<VendorPriceInfo>(2); Var response=client.Update(deletePath,(p)=>p.Doc(new VendorPriceInfo(){vendorName = "test2update..."})); //或 IUpdateRequest<VendorPriceInfo, VendorPriceInfo> request = new UpdateRequest<VendorPriceInfo, VendorPriceInfo>(deletePath) { Doc = new VendorPriceInfo() { priceID = 888, vendorName = "test4update........" } }; var response = client.Update<VendorPriceInfo, VendorPriceInfo>(request);
更新部分字段
IUpdateRequest<VendorPriceInfo, VendorPriceInfoP> request = new UpdateRequest<VendorPriceInfo, VendorPriceInfoP>(deletePath) { Doc = new VendorPriceInfoP() { priceID = 888, vendorName = "test4update........" } }; var response = client.Update(request);
更新部分字段
IUpdateRequest<VendorPriceInfo, object> request = new UpdateRequest<VendorPriceInfo, object>(deletePath) { Doc = new { priceID = 888, vendorName = " test4update........" } }; var response = client.Update(request); //或 client.Update<VendorPriceInfo, object>(deletePath, upt => upt.Doc(new { vendorName = "ptptptptp" }));
注:更新時根據唯一id更新
獲取數據:
var response = client.Get(new DocumentPath<VendorPriceInfo>(0)); //或 var response = client.Get(new DocumentPath<VendorPriceInfo>(0),pd=>pd.Index("test4").Type("v2")); //多個 var response = client.MultiGet(m => m.GetMany<VendorPriceInfo>(new List<long> { 1, 2, 3, 4 }));
注:獲取時根據唯一id獲取
搜索:
說明
基本搜索
var result = client.Search<VendorPriceInfo>( s => s .Explain() //參數可以提供查詢的更多詳情。 .FielddataFields(fs => fs //對指定字段進行分析 .Field(p => p.vendorFullName) .Field(p => p.cbName) ) .From(0) //跳過的數據個數 .Size(50) //返回數據個數 .Query(q => q.Term(p => p.vendorID, 100) // 主要用於精確匹配哪些值,比如數字,日期,布爾值或 not_analyzed的字符串(未經分析的文本數據類型): && q.Term(p => p.vendorName.Suffix("temp"), "姓名") //用於自定義屬性的查詢 (定義方法查看MappingDemo) && q.Bool( //bool 查詢 b => b .Must(mt => mt //所有分句必須全部匹配,與 AND 相同 .TermRange(p => p.Field(f => f.priceID).GreaterThan("0").LessThan("1"))) //指定范圍查找 .Should(sd => sd //至少有一個分句匹配,與 OR 相同 .Term(p => p.priceID, 32915), sd => sd.Terms(t => t.Field(fd => fd.priceID).Terms(new[] {10, 20, 30})),//多值 //|| //sd.Term(p => p.priceID, 1001) //|| //sd.Term(p => p.priceID, 1005) sd => sd.TermRange(tr => tr.GreaterThan("10").LessThan("12").Field(f => f.vendorPrice)) ) .MustNot(mn => mn//所有分句都必須不匹配,與 NOT 相同 .Term(p => p.priceID, 1001) , mn => mn.Bool(//bool 過濾 ,bool 查詢與 bool 過濾相似,用於合並多個查詢子句。不同的是,bool 過濾可以直接給出是否匹配成功, 而bool 查詢要計算每一個查詢子句的 _score (相關性分值)。 bb=>bb.Must(mt=>mt .Match(mc=>mc.Field(fd=>fd.carName).Query("至尊")) )) ) ) )//查詢條件 .Sort(st => st.Ascending(asc => asc.vendorPrice))//排序 .Source(sc => sc.Include(ic => ic .Fields( fd => fd.vendorName, fd => fd.vendorID, fd => fd.priceID, fd => fd.vendorPrice))) //返回特定的字段 ); //TResult var result1 = client.Search<VendorPriceInfo, VendorPriceInfoP>(s => s.Query( q => q.MatchAll() ) .Size(15) );
或
var result = client.Search<VendorPriceInfo>(new SearchRequest() { Sort =new List<ISort> { new SortField { Field = "vendorPrice", Order = SortOrder.Ascending } }, Size = 10, From = 0, Query = new TermQuery() { Field = "priceID", Value = 6 } || new TermQuery( { Field = "priceID", Value = 8 } });
分頁
//分頁最大限制(from+size<=10000)
int pageSize = 10; int pageIndex = 1; var result = client.Search<VendorPriceInfo>(s => s.Query(q => q .MatchAll()) .Size(pageSize) .From((pageIndex - 1) * pageSize) .Sort(st => st.Descending(d => d.priceID)));
掃描和滾屏
string scrollid = ""; var result = client.Search<VendorPriceInfo>(s => s.Query(q => q.MatchAll()) .Size(100) .SearchType(SearchType.Scan) .Scroll("1m"));//scrollid過期時間 //得到滾動掃描的id scrollid = result.ScrollId; //執行滾動掃描得到數據 返回數據量是 result.Shards.Successful*size(查詢成功的分片數*size) result = client.Scroll<VendorPriceInfo>("1m", scrollid); //得到新的id scrollid = result.ScrollId;
查詢條件設置加權
// 在原分值基礎上 設置不同匹配的加成值 具體算法為lucene內部算法
var result = client.Search<VendorPriceInfo>(s => s .Query(q => q.Term(t => t .Field(f => f.cityID).Value(2108).Boost(4)) || q.Term(t => t .Field(f => f.pvcId).Value(2103).Boost(1)) ) .Size(3000) .Sort(st => st.Descending(SortSpecialField.Score)) );
得分控制
//使用functionscore計算得分
var result1 = client.Search<VendorPriceInfo>(s => s .Query(q=>q.FunctionScore(f=>f
//查詢區 .Query(qq => qq.Term(t => t .Field(fd => fd.cityID).Value(2108)) || qq.Term(t => t .Field(fd => fd.pvcId).Value(2103)) ) .Boost(1.0) //functionscore 對分值影響 .BoostMode(FunctionBoostMode.Replace)//計算boost 模式 ;Replace為替換 .ScoreMode(FunctionScoreMode.Sum) //計算score 模式;Sum為累加
//邏輯區 .Functions(fun=>fun .Weight(w => w.Weight(2).Filter(ft => ft .Term(t => t .Field(fd => fd.cityID).Value(2108))))//匹配cityid +2 .Weight(w => w.Weight(1).Filter(ft => ft .Term(t => t .Field(fd => fd.pvcId).Value(2103))))//匹配pvcid +1 ) ) ) .Size(3000) .Sort(st => st.Descending(SortSpecialField.Score).Descending(dsc=>dsc.priceID)) );
//結果中 cityid=2108,得分=2; pvcid=2103 ,得分=1 ,兩者都滿足的,得分=3
查詢字符串-簡單的檢索
var result = client.Search<VendorPriceInfo>(s => s .Query(q => q.QueryString(m => m.Fields(fd=>fd.Field(fdd=>fdd.carName).Field(fdd=>fdd.carGearBox)) .Query("手自一體") ) ) .From(0) .Size(15) );
全文檢索-關鍵詞搜索
var result=client.Search<VendorPriceInfo>(s=>s .Query(q=>q .Match(m=>m.Field(f=>f.carName) .Query("尊貴型") ) ) .From(0) .Size(15) ); //多字段匹配 var result1 = client.Search<VendorPriceInfo>(s => s .Query(q => q .MultiMatch(m => m.Fields(fd=>fd.Fields(f=>f.carName,f=>f.carGearBox)) .Query("尊貴型") ) ) .From(0) .Size(15) );
全文搜索-短語搜索
var result = client.Search<VendorPriceInfo>(s => s .Query(q => q.MatchPhrase(m => m.Field(f => f.carName) .Query("尊貴型") ) ) .From(0) .Size(15) );
坐標點搜索-根據坐標點及距離搜索
const double lat = 39.8694890000; const double lon = 116.4206470000; const double distance = 2000.0; //1 var result = client.Search<VendorPriceInfo>(s => s .Query(q => q .Bool(b => b.Must(m => m .GeoDistance(gd => gd .Location(lat, lon) .Distance(distance, DistanceUnit.Meters) .Field(fd => fd.Location) )) ) ) .From(0) .Size(15) ); //2 var location = new GeoLocation(lat, lon); var distancei = new Distance(distance, DistanceUnit.Meters); var result1 = client.Search<VendorPriceInfo>(s => s .Query(q => q .Bool(b => b.Must(m => m .Exists(e => e.Field(fd => fd.Location)) ) ) && q.GeoDistance(gd => gd .Location(location) .Distance(distancei) .Field(fd => fd.Location) ) ) .From(0) .Size(15) ); //3 var result2 = client.Search<VendorPriceInfo>(s => s .Query(q => q .Bool(b=>b .Must(m=>m.MatchAll()) .Filter(f=>f .GeoDistance(g => g .Name("named_query") .Field(p => p.Location) .DistanceType(GeoDistanceType.Arc) .Location(lat,lon) .Distance("2000.0m") ) ) ) ) .From(0) .Size(15) );
聚合
聚合-基本
var result = client.Search<VendorPriceInfo>(s => s .From(0) .Size(15) .Aggregations(ag=>ag .ValueCount("Count", vc => vc.Field(fd => fd.vendorPrice))//總數 .Sum("vendorPrice_Sum", su => su.Field(fd => fd.vendorPrice))//求和 .Max("vendorPrice_Max", m => m.Field(fd => fd.vendorPrice))//最大值 .Min("vendorPrice_Min", m => m.Field(fd => fd.vendorPrice))//最小值 .Average("vendorPrice_Avg", avg => avg.Field(fd => fd.vendorPrice))//平均值 .Terms("vendorID_group", t => t.Field(fd => fd.vendorID).Size(100))//分組 ) );
聚合-分組
//每個經銷商 的平均報價 var result = client.Search<VendorPriceInfo>(s => s .Size(0) .Aggregations(ag => ag .Terms("vendorID_group", t => t .Field(fd => fd.vendorID) .Size(100) .Aggregations(agg => agg.Average("vendorID_Price_Avg", av => av.Field(fd => fd.vendorPrice))) )//分組 .Cardinality("vendorID_group_count", dy => dy.Field(fd => fd.vendorID))//分組數量 .ValueCount("Count", c => c.Field(fd => fd.vendorID))//總記錄數 ) );
聚合-分組-聚合..
//每個經銷商下 每個品牌 的平均報價 var result = client.Search<VendorPriceInfo>(s => s .Size(0) .Aggregations(ag => ag .Terms("vendorID_group", //vendorID 分組 t => t.Field(fd => fd.vendorID) .Size(100) .Aggregations(agg => agg .Terms("vendorID_cbID_group", //cbID分組 tt => tt.Field(fd => fd.cbID) .Size(50) .Aggregations(aggg => aggg .Average("vendorID_cbID_Price_Avg", av => av.Field(fd => fd.vendorPrice))//Price avg .Max("vendorID_cbID_Price_Max", m => m.Field(fd => fd.vendorPrice))//Price max .Min("vendorID_cbID_Price_Min", m => m.Field(fd => fd.vendorPrice))//Price min .ValueCount("vendorID_cbID_Count", m => m.Field(fd => fd.cbID))//該經銷商對該品牌 報價數 count ) ) .Cardinality("vendorID_cbID_group_count", dy => dy.Field(fd => fd.cbID))//分組數量 .ValueCount("vendorID_Count", c => c.Field(fd => fd.vendorID))//該經銷商的報價數 ) ) .Cardinality("vendorID_group_count",dy=>dy.Field(fd=>fd.vendorID))//分組數量 .ValueCount("Count",c=>c.Field(fd=>fd.priceID))//總記錄數 ) //分組 );
附
官網文檔:
https://www.elastic.co/guide/en/elasticsearch/client/net-api/2.x/introduction.html