ElasticSearch Scroll滚动查询(查询数据超过10000)


ElasticSearch官方默认查询数据是10000条,查询第10001时会报错:Result window is too large, from + size must be less than or equal to...

以下是From/Size查询方法的官方文档:

https://www.elastic.co/guide/en/elasticsearch/reference/6.4/search-request-from-size.html

 

今天我们重点介绍一下:Scroll滚动查询

以下是Scroll官方文档:

https://www.elastic.co/guide/en/elasticsearch/reference/6.4/search-request-scroll.html

 

代码:

@Autowired
private RestHighLevelClient restHighLevelClient;

public void scrollDemo() {

//构造查询条件
SearchRequest searchRequest = new SearchRequest("索引库");
SearchSourceBuilder builder = new SearchSourceBuilder();
//设置查询超时时间
Scroll scroll = new Scroll(TimeValue.timeValueMinutes(1L));
builder.query(QueryBuilders.rangeQuery("字段").gte("开始时间").lte("结束时间"));
//设置最多一次能够取出10000笔数据,从第10001笔数据开始,将开启滚动查询 PS:滚动查询也属于这一次查询,只不过因为一次查不完,分多次查
builder.size(10000);
searchRequest.source(builder);
//将滚动放入
searchRequest.scroll(scroll);
SearchResponse searchResponse = null;
try {
searchResponse = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
} catch (IOException e) {
log.error("查询索引库失败", e.getMessage(), e);
}
SearchHits hits = searchResponse.getHits();
//记录要滚动的ID
String scrollId = searchResponse.getScrollId();

//TODO 对结果集的处理

//滚动查询部分,将从第10001笔数据开始取
SearchHit[] hitsScroll = hits.getHits();
while (hitsScroll != null && hitsScroll.length > 0 ) {
//构造滚动查询条件
SearchScrollRequest searchScrollRequest = new SearchScrollRequest(scrollId);
searchScrollRequest.scroll(scroll);
try {
//响应必须是上面的响应对象,需要对上一层进行覆盖
searchResponse = restHighLevelClient.scroll(searchScrollRequest, RequestOptions.DEFAULT);
} catch (IOException e) {
log.error("滚动查询失败",e.getMessage(),e);
}
scrollId = searchResponse.getScrollId();
hits = searchResponse.getHits();
hitsScroll = hits.getHits();

//TODO 同上面完全一致的结果集处理
}

//清除滚动,否则影响下次查询
ClearScrollRequest clearScrollRequest = new ClearScrollRequest();
clearScrollRequest.addScrollId(scrollId);
ClearScrollResponse clearScrollResponse = null;
try {
clearScrollResponse = restHighLevelClient.clearScroll(clearScrollRequest, RequestOptions.DEFAULT);
} catch (IOException e) {
log.error("滚动查询清除失败",e.getMessage(),e);
}
//清除滚动是否成功
boolean succeeded = clearScrollResponse.isSucceeded();
}

 

当然,除此之外还有一种查询方法:Search After(它与Scroll非常相似,但是其参数是无状态的,始终针对最新版本的搜索器进行解析。如此在后序遍历的过程中根据索引更新删除而改变;导致查询混乱)

详情可见官方文档:

https://www.elastic.co/guide/en/elasticsearch/reference/6.4/search-request-search-after.html


免责声明!

本站转载的文章为个人学习借鉴使用,本站对版权不负任何法律责任。如果侵犯了您的隐私权益,请联系本站邮箱yoyou2525@163.com删除。



 
粤ICP备18138465号  © 2018-2025 CODEPRJ.COM