ES查詢報錯：entity content is too long [142501157] for the configured buffer limit [104857600]

在生產環境批量同步數據的時候，我寫了一個查詢請求，然后直接報錯：entity content is too long [142501157] for the configured buffer limit [104857600]

具體內容如下：

Caused by: org.apache.http.ContentTooLongException: entity content is too long [142501157] for the configured buffer limit [104857600]
        at org.elasticsearch.client.HeapBufferedAsyncResponseConsumer.onEntityEnclosed(HeapBufferedAsyncResponseConsumer.java:76)
        at org.apache.http.nio.protocol.AbstractAsyncResponseConsumer.responseReceived(AbstractAsyncResponseConsumer.java:137)
        at org.apache.http.impl.nio.client.MainClientExec.responseReceived(MainClientExec.java:315)
        at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseReceived(DefaultClientExchangeHandlerImpl.java:151)
        at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.responseReceived(HttpAsyncRequestExecutor.java:315)
        at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:255)
        at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
        at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
        at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
        at java.lang.Thread.run(Thread.java:748)
        Suppressed: org.apache.http.ContentTooLongException: entity content is too long [142501157] for the configured buffer limit [104857600]
                ... 16 more
                Suppressed: org.apache.http.ContentTooLongException: entity content is too long [142501157] for the configured buffer limit [104857600]
                        ... 16 more

在HttpAsyncResponseConsumerFactory類中：

在HeapBufferedAsyncResponseConsumer類中有一個判斷：返回的內容長度大於限定的100MB就會拋出ContentTooLongException異常。

我這里用的依賴是：

            <dependency>
                <groupId>org.elasticsearch.client</groupId>
                <artifactId>elasticsearch-rest-client</artifactId>
                <version>6.6.2</version>
            </dependency>

            <dependency>
                <groupId>org.elasticsearch.client</groupId>
                <artifactId>transport</artifactId>
                <version>6.6.2</version>
            </dependency>

            <dependency>
                <groupId>org.elasticsearch.client</groupId>
                <artifactId>elasticsearch-rest-high-level-client</artifactId>
                <version>6.6.2</version>
            </dependency>

            <dependency>
                <groupId>org.elasticsearch</groupId>
                <artifactId>elasticsearch</artifactId>
                <version>6.6.2</version>
            </dependency>

而我們對es進行查詢操作，或者其他增刪改操作是，都會默認使用如下的RequestOptions.DEFAULT

進入RequestOptions.DEFAULT看它的代碼：這里用了HeapBufferedResponseConsumerFactory.DEFAULT

而這個HeapBufferedResponseConsumerFactory.DEFAULT,就是我們之前看到的，已經被限制了100MB返回值的HeapBufferedResponseConsumerFactory。

所以有兩種方式可以解決這個bug。

方式一：

不使用默認的RequestOptions.DEFAULT，而通過使用自定義RequestOptions的方式（ES官方api已經給我們開放出來了）：

RequestOptions.Builder builder = RequestOptions.DEFAULT.toBuilder();
builder.setHttpAsyncResponseConsumerFactory(
    new HttpAsyncResponseConsumerFactory
    //修改為500MB
    .HeapBufferedResponseConsumerFactory(500 * 1024 * 1024));

方式二：

如果不方便修改怎么辦，例如我們使用的是Spring Data ElasticSearch或者是其他的一些框架，而它默認使用的也是RequestOptions.DEFAULT

這時候我們可以通過java反射進行修改：

        //設置es查詢buffer大小
        RequestOptions requestOptions = RequestOptions.DEFAULT;
        Class<? extends RequestOptions> reqClass = requestOptions.getClass();
        Field reqField = reqClass.getDeclaredField("httpAsyncResponseConsumerFactory");
        reqField.setAccessible(true);
        //去除final
        Field modifiersField = Field.class.getDeclaredField("modifiers");
        modifiersField.setAccessible(true);
        modifiersField.setInt(reqField, reqField.getModifiers() & ~Modifier.FINAL);

        //設置默認的工廠
        reqField.set(requestOptions, new HttpAsyncResponseConsumerFactory() {
            @Override
            public HttpAsyncResponseConsumer<HttpResponse> createHttpAsyncResponseConsumer() {
                //500MB
                return new HeapBufferedAsyncResponseConsumer(5 * 100 * 1024 * 1024);
            }
        });

如果你使用的是springmvc，那么可以把這段代碼加入到攔截器中，這樣可以攔截需要自定義返回值大小的es請求