HttpClient 模擬Curl 調用elasticsearch操作索引

最近需要實現一個功能，批量刪除索引中的doc,約有100W+條，想到了使用httpclient的方式，實現 curl來操作索引。

實現方式很簡單，就是一般發送post請求，比較坑的是，一定要注意，參數后面要加換行\n，否則會報：

HTTP/1.1 400 Bad Request [Warning: 299 Elasticsearch-5.6.9-877a590 "Content type detection for rest requests is deprecated. Specify the content type using the [Content-Type] header." "Thu, 19 Nov 2020 07:39:03 GMT", content-type: application/json; charset=UTF-8, content-length: 235] org.apache.http.conn.BasicManagedEntity@5824a83d

另外，需要注意  contentType的設置，如果是使用es 的批量功能  

如：post  http://10.64.6.32:9200/my_test_index1/product/_bulk

參數：

{"delete":{"_type":"product","_id":"AXWJMZFP321l4tnBy7Z1"}}

{"delete":{"_type":"product","_id":"AXWJMZFP321l4tnBy7Z2"}}

上面參數是要換行的。

另外，對於批量操作content-type需要設置為：text/plain。其它操作一般設置:application/json即可。

 

下面是代碼：

--------------------------參數后面一定要加換行\n-----------------------參數后面一定要加換行\n-----------------------參數后面一定要加換行\n-----------------------參數后面一定要加換行\n--------------------------

public class FileTest {
    public static void main(String[] args) throws IOException {
        // 讀取的CSV文件
         String path = "C://deleteIndexDoc.csv";
         readCsv(path);
    }

    public static void readCsv(String path) {
        try {
            DataInputStream in = new DataInputStream(new FileInputStream(new File(path)));
            CSVReader csvReader = new CSVReader(new InputStreamReader(in, "GBK"), CSVParser.DEFAULT_SEPARATOR,
                    CSVParser.DEFAULT_QUOTE_CHARACTER, CSVParser.DEFAULT_ESCAPE_CHARACTER, 1);
            String[] strs;
            String url = "http://ip:9200/my_test_index/product/_bulk";
            int i = 0;
            StringBuilder deletBuild = new StringBuilder();
             while ((strs = csvReader.readNext()) != null) {
                 i=i+1;
                 String param = "{\"delete\":{\"_id\":\""+strs[0]+"\"}}\n";
                 if(i%100==0)
                 {
                     deletBuild.append(param);
                     sendRequest(url,deletBuild.toString());
                     deletBuild = new StringBuilder();
                     System.out.println("finished num:"+i);
                 }else
                 {
                     deletBuild.append(param);
                 }
             }
             if(deletBuild.length()>0)
             {
                 sendRequest(url,deletBuild.toString());
             }
            csvReader.close();
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    public static void sendRequest(String url,String reqestParm) {
        System.out.println("reqestParm:\n"+reqestParm);
        JSONObject rsp = HttpRequestUtil.httpPost(url, reqestParm, false);
        System.out.println("rsp:"+rsp.toJSONString());
    }
}

HttpRequestUtil.java :

public static JSONObject httpPost(String url,String jsonParam, boolean noNeedResponse){

    //post請求返回結果
    DefaultHttpClient httpClient = new DefaultHttpClient();
    JSONObject jsonResult = null;
    HttpPost method = new HttpPost(url);
    String sdf = jsonParam.toString();
    String contentType = "text/plain";
    //String contentType = "application/json";
    try {
        if (null != jsonParam) {
            //解決中文亂碼問題
            StringEntity entity = new StringEntity(jsonParam.toString(), "utf-8");
            entity.setContentEncoding("UTF-8");
            entity.setContentType(contentType);
            method.setEntity(entity);
        }
        HttpResponse result = httpClient.execute(method);
        url = URLDecoder.decode(url, "UTF-8");
        /**請求發送成功，並得到響應**/
        if (result.getStatusLine().getStatusCode() == 200) {
            String str = "";
            try {
                /**讀取服務器返回過來的json字符串數據**/
                str = EntityUtils.toString(result.getEntity());
                if (noNeedResponse) {
                    return null;
                }
                /**把json字符串轉換成json對象**/
                jsonResult = JSONObject.parseObject(str);
            } catch (Exception e) {
                e.printStackTrace();
                System.out.println("post請求提交失敗1:" + url);
            }
        }
    } catch (IOException e) {
        e.printStackTrace();
        System.out.println("post請求提交失敗:" + url);
    }
    return jsonResult;
}