最近幾天的push:user:req數據寫不到集群，報錯如下：
[2017-06-13T09:31:51,803][DEBUG][o.e.a.b.TransportShardBulkAction] [yunva_etl_es8] [push:user:req][1] failed to execute bulk item (index) index {[push:user:req][/push/user/req][AVyfE_JChh2DyKB0G1Kj], source[{"appId":"500062","yunvaId":"362b103623aa4f75911b374615c6a682","providerId":"1","proviceId":"27","mac":"B0:C4:E7:61:E6:E7","imsi":"460023080802034","imei":"354026057092725","factory":"samsung","model":"GT-S7568","osType":"android","osVersion":15,"networkType":"wifi","ip":"14.152.80.239","sessionId":"20170613091139059173449","time":"2017-06-13 09:11:39.061.+0800"}]} java.lang.IllegalArgumentException: number of documents in the index cannot exceed 2147483519 at org.apache.lucene.index.DocumentsWriterPerThread.reserveOneDoc(DocumentsWriterPerThread.java:208) ~[lucene-core-6.2.1.jar:6.2.1 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - 2016-09-15 05:15:20] at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:215) ~[lucene-core-6.2.1.jar:6.2.1 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - 2016-09-15 05:15:20] at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:478) ~[lucene-core-6.2.1.jar:6.2.1 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - 2016-09-15 05:15:20] at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1562) ~[lucene-core-6.2.1.jar:6.2.1 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - 2016-09-15 05:15:20] at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307) ~[lucene-core-6.2.1.jar:6.2.1 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - 2016-09-15 05:15:20] at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:558) ~[elasticsearch-5.0.1.jar:5.0.1]
搜索了下：
是lucene的文檔數限制，每個分片最大支持2的31次方個文檔數量
https://discuss.elastic.co/t/lucene-max-documents-limit/34761
Lucene max documents limit，Yep, there is a 2^31 limit per shard in Lucene.
查詢當前es分片文檔：
# curl 10.26.241.237:9200/_cat/shards?v index shard prirep state docs store ip node push:user:req 1 p STARTED 2147483519 883.7gb 10.30.136.143 yunva_etl_es8 push:user:req 1 r STARTED 2147483519 883.7gb 10.25.135.215 yunva_etl_es2 push:user:req 3 p STARTED 2147483519 883.7gb 10.45.150.115 yunva_etl_es9 push:user:req 3 r STARTED 2147483519 883.7gb 10.30.136.143 yunva_etl_es8 push:user:req 4 r STARTED 2147483519 883.9gb 10.30.136.143 yunva_etl_es8 push:user:req 4 p STARTED 2147483519 883.9gb 10.174.12.230 yunva_etl_es10 push:user:req 2 p STARTED 2147483519 884gb 10.174.12.230 yunva_etl_es10 push:user:req 2 r STARTED 2147483519 884gb 10.27.78.228 yunva_etl_es5 push:user:req 0 r STARTED 2147483519 883gb 10.45.150.115 yunva_etl_es9 push:user:req 0 p STARTED 2147483519 883gb 10.174.12.230 yunva_etl_es10
查詢出來的2147483519和2的31次方2147483648接近，所以新增加的數據無法插入
臨時的解決辦法：先把kafka的磁盤加到數據多保留一段時間20天
原來的集群數據抽取到hbase中作為冷數據處理，再把kafka里面的數據導入數據到新的集群中