yum -y install openssl-devel zlib-devel bzip2-devel sqlite-devel readline-devel libffi-devel systemtap-sdt-devel
3、下載安裝包

tar zvxf Python-3.7.0.tgz
cd Python-3.7.0
./configure --prefix=/usr/local/python3.7 --enable-optimizations
make && make install

# 編譯完成后，創建軟鏈接文件到執行文件路徑：
ln -s /usr/local/python3/bin/python3 /usr/bin/python3
ln -s /usr/local/python3/bin/pip3 /usr/bin/pip3
# 我們可以清除之前編譯的可執行文件及配置文件 && 清除所有生成的文件：
make clean && make distclean

if [ -z ${PYTHON37_HOME} ]; then
export PYTHON37_HOME=/usr/local/python3.7
export PATH=${PYTHON37_HOME}/bin:${PATH}
fi
6、加載環境變量

2，git2 安裝

centos7系統默認的git安裝版本是1.8，但是在項目構建中發現git版本過低，於是用源碼編譯的方式進行升級.

作者：一介布衣q
鏈接：https://www.imooc.com/article/275738
來源：慕課網
本文原創發布於慕課網，轉載請注明出處，謝謝合作

——————————————————————————————————————————

4，使用ESRally壓測ElasticSearch性能

------------------------------------------------------
 _______ __ _____
 / ____(_)___ ____ _/ / / ___/_________ ________
 / /_ / / __ \/ __ `/ / \__ \/ ___/ __ \/ ___/ _ \
 / __/ / / / / / /_/ / / ___/ / /__/ /_/ / / / __/
/_/ /_/_/ /_/\__,_/_/ /____/\___/\____/_/ \___/
------------------------------------------------------

| Metric | Task | Value | Unit |
|-------------------------------:|---------------------:|----------:|-------:|
| Total indexing time | | 28.0997 | min |
| Total merge time | | 6.84378 | min |
| Total refresh time | | 3.06045 | min |
| Total flush time | | 0.106517 | min |
| Total merge throttle time | | 1.28193 | min |
| Median CPU usage | | 471.6 | % |
| Total Young Gen GC | | 16.237 | s |
| Total Old Gen GC | | 1.796 | s |
| Index size | | 2.60124 | GB |
| Total written | | 11.8144 | GB |
| Heap used for segments | | 14.7326 | MB |
| Heap used for doc values | | 0.115917 | MB |
| Heap used for terms | | 13.3203 | MB |
| Heap used for norms | | 0.0734253 | MB |
| Heap used for points | | 0.5793 | MB |
| Heap used for stored fields | | 0.643608 | MB |
| Segment count | | 97 | |
| Min Throughput | index-append | 31925.2 | docs/s |
| Median Throughput | index-append | 39137.5 | docs/s |
| Max Throughput | index-append | 39633.6 | docs/s |
| 50.0th percentile latency | index-append | 872.513 | ms |
| 90.0th percentile latency | index-append | 1457.13 | ms |
| 99.0th percentile latency | index-append | 1874.89 | ms |
| 100th percentile latency | index-append | 2711.71 | ms |
| 50.0th percentile service time | index-append | 872.513 | ms |
| 90.0th percentile service time | index-append | 1457.13 | ms |
| 99.0th percentile service time | index-append | 1874.89 | ms |
| 100th percentile service time | index-append | 2711.71 | ms |
| ... | ... | ... | ... |
| ... | ... | ... | ... |
| Min Throughput | painless_dynamic | 2.53292 | ops/s |
| Median Throughput | painless_dynamic | 2.53813 | ops/s |
| Max Throughput | painless_dynamic | 2.54401 | ops/s |
| 50.0th percentile latency | painless_dynamic | 172208 | ms |
| 90.0th percentile latency | painless_dynamic | 310401 | ms |
| 99.0th percentile latency | painless_dynamic | 341341 | ms |
| 99.9th percentile latency | painless_dynamic | 344404 | ms |
| 100th percentile latency | painless_dynamic | 344754 | ms |
| 50.0th percentile service time | painless_dynamic | 393.02 | ms |
| 90.0th percentile service time | painless_dynamic | 407.579 | ms |
| 99.0th percentile service time | painless_dynamic | 430.806 | ms |
| 99.9th percentile service time | painless_dynamic | 457.352 | ms |
| 100th percentile service time | painless_dynamic | 459.474 | ms |

----------------------------------
[INFO] SUCCESS (took 2634 seconds)
----------------------------------

在部署完一套ES集群之后，我們肯定想知道這套集群性能如何？是否可以支撐未來業務發展？存不存在性能上的瓶頸？要想有依據的回答這些問題，我們需要通過壓力測試結果中找答案。

介紹

Rally是Elasticsearch的基准測試框架，由官方提供維護。

安裝

安裝Python3.5及以上版本，系統默認可能是2.x版本，如果需要升級請參考《在CentOS7上安裝Python3》。
安裝git1.9及以上版本
安裝esrally pip3 install esrally
配置esrally esrally configure，執行此命令后會在當前用戶根目錄下生成 .rally 目錄，可以 ll ~/.rally這樣來確認。

使用

快速開始

如果想測試當前機器上某個版本單點ES性能，可以像下面這樣：

esrally --distribution-version=6.5.3

# 同樣的如果你想測試其它版本
esrally --distribution-version=6.8.1   --car="4gheap"

當執行上面的命令之后會自動下載對應es版本軟件，並在本地啟動，接着執行測試。這個過程在rally中被稱為比賽，而賽道是用默認的，即geonames。

測試遠程集群

上面的示例不能測試存在的es集群，下面介紹使用方法：

指定跑道和ES集群地址后就可以執行測試。

esrally --pipeline=benchmark-only \
 --track=http_logs \
 --target-hosts=192.168.1.100:9200,192.168.1.101:9200,192.168.1.102:9200 \
 --report-file=/tmp/report_http_logs.md
--track-params="bulk_indexing_clients:96"
 --include-tasks="index-append"
--challenge=append-fast-with-no-conflicts (只測試寫）

 --report-format=csv



備注： esrally list tracks 命令可以查看可用跑道（–track）

可以使用 --report-file=/path/to/your/report.md 將此報告也保存到文件中，並使用 --report-format=csv將其另存為CSV。

修改默認跑道參數

如果直接在默認跑道上修改，會被還原，所以只能通過增加跑道的方式。

在 .rally/benchmarks/tracks 下面創建新的賽道，比如 custom。

在 custom/http_logs/challenges/default.json 文件中調整賽道的配置並保存，例如下面修改default對應的操作，由10個客戶端發起，每個用戶端發出100次操作。

{
 "operation": "default",
 "clients": 10,
 "warmup-iterations": 500,
 "iterations": 100,
 "target-throughput": 100
}

在啟動時指定跑道：esrally --track=custom ....，例如：

esrally --pipeline=benchmark-only \
 --track=custom \
 --track=http_logs \
 --target-hosts=192.168.1.100:9200,192.168.1.101:9200,192.168.1.102:9200 \
 --report-file=/tmp/report_http_logs.md

參考文獻：

https://esrally.readthedocs.io/en/stable/index.html
https://github.com/elastic/rally
比較全的壓測介紹：https://www.jianshu.com/p/c89975b50447
使用docker 運行 esrally（包含離線數據集）：https://www.jianshu.com/p/3a019c135e2a
測試參數解釋：https://www.jianshu.com/p/979f548c233e
討論為什么elasticsearch沒有被壓滿：https://discuss.elastic.co/t/es-benchmark-using-rally-to-stress-a-2-node-setup/150020/6
有壓測結果對比：https://www.jianshu.com/p/e7de3b24f505

數據集：
國內下載慢，可以先執行一遍

esrally --distribution-version=6.5.3 --track=geonames
這樣即使下載測試數據失敗，但是目錄結構都生成好了。可以自行下載bz文件，在

http://benchmarks.elasticsearch.org.s3.amazonaws.com/corpora/geonames/documents.json.bz2
存成默認定義的：
documents-2.json.bz2
documents-2.json.offset
然后直接拷貝過去即可：~/.rally/benchmarks/data/geonames


esrally list tracks

This will show the following list:

 
             Name        Description Documents Compressed Size Uncompressed Size Default Challenge All Challenges ---------- ------------------------------------------------- ----------- ----------------- ------------------- ----------------------- --------------------------- geonames POIs from Geonames 11396505 252.4 MB 3.3 GB append-no-conflicts append-no-conflicts,appe... geopoint Point coordinates from PlanetOSM 60844404 481.9 MB 2.3 GB append-no-conflicts append-no-conflicts,appe... http_logs HTTP server log data 247249096 1.2 GB 31.1 GB append-no-conflicts append-no-conflicts,appe... nested StackOverflow Q&A stored as nested docs 11203029 663.1 MB 3.4 GB nested-search-challenge nested-search-challenge,... noaa Global daily weather measurements from NOAA 33659481 947.3 MB 9.0 GB append-no-conflicts append-no-conflicts,appe... nyc_taxis Taxi rides in New York in 2015 165346692 4.5 GB 74.3 GB append-no-conflicts append-no-conflicts,appe... percolator Percolator benchmark based on AOL queries 2000000 102.7 kB 104.9 MB append-no-conflicts append-no-conflicts,appe... pmc Full text benchmark with academic papers from PMC 574199 5.5 GB 21.7 GB  
            

這個地址里面 https://github.com/elastic/rally-tracks/tree/master/，進入子目錄有各個數據集可配置的參數：

如http_logs：

Parameters （--track-params="bulk_indexing_clients:96"）

This track allows to overwrite the following parameters with Rally 0.8.0+ using --track-params:

bulk_size (default: 5000)
bulk_indexing_clients (default: 8): Number of clients that issue bulk indexing requests.

5,自己定義壓測：

參考：https://esrally.readthedocs.io/en/latest/adding_tracks.html

主要步驟：

1，創建目錄：~/rally-tracks/tutorial

2，手動下載 geonames 壓測數據：　http://download.geonames.org/export/dump/allCountries.zip　。它里面是用ｔａｂ分開的文本文件。需要轉成ｊｓｏｎ格式。

用這段ｐｙｔｈｏｎ代碼轉：

import json

cols = (("geonameid", "int", True),
        ("name", "string", True),
        ("asciiname", "string", False),
        ("alternatenames", "string", False),
        ("latitude", "double", True),
        ("longitude", "double", True),
        ("feature_class", "string", False),
        ("feature_code", "string", False),
        ("country_code", "string", True),
        ("cc2", "string", False),
        ("admin1_code", "string", False),
        ("admin2_code", "string", False),
        ("admin3_code", "string", False),
        ("admin4_code", "string", False),
        ("population", "long", True),
        ("elevation", "int", False),
        ("dem", "string", False),
        ("timezone", "string", False))


def main():
    with open("allCountries.txt", "rt", encoding="UTF-8") as f:
        for line in f:
            tup = line.strip().split("\t")
            record = {}
            for i in range(len(cols)):
                name, type, include = cols[i]
                if tup[i] != "" and include:
                    if type in ("int", "long"):
                        record[name] = int(tup[i])
                    elif type == "double":
                        record[name] = float(tup[i])
                    elif type == "string":
                        record[name] = tup[i]
            print(json.dumps(record, ensure_ascii=False))


if __name__ == "__main__":
    main()

存到剛才~/rally-tracks/tutorial目錄下，python3 toJSON.py > documents.json

３，對於７.０以下的ｅｓ，保持這個成ｉｎｄｅｘ．ｊｓｏｎ

{
  "settings": {
    "index.number_of_replicas": 0
  },
  "mappings": {
    "docs": {
      "dynamic": "strict",
      "properties": {
        "geonameid": {
          "type": "long"
        },
        "name": {
          "type": "text"
        },
        "latitude": {
          "type": "double"
        },
        "longitude": {
          "type": "double"
        },
        "country_code": {
          "type": "text"
        },
        "population": {
          "type": "long"
        }
      }
    }
  }
}

５，再保存一個ｔｒａｃｋ．ｊｓｏｎ

{
  "version": 2,
  "description": "Tutorial benchmark for Rally",
  "indices": [
    {
      "name": "geonames",
      "body": "index.json",
      "types": [ "docs" ]
    }
  ],
  "corpora": [
    {
      "name": "rally-tutorial",
      "documents": [
        {
          "source-file": "documents.json",
          "document-count": 11658903,
          "uncompressed-bytes": 1544799789
        }
      ]
    }
  ],
  "schedule": [
    {
      "operation": {
        "operation-type": "delete-index"
      }
    },
    {
      "operation": {
        "operation-type": "create-index"
      }
    },
    {
      "operation": {
        "operation-type": "cluster-health",
        "request-params": {
          "wait_for_status": "green"
        }
      }
    },
    {
      "operation": {
        "operation-type": "bulk",
        "bulk-size": 5000
      },
      "warmup-time-period": 120,
      "clients": 8
    },
    {
      "operation": {
        "operation-type": "force-merge"
      }
    },
    {
      "operation": {
        "name": "query-match-all",
        "operation-type": "search",
        "body": {
          "query": {
            "match_all": {}
          }
        }
      },
      "clients": 8,
      "warmup-iterations": 1000,
      "iterations": 1000,
      "target-throughput": 100
    }
  ]
}

其中documents 屬性里面的字段值是這么來的：

wc -l documents.json　ｊｓｏｎ個數

stat -f "%z" documents.json　文件大小

７.０以后版本要去掉types 。

６，檢查建立成功沒有：esrally list tracks --track-path=~/rally-tracks/tutorial

７，執行自己的ｔｒａｃｋ：esrally --distribution-version=6.0.0 --track-path=~/rally-tracks/tutorial

--test-mode　來檢測配置文件對否。

這個來生成１０００條數據：head -n 1000 documents.json > documents-1k.json

ESRally壓測ElasticSearch性能 CentOS 7.5 安裝 Python3.7

1，CentOS 7.5 安裝 Python3.7