es sql是一個X-pack組件 ，允許對es執行類似sql的查詢，可以將Elasticsearch SQL理解為一個編譯器，既能理解es，又能理解sql。可以通過利用es，實施大規模實時讀取和處理數據。
sql和es的映射關系
| SQL | Elasticsearch |
| columns | field |
| raw | document |
| table | index |
| catalog or database | cluster實例 |
| cluster | cluster |
先插入一些數據：
PUT /my_index/doc/_bulk {"index":{"_id":"1"}} {"name":"lily","birthday":"2000-01-01","gender":"female"} {"index":{"_id":"2"}} {"name":"kangkang","birthday":"1998-04-01","gender":"male"} {"index":{"_id":"3"}} {"name":"jane","birthday":"1995-02-07","gender":"female"}
SQL REST API
POST /_xpack/sql?format=txt { "query":"select * from my_index where birthday<'1999-01-01' limit 2" }
# format類型有：json,yaml,smile,cbor,txt,csv,tsv
返回結果： birthday | gender | name ------------------------+---------------+--------------- 1998-04-01T00:00:00.000Z|male |kangkang 1995-02-07T00:00:00.000Z|female |jane
POST /_xpack/sql? { "query":"select * from my_index order by birthday desc", "fetch_size":1 # fetch_size 每頁返回多少個結果 } ---------> { "columns": [ { "name": "birthday", "type": "date" }, { "name": "gender", "type": "text" }, { "name": "name", "type": "text" } ], "rows": [ [ "2000-01-01T00:00:00.000Z", "female", "lily" ] ], "cursor": "k4bwAgFz5AFEbkYxWlhKNVZHaGxia1psZEdOb0JRQUFBQUFBQUY3WEZrbGtNa3R5V2s1VVZFTnRORmd3Y21Gd2VHeERMVkVBQUFBQUFBQmUyeFpKWkRKTGNscE9WRlJEYlRSWU1ISmhjSGhzUXkxUkFBQUFBQUFBWHRnV1NXUXlTM0phVGxSVVEyMDBXREJ5WVhCNGJFTXRVUUFBQUFBQUFGN1pGa2xrTWt0eVdrNVVWRU50TkZnd2NtRndlR3hETFZFQUFBQUFBQUJlMmhaSlpESkxjbHBPVkZSRGJUUllNSEpoY0hoc1F5MVL/////DwMBZghiaXJ0aGRheQEAAWYGZ2VuZGVyAAABZgRuYW1lAAA=" } # 該column對象只是第一頁的一部分，當cursor結果中沒有返回時，說明到達最后一頁。 # 可以通過發回cursor字段繼續下一頁。在文本格式的情況下，光標作為Cursorhttp標頭返回。 POST /_xpack/sql?format=json { "cursor": "k4bwAgFz5AFEbkYxWlhKNVZHaGxia1psZEdOb0JRQUFBQUFBQUY3WEZrbGtNa3R5V2s1VVZFTnRORmd3Y21Gd2VHeERMVkVBQUFBQUFBQmUyeFpKWkRKTGNscE9WRlJEYlRSWU1ISmhjSGhzUXkxUkFBQUFBQUFBWHRnV1NXUXlTM0phVGxSVVEyMDBXREJ5WVhCNGJFTXRVUUFBQUFBQUFGN1pGa2xrTWt0eVdrNVVWRU50TkZnd2NtRndlR3hETFZFQUFBQUFBQUJlMmhaSlpESkxjbHBPVkZSRGJUUllNSEpoY0hoc1F5MVL/////DwMBZghiaXJ0aGRheQEAAWYGZ2VuZGVyAAABZgRuYW1lAAA=" } #結果---------> { "rows": [ [ "1998-04-01T00:00:00.000Z", "male", "kangkang" ] ], "cursor": "k4bwAgFz5AFEbkYxWlhKNVZHaGxia1psZEdOb0JRQUFBQUFBQUY3WEZrbGtNa3R5V2s1VVZFTnRORmd3Y21Gd2VHeERMVkVBQUFBQUFBQmUyeFpKWkRKTGNscE9WRlJEYlRSWU1ISmhjSGhzUXkxUkFBQUFBQUFBWHRnV1NXUXlTM0phVGxSVVEyMDBXREJ5WVhCNGJFTXRVUUFBQUFBQUFGN1pGa2xrTWt0eVdrNVVWRU50TkZnd2NtRndlR3hETFZFQUFBQUFBQUJlMmhaSlpESkxjbHBPVkZSRGJUUllNSEpoY0hoc1F5MVL/////DwMBZghiaXJ0aGRheQEAAWYGZ2VuZGVyAAABZgRuYW1lAAA=" } ## -------------再次發回cursor： POST /_xpack/sql?format=json { "cursor": "k4bwAgFz5AFEbkYxWlhKNVZHaGxia1psZEdOb0JRQUFBQUFBQUY3WEZrbGtNa3R5V2s1VVZFTnRORmd3Y21Gd2VHeERMVkVBQUFBQUFBQmUyeFpKWkRKTGNscE9WRlJEYlRSWU1ISmhjSGhzUXkxUkFBQUFBQUFBWHRnV1NXUXlTM0phVGxSVVEyMDBXREJ5WVhCNGJFTXRVUUFBQUFBQUFGN1pGa2xrTWt0eVdrNVVWRU50TkZnd2NtRndlR3hETFZFQUFBQUFBQUJlMmhaSlpESkxjbHBPVkZSRGJUUllNSEpoY0hoc1F5MVL/////DwMBZghiaXJ0aGRheQEAAWYGZ2VuZGVyAAABZgRuYW1lAAA=" } #結果----------------》 { "rows": [] } #接收到最后一頁時，清空es狀態，沒有cursor #要提前清理狀態，可以使用 clear cursor POST _xpack/sql/close { "cursor": "k4bwAgFz5AFEbkYxWlhKNVZHaGxia1psZEdOb0JRQUFBQUFBQUY3NkZrbGtNa3R5V2s1VVZFTnRORmd3Y21Gd2VHeERMVkVBQUFBQUFBQmVfaFpKWkRKTGNscE9WRlJEYlRSWU1ISmhjSGhzUXkxUkFBQUFBQUFBWHZzV1NXUXlTM0phVGxSVVEyMDBXREJ5WVhCNGJFTXRVUUFBQUFBQUFGNzhGa2xrTWt0eVdrNVVWRU50TkZnd2NtRndlR3hETFZFQUFBQUFBQUJlX1JaSlpESkxjbHBPVkZSRGJUUllNSEpoY0hoc1F5MVL/////DwMBZghiaXJ0aGRheQEAAWYGZ2VuZGVyAAABZgRuYW1lAAA=" } #結果——-----------------> { "succeeded": true }
通過filter參數可以指定es的Query DSL來過濾
POST _xpack/sql?format=txt { "query":"select * from my_index order by birthday desc", "filter":{ "term": { "name": "kangkang" } }, "fetch_size":1 }
# 除了query和cursor字段外 請求還可以包括fetch_size和time_zone
# fetch_size 每頁返回多少個結果
# time_zone 日期函數和日期解析的時區，默認為utc
SQL Translate API
sql translate api接受json文檔中的sql並將其轉換為es查詢。
POST _xpack/sql/translate { "query":"select * from my_index order by birthday", "fetch_size":3 } #結果-----------------> { "size": 3, "_source": { "includes": [ "gender", "name" ], "excludes": [] }, "docvalue_fields": [ "birthday" ], "sort": [ { "birthday": { "order": "asc" } } ] }
SQL CLI
可以用命令行形式，在x-pack的bin目錄執行查詢語句：
# ./elasticsearch-sql-cli
sql> select * from my_index where birthday<'1999-01-01';
birthday | gender | name
------------------------+---------------+---------------
1998-04-01T00:00:00.000Z|male |kangkang
1995-02-07T00:00:00.000Z|female |jane
SQL JDBC
將jdbc調用轉化為es sql
SQL 語句
- describe table
# DESC table # DESCRIBE table POST _xpack/sql?format=txt { "query":"describe my_index" } ----------> column | type ---------------+--------------- birthday |TIMESTAMP gender |VARCHAR gender.keyword |VARCHAR name |VARCHAR name.keyword |VARCHAR
- select
# SELECT select_expr [, ...] [ FROM table_name ] [ WHERE condition ] [ GROUP BY grouping_element [, ...] ] [ HAVING condition] [ ORDER BY expression [ ASC | DESC ] [, ...] ] [ LIMIT [ count ] ]
- show columns
#SHOW COLUMNS [ FROM | IN ] ? table POST _xpack/sql?format=txt { "query":"show columns in my_index" }
column | type
---------------+---------------
birthday |TIMESTAMP
gender |VARCHAR
gender.keyword |VARCHAR
name |VARCHAR
name.keyword |VARCHAR - show functions
#SHOW FUNCTIONS [ LIKE? pattern? ]? POST _xpack/sql?format=txt { "query":"show functions like 'sum%'" } name | type ---------------+--------------- SUM |AGGREGATE SUM_OF_SQUARES |AGGREGATE
- show tables
# SHOW TABLES [ LIKE? pattern? ]? POST _xpack/sql?format=txt { "query":"show tables like 'my_index'" } #------------------------> name | type ---------------+--------------- my_index |BASE TABLE
functions and operators
- 比較運算符： = , < , <= , > , >=, 不等於 <> != <=> ， between，is null/is not null
- 邏輯運算符： AND ,OR ,NOT
- 數字運算符： + - * / %
POST _xpack/sql?format=txt { "query":"select 1+1 as x" } ----------> x --------------- 2
- 數學函數： abs(絕對值), crbt(立方根)，round(四舍五入)....
POST _xpack/sql?format=txt { "query":"select abs(age) from test_index " } ---------> ABS(age) --------------- 27
- 時間和日期函數： year, month, week, doy, dow, hour ,minute_of_day, minute,second，extract
POST _xpack/sql?format=txt { "query":"select year(cast('2018-07-12' as timestamp )) as year" } #從日期中提取年份 -------> year --------------- 2018
- 聚合： avg , count , count(distinct) , max , min , sum
POST _xpack/sql/?format=txt { "query":"select avg(age) as avg from test_index" } POST _xpack/sql?format=txt { "query":"select count(distinct age) as count from test_index" } #不同值的個數