寫在前面：kestrel當前發展很快，官方文檔更新也不及時，比如這個：https://opencybersecurityalliance.org/posts/kestrel-2021-07-26/，巨坑！用最新版本的kestrel，里面的語法都出現解析錯誤，你說尷尬不。。。沒辦法，自己看源碼搞吧，源碼測試里僅僅有單元測試的，沒有端到端的測試，只能去看底層源碼揣摩使用方法。。。總之，我最終修改了elastic_ecs模塊下stix_transmission/api_client.py的源碼才搞定。
本文目標：kestrel對接elasticsearch，最終能夠使用kestrel查詢出ES的數據來。最終效果圖：
kestrel查詢腳本：
browsers = GET process FROM stixshifter://host110 WHERE [process:name IN ('firefox.exe', 'chrome.exe')] START t'2021-01-01T00:00:00Z' STOP t'2021-12-31T00:00:00Z'
DISP browsers ATTR name, pid
輸出：
name pid chrome.exe 12132112121 firefox.exe 121321 firefox.exe 9121321
ES數據源，主要寫入了幾條：
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110/_doc/1 -d '{
"process": {
"name": "firefox.exe",
"content": "I hava a friend who loves smile, gymbo is his name",
"pid": "121321"},
"@timestamp": "2021-11-02T14:44:23.453+0000"
}'
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110/_doc/2 -d '{
"process": {
"name": "chrome.exe",
"content": "I hava a friend who loves smile, gymbo is his name",
"pid": "12132112121"},
"@timestamp": "2021-11-02T14:44:23.453+0000"
}'
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110/_doc/3 -d '{
"process": {
"name": "twitter.exe",
"content": "I hava a friend who loves smile, gymbo is his name",
"pid": "1213211242123"},
"@timestamp": "2021-11-02T14:44:23.453+0000"
}'
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110/_doc/4 -d '{
"process": {
"name": "firefox.exe",
"content": "I hava a friend who loves smile, gymbo is his name",
"pid": "9121321"},
"@timestamp": "2021-11-02T14:44:23.453+0000"
}'
在已經安裝好了kestrel的前提下，操作如下：
第一步，設置ES的api_key，可以參考：https://blog.csdn.net/UbuntuTouch/article/details/107181440，我的ES配置：
node.name: node-1 network.host: 0.0.0.0 http.port: 9200 cluster.initial_master_nodes: ["node-1"] xpack.security.enabled: true xpack.security.authc.api_key.enabled: true xpack.security.http.ssl.enabled: true xpack.security.http.ssl.keystore.path: /home/es_user/elasticsearch-7.15.1/config/elastic-stack-ca.p12 xpack.security.http.ssl.truststore.path: /home/es_user/elasticsearch-7.15.1/config/elastic-stack-ca.p12 xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.keystore.path: /home/es_user/elasticsearch-7.15.1/config/elastic-stack-ca.p12 xpack.security.transport.ssl.truststore.path: /home/es_user/elasticsearch-7.15.1/config/elastic-stack-ca.p12 http.cors.enabled: true http.cors.allow-origin: "*"
然后就是生成api_key了。
第二步：安裝stix_shifter好以后，設置connector環境變量。
pip install stix-shifter-modules-elastic_ecs
export STIXSHIFTER_HOST110_CONFIG='{"auth":{"id":"ZT3QznwBhSK3ri59dnDv", "api_key":"oY8lmKTpTOOXxNqcwJuiqA"}}'
export STIXSHIFTER_HOST110_CONNECTION='{"host":"localhost", "port":9200, "indices":"host110"}'
export STIXSHIFTER_HOST110_CONNECTOR=elastic_ecs
補充說下stix_shifter的用途，本質上是將kestrel lang對ES數據的查詢語句轉換為ES的查詢語法。
例如，我的腳本中：
GET process FROM stixshifter://host110 WHERE [process:name IN ('firefox.exe', 'chrome.exe')] START t'2021-01-01T00:00:00Z' STOP t'2021-12-31T00:00:00Z'
這條語句會被stix_shifter轉換為如下ES查詢：
{'Content-Type': 'application/json'} search data==> {'_source': {'includes': ['@timestamp', 'source.*', 'destination.*', 'event.*', 'client.*', 'server.*', 'host.*', 'network.*', 'process.*', 'user.*', 'file.*', 'url.*', 'registry.*', 'dns.*']}, 'query': {'query_string': {'query': '(process.pid : ("12132112121" OR "9121321" OR "121321") OR process.ppid : ("12132112121" OR "9121321" OR "121321") OR process.parent.pid : ("12132112121" OR "9121321" OR "121321") OR process.parent.ppid : ("12132112121" OR "9121321" OR "121321")) AND (@timestamp:["2021-01-01T00:00:00.000Z" TO "2021-12-31T00:00:00.000Z"])'}}}
當然要通過追蹤kestrel源碼追蹤分析才知道。
第三步：kestrel在對接ES自簽名證書的時候，有bug會一致報這樣錯誤，真是蛋疼啊，我是通過源碼分析才發現的。
錯誤如下：
File "/root/bone/huntingspace/lib/python3.8/site-packages/kestrel_datasource_stixshifter/interface.py", line 151, in query
raise DataSourceError(
kestrel.exceptions.DataSourceError: [ERROR] DataSourceError: data source internal error: STIX-shifter transmission.results() failed. please test data source manually.
然后你深入源碼分析才知道是這個錯誤：
"Wrong certificate: HTTPSConnectionPool(host='localhost', port=9200): Max retries exceeded with url: /_cluster/health?pretty (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1131)')))", 'code': 'authentication_fail'}
就是因為自簽名證書的問題，解決方式如下：
修改源碼：site-packages/stix_shifter_modules/elastic_ecs/stix_transmission/api_client.py
42 self.client = RestApiClient(connection.get('host'),
43 connection.get('port'),
44 headers,
45 url_modifier_function=url_modifier_function,
46 cert_verify= connection.get('selfSignedCert', True),
47 sni=connection.get('sni', None)
48 )
將46行的True修改為False就好了！
第四步，ES寫入mapping和數據，一個bash腳本：
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110?pretty
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110?pretty -d '{"mappings": {
"properties": {
"process": {
"type": "object",
"properties": {
"@timestamp": {"type": "date"},
"name": {"type": "text"},
"pid": {"type": "text"},
"content": {"type": "text"}
}
}}}}'
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110/_doc/1 -d '{
"process": {
"name": "firefox.exe",
"content": "I hava a friend who loves smile, gymbo is his name",
"pid": "121321"},
"@timestamp": "2021-11-02T14:44:23.453+0000"
}'
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110/_doc/2 -d '{
"process": {
"name": "chrome.exe",
"content": "I hava a friend who loves smile, gymbo is his name",
"pid": "12132112121"},
"@timestamp": "2021-11-02T14:44:23.453+0000"
}'
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110/_doc/3 -d '{
"process": {
"name": "twitter.exe",
"content": "I hava a friend who loves smile, gymbo is his name",
"pid": "1213211242123"},
"@timestamp": "2021-11-02T14:44:23.453+0000"
}'
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110/_doc/4 -d '{
"process": {
"name": "firefox.exe",
"content": "I hava a friend who loves smile, gymbo is his name",
"pid": "9121321"},
"@timestamp": "2021-11-02T14:44:23.453+0000"
}'
第5步： 使用最初的kestrel腳本運行即可出現目標結果了。