作者：Syn良子 出處：https://www.cnblogs.com/cssdongl/p/9715735.html 轉載請注明出處
Druid的數據采集格式
Druid可以采集非標准化的數據諸如JSON,CSV或者以某種分隔符隔開的TSV格式,當然還支持自定義格式.雖然大部分的文檔使用JSON格式,但是通過druid來配置支持其他的限定格式也不是很難.
當前支持的格式化數據
- 列表項
JSON
{"timestamp": "2013-08-31T01:02:33Z", "page": "Gypsy Danger", "language" : "en", "user" : "nuclear", "unpatrolled" : "true", "newPage" : "true", "robot": "false", "anonymous": "false", "namespace":"article", "continent":"North America", "country":"United States", "region":"Bay Area", "city":"San Francisco", "added": 57, "deleted": 200, "delta": -143}
CSV
2013-08-31T01:02:33Z,"Gypsy Danger","en","nuclear","true","true","false","false","article","North America","United States","Bay Area","San Francisco",57,200,-143
TSV
2013-08-31T01:02:33Z "Gypsy Danger" "en" "nuclear" "true" "true" "false" "false" "article" "North America" "United States" "Bay Area" "San Francisco" 57 200 -143
需要注意的是CSV,TSV不能包含列頭,這點在數據采集的時候一定要注意
自定義格式
Druid支持使用正則解析和JavaScript來自定義數據格式.但是這種方式並沒有自己實現的Java解析器或者額外的流式處理工具效率更高.
配置數據采集的schema
什么是data schema?其實就是Druid的index數據攝取任務需要的數據源的描述的元數據.它主要描述要采集的數據類型,數據由哪些列構成,哪些是指標列,哪些是維度列,時間的粒度等.
以CSV格式舉例
"parseSpec": {
"format" : "csv",
"timestampSpec" : {
"column" : "timestamp"
},
"columns" : ["timestamp","page","language","user","unpatrolled","newPage","robot","anonymous","namespace","continent","country","region","city","added","deleted","delta"],
"dimensionsSpec" : {
"dimensions" : ["page","language","user","unpatrolled","newPage","robot","anonymous","namespace","continent","country","region","city"]
}}
parseSpec指明了數據源格式,這里是format中表明是CSV格式,然后說明時間戳字段名是timestamp,數據字段名是columns里面那一堆,dimensionsSpec則代表哪些字段可以作為維度.
參考資料:Druid的數據格式