linux 下jq的使用

安裝：
yum install jq -y
文檔：
https://stedolan.github.io/jq/manual/
更多:
https://blog.csdn.net/Cheat1173010256/article/details/118230562

數據源
333.json

[{
    "name": "站長工具",
    "url": "http://tool.chinaz.com",
    "address": {
      "city": "廈門",
      "country": "中國"
    },
    "arrayBrowser": [{
        "name": "Google",
        "url": "http://www.google.com"
      },
      {
        "name": "Baidu",
        "url": "http://www.baidu.com"
      }
    ]
  },
  {
    "name": "站長之家",
    "url": "http://tool.zzhome.com",
    "address": {
      "city": "大連",
      "country": "中國"
    },
    "arrayBrowser": [{
        "name": "360",
        "url": "http://www.so.com"
      },
      {
        "name": "bing",
        "url": "http://www.bing.com"
      }
    ],
    "127.0.0.1": {
      "error": 200,
      "msg": "ok"
    }
  }
]

例子基礎

haima@haima-PC:~/Desktop$ cat 333.json |jq '.'
[
  {
    "name": "站長工具",
    "url": "http://tool.chinaz.com",
    "address": {
      "city": "廈門",
      "country": "中國"
    },
    "arrayBrowser": [
      {
        "name": "Google",
        "url": "http://www.google.com"
      },
      {
        "name": "Baidu",
        "url": "http://www.baidu.com"
      }
    ]
  },
  {
    "name": "站長之家",
    "url": "http://tool.zzhome.com",
    "address": {
      "city": "大連",
      "country": "中國"
    },
    "arrayBrowser": [
      {
        "name": "360",
        "url": "http://www.so.com"
      },
      {
        "name": "bing",
        "url": "http://www.bing.com"
      }
    ],
    "127.0.0.1": {
      "error": 200,
      "msg": "ok"
    }
  }
]

管道線 |
jq支持管道線 |，它如同linux命令中的管道線——把前面命令的輸出當作是后面命令的輸入。如下命令把.[0]作為{…}的輸入，進而訪問嵌套的屬性，如.name和.address.city。
觀察如下幾個命令，通過改變|前后的輸入和輸出來達到不同的效果：

cat json.txt | jq '.[0] | {name:.name,city:.address.city}'

{
  "name": "站長工具",
  "city": "廈門"
}

cat json.txt | jq '.[0] | {name:.arrayBrowser[1].name,city:.address.city}'

{
  "name": "Baidu",
  "city": "廈門"
}

cat json.txt | jq ".[] | {name:.arrayBrowser[1].name,city:.address.city}"

{
  "name": "Baidu",
  "city": "廈門"
}
{
  "name": "bing",
  "city": "大連"
}

[]
如果希望把jq的輸出當作一個數組，可以在前后加上[]：

cat json.txt | jq "[.[] | {name:.arrayBrowser[1].name,city:.address.city}]"
[
  {
    "name": "Baidu",
    "city": "廈門"
  },
  {
    "name": "bing",
    "city": "大連"
  }
]

自定義key
在{}中，冒號前面的名字是映射的名稱，你可以任意修改，如：

```sh
cat json.txt | jq "[.[] | {name_001:.arrayBrowser[1].name,city_002:.address.city}]"

[
  {
    "name_001": "Baidu",
    "city_002": "廈門"
  },
  {
    "name_001": "bing",
    "city_002": "大連"
  }
]

原文鏈接：https://blog.csdn.net/qq_26502245/article/details/100191694

取0號單元下的元素

haima@haima-PC:~/Desktop$ cat 333.json |jq '.[0].name'
"站長工具"
haima@haima-PC:~/Desktop$ cat 333.json |jq '.[0].address'
{
  "city": "廈門",
  "country": "中國"
}
haima@haima-PC:~/Desktop$ cat 333.json |jq '.[0].address.city'
"廈門"
haima@haima-PC:~/Desktop$ cat 333.json |jq '.[0].address["city"]'
"廈門"
haima@haima-PC:~/Desktop$ cat 333.json |jq ".[0].address[\"city\"]"
"廈門"
haima@haima-PC:~/Desktop$ cat 333.json |jq '.[1]["127.0.0.1"]'
{
  "error": 200,
  "msg": "ok"
}
haima@haima-PC:~/Desktop$ cat 333.json | jq ".[0]" | jq 'keys' #獲取0號單元的所有key
[
  "address",
  "arrayBrowser",
  "name",
  "url"
]
haima@haima-PC:~/Desktop$ cat 333.json | jq ".[0].address" # 獲取0號單元的address
{
  "city": "廈門",
  "country": "中國"
}
haima@haima-PC:~/Desktop$ cat 333.json | jq ".[0].address" | jq 'keys' #獲取0號單元的address里的所有 key(嵌套提取)
[
  "city",
  "country"
]
haima@haima-PC:~/Desktop$ cat 333.json | jq ".[0]" | jq 'has("name")' #判斷是否有某個 key
true
haima@haima-PC:~/Desktop$ cat 333.json | jq ".[0]" | jq 'has("test")' #判斷是否有某個 key
false

壓縮json

haima@haima-PC:~/Desktop$ cat 333.json | jq -c . 
[{"name":"站長工具","url":"http://tool.chinaz.com","address":{"city":"廈門","country":"中國"},"arrayBrowser":[{"name":"Google","url":"http://www.google.com"},{"name":"Baidu","url":"http://www.baidu.com"}]},{"name":"站長之家","url":"http://tool.zzhome.com","address":{"city":"大連","country":"中國"},"arrayBrowser":[{"name":"360","url":"http://www.so.com"},{"name":"bing","url":"http://www.bing.com"}],"127.0.0.1":{"error":200,"msg":"ok"}}]

例子二:

數據源 22.json

{"ip":"34.120.75.75"}
{"ip":"35.193.30.131"}
{"ip":"35.227.228.135"}
{"ip":"130.211.21.179"}
{"ip":"35.192.215.111"}
{"ip":"35.214.27.221"}
{"ip":"34.120.136.4"}
{"ip":"35.209.185.188"}
{"ip":"34.121.213.115"}
{"ip":"34.98.116.47"}

取出ip

$ cat 22.json |jq '.ip'
"34.120.75.75"
"35.193.30.131"
"35.227.228.135"
"130.211.21.179"
"35.192.215.111"
"35.214.27.221"
"34.120.136.4"
"35.209.185.188"
"34.121.213.115"
"34.98.116.47"

取出ip並去除雙引號 (需要安裝jq)

$ cat 22.json |jq '.ip' | sed 's/\"//g'
34.120.75.75
35.193.30.131
35.227.228.135
130.211.21.179
35.192.215.111
35.214.27.221
34.120.136.4
35.209.185.188
34.121.213.115
34.98.116.47

排序去重后,取出ip並去除雙引號 (需要安裝jq)

$  cat 22.json |sort | uniq  | sort -n | jq '.ip' | sed 's/\"//g'
34.120.241.13
34.125.140.101
34.126.67.33
34.65.22.11
34.77.137.149
34.85.82.186
34.86.160.16
34.90.193.87
34.92.44.105
34.93.161.137
34.94.248.178

解析：

s 為替換

以g結尾表示的是:全局性,意即”替代文本取代正則表達式中每一個匹配的”

直接改變的話用i：

sed i 's/\"//g'  11.txt

另外輸出的話：

sed 's/\"//g'  aa.txt > 11.clean.txt

例子:
取出port和value ,組裝json格式

{"took":648,"timed_out":false,"_shards":{"total":50,"successful":50,"skipped":0,"failed":0},"hits":{"total":9893731,"max_score":1.0,"hits":[{"_index":"fofapro_service","_type":"order","_id":"1.169.157.54:80","_score":1.0,"_source":{"port":80}}]},"aggregations":{"distinct_ips":{"value":9883243}}}
{"took":648,"timed_out":false,"_shards":{"total":50,"successful":50,"skipped":0,"failed":0},"hits":{"total":9893731,"max_score":1.0,"hits":[{"_index":"fofapro_service","_type":"order","_id":"1.169.157.54:80","_score":1.0,"_source":{"port":81}}]},"aggregations":{"distinct_ips":{"value":9883243}}}

命令:

$ cat es_port.json | jq '{port:.hits.hits[0]._source.port , count:.aggregations.distinct_ips.value}' -c | awk -F'port":' '{print $2}' | awk -F',"count":' '{print "{\"" $1"\":\""$2}' | sed 's/}/"}/g'
{"80":"9883243"}
{"81":"9883243"}

-c ：一行顯示

4 JSON parse array

   cat json_raw.txt | jq '.employees[1].name'
   "Laura"

5 內建函數
jq還有一些內建函數如 key，has
key是用來獲取JSON中的key元素的：

cat json_raw.txt | jq 'keys'
[
  "employees",
  "location",
  "name"
]

has是用來是判斷是否存在某個key：

cat json_raw.txt | jq 'has("name")'
true

cat json_raw.txt | jq 'has("noexisted")'
false