Category: jq

jq

jqWhat is jq?

jq can transform JSON in various ways, by selecting, iterating, reducing and otherwise mangling JSON documents.

 

Basically, jq is your sed for json :D.

 

Can be found at https://stedolan.github.io/jq/

 

I randomly discovered jq recently and it seems to be quite the program I hoped it would be, in terms of helping me with json.

Ok, enough of the humanly talks, let’s get do something practical.

[note, I was in a rush so geshi is a bit screwed up, I selected most of these source as python instead of json or bash, bear with me :p]

Suppose i have a json file called output.json:

 

s3lv3n-mc-burger:jqtest selven$ cat output.json 
{"total":4,"notifications":[{"severity":"urgent","details":{"indices":69,"index_ranges":68},"type":"index_ranges_recalculation","timestamp":"2015-09-25T07:50:30.357Z","node_id":"testnode"},{"severity":"normal","type":"es_cluster_green","timestamp":"2015-09-25T06:45:01.933Z","node_id":"testnode"},{"severity":"urgent","type":"journal_uncommitted_messages_deleted","timestamp":"2015-09-24T16:28:29.122Z","node_id":"testnode"},{"severity":"urgent","details":{"journal_utilization_percentage":96.0},"type":"journal_utilization_too_high","timestamp":"2015-09-24T16:26:04.121Z","node_id":"testnode"}]}

 

s3lv3n-mc-burger:jqtest selven$ cat output.json | jq .
{
"total": 4,
"notifications": [
{
"severity": "urgent",
"details": {
"indices": 69,
"index_ranges": 68
},
"type": "index_ranges_recalculation",
"timestamp": "2015-09-25T07:50:30.357Z",
"node_id": "testnode"
},
{
"severity": "normal",
"type": "es_cluster_green",
"timestamp": "2015-09-25T06:45:01.933Z",
"node_id": "testnode"
},
{
"severity": "urgent",
"type": "journal_uncommitted_messages_deleted",
"timestamp": "2015-09-24T16:28:29.122Z",
"node_id": "testnode"
},
{
"severity": "urgent",
"details": {
"journal_utilization_percentage": 96
},
"type": "journal_utilization_too_high",
"timestamp": "2015-09-24T16:26:04.121Z",
"node_id": "testnode"
}
]
}

🙂 Yup, you now have a prettified JSON output, but hey, you are not here just for that.

 

How many notifications do i have in that JSON

s3lv3n-mc-burger:jqtest selven$ cat output.json | jq .total
4

 

awesome!

How about the contents of notifications?

s3lv3n-mc-burger:jqtest selven$ cat output.json | jq '.notifications '
[
  {
    "severity": "urgent",
    "details": {
      "indices": 69,
      "index_ranges": 68
    },
    "type": "index_ranges_recalculation",
    "timestamp": "2015-09-25T07:50:30.357Z",
    "node_id": "testnode"
  },
  {
    "severity": "green",
    "type": "es_cluster_green",
    "timestamp": "2015-09-25T06:45:01.933Z",
    "node_id": "testnode"
  },
  {
    "severity": "urgent",
    "type": "journal_uncommitted_messages_deleted",
    "timestamp": "2015-09-24T16:28:29.122Z",
    "node_id": "testnode"
  },
  {
    "severity": "urgent",
    "details": {
      "journal_utilization_percentage": 96
    },
    "type": "journal_utilization_too_high",
    "timestamp": "2015-09-24T16:26:04.121Z",
    "node_id": "testnode"
  }
]

 

Now what if we want to get only notifications which are urgent?

s3lv3n-mc-burger:jqtest selven$ cat output.json | jq '.notifications| .[]|select(.severity == "urgent")'
{
  "severity": "urgent",
  "details": {
    "indices": 69,
    "index_ranges": 68
  },
  "type": "index_ranges_recalculation",
  "timestamp": "2015-09-25T07:50:30.357Z",
  "node_id": "testnode"
}
{
  "severity": "urgent",
  "type": "journal_uncommitted_messages_deleted",
  "timestamp": "2015-09-24T16:28:29.122Z",
  "node_id": "testnode"
}
{
  "severity": "urgent",
  "details": {
    "journal_utilization_percentage": 96
  },
  "type": "journal_utilization_too_high",
  "timestamp": "2015-09-24T16:26:04.121Z",
  "node_id": "testnode"
}

 

Let’s access them with indexes

s3lv3n-mc-burger:jqtest selven$ cat output.json | jq '.notifications| .[0]'
{
  "severity": "urgent",
  "details": {
    "indices": 69,
    "index_ranges": 68
  },
  "type": "index_ranges_recalculation",
  "timestamp": "2015-09-25T07:50:30.357Z",
  "node_id": "testnode"
}
s3lv3n-mc-burger:jqtest selven$ cat output.json | jq '.notifications| .[1]'
{
  "severity": "green",
  "type": "es_cluster_green",
  "timestamp": "2015-09-25T06:45:01.933Z",
  "node_id": "testnode"
}
s3lv3n-mc-burger:jqtest selven$ cat output.json | jq '.notifications| .[2]'
{
  "severity": "urgent",
  "type": "journal_uncommitted_messages_deleted",
  "timestamp": "2015-09-24T16:28:29.122Z",
  "node_id": "testnode"
}
s3lv3n-mc-burger:jqtest selven$ cat output.json | jq '.notifications| .[4]'
null
s3lv3n-mc-burger:jqtest selven$ cat output.json | jq '.notifications| .[3]'
{
  "severity": "urgent",
  "details": {
    "journal_utilization_percentage": 96
  },
  "type": "journal_utilization_too_high",
  "timestamp": "2015-09-24T16:26:04.121Z",
  "node_id": "testnode"
}

 

How about getting the journal utilization percentage from the third notification?

 

s3lv3n-mc-burger:jqtest selven$ cat output.json | jq '.notifications| .[3].details.journal_utilization_percentage'
96

 

Note that the above can also be done this way

cat output.json | jq '.notifications[3].details.journal_utilization_percentage'

 

 

 

And here we end it with a beautiful one liner:

s3lv3n-mc-burger:jqtest selven$ all_json=`cat output.json` && notifs=`echo $all_json| jq .total` && urgencies=`echo $all_json | jq '.notifications | .[]| select(.severity == "urgent")| .type'`; if  [ $? -eq 0 ]; then clear; echo "Number of notification: $notifs"; echo " With pressing matters as follows: "; echo $urgencies; fi

 

Output is:

Number of notification: 4
With pressing matters as follows:
“index_ranges_recalculation” “journal_uncommitted_messages_deleted” “journal_utilization_too_high”

 

If you want to get only the keys

 

s3lv3n-mc-burger:jqtest selven$ cat output.json| jq 'keys'

Now isn’t that awesome 😀

Note that I have written this in 15 minutes, I just decided to learn jq without the tutorial, and was too much in a hurry to start firing up with jq :p.

 

Special Characters can fuck your jq filter

s3lv3n-mc-burger:jq selven$ cat test
{
"SyIf43UmQI6xC-bibitBgVkQ": {
        "valuea": 12,
        "valueb": 13
    },
"blah": 14,
"withoutminus": {
        "valuec": 15, 
        "valued": 16
    }
}
s3lv3n-mc-burger:jq selven$ cat test|jq '.SyIf43UmQI6xC-bibitBgVkQ'
jq: error: eCuBgVkQ/0 is not defined at , line 1:
.SyIf43UmQI6xC-bibitBgVkQ               
jq: 1 compile error
s3lv3n-mc-burger:jq selven$ var="SyIf43UmQI6xC-bibitBgVkQ";cat test|jq '.["'$var'"]'
{
  "valuea": 12,
  "valueb": 13
}
 
s3lv3n-mc-burger:jq selven$ Holy shit it works!

 

Happy jq discovery. There are several better ways to do the above, I have just :p been brutal above.