jq: The JSON CLI tool

Thanks to lodash, manipulating JSON from JavaScript is easy. Sometimes we’re working with chunks of JSON and it’s easier to deal with it from the shell. Scenarios like this include digging through log output and that (sometimes painful) time when you are first interacting with a new API.

Working with JSON in the shell is quicker and easier than trying to create a temporary file in your project or IDE, then writing some JavaScript code to open that file, parse its contents and scan for whatever it is you are looking for. Those of you that are skilled with the Node.js shell may disagree, but I’m out to optimize your workflow too!

Hello jq

jq is a CLI (command line interface) for working with JSON. It provides shell friendly features (piping, output redirection, etc) that make it feel as if it’s part of your operating system.

To get started, install jq (on OSX)

brew install jq

Or install it directly from the site

Running the jq command by itself will provide some usage info, but not a ton. You’re going to want to head to the online manual, or the man page (man jq).

Let’s see what jq can do!

jq is normally invoked by “piping” JSON data to it (using the | operator). This can be as simple as an echo command. Here we’ll use the simplest filter ., “which copies jq’s input to its output”.

echo '{"name": "Brian"}' | jq '.'
  "name": "Brian"

By default, jq will pretty print and colorize its output. Neat!

Because it works with pipes, you can also use commands like curl to get an API response, then pass it through jq like so:

curl https://swapi.co/api/people/1/ | jq '.'
# Run this yourself to see output, it's too long to post here

As you can see this can be very useful when you need a quick sanity check of an API response structure, or if you forget if the key name is eyeColor or eye_color.

jq does more than pretty formatting! It let’s you slice & dice the JSON in many ways. For instance, let’s ask jq to return Luke’s eye_color

curl https://swapi.co/api/people/1/ | jq '.eye_color'


Thankfully, similar to lodash.get, if we ask for a path that doesn’t exist, it doesn’t die a horrible death

curl https://swapi.co/api/people/ | jq '.does.not.exist'

Let’s get more advanced

What if we wanted to get the eye_color of all the people (on the first page of results)? Again, jq provides a syntax similar to lodash.get:

curl https://swapi.co/api/people/ | jq '.results[].eye_color'

OK, let’s see if we can make it work a little harder. Let’s get the name of all the people (on the first page of results) that have and eye_color of blue?

curl https://swapi.co/api/people/ | jq '.results[] | select(.eye_color == "blue") .name'
"Luke Skywalker"
"Owen Lars"
"Beru Whitesun lars"

Sweet! This is where the power of jq really shines. There are many more powerful filters, operators and functions I suggest you explore. Hopefully I was able to whet your appetite as this is just the tip of the iceberg of things you can do with jq.

One more

One last quick tip - this is the one I use for making sense of my JSON log messages all the time. If you are in OSX, you have the ability to access the clipboard from the shell using pbcopy and pbpaste. So when I am working with JSON logs, from an application like Splunk, I copy the relevant JSON structure to the clipboard, then run it through jq like this:

pbpaste | jq '.'

The pretty printing alone is super useful. On top of that, you have all the great filtering commands to coerce an intimidating log message into a more approachable piece of data!

Special thanks to https://swapi.co/ for providing a fun API to play with!

Working with JSON in JavaScript

As software developers, we spend a lot of time parsing JSON, like, A LOT. There are many tools and libraries available to make our development lives easier. This is the first of a 3 part series where I’ll dig a bit into tools and libraries that I use every day.

Enter lodash

lodash is a phenomenal JavaScript library that provides incredibly useful functions, most of which should exist in the standard library. So what can it do for JSON? Let’s take a look at the get, set and has functions.

First up, lodash.get is the standard for picking through JSON data, or, in fact, any JavaScript object. It allows for easy and safe traversal of nested objects and arrays. get is “safe”, meaning you won’t get the dreaded Cannot read property 'whatever' of undefined that can occur when some nested properties are missing. In this case, get will simply return undefined.

Let’s look at some examples:

const object = {
  "blog": {
    "name": "The Brian Olore Story",
    "articles": [
        "title": "First Post",
        "published": true,
        "tags": ["new", "blog", "post"]
        "title": "Why you should TDD",
        "published": false,
        "tags": ["tdd", "dev"]

// simple path traversal
_.get(object, 'blog.name');
// => 'The Brian Olore Story'

// traverse into arrays
_.get(object, 'blog.articles[1].tags');
// => ["tdd", "dev"]

// path doesn't exist
_.get(object, 'blog.name.rating');
// => undefined

// provide a 3rd parameter to be returned if any part of the path is undefined
_.get(object, 'blog.rating', 100); 
// => 100

Next, let’s take a look at lodash.set. set works the same way as get, but will modify the object passed in. The 3rd parameter is a value to insert/update into the given object. Similar to get, it’s “safe”, meaning: if a portion of the path doesn’t exist, it will be created. Remember: set mutates the object passed in.

// change the blog name
_.set(object, 'blog.name', 'My Story');

// set the rating (that didn't previously exist)
_.set(object, 'blog.rating', 88);

// create a whole new path
_.set(object, 'blog.followers.count', 21);

Finally, while slightly less used and often overlooked, lodash.has returns a boolean if the path provided exists.

// has a name?
_.has(object, 'blog.name');
// => true

// has a rating? (that doesn't exist)
_.has(object, 'blog.rating');
// => false

// has a deeply nested value (that doesn't exist)
_.has(object, 'blog.followers[0].name');
// => false

Another great thing about lodash is that you can choose to bring in all lodash functions, or just individual functions.

const _ = require('lodash');  // all lodash functions (npm install lodash)

const _get = require('lodash.get');  // just the get function (npm install lodash.get)

What tools do you use to traverse the gobs of JSON data we deal with every day?