Using jq to Consume JSON in the Shell This article fs a tutorial on using jq as a JSON parser and fetching information about the weather from different cities, ‘Web APIs. Ifyou try to find the API documentation of popular service, chances are that the API will respond in JSON format. Many mainstream languages even have JSON parsers built in, But when it comes to shell scripting, there is no inbuilt JSON parser, and the only hacker way of Processing JSON is with a combination of awk and sed, which are very painful to use, ‘There are many JSON parsers apart from jq but, inthis article, we will focus only on this option, J SON has become the most prevalent way of consuming, Installation Jaisa single binary program with no dependencies, so installation is as simple as downloading the binary from huups://stedolan.github.iovje, copying the binary in /bin or /usr/bin and setting permissions. Many Linux distributions provide jq in the repositories, so installing jq is as easy as using the following commands: sudo apt install jq 0: sudo pacnan -S jq Installation instructions may vary depending upon the distribution. Detailed instructions are available at htips:// stedolan.github.io/jq/download/, Usage For this demonstration, version 1.5 of jq was used. All the code examples are available at https://github.com/ Jatindhankhar/jq-tutorial jq can be used in conjunction With other tools like cat and cut, by piping, or be used to directly read from the file, although the former is more popular in practice. When working with jq, two fantastic resources can be used. The first one is the documentation at ‘utps://stedolan.github.io/jq/manual, and the second is the Online Playground (hetps://iqplay.org/) where one can play with jg and even share the snippets. Throughout this article, we will use different API endpoints of the MetaWeather API (htps://www. metaweather:con/api). The simplest use of jq is to pretty format JSON data. Let's fetch the list of cities that contain the word ‘new? in them, and then use this information to further fetch details of particular city, as follows: Curl -sS https://ww.netaweather.coa/api/Location/ search/2query=new ‘The above command will fetch all cities containing ‘new? {in their name. At this point, the output is not formatted. [{"title": "New York", "location_type":"city","woeid":24 50115, "Iatt_long”:"40.71456, -74,007128"}, (*title": "Wow Delhi", “location_type": "city", "woeid”:28743736, "latt_long" {6 | DECEMBER 2017 | OPEN SOURCE FOR YOU | www OpensoutceFot.com‘The screenshot shown in Figure 1 compares the output of both commands Now that we have some data to work upon, we can use jg to filter the keys. The simplest filter available is ‘." which does nothing and filters the whole document as it is. Filters are passed to jq in single quotes. By looking at the output, we can see that all the objects are trapped inside a JSON array. To filter out the array, we use [J , which will display all items inside an array. To target a specific item by index, we place the index number inside (0}. ‘To display the fist item, use the following code: How To i To display only the available cities, we add another filter, ‘which is the key name itself (in our case, .title). We can combine multiple filters using the | (pipe) operator. Here we combine the] filter with tite in this way: | title. For simple queries, we can avoid the | operator and rewrite itas.[] tite, but we will use the | operator to combine queries, But what if we want to display multiple keys together? Just separate them by 4. Now, let's display the city along with its ID (woetd): Figure 1: Output comparison ‘wave OpenSourceForU.com | OPEN SOURCEFORYOU | DECEMBER 2017 67 aa!Figure 2: Basic filters ‘The output looks good, but what if we format the output and prin it on a single line? For that we can use string interpolation. To use keys inside a string pattern, we use backslash and parentheses so that they are not executed In our case, JSON is small, bt if it is too big and we need to filter it based on a key value (like display the information for New Delhi), jq provides the select keyword for that operation. ‘Now that we have the Where on Earth ID (wwoeid) for New Delhi, we can retrieve more information about New Delhi using the endpoint https://www.metaweather.com/api/ location/woeidd. 668 | DECEMBER 2017 | OPEN SOURCE-FOR YOU The JSON structure for this endpoint looks like what's shown in Figure 3. Consolidated_weather contains an array of JSON objects ‘with weather information, and the sources key contains an array of JSON objects from which particular weather information was fetched, This time, let’s store JSON in a file named weather, {son instead of ditectly piping data. This will help us opeaetane 6 0° 33 aa 33 Figure 3: JSON structure wv. OperSouceFarcomavoid making an API call every time we want to perform an ‘operation and, instead, we can use the saved JSON. on/28043736/ > weather Now we can use jq in the format jq ‘ters’ weatherjson and we can also load filters from a fle using the -f parameter. The command is jq-ffilters.txt weatherjson, but we can just load the JSON file and pass filters in the command line. Lets list the weather followed by the source name. Since both sources and consolidated weather is of the same length (get the length using the length filter), we can use range to generate an index and use string interpolation. There is transpose and map inbuilt as well. Covering all of them won't be possible in a single article. Jq ‘range(9;({-sourcesf]] | length) as si | (consolidated weather [i (-sourees[si] ate 28¢ pred het office Opentieathervap predicts Clea World Weather online Yahoo cl ‘There are so many functions and filters but we will use sort_by and date functions, and end this article by printing the forecast for each day in ascending order. This function takes value via the Pipe def fornat_date(x) ptine(“*Y 1) operator a-Si") | mktine | strftine("%a - xa, def print 1 1 Location: \(. tithe Coordinates \(,latt_tong def print. data (fortat_date( applicable date) \t\t I Mumidity : .\( hunidity)\t\e | Weather state: \(,weather_state_nane)\t\t\t def process. weather data: | sort_by(-applicable date){] | print-data; ‘av OperSourceFarUcom Figure 4: Final output as $root | print location, process weather data) (consolidated weather | Save the above code as filtertt sort_by sons the value by data. format_date takes dates as Parameters and extracts short day names, dates and months. pprint_location and print_data do not take any parameter, and can be applied after the pipe operator; and the default Parameter for a parameterless function will be ‘." jg -f filter.txt weather. json -r -F will return a raw string, The output is shown in Figure 4 hope this article has given you an overview of all that jq ‘can achieve, If you ate looking for a tool that is easy to use in shell scripts, can help you out; so give ita tr. TOMO} Gecmiess 1] httosy/stedolan github jo/jamanual/v1.5/ [2] nips//sithub.comjatinahankharig-tutoral [2] htips:/mw.metaveathercom/api/ [al hups:/aplay.ora/ Seb The author loves working with modem CH, Ruby, JavaScript ‘and Haskell. He can be reached at jatin@jatinahankhari. OPEN SOURCE FOR YOU | DECEMBER 2017 | 69 7A eee ll Le ll