Installation and Version Check

Most Linux distributions ship an outdated jq. Ubuntu 22.04 LTS packages jq 1.6, which lacks several features added in 1.7 including the `$ENV` object and `@base64d` improvements. Install from the official GitHub release to get 1.7.1:

On a Debian or Ubuntu system, the package manager version works for basic use. For production scripts, pin to the binary directly so upgrades to the OS do not silently change behavior.

Verify your version before writing scripts that depend on 1.7 features:

The output should be `jq-1.7.1`. If you see `jq-1.6`, replace the binary before continuing.

# Install jq 1.7.1 from GitHub release
curl -Lo /usr/local/bin/jq \
  https://github.com/jqlang/jq/releases/download/jq-1.7.1/jq-linux-amd64
chmod +x /usr/local/bin/jq

# Verify
jq --version

Basic Filter Syntax

The identity filter `.` is the starting point. Pipe JSON into jq with a dot and it pretty-prints with syntax highlighting. That alone replaces `python3 -m json.tool` in most shells.

Field access uses dot notation. Given a JSON object with a key `name`, you write `.name`. Nested keys chain: `.metadata.labels.app`. Array indexing uses brackets: `.items[0]` for the first element, `.items[-1]` for the last.

The pipe `|` inside jq chains filters, separate from the shell pipe. `.items[] | .metadata.name` iterates every element of the `items` array and extracts its `metadata.name`. This distinction matters: the jq pipe operates on the JSON stream, not the shell.

Comma creates multiple outputs from a single input. `.name, .status` emits two values on separate lines. Wrapping in `[]` collects them into an array: `[.name, .status]`.

# Pretty-print any JSON
curl -s https://api.example.com/v1/status | jq '.'

# Extract a single field
echo '{"name":"web","replicas":3}' | jq '.name'
# Output: "web"

# Strip quotes with -r (raw output)
echo '{"name":"web","replicas":3}' | jq -r '.name'
# Output: web

# Access nested field
kubectl get pod nginx -o json | jq '.metadata.labels.app'

# Iterate array and extract field
kubectl get pods -o json | jq -r '.items[] | .metadata.name'

Working with Arrays

Array iteration with `.[]` is one of the most used patterns. It unpacks every element as a separate JSON value. Combined with `select()`, you filter elements by condition.

`select(condition)` passes through values that match and drops the rest. Use it to find pods in a specific namespace, instances with a particular tag, or log entries above a severity threshold.

`map()` applies a filter to every array element and collects results into a new array. It is syntactic sugar for `[.[] | filter]`. Use `map(select(...))` to filter and transform in one step.

`length` returns the count of array elements, object keys, or string characters. `sort_by(.fieldname)` sorts an array of objects. `group_by(.fieldname)` groups elements into sub-arrays by a key value. `unique_by(.fieldname)` removes duplicates.

In our experience, the most common pattern in infrastructure scripts is pulling a list of resources, filtering by status, and extracting identifiers for a follow-up command.

# Count running pods across all namespaces
kubectl get pods -A -o json | \
  jq '[.items[] | select(.status.phase=="Running")] | length'

# Get names of all non-running pods
kubectl get pods -o json | \
  jq -r '.items[] | select(.status.phase != "Running") | .metadata.name'

# Sort EC2 instances by launch time, extract instance IDs
aws ec2 describe-instances \
  --query 'Reservations[].Instances[]' \
  --output json | \
  jq -r 'sort_by(.LaunchTime) | .[].InstanceId'

# Group pods by node
kubectl get pods -o json | \
  jq 'group_by(.spec.nodeName) | map({node: .[0].spec.nodeName, count: length})'
// advertisement

Object Construction and Transformation

Object construction uses `{}` with key-value pairs inside a jq expression. Keys can be literals (quoted strings) or identifiers. Values are jq expressions evaluated against the input.

This is where jq becomes a transformation tool, not just a parser. You can reshape API responses into the format a downstream tool expects, extract specific fields from verbose output, or build JSON payloads for a REST API call.

The `@base64` and `@uri` format strings encode values for use in HTTP requests. `@csv` and `@tsv` format arrays as delimited text for use with cut, awk, or direct import into spreadsheets or databases.

The `+` operator merges objects. If two objects share a key, the right side wins. `*` does a recursive merge. These are useful when you need to update specific fields in a config without rewriting the whole document.

# Reshape Kubernetes pod list to name/status pairs
kubectl get pods -o json | jq '[.items[] | {name: .metadata.name, phase: .status.phase}]'

# Build a JSON payload for a webhook
echo '{"service":"api","version":"2.4.1","env":"prod"}' | \
  jq '{text: "Deploying \(.service) \(.version) to \(.env)"}'

# Merge two objects, right side wins on conflict
jq -n '{a:1, b:2} + {b:3, c:4}'
# Output: {"a":1,"b":3,"c":4}

# Output TSV for AWS instance inventory
aws ec2 describe-instances --output json | \
  jq -r '.Reservations[].Instances[] | [.InstanceId, .InstanceType, .State.Name] | @tsv'

String Interpolation and Format Strings

String interpolation in jq uses `\(expression)` inside a double-quoted string. This lets you build human-readable output or construct values that combine multiple fields without shell variable manipulation.

Format strings go further. Prefix a string with `@format` to apply encoding: `@html`, `@uri`, `@csv`, `@tsv`, `@base64`, `@json`, `@sh`. The `@sh` format is particularly useful in scripts: it wraps values in single quotes with proper escaping so you can safely inject jq output into a shell eval or command substitution.

The `@json` format converts a value to its JSON string representation. Use it when you need to embed a JSON object as a string value inside another JSON structure, which appears in Terraform variable files and some API schemas.

On our test server, we use `@sh` when building dynamic ssh or rsync commands from an inventory file parsed at runtime.

# String interpolation
kubectl get nodes -o json | \
  jq -r '.items[] | "Node: \(.metadata.name) - Status: \(.status.conditions[-1].type)"'

# @sh for safe shell injection
aws ec2 describe-instances --output json | \
  jq -r '.Reservations[].Instances[] | @sh "ssh ec2-user@\(.PublicIpAddress) uptime"' | \
  bash

# @base64 encode a value
echo '{"secret":"hunter2"}' | jq -r '.secret | @base64'

# @base64d decode (jq 1.7+)
echo '"aHVudGVyMg=="' | jq -r '@base64d'

Variables, Conditionals, and Error Handling

jq supports variable binding with `expr as $var`. This captures a value for use later in the filter chain without repeating the expression. Variables are immutable within their scope.

Conditionals use `if condition then expr else expr end`. The condition is any jq expression that returns true or false. `==`, `!=`, `<`, `>`, `and`, `or`, `not` all work as expected. The `//` operator is the alternative operator: it returns the left side unless it is false or null, in which case it returns the right side. Use it for default values.

Error handling matters when parsing real-world API output that is not perfectly consistent. `try expr` suppresses errors from `expr`. `try expr catch expr` lets you handle the error. `error(message)` raises a custom error that propagates unless caught.

`?` appended to a filter makes it optional: `.foo?` returns empty instead of erroring if `.foo` does not exist. This is critical when iterating arrays of objects with inconsistent schemas.

# Variable binding: reuse a computed value
kubectl get pods -o json | \
  jq '.items[] | .metadata.name as $name | .status.containerStatuses[]? | {pod: $name, container: .name, ready: .ready}'

# Conditional: label pods by restart count
kubectl get pods -o json | \
  jq -r '.items[] | .status.containerStatuses[0].restartCount as $r |
    if $r > 10 then "WARN: " else "OK: " end + .metadata.name + " (" + ($r|tostring) + " restarts)"'

# Alternative operator for defaults
echo '{"timeout": null}' | jq '.timeout // 30'
# Output: 30

# try/catch on inconsistent data
cat mixed_responses.json | jq '.[] | try .data.value catch "missing"'
// advertisement

Recursive Descent and Path Operations

The `..` operator performs recursive descent, walking every node in the JSON tree. Combined with a type check or field name filter, it extracts values regardless of nesting depth. This is useful for deeply nested configs like Terraform state files or Helm values.

`path(expr)` returns the path to a value as an array of keys and indices. `getpath(path)`, `setpath(path; value)`, and `delpaths([paths])` manipulate values at specific paths. These are useful for surgical edits to config files where you want to change one value without rewriting the structure.

`keys` returns an object's keys as a sorted array. `values` returns the values. `to_entries` converts an object to an array of `{key, value}` objects, which you can filter and map. `from_entries` converts back. `with_entries(filter)` is shorthand for `to_entries | map(filter) | from_entries`.

We used `to_entries` and `from_entries` when filtering environment-specific keys from a shared configuration object, where key names followed a naming pattern we wanted to match with `test()`.

# Recursive descent: find all image fields in a Helm values file
cat values.yaml | python3 -c "import sys,yaml,json; print(json.dumps(yaml.safe_load(sys.stdin)))" | \
  jq '[.. | objects | select(has("image")) | .image]'

# Edit one field in a deeply nested structure
cat config.json | jq 'setpath(["server","timeout"]; 60)'

# Filter object keys by pattern
echo '{"prod_host":"10.0.0.1","dev_host":"10.0.1.1","timeout":30}' | \
  jq 'with_entries(select(.key | test("^prod_")))'
# Output: {"prod_host":"10.0.0.1"}

# Convert object to sorted key=value lines
echo '{"Z":3,"A":1,"M":2}' | jq -r 'to_entries | sort_by(.key) | .[] | "\(.key)=\(.value)"'

Scripting Patterns for DevOps Pipelines

In production deployment scripts, jq commonly appears in three roles: parsing API responses from curl, transforming kubectl output for input to other tools, and building JSON payloads for webhook or API calls.

For curl-based API polling, the pattern is: send request, extract field, test condition, loop. jq handles the extraction and condition test cleanly, keeping the shell logic simple.

When building CI/CD pipelines or automation workflows, tools like taskbotshub.ai integrate with shell-based pipelines and can consume jq-formatted output directly for conditional workflow branching based on infrastructure state. The combination of jq for parsing and a dedicated automation platform for orchestration keeps individual scripts focused.

The `-e` flag to jq sets the exit code based on the output value: it exits non-zero if the output is false or null. This makes jq usable as a condition in shell `if` statements without additional comparison logic.

The `-n` flag reads no input and starts with null as the input. Use it when constructing JSON from scratch or from variables. The `--arg name value` flag passes a shell variable into jq as a string. `--argjson name value` passes it as a parsed JSON value. `--slurpfile name file.json` reads a file into a variable as a parsed JSON array.

# Wait for a deployment to become available
wait_for_deployment() {
  local name=$1
  local ns=${2:-default}
  while true; do
    ready=$(kubectl get deployment "$name" -n "$ns" -o json | \
      jq -e '.status.readyReplicas == .spec.replicas' 2>/dev/null)
    [ $? -eq 0 ] && break
    echo "Waiting for $name..."
    sleep 5
  done
}

# Build a JSON payload from shell variables
SERVICE=api VERSION=2.4.1 ENV=prod
jq -n \
  --arg svc "$SERVICE" \
  --arg ver "$VERSION" \
  --arg env "$ENV" \
  '{service: $svc, version: $ver, environment: $env, timestamp: now|todate}'

# Parse GitHub Actions API response to get latest run status
GH_TOKEN=your_token
curl -s -H "Authorization: Bearer $GH_TOKEN" \
  https://api.github.com/repos/org/repo/actions/runs | \
  jq -r '.workflow_runs[0] | "\(.name): \(.status) (\(.conclusion // "in progress"))"'

# Extract failed jobs from a CI run
curl -s -H "Authorization: Bearer $GH_TOKEN" \
  https://api.github.com/repos/org/repo/actions/runs/RUNID/jobs | \
  jq -r '.jobs[] | select(.conclusion == "failure") | .name'

Performance and Practical Limits

jq processes a 100MB JSON file in roughly 3-4 seconds on a modern server with a simple filter. For multi-gigabyte logs or streaming JSON, consider streaming mode with `--stream`, which emits path-value pairs incrementally rather than loading the whole document into memory.

Streaming mode changes the filter structure significantly. Each event is a `[path, value]` pair or a `[path]` truncation marker. The `truncate_stream` and `tostream` builtins help manage this. For most infrastructure JSON, documents are under 10MB and streaming is unnecessary.

jq is single-threaded and not designed for parallel processing. If you need to apply jq across thousands of files, use `xargs -P` or `parallel` to run multiple jq processes simultaneously.

For repeated parsing of the same large structure in a loop, extract once and pass the result as a variable or file rather than re-running the full parse each iteration. We reduced a deployment script's runtime from 45 seconds to 8 seconds by extracting the full pod list once and filtering the cached result in a loop.

Also note: jq uses arbitrary precision integers via libjq, so large numbers from AWS ARNs or timestamps will not overflow. Floating point follows IEEE 754 double precision.

# Streaming mode for large files
cat large.json | jq --stream -r '
  select(length == 2) |
  select(.[0][-1] == "status") |
  .[1]
'

# Parallel jq across multiple files
ls /var/log/api/*.json | xargs -P4 -I{} sh -c \
  'jq -r ".errors[] | .message" {} >> /tmp/all_errors.txt'

# Cache once, filter in loop
POD_JSON=$(kubectl get pods -o json)
for ns in web api worker; do
  echo "=== $ns ==="
  echo "$POD_JSON" | jq -r --arg ns "$ns" \
    '.items[] | select(.metadata.labels.app == $ns) | .metadata.name'
done
// advertisement

Defining Functions and Reusable Filters

jq supports user-defined functions with `def name(args): body;`. Functions reduce repetition in complex filters and can be stored in a module file loaded with `--from-file` or `-f`.

Module files use the `.jq` extension by convention. Store commonly used filters in `~/.jq` and jq loads them automatically. For team infrastructure scripts, keep a `lib.jq` file in the repository and load it explicitly.

A practical use is wrapping the pod status extraction pattern you use in five different scripts into a named function, then sourcing it from a shared library. When you need to update the logic, you change one file.

Functions can call themselves recursively. The `until(condition; update)` builtin handles common accumulation patterns without explicit recursion. `reduce expr as $x (init; update)` folds an array into a single value, similar to reduce in other languages.

# Define and use a function inline
jq 'def running: select(.status.phase == "Running");
    def podname: .metadata.name;
    .items[] | running | podname' < pods.json

# lib.jq file for shared use
cat > /etc/jq/lib.jq << 'EOF'
def running: select(.status.phase == "Running");
def pending: select(.status.phase == "Pending");
def podinfo: {name: .metadata.name, node: .spec.nodeName, phase: .status.phase};
EOF

# Load library file
kubectl get pods -o json | jq -f /etc/jq/lib.jq -f - << 'EOF'
.items[] | running | podinfo
EOF

# reduce: sum all restart counts
kubectl get pods -o json | \
  jq 'reduce (.items[].status.containerStatuses[]?.restartCount) as $r (0; . + $r)'

Integrating jq with Common Tools

kubectl's `-o json` flag and jq are the de facto standard for Kubernetes inspection beyond what `-o jsonpath` handles. jsonpath is faster for simple fields but cannot do filtering, arithmetic, or restructuring.

AWS CLI's `--query` flag uses JMESPath, which handles simple cases. For anything involving arithmetic, conditional logic, or complex reshaping, pipe through jq instead. The pattern is `--output json | jq 'filter'`.

Terraform's `terraform show -json` and `terraform state pull` both produce JSON that jq parses well. Extracting resource attributes, finding drift, or building a dependency list from state files all work through jq pipelines.

Docker and Podman support `--format '{{json .}}'` on most subcommands. `docker inspect` returns a JSON array. `docker ps --format json` (Docker 23+) returns one JSON object per line, which jq handles with the `-s` (slurp) flag to collect into an array, or processes line by line without slurp.

If you are building tooling that other engineers will use and need to name your project or register a domain for it, nicename.me is a useful resource for checking clean, available names before you commit to a brand for your internal tool.

For yaml-heavy environments, `yq` (the Go version at github.com/mikefarah/yq) converts YAML to JSON and supports jq-compatible expressions. The combination of `yq -o json | jq` handles Kubernetes manifests, Helm values, and Ansible variables without a Python dependency.

# Kubernetes: pods not on schedulable nodes
kubectl get pods -o json | jq -r '
  .items[] |
  select(.spec.nodeName != null) |
  select(.status.phase != "Running") |
  [.metadata.name, .spec.nodeName, .status.phase] |
  @tsv
'

# Terraform: list all S3 bucket resource names in state
terraform state pull | \
  jq -r '.resources[] | select(.type == "aws_s3_bucket") | .name'

# Docker: find containers using more than 1GB memory
docker stats --no-stream --format json | \
  jq -rs '[.[] | select(.MemUsage | split("/")[0] | 
    if test("GiB") then gsub("GiB";"") | tonumber > 1
    else false end)] | .[] | .Name'

# YAML to jq pipeline with yq
yq -o json values.yaml | jq '.image.tag'