jq JSON Processing

jq JSON 数据处理

Expert knowledge for processing, querying, and transforming JSON data using jq, the lightweight and flexible command-line JSON processor.

本文提供使用jq（轻量灵活的命令行JSON处理器）进行JSON数据处理、查询和转换的专业知识。

Core Expertise

核心能力

JSON Operations

Query and filter JSON with path expressions
Transform JSON structure and shape
Combine, merge, and split JSON data
Validate JSON syntax and structure

Data Extraction

Extract specific fields from JSON objects
Filter arrays based on conditions
Navigate nested JSON structures
Handle null values and missing keys

JSON 操作

使用路径表达式查询和过滤JSON
转换JSON的结构和形态
合并、拆分JSON数据
验证JSON语法和结构

数据提取

从JSON对象中提取特定字段
根据条件过滤数组
遍历嵌套的JSON结构
处理空值和缺失的键

Essential Commands

常用命令

Basic Querying

基础查询

bash

undefined

bash

undefined

Pretty-print JSON

jq '.' file.json

Extract specific field

jq '.fieldName' file.json

Extract nested field

jq '.user.email' file.json

Extract array element

jq '.[0]' file.json jq '.items[2]' file.json

undefined

jq '.[0]' file.json jq '.items[2]' file.json

undefined

Array Operations

数组操作

bash

undefined

bash

undefined

Get array length

jq '.items | length' file.json

Map over array

jq '.items | map(.name)' file.json

Filter array

jq '.items[] | select(.active == true)' file.json jq '.users[] | select(.age > 18)' file.json

Sort array

jq '.items | sort_by(.name)' file.json jq '.items | sort_by(.date) | reverse' file.json

Get first/last elements

jq '.items | first' file.json jq '.items | last' file.json

Unique values

jq '.tags | unique' file.json

undefined

jq '.tags | unique' file.json

undefined

Object Operations

对象操作

bash

undefined

bash

undefined

Get all keys

jq 'keys' file.json jq '.config | keys' file.json

Get all values

jq '.[] | values' file.json

Select specific fields

jq '{name, email}' file.json jq '{name: .fullName, id: .userId}' file.json

Add field

jq '. + {newField: "value"}' file.json

Delete field

jq 'del(.password)' file.json

Merge objects

jq '. * {updated: true}' file.json

undefined

jq '. * {updated: true}' file.json

undefined

Filtering and Conditions

过滤与条件判断

bash

undefined

bash

undefined

Select with conditions

jq 'select(.status == "active")' file.json jq '.[] | select(.price < 100)' file.json

Multiple conditions (AND)

jq '.[] | select(.active and .verified)' file.json jq '.[] | select(.age > 18 and .country == "US")' file.json

Multiple conditions (OR)

jq '.[] | select(.type == "admin" or .type == "moderator")' file.json

Exists / has field

jq '.[] | select(has("email"))' file.json jq 'select(.optional != null)' file.json

Not condition

jq '.[] | select(.status != "deleted")' file.json

undefined

jq '.[] | select(.status != "deleted")' file.json

undefined

String Operations

字符串操作

bash

undefined

bash

undefined

String interpolation

jq '"Hello, (.name)"' file.json

Convert to string

jq '.id | tostring' file.json

String contains

jq '.[] | select(.email | contains("@gmail.com"))' file.json

String starts/ends with

jq '.[] | select(.name | startswith("A"))' file.json jq '.[] | select(.file | endswith(".json"))' file.json

Split string

jq '.path | split("/")' file.json

Join array to string

jq '.tags | join(", ")' file.json

Lowercase/uppercase

jq '.name | ascii_downcase' file.json jq '.name | ascii_upcase' file.json

undefined

jq '.name | ascii_downcase' file.json jq '.name | ascii_upcase' file.json

undefined

Pipes and Composition

管道与组合操作

bash

undefined

bash

undefined

Chain operations

jq '.items | map(.name) | sort | unique' file.json

Multiple filters

jq '.users[] | select(.active) | select(.age > 18) | .email' file.json

Group by

jq 'group_by(.category)' file.json jq 'group_by(.status) | map({status: .[0].status, count: length})' file.json

undefined

jq 'group_by(.category)' file.json jq 'group_by(.status) | map({status: .[0].status, count: length})' file.json

undefined

Output Formatting

输出格式化

bash

undefined

bash

undefined

Compact output (no pretty-print)

jq -c '.' file.json

Raw output (no quotes for strings)

jq -r '.message' file.json jq -r '.items[] | .name' file.json

Output as tab-separated values

jq -r '.[] | [.name, .age, .email] | @tsv' file.json

Output as CSV

jq -r '.[] | [.name, .age, .email] | @csv' file.json

Output as JSON array on one line

jq -c '[.items[]]' file.json

undefined

jq -c '[.items[]]' file.json

undefined

Input/Output Options

输入/输出选项

bash

undefined

bash

undefined

Read from stdin

cat file.json | jq '.items' curl -s https://api.example.com/data | jq '.results'

Multiple input files

jq -s '.' file1.json file2.json # Slurp into array

Write to file

jq '.filtered' input.json > output.json

In-place edit (use sponge from moreutils)

jq '.updated = true' file.json | sponge file.json

undefined

jq '.updated = true' file.json | sponge file.json

undefined

Advanced Patterns

高级模式

bash

undefined

bash

undefined

Recursive descent

jq '.. | .email? // empty' file.json

Reduce (sum, accumulate)

jq '[.items[].price] | add' file.json jq 'reduce .items[] as $item (0; . + $item.price)' file.json

Variable assignment

jq '.items[] | . as $item | $item.name + " - " + ($item.price | tostring)' file.json

Conditional (if-then-else)

jq '.items[] | if .price > 100 then "expensive" else "affordable" end' file.json jq 'if .error then .error else .data end' file.json

Try-catch for error handling

jq '.items[] | try .field catch "not found"' file.json

Flatten nested arrays

jq '.items | flatten' file.json jq '.items | flatten(1)' file.json # Flatten one level

undefined

jq '.items | flatten' file.json jq '.items | flatten(1)' file.json # Flatten one level

undefined

Real-World Examples

实际应用示例

bash

undefined

bash

undefined

Extract all emails from nested structure

jq '.. | .email? // empty' users.json

Get unique list of all tags across items

jq '[.items[].tags[]] | unique' data.json

Count items by status

jq 'group_by(.status) | map({status: .[0].status, count: length})' items.json

Transform API response to simple list

jq '.results[] | {id, name: .full_name, active: .is_active}' response.json

Filter GitHub workflow runs (recent failures)

gh run list --json status,conclusion,name,createdAt |
jq '.[] | select(.conclusion == "failure") | {name, createdAt}'

Extract package.json dependencies with versions

jq '.dependencies | to_entries | map("(.key)@(.value)")' package.json

Merge two JSON files

jq -s '.[0] * .[1]' base.json override.json

Create summary from log data

jq 'group_by(.level) | map({level: .[0].level, count: length, samples: [.[].message][:3]})' logs.json

undefined

jq 'group_by(.level) | map({level: .[0].level, count: length, samples: [.[].message][:3]})' logs.json

undefined

Best Practices

最佳实践

Query Construction

Start simple, build complexity incrementally
Test filters on small datasets first
Use
```
-c
```
flag for compact output in scripts
Use
```
-r
```
flag for raw strings (no quotes)

Performance

Use
```
select()
```
early in pipeline to reduce data
Use targeted queries with specific paths for large files
Stream large JSON with
```
--stream
```
flag
Consider
```
jq -c
```
for faster processing

Error Handling

Use
```
?
```
operator for optional access:
```
.field?
```
Use
```
// empty
```
to filter out nulls/errors
Use
```
try-catch
```
for graceful error handling
Check for field existence with
```
has("field")
```

Readability

Break complex queries into multiple steps
Use variables with
```
as $var
```
for clarity
Add comments in shell scripts
Format multi-line jq programs for readability

查询构建

从简单查询开始，逐步增加复杂度
先在小数据集上测试过滤器
在脚本中使用
```
-c
```
参数获取紧凑输出
使用
```
-r
```
参数获取原始字符串（无引号）

性能优化

在管道中尽早使用
```
select()
```
减少数据量
对大文件使用带具体路径的定向查询
使用
```
--stream
```
参数处理流式大JSON
考虑使用
```
jq -c
```
提升处理速度

错误处理

使用
```
?
```
操作符进行可选访问：
```
.field?
```
使用
```
// empty
```
过滤空值/错误
使用
```
try-catch
```
实现优雅的错误处理
使用
```
has("field")
```
检查字段是否存在

可读性

将复杂查询拆分为多个步骤
使用
```
as $var
```
定义变量提升清晰度
在Shell脚本中添加注释
对多行jq程序进行格式化以提升可读性

Common Patterns

常见使用模式

API Response Processing

API响应处理

bash

undefined

bash

undefined

GitHub API: Get PR titles and authors

gh pr list --json title,author,number |
jq -r '.[] | "#(.number) - (.title) by @(.author.login)"'

REST API: Extract and flatten pagination

curl -s "https://api.example.com/items" |
jq '.data.items[] | {id, name, status}'

undefined

curl -s "https://api.example.com/items" |
jq '.data.items[] | {id, name, status}'

undefined

Configuration Files

配置文件处理

bash

undefined

bash

undefined

Extract environment-specific config

jq '.environments.production' config.json

Update configuration value

jq '.settings.timeout = 30' config.json > config.updated.json

jq '.settings.timeout = 30' config.json > config.update.json

Merge base config with environment overrides

jq -s '.[0] * .[1]' base-config.json prod-config.json

undefined

jq -s '.[0] * .[1]' base-config.json prod-config.json

undefined

Log Analysis

日志分析

bash

undefined

bash

undefined

Count errors by type

jq 'select(.level == "error") | .type' logs.json | sort | uniq -c

Extract error messages with timestamps

jq -r 'select(.level == "error") | "(.timestamp) - (.message)"' logs.json

Group by hour and count

jq -r '.timestamp | split("T")[1] | split(":")[0]' logs.json | sort | uniq -c

undefined

jq -r '.timestamp | split("T")[1] | split(":")[0]' logs.json | sort | uniq -c

undefined

Data Transformation

数据转换

bash

undefined

bash

undefined

CSV to JSON (with headers)

jq -R -s 'split("\n") | .[1:] | map(split(",")) | map({name: .[0], age: .[1], email: .[2]})' data.csv

JSON to CSV

jq -r '.[] | [.name, .age, .email] | @csv' data.json

Flatten nested structure

jq '[.items[] | {id, name, category: .meta.category}]' nested.json

undefined

jq '[.items[] | {id, name, category: .meta.category}]' nested.json

undefined

Troubleshooting

问题排查

Invalid JSON

无效JSON

bash

undefined

bash

undefined

Validate JSON syntax

jq empty file.json # Returns exit code 0 if valid

Find syntax errors

jq '.' file.json 2>&1 | grep "parse error"

undefined

jq '.' file.json 2>&1 | grep "parse error"

undefined

Empty Results

无结果返回

bash

undefined

bash

undefined

Debug: Print entire structure

jq '.' file.json

Debug: Check field existence

jq 'keys' file.json jq 'type' file.json # Check if array, object, etc.

Debug: Show all values

jq '.. | scalars' file.json

undefined

jq '.. | scalars' file.json

undefined

Type Errors

类型错误

bash

undefined

bash

undefined

Check field types

jq '.field | type' file.json

Convert types safely

jq '.id | tonumber' file.json jq '.count | tostring' file.json

Handle mixed types

jq '.items[] | if type == "array" then .[] else . end' file.json

undefined

jq '.items[] | if type == "array" then .[] else . end' file.json

undefined

Performance Issues

性能问题

bash

undefined

bash

undefined

Stream large files

jq --stream '.' large-file.json

Process line by line

cat large.json | jq -c '.[]' | while read -r line; do echo "$line" | jq '.field' done

undefined

cat large.json | jq -c '.[]' | while read -r line; do echo "$line" | jq '.field' done

undefined

Integration with Other Tools

与其他工具集成

bash

undefined

bash

undefined

With curl (API calls)

curl -s "https://api.github.com/users/octocat" | jq '.name, .bio'

With gh CLI (GitHub operations)

gh api repos/owner/repo/issues | jq '.[] | {number, title, state}'

With find (batch processing)

find . -name "package.json" -exec jq '.version' {} ;

With xargs (parallel processing)

cat urls.txt | xargs -I {} curl -s {} | jq '.data'

With yq (YAML to JSON conversion)

yq eval -o=json file.yaml | jq '.specific.field'

undefined

yq eval -o=json file.yaml | jq '.specific.field'

undefined

Quick Reference

速查指南

Operators

操作符

```
.field
```
- Access field
```
.[]
```
- Iterate array/object
```
|
```
- Pipe (chain operations)
```
,
```
- Multiple outputs
```
?
```
- Optional (suppress errors)
```
//
```
- Alternative operator (default value)

```
.field
```
- 访问字段
```
.[]
```
- 遍历数组/对象
```
|
```
- 管道（链式操作）
```
,
```
- 多输出
```
?
```
- 可选访问（抑制错误）
```
//
```
- 替代操作符（默认值）

Functions

函数

```
keys
```
,
```
values
```
- Object keys/values
```
length
```
- Array/object/string length
```
map()
```
,
```
select()
```
- Array operations
```
sort
```
,
```
sort_by()
```
- Sorting
```
group_by()
```
- Grouping
```
unique
```
- Remove duplicates
```
add
```
- Sum numbers or concatenate
```
has()
```
- Check field existence
```
type
```
- Get value type

```
keys
```
,
```
values
```
- 对象键/值
```
length
```
- 数组/对象/字符串长度
```
map()
```
,
```
select()
```
- 数组操作
```
sort
```
,
```
sort_by()
```
- 排序
```
group_by()
```
- 分组
```
unique
```
- 去重
```
add
```
- 数值求和或字符串拼接
```
has()
```
- 检查字段存在性
```
type
```
- 获取值类型

Type Conversions

类型转换

```
tostring
```
,
```
tonumber
```
- Convert types
```
@csv
```
,
```
@tsv
```
,
```
@json
```
- Format output
```
split()
```
,
```
join()
```
- String/array conversion

```
tostring
```
,
```
tonumber
```
- 类型转换
```
@csv
```
,
```
@tsv
```
,
```
@json
```
- 输出格式化
```
split()
```
,
```
join()
```
- 字符串/数组转换

Installation

安装方法

bash

undefined

bash

undefined

macOS (Homebrew)

brew install jq

Ubuntu/Debian

sudo apt-get install jq

Verify installation

jq --version

undefined

jq --version

undefined

Resources

参考资源

Official Manual: https://stedolan.github.io/jq/manual/
jq Playground: https://jqplay.org/
Tutorial: https://stedolan.github.io/jq/tutorial/

官方手册: https://stedolan.github.io/jq/manual/
jq 在线 playground: https://jqplay.org/
教程: https://stedolan.github.io/jq/tutorial/

jq-json-processing

Original

Translation

jq JSON Processing

jq JSON 数据处理

Core Expertise

核心能力

Essential Commands

常用命令

Basic Querying

基础查询

Pretty-print JSON

Pretty-print JSON

Extract specific field

Extract specific field

Extract nested field

Extract nested field

Extract array element

Extract array element

Array Operations

数组操作

Get array length

Get array length

Map over array

Map over array

Filter array

Filter array

Sort array

Sort array

Get first/last elements

Get first/last elements

Unique values

Unique values

Object Operations

对象操作

Get all keys

Get all keys

Get all values

Get all values

Select specific fields

Select specific fields

Add field

Add field

Delete field

Delete field

Merge objects

Merge objects

Filtering and Conditions

过滤与条件判断

Select with conditions

Select with conditions

Multiple conditions (AND)

Multiple conditions (AND)

Multiple conditions (OR)

Multiple conditions (OR)

Exists / has field

Exists / has field

Not condition

Not condition

String Operations

字符串操作

String interpolation

String interpolation

Convert to string

Convert to string

String contains

String contains

String starts/ends with

String starts/ends with

Split string

Split string

Join array to string

Join array to string

Lowercase/uppercase

Lowercase/uppercase

Pipes and Composition

管道与组合操作

Chain operations

Chain operations

Multiple filters