loki-logging

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Grafana Loki

Aggregate and query logs with Grafana Loki, the Prometheus-inspired logging system.

使用受Prometheus启发的日志系统Grafana Loki来聚合和查询日志。

When to Use This Skill

适用场景

Use this skill when:

Implementing cost-effective log aggregation
Building logging for Kubernetes environments
Integrating logs with Grafana dashboards
Querying logs with label-based filtering
Preferring lighter-weight alternative to ELK

在以下场景中使用本技能：

实现高性价比的日志聚合
为Kubernetes环境搭建日志系统
将日志与Grafana仪表盘集成
通过基于标签的过滤查询日志
偏好轻量级替代方案而非ELK栈

Prerequisites

前置条件

Docker or Kubernetes
Grafana for visualization
Promtail or other log shipper

Docker或Kubernetes环境
用于可视化的Grafana
Promtail或其他日志采集器

Architecture Overview

架构概述

┌─────────────┐     ┌──────────┐     ┌──────────┐
│ Application │────▶│ Promtail │────▶│   Loki   │
└─────────────┘     └──────────┘     └──────────┘
                                          │
                                          ▼
                                     ┌──────────┐
                                     │ Grafana  │
                                     └──────────┘

┌─────────────┐     ┌──────────┐     ┌──────────┐
│ Application │────▶│ Promtail │────▶│   Loki   │
└─────────────┘     └──────────┘     └──────────┘
                                          │
                                          ▼
                                     ┌──────────┐
                                     │ Grafana  │
                                     └──────────┘

Docker Deployment

Docker部署

yaml

undefined

yaml

undefined

docker-compose.yml

version: '3.8'

services: loki: image: grafana/loki:2.9.0 ports: - "3100:3100" volumes: - ./loki-config.yaml:/etc/loki/local-config.yaml - loki-data:/loki command: -config.file=/etc/loki/local-config.yaml

promtail: image: grafana/promtail:2.9.0 volumes: - ./promtail-config.yaml:/etc/promtail/config.yaml - /var/log:/var/log:ro - /var/lib/docker/containers:/var/lib/docker/containers:ro command: -config.file=/etc/promtail/config.yaml

grafana: image: grafana/grafana:10.2.0 ports: - "3000:3000" volumes: - grafana-data:/var/lib/grafana - ./grafana/provisioning:/etc/grafana/provisioning environment: - GF_AUTH_ANONYMOUS_ENABLED=true - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin

volumes: loki-data: grafana-data:

undefined

version: '3.8'

services: loki: image: grafana/loki:2.9.0 ports: - "3100:3100" volumes: - ./loki-config.yaml:/etc/loki/local-config.yaml - loki-data:/loki command: -config.file=/etc/loki/local-config.yaml

volumes: loki-data: grafana-data:

undefined

Loki Configuration

Loki配置

yaml

undefined

yaml

undefined

loki-config.yaml

auth_enabled: false

server: http_listen_port: 3100

common: path_prefix: /loki storage: filesystem: chunks_directory: /loki/chunks rules_directory: /loki/rules replication_factor: 1 ring: kvstore: store: inmemory

schema_config: configs: - from: 2020-10-24 store: boltdb-shipper object_store: filesystem schema: v11 index: prefix: index_ period: 24h

storage_config: boltdb_shipper: active_index_directory: /loki/index cache_location: /loki/cache shared_store: filesystem

limits_config: reject_old_samples: true reject_old_samples_max_age: 168h max_query_series: 5000 max_query_parallelism: 2

chunk_store_config: max_look_back_period: 168h

table_manager: retention_deletes_enabled: true retention_period: 168h

undefined

auth_enabled: false

server: http_listen_port: 3100

common: path_prefix: /loki storage: filesystem: chunks_directory: /loki/chunks rules_directory: /loki/rules replication_factor: 1 ring: kvstore: store: inmemory

schema_config: configs: - from: 2020-10-24 store: boltdb-shipper object_store: filesystem schema: v11 index: prefix: index_ period: 24h

storage_config: boltdb_shipper: active_index_directory: /loki/index cache_location: /loki/cache shared_store: filesystem

limits_config: reject_old_samples: true reject_old_samples_max_age: 168h max_query_series: 5000 max_query_parallelism: 2

chunk_store_config: max_look_back_period: 168h

table_manager: retention_deletes_enabled: true retention_period: 168h

undefined

Promtail Configuration

Promtail配置

yaml

undefined

yaml

undefined

promtail-config.yaml

server: http_listen_port: 9080 grpc_listen_port: 0

positions: filename: /tmp/positions.yaml

clients:

url: http://loki:3100/loki/api/v1/push

scrape_configs:

System logs

job_name: system static_configs:
- targets:
  - localhost labels: job: varlogs path: /var/log/*.log

Docker container logs

job_name: docker docker_sd_configs:
- host: unix:///var/run/docker.sock refresh_interval: 5s relabel_configs:
- source_labels: ['__meta_docker_container_name'] regex: '/(.*)' target_label: 'container'
- source_labels: ['__meta_docker_container_log_stream'] target_label: 'stream'

Application logs with parsing

job_name: application static_configs:
- targets:
  - localhost labels: job: application path: /var/log/app/*.log pipeline_stages:
- json: expressions: level: level message: message timestamp: timestamp
- labels: level:
- timestamp: source: timestamp format: RFC3339

undefined

server: http_listen_port: 9080 grpc_listen_port: 0

positions: filename: /tmp/positions.yaml

clients:

url: http://loki:3100/loki/api/v1/push

scrape_configs:

System logs

job_name: system static_configs:
- targets:
  - localhost labels: job: varlogs path: /var/log/*.log

Docker container logs

job_name: docker docker_sd_configs:
- host: unix:///var/run/docker.sock refresh_interval: 5s relabel_configs:
- source_labels: ['__meta_docker_container_name'] regex: '/(.*)' target_label: 'container'
- source_labels: ['__meta_docker_container_log_stream'] target_label: 'stream'

Application logs with parsing

job_name: application static_configs:
- targets:
  - localhost labels: job: application path: /var/log/app/*.log pipeline_stages:
- json: expressions: level: level message: message timestamp: timestamp
- labels: level:
- timestamp: source: timestamp format: RFC3339

undefined

Kubernetes Deployment

Kubernetes部署

bash

undefined

bash

undefined

Using Helm

helm repo add grafana https://grafana.github.io/helm-charts helm install loki grafana/loki-stack
--namespace monitoring
--create-namespace
--set grafana.enabled=true
--set promtail.enabled=true

undefined

helm repo add grafana https://grafana.github.io/helm-charts helm install loki grafana/loki-stack
--namespace monitoring
--create-namespace
--set grafana.enabled=true
--set promtail.enabled=true

undefined

Promtail DaemonSet

yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: promtail
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: promtail
  template:
    metadata:
      labels:
        app: promtail
    spec:
      containers:
        - name: promtail
          image: grafana/promtail:2.9.0
          args:
            - -config.file=/etc/promtail/promtail.yaml
          volumeMounts:
            - name: config
              mountPath: /etc/promtail
            - name: varlog
              mountPath: /var/log
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
      volumes:
        - name: config
          configMap:
            name: promtail-config
        - name: varlog
          hostPath:
            path: /var/log
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers

yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: promtail
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: promtail
  template:
    metadata:
      labels:
        app: promtail
    spec:
      containers:
        - name: promtail
          image: grafana/promtail:2.9.0
          args:
            - -config.file=/etc/promtail/promtail.yaml
          volumeMounts:
            - name: config
              mountPath: /etc/promtail
            - name: varlog
              mountPath: /var/log
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
      volumes:
        - name: config
          configMap:
            name: promtail-config
        - name: varlog
          hostPath:
            path: /var/log
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers

LogQL Queries

LogQL查询

Basic Queries

基础查询

logql

undefined

logql

undefined

All logs from a job

{job="application"}

Filter by label

{job="application", level="error"}

Multiple labels

{namespace="production", container="api"}

Regex match

{job=~"app.*"}

undefined

{job=~"app.*"}

undefined

Log Pipeline

日志处理流水线

logql

undefined

logql

undefined

Filter by content

{job="application"} |= "error"

Exclude content

{job="application"} != "debug"

Regex filter

{job="application"} |~ "user_id=[0-9]+"

JSON parsing

{job="application"} | json | level="error"

Line format

{job="application"} | json | line_format "{{.level}}: {{.message}}"

undefined

{job="application"} | json | line_format "{{.level}}: {{.message}}"

undefined

Metric Queries

指标查询

logql

undefined

logql

undefined

Count logs per second

count_over_time({job="application"}[5m])

Rate of errors

rate({job="application", level="error"}[5m])

Sum by label

sum by (level) (count_over_time({job="application"}[5m]))

Top services by error count

topk(5, sum by (service) (count_over_time({level="error"}[1h])))

undefined

topk(5, sum by (service) (count_over_time({level="error"}[1h])))

undefined

Aggregations

聚合查询

logql

undefined

logql

undefined

Average log line length

avg_over_time({job="application"} | unwrap line_length [5m])

Percentile of numeric field

quantile_over_time(0.95, {job="application"} | json | unwrap response_time [5m])

Error percentage

sum(rate({job="application", level="error"}[5m])) / sum(rate({job="application"}[5m])) * 100

undefined

sum(rate({job="application", level="error"}[5m])) / sum(rate({job="application"}[5m])) * 100

undefined

Pipeline Stages

流水线阶段

yaml

undefined

yaml

undefined

promtail-config.yaml

pipeline_stages:

Parse JSON logs

json: expressions: level: level message: msg trace_id: trace_id

Extract with regex

regex: expression: 'user_id=(?P<user_id>\d+)'

Add labels from parsed fields

labels: level: user_id:

Modify timestamp

timestamp: source: timestamp format: '2006-01-02T15:04:05.000Z'

Filter logs

match: selector: '{level="debug"}' action: drop

Add static labels

static_labels: environment: production

Modify log line

template: source: message template: '{{ ToUpper .Value }}'

undefined

pipeline_stages:

Parse JSON logs

json: expressions: level: level message: msg trace_id: trace_id

Extract with regex

regex: expression: 'user_id=(?P<user_id>\d+)'

Add labels from parsed fields

labels: level: user_id:

Modify timestamp

timestamp: source: timestamp format: '2006-01-02T15:04:05.000Z'

Filter logs

match: selector: '{level="debug"}' action: drop

Add static labels

static_labels: environment: production

Modify log line

template: source: message template: '{{ ToUpper .Value }}'

undefined

Grafana Integration

Grafana集成

Data Source Configuration

数据源配置

yaml

undefined

yaml

undefined

grafana/provisioning/datasources/loki.yaml

apiVersion: 1

datasources:

name: Loki type: loki access: proxy url: http://loki:3100 isDefault: false jsonData: maxLines: 1000

undefined

apiVersion: 1

datasources:

name: Loki type: loki access: proxy url: http://loki:3100 isDefault: false jsonData: maxLines: 1000

undefined

Dashboard Panel

仪表盘面板

json

{
  "title": "Application Logs",
  "type": "logs",
  "datasource": "Loki",
  "targets": [
    {
      "expr": "{job=\"application\"} | json",
      "refId": "A"
    }
  ],
  "options": {
    "showTime": true,
    "showLabels": true,
    "wrapLogMessage": true
  }
}

json

{
  "title": "Application Logs",
  "type": "logs",
  "datasource": "Loki",
  "targets": [
    {
      "expr": "{job=\"application\"} | json",
      "refId": "A"
    }
  ],
  "options": {
    "showTime": true,
    "showLabels": true,
    "wrapLogMessage": true
  }
}

Recording Rules

记录规则

yaml

undefined

yaml

undefined

loki-rules.yaml

groups:

name: error_rates interval: 1m rules:
- record: job:log_errors:rate5m expr: | sum by (job) (rate({level="error"}[5m]))

undefined

groups:

name: error_rates interval: 1m rules:
- record: job:log_errors:rate5m expr: | sum by (job) (rate({level="error"}[5m]))

undefined

Alerting

告警配置

yaml

undefined

yaml

undefined

loki-alerts.yaml

groups:

name: log_alerts rules:
- alert: HighErrorRate expr: | sum(rate({level="error"}[5m])) > 10 for: 5m labels: severity: critical annotations: summary: "High error rate in logs" description: "Error rate is {{ $value }} errors/second"

undefined

groups:

name: log_alerts rules:
- alert: HighErrorRate expr: | sum(rate({level="error"}[5m])) > 10 for: 5m labels: severity: critical annotations: summary: "High error rate in logs" description: "Error rate is {{ $value }} errors/second"

undefined

Common Issues

常见问题

Issue: High Memory Usage

问题：内存占用过高

Problem: Loki consuming too much memory Solution: Reduce max_query_series, limit query time range

问题描述：Loki占用过多内存 解决方案：减小max_query_series参数值，限制查询时间范围

Issue: Logs Not Appearing

问题：日志未显示

Problem: Promtail not shipping logs Solution: Check positions file, verify file paths, check label configuration

问题描述：Promtail未发送日志 解决方案：检查positions文件，验证文件路径，检查标签配置

Issue: Query Timeout

问题：查询超时

Problem: LogQL queries timing out Solution: Add more specific label filters, reduce time range

问题描述：LogQL查询超时 解决方案：添加更具体的标签过滤条件，缩小时间范围

Issue: Ingestion Rate Limit

问题：Ingestion速率限制

Problem: Logs being dropped Solution: Increase per_stream_rate_limit in limits_config

问题描述：日志被丢弃 解决方案：在limits_config中增大per_stream_rate_limit参数值

Best Practices

最佳实践

Use meaningful labels (avoid high cardinality)
Filter by labels before log content
Parse logs at collection time with Promtail
Set appropriate retention periods
Use recording rules for common queries
Implement proper multitenancy for large deployments
Monitor Loki's own metrics
Use chunk caching for better performance

使用有意义的标签（避免高基数标签）
在过滤日志内容前先通过标签过滤
在采集阶段使用Promtail解析日志
设置合适的日志保留周期
为常用查询配置记录规则
为大型部署实现适当的多租户机制
监控Loki自身的指标
使用块缓存提升性能