Back to Details

pyroscope

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Grafana Pyroscope Skill

Grafana Pyroscope Skill

Comprehensive guide for Grafana Pyroscope - the open-source continuous profiling platform for analyzing application performance at the code level.
本指南是关于Grafana Pyroscope的全面介绍，它是一款开源持续剖析平台，用于在代码层面分析应用性能。

What is Pyroscope?

什么是Pyroscope？

Pyroscope is a horizontally-scalable, highly-available, multi-tenant continuous profiling system that:
  • Collects profiling data continuously with minimal overhead (~2-5% CPU)
  • Provides code-level visibility with source-line granularity
  • Stores compressed profiles in object storage (S3, GCS, Azure Blob)
  • Integrates with Grafana for correlating profiles with metrics, logs, and traces
  • Supports multiple languages - Go, Java, Python, .NET, Ruby, Node.js, Rust
Pyroscope是一个可水平扩展、高可用、多租户的持续剖析系统，具备以下特性：
  • 持续收集剖析数据，开销极低（约2-5% CPU占用）
  • 提供代码级可见性，支持源码行粒度
  • 以压缩格式存储剖析数据至对象存储（S3、GCS、Azure Blob）
  • 与Grafana集成，可将剖析数据与指标、日志、追踪数据关联分析
  • 支持多语言 - Go、Java、Python、.NET、Ruby、Node.js、Rust

Architecture Overview

架构概述

Core Components

核心组件

ComponentPurpose
DistributorValidates and routes incoming profiles to ingesters
IngesterBuffers profiles in memory, compresses and writes to storage
QuerierRetrieves and processes profile data for analysis
Query FrontendHandles query requests, caching, and scheduling
Query SchedulerManages per-tenant query queues
Store GatewayProvides access to long-term profile storage
CompactorMerges blocks, manages retention, handles deletion
组件用途
Distributor验证并将传入的剖析数据路由至Ingester
Ingester在内存中缓冲剖析数据，压缩后写入存储
Querier检索并处理剖析数据以进行分析
Query Frontend处理查询请求、缓存及调度
Query Scheduler管理多租户查询队列
Store Gateway提供对长期剖析数据存储的访问
Compactor合并数据块、管理数据保留周期、处理数据删除

Data Flow

数据流

Write Path:
text
SDK/Alloy → Distributor → Ingester → Object Storage
                                   ↓
                             Blocks + Indexes
Read Path:
text
Query → Query Frontend → Query Scheduler → Querier
                                             ↓
                                    Ingesters + Store Gateway
写入路径：
text
SDK/Alloy → Distributor → Ingester → Object Storage
                                   ↓
                             Blocks + Indexes
读取路径：
text
Query → Query Frontend → Query Scheduler → Querier
                                             ↓
                                    Ingesters + Store Gateway

Deployment Modes

部署模式

1. Monolithic Mode (
-target=all
)

1. 单体模式 (
-target=all
)

  • All components in single process
  • Best for: Development, small-scale deployments
  • Query URL: 
    http://pyroscope:4040/
  • 所有组件运行在单个进程中
  • 适用场景：开发环境、小规模部署
  • 查询URL：
    http://pyroscope:4040/

2. Microservices Mode (Production)

2. 微服务模式（生产环境）

  • Each component runs independently
  • Horizontally scalable
  • Query URL: 
    http://pyroscope-querier:4040/
yaml
undefined
  • 每个组件独立运行
  • 支持水平扩展
  • 查询URL：
    http://pyroscope-querier:4040/
yaml
undefined

Microservices deployment

Microservices deployment

architecture: microservices: enabled: true
querier: replicas: 3 distributor: replicas: 2 ingester: replicas: 3 compactor: replicas: 3 storeGateway: replicas: 3
undefined
architecture: microservices: enabled: true
querier: replicas: 3 distributor: replicas: 2 ingester: replicas: 3 compactor: replicas: 3 storeGateway: replicas: 3
undefined

Quick Start - Kubernetes Helm

快速开始 - Kubernetes Helm

Add Repository

添加仓库

bash
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
bash
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

Install Single Binary

安装单体二进制版本

bash
kubectl create namespace pyroscope
helm install pyroscope grafana/pyroscope -n pyroscope
bash
kubectl create namespace pyroscope
helm install pyroscope grafana/pyroscope -n pyroscope

Install Microservices Mode

安装微服务模式

bash
curl -Lo values-micro-services.yaml \
  https://raw.githubusercontent.com/grafana/pyroscope/main/operations/pyroscope/helm/pyroscope/values-micro-services.yaml

helm install pyroscope grafana/pyroscope \
  -n pyroscope \
  --values values-micro-services.yaml
bash
curl -Lo values-micro-services.yaml \
  https://raw.githubusercontent.com/grafana/pyroscope/main/operations/pyroscope/helm/pyroscope/values-micro-services.yaml

helm install pyroscope grafana/pyroscope \
  -n pyroscope \
  --values values-micro-services.yaml

Profile Types

剖析类型

TypeDescriptionLanguages
CPUWall/CPU time consumptionAll
MemoryAllocation objects/space, heapGo, Java, .NET
GoroutineConcurrent goroutinesGo
MutexLock contention (count/duration)Go, Java, .NET
BlockThread blocking/delaysGo
ExceptionsException trackingPython
类型描述支持语言
CPU墙钟/CPU时间消耗所有语言
Memory分配对象/空间、堆内存Go、Java、.NET
Goroutine并发协程统计Go
Mutex锁竞争（次数/时长）Go、Java、.NET
Block线程阻塞/延迟Go
Exceptions异常追踪Python

Client Configuration Methods

客户端配置方式

Method 1: SDK Instrumentation (Push Mode)

方式1：SDK埋点（推送模式）

Go SDK:
go
import "github.com/grafana/pyroscope-go"

pyroscope.Start(pyroscope.Config{
    ApplicationName: "my-app",
    ServerAddress:   "http://pyroscope:4040",
    ProfileTypes: []pyroscope.ProfileType{
        pyroscope.ProfileCPU,
        pyroscope.ProfileAllocObjects,
        pyroscope.ProfileAllocSpace,
        pyroscope.ProfileInuseObjects,
        pyroscope.ProfileInuseSpace,
        pyroscope.ProfileGoroutines,
        pyroscope.ProfileMutexCount,
        pyroscope.ProfileMutexDuration,
        pyroscope.ProfileBlockCount,
        pyroscope.ProfileBlockDuration,
    },
    Tags: map[string]string{
        "env": "production",
    },
})
Java SDK:
java
PyroscopeAgent.start(
    new Config.Builder()
        .setApplicationName("my-app")
        .setServerAddress("http://pyroscope:4040")
        .setProfilingEvent(EventType.ITIMER)
        .setFormat(Format.JFR)
        .build()
);
Python SDK:
python
import pyroscope

pyroscope.configure(
    application_name="my-app",
    server_address="http://pyroscope:4040",
    tags={"env": "production"},
)
Go SDK：
go
import "github.com/grafana/pyroscope-go"

pyroscope.Start(pyroscope.Config{
    ApplicationName: "my-app",
    ServerAddress:   "http://pyroscope:4040",
    ProfileTypes: []pyroscope.ProfileType{
        pyroscope.ProfileCPU,
        pyroscope.ProfileAllocObjects,
        pyroscope.ProfileAllocSpace,
        pyroscope.ProfileInuseObjects,
        pyroscope.ProfileInuseSpace,
        pyroscope.ProfileGoroutines,
        pyroscope.ProfileMutexCount,
        pyroscope.ProfileMutexDuration,
        pyroscope.ProfileBlockCount,
        pyroscope.ProfileBlockDuration,
    },
    Tags: map[string]string{
        "env": "production",
    },
})
Java SDK：
java
PyroscopeAgent.start(
    new Config.Builder()
        .setApplicationName("my-app")
        .setServerAddress("http://pyroscope:4040")
        .setProfilingEvent(EventType.ITIMER)
        .setFormat(Format.JFR)
        .build()
);
Python SDK：
python
import pyroscope

pyroscope.configure(
    application_name="my-app",
    server_address="http://pyroscope:4040",
    tags={"env": "production"},
)

Method 2: Grafana Alloy (Pull Mode)

方式2：Grafana Alloy（拉取模式）

Auto-instrumentation via Annotations:
yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    metadata:
      annotations:
        profiles.grafana.com/cpu.scrape: "true"
        profiles.grafana.com/cpu.port: "8080"
        profiles.grafana.com/memory.scrape: "true"
        profiles.grafana.com/memory.port: "8080"
        profiles.grafana.com/goroutine.scrape: "true"
        profiles.grafana.com/goroutine.port: "8080"
Alloy Configuration:
river
pyroscope.scrape "default" {
  targets = discovery.kubernetes.pods.targets
  forward_to = [pyroscope.write.default.receiver]

  profiling_config {
    profile.process_cpu { enabled = true }
    profile.memory { enabled = true }
    profile.goroutine { enabled = true }
  }
}

pyroscope.write "default" {
  endpoint {
    url = "http://pyroscope:4040"
  }
}
通过注解自动埋点：
yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    metadata:
      annotations:
        profiles.grafana.com/cpu.scrape: "true"
        profiles.grafana.com/cpu.port: "8080"
        profiles.grafana.com/memory.scrape: "true"
        profiles.grafana.com/memory.port: "8080"
        profiles.grafana.com/goroutine.scrape: "true"
        profiles.grafana.com/goroutine.port: "8080"
Alloy配置：
river
pyroscope.scrape "default" {
  targets = discovery.kubernetes.pods.targets
  forward_to = [pyroscope.write.default.receiver]

  profiling_config {
    profile.process_cpu { enabled = true }
    profile.memory { enabled = true }
    profile.goroutine { enabled = true }
  }
}

pyroscope.write "default" {
  endpoint {
    url = "http://pyroscope:4040"
  }
}

Method 3: eBPF Profiling (Linux)

方式3：eBPF剖析（Linux）

For compiled languages (C/C++, Go, Rust):
river
pyroscope.ebpf "default" {
  forward_to = [pyroscope.write.default.receiver]
  targets = discovery.kubernetes.pods.targets
}
适用于编译型语言（C/C++、Go、Rust）：
river
pyroscope.ebpf "default" {
  forward_to = [pyroscope.write.default.receiver]
  targets = discovery.kubernetes.pods.targets
}

Storage Configuration

存储配置

Azure Blob Storage

Azure Blob存储

yaml
pyroscope:
  config:
    storage:
      backend: azure
      azure:
        container_name: pyroscope-data
        account_name: mystorageaccount
        account_key: ${AZURE_ACCOUNT_KEY}
yaml
pyroscope:
  config:
    storage:
      backend: azure
      azure:
        container_name: pyroscope-data
        account_name: mystorageaccount
        account_key: ${AZURE_ACCOUNT_KEY}

AWS S3

AWS S3

yaml
pyroscope:
  config:
    storage:
      backend: s3
      s3:
        bucket_name: pyroscope-data
        region: us-east-1
        endpoint: s3.us-east-1.amazonaws.com
        access_key_id: ${AWS_ACCESS_KEY_ID}
        secret_access_key: ${AWS_SECRET_ACCESS_KEY}
yaml
pyroscope:
  config:
    storage:
      backend: s3
      s3:
        bucket_name: pyroscope-data
        region: us-east-1
        endpoint: s3.us-east-1.amazonaws.com
        access_key_id: ${AWS_ACCESS_KEY_ID}
        secret_access_key: ${AWS_SECRET_ACCESS_KEY}

Google Cloud Storage

Google Cloud存储

yaml
pyroscope:
  config:
    storage:
      backend: gcs
      gcs:
        bucket_name: pyroscope-data
        # Uses GOOGLE_APPLICATION_CREDENTIALS
yaml
pyroscope:
  config:
    storage:
      backend: gcs
      gcs:
        bucket_name: pyroscope-data
        # Uses GOOGLE_APPLICATION_CREDENTIALS

Grafana Integration

Grafana集成

Data Source Configuration

数据源配置

yaml
apiVersion: 1
datasources:
  - name: Pyroscope
    type: grafana-pyroscope-datasource
    access: proxy
    url: http://pyroscope-querier:4040
    isDefault: false
    editable: true
yaml
apiVersion: 1
datasources:
  - name: Pyroscope
    type: grafana-pyroscope-datasource
    access: proxy
    url: http://pyroscope-querier:4040
    isDefault: false
    editable: true

Trace-to-Profile Linking

追踪-剖析关联

Enable span profiles to correlate traces with profiles:
Go with OpenTelemetry:
go
import (
    "github.com/grafana/pyroscope-go"
    otelpyroscope "github.com/grafana/otel-profiling-go"
)

tp := trace.NewTracerProvider(
    trace.WithSpanProcessor(otelpyroscope.NewSpanProcessor()),
)
Requirements:
  • Minimum span duration: 20ms
  • Supported: Go, Java, .NET, Python, Ruby
启用Span剖析以关联追踪数据与剖析数据：
Go + OpenTelemetry：
go
import (
    "github.com/grafana/pyroscope-go"
    otelpyroscope "github.com/grafana/otel-profiling-go"
)

tp := trace.NewTracerProvider(
    trace.WithSpanProcessor(otelpyroscope.NewSpanProcessor()),
)
要求：
  • 最小Span时长：20ms
  • 支持语言：Go、Java、.NET、Python、Ruby

Resource Requirements

资源需求

Single Binary (Development)

单体二进制版本（开发环境）

yaml
resources:
  requests:
    cpu: 500m
    memory: 512Mi
  limits:
    cpu: 1
    memory: 2Gi
yaml
resources:
  requests:
    cpu: 500m
    memory: 512Mi
  limits:
    cpu: 1
    memory: 2Gi

Microservices (Production)

微服务版本（生产环境）

ComponentCPU RequestMemory RequestMemory Limit
Distributor500m256Mi1Gi
Ingester18Gi16Gi
Querier100m256Mi1Gi
Query Frontend100m256Mi1Gi
Compactor18Gi16Gi
Store Gateway18Gi16Gi
组件CPU请求内存请求内存限制
Distributor500m256Mi1Gi
Ingester1核8Gi16Gi
Querier100m256Mi1Gi
Query Frontend100m256Mi1Gi
Compactor1核8Gi16Gi
Store Gateway1核8Gi16Gi

Common Helm Values

常用Helm配置值

yaml
undefined
yaml
undefined

Production values

Production values

architecture: microservices: enabled: true
pyroscope: persistence: enabled: true size: 50Gi
config: storage: backend: s3 s3: bucket_name: pyroscope-prod region: us-east-1
architecture: microservices: enabled: true
pyroscope: persistence: enabled: true size: 50Gi
config: storage: backend: s3 s3: bucket_name: pyroscope-prod region: us-east-1

High availability

High availability

ingester: replicas: 3 terminationGracePeriodSeconds: 600
querier: replicas: 3
distributor: replicas: 2
compactor: replicas: 3 terminationGracePeriodSeconds: 1200
storeGateway: replicas: 3
ingester: replicas: 3 terminationGracePeriodSeconds: 600
querier: replicas: 3
distributor: replicas: 2
compactor: replicas: 3 terminationGracePeriodSeconds: 1200
storeGateway: replicas: 3

Pod disruption budget

Pod disruption budget

podDisruptionBudget: enabled: true maxUnavailable: 1
podDisruptionBudget: enabled: true maxUnavailable: 1

Topology spread

Topology spread

topologySpreadConstraints:
  • maxSkew: 1 topologyKey: kubernetes.io/hostname whenUnsatisfiable: DoNotSchedule
topologySpreadConstraints:
  • maxSkew: 1 topologyKey: kubernetes.io/hostname whenUnsatisfiable: DoNotSchedule

Monitoring

Monitoring

serviceMonitor: enabled: true
serviceMonitor: enabled: true

Alloy for profile collection

Alloy for profile collection

alloy: enabled: true
undefined
alloy: enabled: true
undefined

API Endpoints

API端点

Ingestion

数据写入

bash
undefined
bash
undefined

Push profiles (Connect API)

Push profiles (Connect API)

POST /push.v1.PusherService/Push
POST /push.v1.PusherService/Push

Legacy HTTP (pprof, JFR formats)

Legacy HTTP (pprof, JFR formats)

POST /ingest
undefined
POST /ingest
undefined

Query

数据查询

bash
undefined
bash
undefined

Merged profile

Merged profile

POST /querier.v1.QuerierService/SelectMergeProfile
POST /querier.v1.QuerierService/SelectMergeProfile

Flame graph data

Flame graph data

POST /querier.v1.QuerierService/SelectMergeStacktraces
POST /querier.v1.QuerierService/SelectMergeStacktraces

Available labels

Available labels

POST /querier.v1.QuerierService/LabelNames
POST /querier.v1.QuerierService/LabelNames

Profile types

Profile types

POST /querier.v1.QuerierService/ProfileTypes
POST /querier.v1.QuerierService/ProfileTypes

Legacy render

Legacy render

GET /pyroscope/render?query={}&from=now-1h&until=now
undefined
GET /pyroscope/render?query={}&from=now-1h&until=now
undefined

System

系统监控

bash
undefined
bash
undefined

Readiness

就绪检查

GET /ready
GET /ready

Configuration

配置查询

GET /config
GET /config

Metrics

指标查询

GET /metrics
undefined
GET /metrics
undefined

Troubleshooting

问题排查

Diagnostic Commands

诊断命令

bash
undefined
bash
undefined

Check pod status

检查Pod状态

kubectl get pods -n pyroscope -l app.kubernetes.io/name=pyroscope
kubectl get pods -n pyroscope -l app.kubernetes.io/name=pyroscope

View ingester logs

查看Ingester日志

kubectl logs -n pyroscope -l app.kubernetes.io/component=ingester --tail=100
kubectl logs -n pyroscope -l app.kubernetes.io/component=ingester --tail=100

Check ring status

检查Ring状态

kubectl exec -it pyroscope-0 -n pyroscope --
curl http://localhost:4040/ingester/ring
kubectl exec -it pyroscope-0 -n pyroscope --
curl http://localhost:4040/ingester/ring

Verify readiness

验证就绪状态

kubectl exec -it pyroscope-0 -n pyroscope --
curl http://localhost:4040/ready
kubectl exec -it pyroscope-0 -n pyroscope --
curl http://localhost:4040/ready

Check configuration

查看配置

kubectl exec -it pyroscope-0 -n pyroscope --
curl http://localhost:4040/config
undefined
kubectl exec -it pyroscope-0 -n pyroscope --
curl http://localhost:4040/config
undefined

Common Issues

常见问题

1. Ingester OOM:
yaml
ingester:
  resources:
    limits:
      memory: 16Gi
2. Storage Authentication Failed:
bash
undefined
1. Ingester内存溢出：
yaml
ingester:
  resources:
    limits:
      memory: 16Gi
2. 存储认证失败：
bash
undefined

Azure - verify RBAC

Azure - 验证RBAC权限

az role assignment create
--role "Storage Blob Data Contributor"
--assignee-object-id <principal-id>
--scope <storage-scope>

**3. High Cardinality Labels:**

```yaml
az role assignment create
--role "Storage Blob Data Contributor"
--assignee-object-id <principal-id>
--scope <storage-scope>

**3. 高基数标签：**

```yaml

Limit label cardinality

限制标签基数

pyroscope: config: validation: max_label_names_per_series: 25

**4. Query Timeout:**

```yaml
pyroscope:
  config:
    querier:
      query_timeout: 5m
      max_concurrent: 8
pyroscope: config: validation: max_label_names_per_series: 25

**4. 查询超时：**

```yaml
pyroscope:
  config:
    querier:
      query_timeout: 5m
      max_concurrent: 8

Reference Documentation

参考文档

For detailed configuration by topic:
  • Helm Deployment: Complete Helm values reference
  • Architecture: Component details and scaling
  • SDK Instrumentation: Language SDK guides
  • Troubleshooting: Common issues and diagnostics
按主题分类的详细配置说明：
  • Helm部署：完整Helm配置值参考
  • 架构：组件细节与扩展指南
  • SDK埋点：各语言SDK指南
  • 问题排查：常见问题与诊断方法

External Resources

外部资源