otel-logging-patterns
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChineseOpenTelemetry Logging Patterns
OpenTelemetry 日志模式
Table of Contents
目录
Purpose
用途
This skill provides production-grade OpenTelemetry logging patterns for Python applications. It covers:
- OTEL Logging Architecture: Provider and processor configuration
- Trace Correlation: Automatic trace context injection into logs
- Structured Logging: Integration with structlog for context preservation
- Log Exporters: OTLP, Jaeger, console, and file exporters
- Error Handling: Comprehensive exception and error logging
- Testing Patterns: Unit and integration tests for logging infrastructure
- Performance: Optimization tips to minimize logging overhead
This enables production observability where logs, traces, and metrics are correlated through trace IDs for efficient debugging and monitoring.
本方案为Python应用提供生产级别的OpenTelemetry日志实现模式,涵盖以下内容:
- OTEL日志架构:日志提供者与处理器配置
- 追踪关联:自动将追踪上下文注入日志
- 结构化日志:与structlog集成以保留上下文信息
- 日志导出器:OTLP、Jaeger、控制台及文件导出器
- 错误处理:全面的异常与错误日志记录
- 测试模式:日志基础设施的单元测试与集成测试
- 性能优化:减少日志开销的优化技巧
通过Trace ID关联日志、追踪与指标数据,实现生产环境的可观测性,从而提升调试与监控效率。
Quick Start
快速开始
For this project, use the authoritative OTEL logging module. Get started in 3 simple steps:
- Initialize OTEL at application startup (once):
python
from app.core.monitoring.otel_logger import initialize_otel_logger针对本项目,使用官方OTEL日志模块,通过3个简单步骤快速上手:
- 在应用启动时初始化OTEL(仅需执行一次):
python
from app.core.monitoring.otel_logger import initialize_otel_loggerCall once at app startup
应用启动时调用一次
initialize_otel_logger(
log_level="INFO",
enable_console=True,
enable_otlp=True,
otlp_endpoint="localhost:4317"
)
2. **Get logger and tracer in each module**:
```python
from app.core.monitoring.otel_logger import logger, get_tracerinitialize_otel_logger(
log_level="INFO",
enable_console=True,
enable_otlp=True,
otlp_endpoint="localhost:4317"
)
2. **在每个模块中获取日志器与追踪器**:
```python
from app.core.monitoring.otel_logger import logger, get_tracerModule-level initialization
模块级初始化
logger = logger(name)
tracer = get_tracer(name)
3. **Use trace_span for operations**:
```python
from app.core.monitoring.otel_logger import trace_span, logger
logger = logger(__name__)logger = logger(name)
tracer = get_tracer(name)
3. **使用trace_span封装操作**:
```python
from app.core.monitoring.otel_logger import trace_span, logger
logger = logger(__name__)Logs automatically include trace_id and span_id
日志将自动包含trace_id和span_id
with trace_span("process_order", order_id="12345") as span:
logger.info("processing_order", order_id="12345")
# Do work...
logger.info("order_processed", result_count=5)
That's it! All logs automatically include trace context, and spans are created with attributes.
**Key Benefits**:
- ✅ Single entry point: `app/core/monitoring/otel_logger.py`
- ✅ No direct imports of structlog or opentelemetry needed
- ✅ Automatic trace context propagation
- ✅ Automatic exception handling in spans
- ✅ Works with async and sync functionswith trace_span("process_order", order_id="12345") as span:
logger.info("processing_order", order_id="12345")
# 执行业务操作...
logger.info("order_processed", result_count=5)
完成以上步骤即可!所有日志将自动包含追踪上下文,并且Span会附带属性信息。
**核心优势**:
- ✅ 统一入口:`app/core/monitoring/otel_logger.py`
- ✅ 无需直接导入structlog或opentelemetry
- ✅ 自动追踪上下文传播
- ✅ Span中自动处理异常
- ✅ 支持异步与同步函数Instructions
操作步骤
Step 1: Configure OTEL Logging Provider
步骤1:配置OTEL日志提供者
Set up the core OpenTelemetry logging infrastructure with provider and exporter configuration.
Basic setup (console exporter for development):
python
undefined搭建核心OpenTelemetry日志基础设施,配置日志提供者与导出器。
基础配置(开发环境使用控制台导出器):
python
undefinedapp/shared/otel_config.py
app/shared/otel_config.py
from opentelemetry import logs
from opentelemetry.sdk.logs import LoggerProvider
from opentelemetry.sdk.logs.export import ConsoleLogExporter, SimpleLogRecordExporter
from opentelemetry.sdk.resources import Resource
def setup_console_logging(service_name: str) -> LoggerProvider:
"""Set up OTEL logging with console exporter."""
resource = Resource.create({"service.name": service_name})
logger_provider = LoggerProvider(resource=resource)
exporter = ConsoleLogExporter()
processor = SimpleLogRecordExporter(exporter)
logger_provider.add_log_record_processor(processor)
logs.set_logger_provider(logger_provider)
return logger_provider
**For production OTLP/Jaeger setup**, see [references/advanced-patterns.md](references/advanced-patterns.md#otelconfig-class)from opentelemetry import logs
from opentelemetry.sdk.logs import LoggerProvider
from opentelemetry.sdk.logs.export import ConsoleLogExporter, SimpleLogRecordExporter
from opentelemetry.sdk.resources import Resource
def setup_console_logging(service_name: str) -> LoggerProvider:
"""设置带有控制台导出器的OTEL日志系统。"""
resource = Resource.create({"service.name": service_name})
logger_provider = LoggerProvider(resource=resource)
exporter = ConsoleLogExporter()
processor = SimpleLogRecordExporter(exporter)
logger_provider.add_log_record_processor(processor)
logs.set_logger_provider(logger_provider)
return logger_provider
**生产环境OTLP/Jaeger配置**,请参考[references/advanced-patterns.md](references/advanced-patterns.md#otelconfig-class)Step 2: Integrate Structured Logging Library
步骤2:集成结构化日志库
Set up structlog with OTEL trace context integration:
python
undefined设置structlog并与OTEL追踪上下文集成:
python
undefinedapp/shared/logging_setup.py
app/shared/logging_setup.py
import structlog
from opentelemetry.instrumentation.logging import LoggingInstrumentor
def setup_structlog() -> None:
"""Configure structlog with OTEL trace context integration."""
# Enable OTEL logging instrumentation
LoggingInstrumentor().instrument()
# Configure structlog
structlog.configure(
processors=[
structlog.contextvars.merge_contextvars,
structlog.processors.TimeStamper(fmt="iso"),
structlog.processors.ExceptionRenderer(),
structlog.processors.JSONRenderer(),
],
logger_factory=structlog.logging.LoggerFactory(),
cache_logger_on_first_use=True,
)def get_logger(name: str):
"""Get a logger instance with trace context support."""
return structlog.logger(name)
**For advanced structlog configuration**, see [references/advanced-patterns.md](references/advanced-patterns.md#structlog-configuration)import structlog
from opentelemetry.instrumentation.logging import LoggingInstrumentor
def setup_structlog() -> None:
"""配置structlog并集成OTEL追踪上下文。"""
# 启用OTEL日志埋点
LoggingInstrumentor().instrument()
# 配置structlog
structlog.configure(
processors=[
structlog.contextvars.merge_contextvars,
structlog.processors.TimeStamper(fmt="iso"),
structlog.processors.ExceptionRenderer(),
structlog.processors.JSONRenderer(),
],
logger_factory=structlog.logging.LoggerFactory(),
cache_logger_on_first_use=True,
)def get_logger(name: str):
"""获取支持追踪上下文的日志器实例。"""
return structlog.logger(name)
**进阶structlog配置**,请参考[references/advanced-patterns.md](references/advanced-patterns.md#structlog-configuration)Step 3: Add Trace Context Propagation
步骤3:添加追踪上下文传播
Ensure trace context flows through logs automatically:
python
undefined确保追踪上下文自动在日志中传递:
python
undefinedapp/shared/observability.py
app/shared/observability.py
from contextvars import ContextVar
from opentelemetry import trace
from contextvars import ContextVar
from opentelemetry import trace
Context variables for request tracking
用于请求追踪的上下文变量
request_id_var: ContextVar[str | None] = ContextVar("request_id", default=None)
user_id_var: ContextVar[str | None] = ContextVar("user_id", default=None)
class ObservabilityContext:
"""Manage observability context (trace IDs, request IDs, user IDs)."""
@staticmethod
def set_request_id(request_id: str) -> None:
"""Set request ID in context."""
request_id_var.set(request_id)
@staticmethod
def get_tracer(name: str):
"""Get tracer instance."""
return trace.get_tracer(name)
@staticmethod
def set_span_attribute(key: str, value: any) -> None:
"""Set attribute on current span."""
span = trace.get_current_span()
if span.is_recording():
span.set_attribute(key, value)
**For complete ObservabilityContext class**, see [references/advanced-patterns.md](references/advanced-patterns.md#observability-context)request_id_var: ContextVar[str | None] = ContextVar("request_id", default=None)
user_id_var: ContextVar[str | None] = ContextVar("user_id", default=None)
class ObservabilityContext:
"""管理可观测性上下文(Trace ID、请求ID、用户ID)。"""
@staticmethod
def set_request_id(request_id: str) -> None:
"""在上下文中设置请求ID。"""
request_id_var.set(request_id)
@staticmethod
def get_tracer(name: str):
"""获取追踪器实例。"""
return trace.get_tracer(name)
@staticmethod
def set_span_attribute(key: str, value: any) -> None:
"""为当前Span设置属性。"""
span = trace.get_current_span()
if span.is_recording():
span.set_attribute(key, value)
**完整ObservabilityContext类**,请参考[references/advanced-patterns.md](references/advanced-patterns.md#observability-context)Step 4: Set Up Log Exporters
步骤4:设置日志导出器
Configure different exporters for different environments:
python
undefined针对不同环境配置不同的导出器:
python
undefinedapp/config.py
app/config.py
from pydantic import Field
from pydantic_settings import BaseSettings
class Config(BaseSettings):
"""Configuration for OTEL logging."""
otel_enabled: bool = Field(default=True)
otel_exporter_type: str = Field(default="console") # 'otlp', 'jaeger', or 'console'
otel_otlp_endpoint: str = Field(default="localhost:4317")
otel_jaeger_host: str = Field(default="localhost")
otel_jaeger_port: int = Field(default=6831)
Then use config in main:
```pythonfrom pydantic import Field
from pydantic_settings import BaseSettings
class Config(BaseSettings):
"""OTEL日志系统配置。"""
otel_enabled: bool = Field(default=True)
otel_exporter_type: str = Field(default="console") # 'otlp', 'jaeger', 或 'console'
otel_otlp_endpoint: str = Field(default="localhost:4317")
otel_jaeger_host: str = Field(default="localhost")
otel_jaeger_port: int = Field(default=6831)
然后在主程序中使用配置:
```pythonmain.py
main.py
from app.config import Config
from app.shared.logging_setup import setup_structlog, get_logger
from app.shared.otel_config import OTELConfig
async def main() -> None:
"""Main entry point."""
config = Config()
setup_structlog()
if config.otel_enabled:
otel_config = OTELConfig(
service_name="my-service",
exporter_type=config.otel_exporter_type,
otlp_endpoint=config.otel_otlp_endpoint,
)
otel_config.setup_logging()
otel_config.setup_tracing()
logger = get_logger(__name__)
logger.info("service_started")
**For complete exporter configuration**, see [references/advanced-patterns.md](references/advanced-patterns.md#exporter-configuration)from app.config import Config
from app.shared.logging_setup import setup_structlog, get_logger
from app.shared.otel_config import OTELConfig
async def main() -> None:
"""主入口函数。"""
config = Config()
setup_structlog()
if config.otel_enabled:
otel_config = OTELConfig(
service_name="my-service",
exporter_type=config.otel_exporter_type,
otlp_endpoint=config.otel_otlp_endpoint,
)
otel_config.setup_logging()
otel_config.setup_tracing()
logger = get_logger(__name__)
logger.info("service_started")
**完整导出器配置**,请参考[references/advanced-patterns.md](references/advanced-patterns.md#exporter-configuration)Step 5: Implement Instrumentation
步骤5:实现埋点
Add tracing and logging to key application flows:
python
undefined为关键业务流程添加追踪与日志:
python
undefinedapp/use_cases/extract_orders.py
app/use_cases/extract_orders.py
from opentelemetry import trace
from app.shared.logging_setup import get_logger
class ExtractOrdersUseCase:
"""Use case with observability."""
def __init__(self, gateway, publisher) -> None:
self.gateway = gateway
self.publisher = publisher
self.logger = get_logger(__name__)
self.tracer = trace.get_tracer(__name__)
async def execute(self) -> int:
"""Execute with tracing."""
with self.tracer.start_as_current_span("extract_orders") as span:
self.logger.info("starting_extraction")
try:
orders = await self.gateway.fetch_all_orders()
self.logger.info("orders_fetched", count=len(orders))
for order in orders:
await self.publisher.publish(order)
span.set_attribute("orders_processed", len(orders))
self.logger.info("extraction_completed", total=len(orders))
return len(orders)
except Exception as e:
self.logger.error("extraction_failed", error=str(e))
span.record_exception(e)
raise
**For advanced instrumentation patterns**, see [examples/examples.md](examples/examples.md#fastapi-instrumentation)from opentelemetry import trace
from app.shared.logging_setup import get_logger
class ExtractOrdersUseCase:
"""带有可观测性的用例。"""
def __init__(self, gateway, publisher) -> None:
self.gateway = gateway
self.publisher = publisher
self.logger = get_logger(__name__)
self.tracer = trace.get_tracer(__name__)
async def execute(self) -> int:
"""通过追踪执行用例。"""
with self.tracer.start_as_current_span("extract_orders") as span:
self.logger.info("starting_extraction")
try:
orders = await self.gateway.fetch_all_orders()
self.logger.info("orders_fetched", count=len(orders))
for order in orders:
await self.publisher.publish(order)
span.set_attribute("orders_processed", len(orders))
self.logger.info("extraction_completed", total=len(orders))
return len(orders)
except Exception as e:
self.logger.error("extraction_failed", error=str(e))
span.record_exception(e)
raise
**进阶埋点模式**,请参考[examples/examples.md](examples/examples.md#fastapi-instrumentation)Step 6: Add Error and Exception Logging
步骤6:添加错误与异常日志
Implement comprehensive error logging with context:
python
undefined实现包含上下文信息的全面错误日志:
python
undefinedapp/error_handlers.py
app/error_handlers.py
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
from app.shared.logging_setup import get_logger
from app.shared.observability import ObservabilityContext
logger = get_logger(name)
async def global_exception_handler(request: Request, exc: Exception) -> JSONResponse:
"""Global exception handler with OTEL logging."""
request_id = ObservabilityContext.get_request_id()
logger.error(
"unhandled_exception",
error=str(exc),
error_type=type(exc).__name__,
request_id=request_id,
path=request.url.path,
method=request.method,
exc_info=True, # Include full stack trace
)
return JSONResponse(
status_code=500,
content={"detail": "Internal server error", "request_id": request_id},
)def setup_error_handlers(app: FastAPI) -> None:
"""Register error handlers."""
app.add_exception_handler(Exception, global_exception_handler)
**For complete error handling patterns**, see [examples/examples.md](examples/examples.md#error-handling)from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
from app.shared.logging_setup import get_logger
from app.shared.observability import ObservabilityContext
logger = get_logger(name)
async def global_exception_handler(request: Request, exc: Exception) -> JSONResponse:
"""带有OTEL日志的全局异常处理器。"""
request_id = ObservabilityContext.get_request_id()
logger.error(
"unhandled_exception",
error=str(exc),
error_type=type(exc).__name__,
request_id=request_id,
path=request.url.path,
method=request.method,
exc_info=True, # 包含完整堆栈跟踪
)
return JSONResponse(
status_code=500,
content={"detail": "内部服务器错误", "request_id": request_id},
)def setup_error_handlers(app: FastAPI) -> None:
"""注册错误处理器。"""
app.add_exception_handler(Exception, global_exception_handler)
**完整错误处理模式**,请参考[examples/examples.md](examples/examples.md#error-handling)Requirements
依赖要求
Production OTEL logging requires:
- - OTEL API
opentelemetry-api>=1.22.0 - - OTEL SDK with logging support
opentelemetry-sdk>=1.22.0 - - OTLP exporter for gRPC
opentelemetry-exporter-otlp>=0.43b0 - - Logging instrumentation
opentelemetry-instrumentation-logging>=0.43b0 - - Structured logging library
structlog>=23.2.0 - - Configuration management
pydantic>=2.5.0 - Python 3.11+ with type checking
生产环境OTEL日志系统需要以下依赖:
- - OTEL API
opentelemetry-api>=1.22.0 - - 带日志支持的OTEL SDK
opentelemetry-sdk>=1.22.0 - - gRPC协议的OTLP导出器
opentelemetry-exporter-otlp>=0.43b0 - - 日志埋点工具
opentelemetry-instrumentation-logging>=0.43b0 - - 结构化日志库
structlog>=23.2.0 - - 配置管理工具
pydantic>=2.5.0 - Python 3.11+ 并支持类型检查
Common Patterns
通用模式
Trace Context Propagation
追踪上下文传播
Logs automatically include trace_id and span_id when logged within a span:
python
from opentelemetry import trace
from app.shared.logging_setup import get_logger
logger = get_logger(__name__)
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("process_payment"):
logger.info("payment_started", customer_id="cust-123")
logger.info("payment_completed", amount=99.99)
# All logs have same trace_id and span_id在Span范围内记录的日志将自动包含trace_id和span_id:
python
from opentelemetry import trace
from app.shared.logging_setup import get_logger
logger = get_logger(__name__)
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("process_payment"):
logger.info("payment_started", customer_id="cust-123")
logger.info("payment_completed", amount=99.99)
# 所有日志将包含相同的trace_id和span_idContext Variables
上下文变量
Use context variables for request-scoped data:
python
from app.shared.observability import ObservabilityContext
ObservabilityContext.set_request_id("req-12345")
ObservabilityContext.set_user_id("user-456")
logger.info("user_action", action="login")使用上下文变量存储请求范围的数据:
python
from app.shared.observability import ObservabilityContext
ObservabilityContext.set_request_id("req-12345")
ObservabilityContext.set_user_id("user-456")
logger.info("user_action", action="login")Log includes request_id and user_id automatically
日志将自动包含request_id和user_id
**For complete pattern examples**, see [examples/examples.md](examples/examples.md)
**完整模式示例**,请参考[examples/examples.md](examples/examples.md)Testing OTEL Logging
OTEL日志测试
Unit Testing
单元测试
Test logging configuration without external dependencies:
python
import pytest
from opentelemetry import logs
from opentelemetry.sdk.logs import LoggerProvider
def test_logging_configuration():
"""Test OTEL logging provider setup."""
logger_provider = LoggerProvider()
assert logger_provider is not None
logs.set_logger_provider(logger_provider)
assert logs.get_logger_provider() == logger_provider在无外部依赖的情况下测试日志配置:
python
import pytest
from opentelemetry import logs
from opentelemetry.sdk.logs import LoggerProvider
def test_logging_configuration():
"""测试OTEL日志提供者的设置。"""
logger_provider = LoggerProvider()
assert logger_provider is not None
logs.set_logger_provider(logger_provider)
assert logs.get_logger_provider() == logger_providerIntegration Testing
集成测试
Test log export with real exporters:
python
@pytest.fixture
def in_memory_log_exporter():
"""In-memory exporter for testing."""
class InMemoryExporter:
def __init__(self):
self.records = []
def emit(self, log_records):
self.records.extend(log_records)
return InMemoryExporter()
def test_trace_context_in_logs(in_memory_log_exporter):
"""Verify trace context is included in logs."""
# Setup logger and tracer
# Log within span
# Verify trace context in exported logs
passFor comprehensive testing patterns, see examples/examples.md
使用真实导出器测试日志导出:
python
@pytest.fixture
def in_memory_log_exporter():
"""用于测试的内存导出器。"""
class InMemoryExporter:
def __init__(self):
self.records = []
def emit(self, log_records):
self.records.extend(log_records)
return InMemoryExporter()
def test_trace_context_in_logs(in_memory_log_exporter):
"""验证日志中包含追踪上下文。"""
# 配置日志器与追踪器
# 在Span范围内记录日志
# 验证导出的日志中包含追踪上下文
pass全面测试模式,请参考examples/examples.md
Troubleshooting
故障排查
Common Issues and Solutions
常见问题与解决方案
Issue: Trace IDs not appearing in logs
- Ensure is called before logging
LoggingInstrumentor().instrument() - Verify logs are being created within a span context
- Check that OTEL trace provider is set before creating loggers
Issue: OTLP exporter connection refused
- Verify OTEL Collector is running on specified endpoint
- Check endpoint configuration (default: localhost:4317)
- Use console exporter in development
- Check firewall rules for gRPC port 4317
Issue: High logging overhead
- Use instead of
BatchLogRecordExporterSimpleLogRecordExporter - Increase batch size for fewer exports (e.g., 512 records)
- Filter verbose DEBUG logs in production
Issue: Logs not appearing in output
- Verify logger is initialized with
setup_structlog() - Check Python logging level (INFO, WARNING, ERROR)
- Verify structlog configuration has JSONRenderer processor
For detailed troubleshooting, see references/advanced-patterns.md
问题:Trace ID未出现在日志中
- 确保在记录日志前调用了
LoggingInstrumentor().instrument() - 验证日志是在Span上下文内创建的
- 检查OTEL追踪提供者是否在创建日志器之前已设置
问题:OTLP导出器连接被拒绝
- 验证OTEL Collector是否在指定端点运行
- 检查端点配置(默认:localhost:4317)
- 开发环境使用控制台导出器
- 检查gRPC端口4317的防火墙规则
问题:日志开销过高
- 使用替代
BatchLogRecordExporterSimpleLogRecordExporter - 增加批量大小以减少导出次数(例如:512条记录)
- 生产环境过滤冗余的DEBUG日志
问题:日志未显示在输出中
- 验证已通过初始化日志器
setup_structlog() - 检查Python日志级别(INFO、WARNING、ERROR)
- 验证structlog配置包含JSONRenderer处理器
详细故障排查指南,请参考references/advanced-patterns.md
Supporting Resources
参考资源
| Resource | Purpose |
|---|---|
| 10+ complete examples covering FastAPI, Kafka, background jobs, testing |
| Performance tuning, custom exporters, trace propagation details |
| 资源 | 用途 |
|---|---|
| 10+个完整示例,涵盖FastAPI、Kafka、后台任务、测试等场景 |
| 性能调优、自定义导出器、追踪传播细节 |