log-analyzer

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Log Analyzer Skill

日志分析技能

Overview

概述

This skill helps you effectively analyze application logs to diagnose issues, track errors, and understand system behavior. Covers log searching, pattern detection, structured logging, and integration with monitoring tools.
本技能可帮助你高效分析应用日志,以诊断问题、追踪错误并了解系统行为。涵盖日志搜索、模式检测、结构化日志记录以及与监控工具的集成。

Log Analysis Philosophy

日志分析理念

Principles

原则

  1. Structure over text: Structured logs are easier to analyze
  2. Context matters: Include relevant metadata
  3. Levels have meaning: Use appropriate severity levels
  4. Correlation is key: Link related events across services
  1. 结构化优先于文本:结构化日志更易于分析
  2. 上下文至关重要:包含相关元数据
  3. 日志级别具有意义:使用合适的严重级别
  4. 关联是关键:跨服务关联相关事件

Log Levels

日志级别

LevelWhen to UseExample
ERRORSomething failed, needs attentionDatabase connection failed
WARNUnexpected but handledRetry succeeded after failure
INFONormal operation milestonesUser signed up
DEBUGDetailed diagnostic infoQuery executed in 50ms
TRACEVery detailed, usually disabledFunction entered/exited
级别使用场景示例
ERROR发生故障,需要关注数据库连接失败
WARN意外但已处理重试失败后成功
INFO正常运行里程碑用户注册成功
DEBUG详细诊断信息查询执行耗时50ms
TRACE极详细信息,通常禁用函数进入/退出

Structured Logging

结构化日志

Winston Configuration (Node.js)

Winston配置(Node.js)

typescript
// src/lib/logger.ts
import winston from 'winston';

const logger = winston.createLogger({
  level: process.env.LOG_LEVEL || 'info',
  format: winston.format.combine(
    winston.format.timestamp(),
    winston.format.errors({ stack: true }),
    winston.format.json()
  ),
  defaultMeta: {
    service: 'my-app',
    environment: process.env.NODE_ENV,
  },
  transports: [
    // Console for development
    new winston.transports.Console({
      format: process.env.NODE_ENV === 'development'
        ? winston.format.combine(
            winston.format.colorize(),
            winston.format.simple()
          )
        : winston.format.json(),
    }),
  ],
});

// Add request context
export function createRequestLogger(requestId: string, userId?: string) {
  return logger.child({
    requestId,
    userId,
  });
}

export { logger };
typescript
// src/lib/logger.ts
import winston from 'winston';

const logger = winston.createLogger({
  level: process.env.LOG_LEVEL || 'info',
  format: winston.format.combine(
    winston.format.timestamp(),
    winston.format.errors({ stack: true }),
    winston.format.json()
  ),
  defaultMeta: {
    service: 'my-app',
    environment: process.env.NODE_ENV,
  },
  transports: [
    // Console for development
    new winston.transports.Console({
      format: process.env.NODE_ENV === 'development'
        ? winston.format.combine(
            winston.format.colorize(),
            winston.format.simple()
          )
        : winston.format.json(),
    }),
  ],
});

// Add request context
export function createRequestLogger(requestId: string, userId?: string) {
  return logger.child({
    requestId,
    userId,
  });
}

export { logger };

Pino Logger (High Performance)

Pino Logger(高性能)

typescript
// src/lib/logger.ts
import pino from 'pino';

export const logger = pino({
  level: process.env.LOG_LEVEL || 'info',
  transport: process.env.NODE_ENV === 'development'
    ? {
        target: 'pino-pretty',
        options: {
          colorize: true,
          translateTime: 'SYS:standard',
        },
      }
    : undefined,
  base: {
    service: 'my-app',
    env: process.env.NODE_ENV,
  },
  redact: {
    paths: ['password', 'token', 'apiKey', '*.password', '*.token'],
    censor: '[REDACTED]',
  },
});

// Request-scoped logger
export function requestLogger(requestId: string, userId?: string) {
  return logger.child({ requestId, userId });
}
typescript
// src/lib/logger.ts
import pino from 'pino';

export const logger = pino({
  level: process.env.LOG_LEVEL || 'info',
  transport: process.env.NODE_ENV === 'development'
    ? {
        target: 'pino-pretty',
        options: {
          colorize: true,
          translateTime: 'SYS:standard',
        },
      }
    : undefined,
  base: {
    service: 'my-app',
    env: process.env.NODE_ENV,
  },
  redact: {
    paths: ['password', 'token', 'apiKey', '*.password', '*.token'],
    censor: '[REDACTED]',
  },
});

// Request-scoped logger
export function requestLogger(requestId: string, userId?: string) {
  return logger.child({ requestId, userId });
}

Logging Best Practices

日志记录最佳实践

typescript
// DO: Include context
logger.info('User signed up', {
  userId: user.id,
  email: user.email,
  plan: 'free',
  source: 'web',
});

// DO: Log errors with stack traces
logger.error('Payment failed', {
  error: err.message,
  stack: err.stack,
  userId: user.id,
  amount: 99.99,
  provider: 'stripe',
});

// DO: Use appropriate levels
logger.debug('Database query', {
  query: 'SELECT * FROM users WHERE id = ?',
  duration: 45,
  rows: 1,
});

// DON'T: Log sensitive data
// BAD: logger.info('Login', { password: user.password })

// DON'T: Use string concatenation
// BAD: logger.info('User ' + user.id + ' signed up')
typescript
// DO: Include context
logger.info('User signed up', {
  userId: user.id,
  email: user.email,
  plan: 'free',
  source: 'web',
});

// DO: Log errors with stack traces
logger.error('Payment failed', {
  error: err.message,
  stack: err.stack,
  userId: user.id,
  amount: 99.99,
  provider: 'stripe',
});

// DO: Use appropriate levels
logger.debug('Database query', {
  query: 'SELECT * FROM users WHERE id = ?',
  duration: 45,
  rows: 1,
});

// DON'T: Log sensitive data
// BAD: logger.info('Login', { password: user.password })

// DON'T: Use string concatenation
// BAD: logger.info('User ' + user.id + ' signed up')

Log Searching & Filtering

日志搜索与过滤

Command Line (grep, jq)

命令行工具(grep、jq)

bash
undefined
bash
undefined

Search JSON logs with jq

Search JSON logs with jq

cat logs.json | jq 'select(.level == "error")'
cat logs.json | jq 'select(.level == "error")'

Search for specific user

Search for specific user

cat logs.json | jq 'select(.userId == "usr_123")'
cat logs.json | jq 'select(.userId == "usr_123")'

Search by time range

Search by time range

cat logs.json | jq 'select(.timestamp >= "2024-01-15T10:00:00")'
cat logs.json | jq 'select(.timestamp >= "2024-01-15T10:00:00")'

Count errors by type

Count errors by type

cat logs.json | jq 'select(.level == "error") | .error.code' | sort | uniq -c
cat logs.json | jq 'select(.level == "error") | .error.code' | sort | uniq -c

Extract specific fields

Extract specific fields

cat logs.json | jq '{timestamp, level, message, userId}'
cat logs.json | jq '{timestamp, level, message, userId}'

Search text logs

Search text logs

grep -i "error" logs.txt grep -E "error|warn" logs.txt grep -A5 "ERROR" logs.txt # 5 lines after match grep -B3 "ERROR" logs.txt # 3 lines before match
undefined
grep -i "error" logs.txt grep -E "error|warn" logs.txt grep -A5 "ERROR" logs.txt # 5 lines after match grep -B3 "ERROR" logs.txt # 3 lines before match
undefined

Vercel Log Analysis

Vercel日志分析

bash
undefined
bash
undefined

View live logs

View live logs

vercel logs --follow
vercel logs --follow

Filter by level

Filter by level

vercel logs --level error
vercel logs --level error

Search specific timeframe

Search specific timeframe

vercel logs --since 2h
vercel logs --since 2h

Filter by deployment

Filter by deployment

vercel logs --deployment-url https://myapp-abc123.vercel.app
vercel logs --deployment-url https://myapp-abc123.vercel.app

Output JSON for processing

Output JSON for processing

vercel logs --output json | jq 'select(.level == "error")'
undefined
vercel logs --output json | jq 'select(.level == "error")'
undefined

Supabase Log Analysis

Supabase日志分析

sql
-- Query Postgres logs
SELECT *
FROM postgres_logs
WHERE timestamp > now() - interval '1 hour'
  AND error_severity = 'ERROR'
ORDER BY timestamp DESC
LIMIT 100;

-- Query auth logs
SELECT *
FROM auth.audit_log_entries
WHERE timestamp > now() - interval '24 hours'
  AND payload->>'action' = 'login'
ORDER BY timestamp DESC;

-- Find slow queries
SELECT
  query,
  calls,
  mean_exec_time,
  total_exec_time
FROM pg_stat_statements
ORDER BY mean_exec_time DESC
LIMIT 20;
sql
-- Query Postgres logs
SELECT *
FROM postgres_logs
WHERE timestamp > now() - interval '1 hour'
  AND error_severity = 'ERROR'
ORDER BY timestamp DESC
LIMIT 100;

-- Query auth logs
SELECT *
FROM auth.audit_log_entries
WHERE timestamp > now() - interval '24 hours'
  AND payload->>'action' = 'login'
ORDER BY timestamp DESC;

-- Find slow queries
SELECT
  query,
  calls,
  mean_exec_time,
  total_exec_time
FROM pg_stat_statements
ORDER BY mean_exec_time DESC
LIMIT 20;

Pattern Detection

模式检测

Common Error Patterns

常见错误模式

typescript
// Error pattern analyzer
interface ErrorPattern {
  pattern: RegExp;
  category: string;
  severity: 'critical' | 'high' | 'medium' | 'low';
  suggestion: string;
}

const errorPatterns: ErrorPattern[] = [
  {
    pattern: /ECONNREFUSED|connection refused/i,
    category: 'connectivity',
    severity: 'critical',
    suggestion: 'Check if the target service is running and accessible',
  },
  {
    pattern: /timeout|ETIMEDOUT/i,
    category: 'timeout',
    severity: 'high',
    suggestion: 'Increase timeout or check for slow operations',
  },
  {
    pattern: /out of memory|heap|OOM/i,
    category: 'memory',
    severity: 'critical',
    suggestion: 'Increase memory allocation or fix memory leak',
  },
  {
    pattern: /rate.?limit|429|too many requests/i,
    category: 'rate-limiting',
    severity: 'medium',
    suggestion: 'Implement backoff strategy or increase rate limits',
  },
  {
    pattern: /unauthorized|401|invalid.?token/i,
    category: 'auth',
    severity: 'medium',
    suggestion: 'Check authentication credentials or token expiry',
  },
  {
    pattern: /not.?found|404/i,
    category: 'not-found',
    severity: 'low',
    suggestion: 'Verify resource exists or check URL',
  },
];

function categorizeError(message: string): ErrorPattern | null {
  for (const pattern of errorPatterns) {
    if (pattern.pattern.test(message)) {
      return pattern;
    }
  }
  return null;
}
typescript
// Error pattern analyzer
interface ErrorPattern {
  pattern: RegExp;
  category: string;
  severity: 'critical' | 'high' | 'medium' | 'low';
  suggestion: string;
}

const errorPatterns: ErrorPattern[] = [
  {
    pattern: /ECONNREFUSED|connection refused/i,
    category: 'connectivity',
    severity: 'critical',
    suggestion: 'Check if the target service is running and accessible',
  },
  {
    pattern: /timeout|ETIMEDOUT/i,
    category: 'timeout',
    severity: 'high',
    suggestion: 'Increase timeout or check for slow operations',
  },
  {
    pattern: /out of memory|heap|OOM/i,
    category: 'memory',
    severity: 'critical',
    suggestion: 'Increase memory allocation or fix memory leak',
  },
  {
    pattern: /rate.?limit|429|too many requests/i,
    category: 'rate-limiting',
    severity: 'medium',
    suggestion: 'Implement backoff strategy or increase rate limits',
  },
  {
    pattern: /unauthorized|401|invalid.?token/i,
    category: 'auth',
    severity: 'medium',
    suggestion: 'Check authentication credentials or token expiry',
  },
  {
    pattern: /not.?found|404/i,
    category: 'not-found',
    severity: 'low',
    suggestion: 'Verify resource exists or check URL',
  },
];

function categorizeError(message: string): ErrorPattern | null {
  for (const pattern of errorPatterns) {
    if (pattern.pattern.test(message)) {
      return pattern;
    }
  }
  return null;
}

Log Aggregation Script

日志聚合脚本

typescript
// scripts/analyze-logs.ts
import * as readline from 'readline';
import * as fs from 'fs';

interface LogEntry {
  timestamp: string;
  level: string;
  message: string;
  error?: {
    message: string;
    code?: string;
  };
  [key: string]: any;
}

interface LogSummary {
  totalEntries: number;
  byLevel: Record<string, number>;
  errorMessages: Record<string, number>;
  timeRange: {
    start: string;
    end: string;
  };
}

async function analyzeLogs(filePath: string): Promise<LogSummary> {
  const summary: LogSummary = {
    totalEntries: 0,
    byLevel: {},
    errorMessages: {},
    timeRange: { start: '', end: '' },
  };

  const fileStream = fs.createReadStream(filePath);
  const rl = readline.createInterface({
    input: fileStream,
    crlfDelay: Infinity,
  });

  for await (const line of rl) {
    try {
      const entry: LogEntry = JSON.parse(line);
      summary.totalEntries++;

      // Count by level
      summary.byLevel[entry.level] = (summary.byLevel[entry.level] || 0) + 1;

      // Track error messages
      if (entry.level === 'error' && entry.error?.message) {
        const msg = entry.error.message.substring(0, 100);
        summary.errorMessages[msg] = (summary.errorMessages[msg] || 0) + 1;
      }

      // Track time range
      if (!summary.timeRange.start || entry.timestamp < summary.timeRange.start) {
        summary.timeRange.start = entry.timestamp;
      }
      if (!summary.timeRange.end || entry.timestamp > summary.timeRange.end) {
        summary.timeRange.end = entry.timestamp;
      }
    } catch {
      // Skip non-JSON lines
    }
  }

  return summary;
}

// Usage
analyzeLogs('logs.json').then(console.log);
typescript
// scripts/analyze-logs.ts
import * as readline from 'readline';
import * as fs from 'fs';

interface LogEntry {
  timestamp: string;
  level: string;
  message: string;
  error?: {
    message: string;
    code?: string;
  };
  [key: string]: any;
}

interface LogSummary {
  totalEntries: number;
  byLevel: Record<string, number>;
  errorMessages: Record<string, number>;
  timeRange: {
    start: string;
    end: string;
  };
}

async function analyzeLogs(filePath: string): Promise<LogSummary> {
  const summary: LogSummary = {
    totalEntries: 0,
    byLevel: {},
    errorMessages: {},
    timeRange: { start: '', end: '' },
  };

  const fileStream = fs.createReadStream(filePath);
  const rl = readline.createInterface({
    input: fileStream,
    crlfDelay: Infinity,
  });

  for await (const line of rl) {
    try {
      const entry: LogEntry = JSON.parse(line);
      summary.totalEntries++;

      // Count by level
      summary.byLevel[entry.level] = (summary.byLevel[entry.level] || 0) + 1;

      // Track error messages
      if (entry.level === 'error' && entry.error?.message) {
        const msg = entry.error.message.substring(0, 100);
        summary.errorMessages[msg] = (summary.errorMessages[msg] || 0) + 1;
      }

      // Track time range
      if (!summary.timeRange.start || entry.timestamp < summary.timeRange.start) {
        summary.timeRange.start = entry.timestamp;
      }
      if (!summary.timeRange.end || entry.timestamp > summary.timeRange.end) {
        summary.timeRange.end = entry.timestamp;
      }
    } catch {
      // Skip non-JSON lines
    }
  }

  return summary;
}

// Usage
analyzeLogs('logs.json').then(console.log);

Request Tracing

请求追踪

Request ID Middleware

请求ID中间件

typescript
// src/middleware.ts
import { NextResponse } from 'next/server';
import type { NextRequest } from 'next/server';
import { v4 as uuidv4 } from 'uuid';

export function middleware(request: NextRequest) {
  const requestId = request.headers.get('x-request-id') || uuidv4();

  const response = NextResponse.next();
  response.headers.set('x-request-id', requestId);

  return response;
}
typescript
// src/middleware.ts
import { NextResponse } from 'next/server';
import type { NextRequest } from 'next/server';
import { v4 as uuidv4 } from 'uuid';

export function middleware(request: NextRequest) {
  const requestId = request.headers.get('x-request-id') || uuidv4();

  const response = NextResponse.next();
  response.headers.set('x-request-id', requestId);

  return response;
}

Tracing Context

追踪上下文

typescript
// src/lib/tracing.ts
import { AsyncLocalStorage } from 'async_hooks';

interface TraceContext {
  requestId: string;
  userId?: string;
  startTime: number;
}

const asyncLocalStorage = new AsyncLocalStorage<TraceContext>();

export function withTrace<T>(
  context: TraceContext,
  fn: () => T | Promise<T>
): T | Promise<T> {
  return asyncLocalStorage.run(context, fn);
}

export function getTrace(): TraceContext | undefined {
  return asyncLocalStorage.getStore();
}

// Usage in logger
export function log(level: string, message: string, data?: object) {
  const trace = getTrace();
  console.log(JSON.stringify({
    timestamp: new Date().toISOString(),
    level,
    message,
    requestId: trace?.requestId,
    userId: trace?.userId,
    duration: trace ? Date.now() - trace.startTime : undefined,
    ...data,
  }));
}
typescript
// src/lib/tracing.ts
import { AsyncLocalStorage } from 'async_hooks';

interface TraceContext {
  requestId: string;
  userId?: string;
  startTime: number;
}

const asyncLocalStorage = new AsyncLocalStorage<TraceContext>();

export function withTrace<T>(
  context: TraceContext,
  fn: () => T | Promise<T>
): T | Promise<T> {
  return asyncLocalStorage.run(context, fn);
}

export function getTrace(): TraceContext | undefined {
  return asyncLocalStorage.getStore();
}

// Usage in logger
export function log(level: string, message: string, data?: object) {
  const trace = getTrace();
  console.log(JSON.stringify({
    timestamp: new Date().toISOString(),
    level,
    message,
    requestId: trace?.requestId,
    userId: trace?.userId,
    duration: trace ? Date.now() - trace.startTime : undefined,
    ...data,
  }));
}

Error Tracking Integration

错误追踪集成

Sentry Integration

Sentry集成

typescript
// src/lib/sentry.ts
import * as Sentry from '@sentry/nextjs';

Sentry.init({
  dsn: process.env.SENTRY_DSN,
  environment: process.env.NODE_ENV,
  tracesSampleRate: 0.1, // 10% of transactions
  beforeSend(event, hint) {
    // Filter out specific errors
    const error = hint.originalException;
    if (error instanceof Error && error.message.includes('Network')) {
      return null;
    }
    return event;
  },
});

// Capture with context
export function captureError(error: Error, context?: Record<string, any>) {
  Sentry.withScope((scope) => {
    if (context) {
      scope.setExtras(context);
    }
    Sentry.captureException(error);
  });
}

// Usage
try {
  await riskyOperation();
} catch (error) {
  captureError(error, {
    userId: user.id,
    action: 'payment',
    amount: 99.99,
  });
  throw error;
}
typescript
// src/lib/sentry.ts
import * as Sentry from '@sentry/nextjs';

Sentry.init({
  dsn: process.env.SENTRY_DSN,
  environment: process.env.NODE_ENV,
  tracesSampleRate: 0.1, // 10% of transactions
  beforeSend(event, hint) {
    // Filter out specific errors
    const error = hint.originalException;
    if (error instanceof Error && error.message.includes('Network')) {
      return null;
    }
    return event;
  },
});

// Capture with context
export function captureError(error: Error, context?: Record<string, any>) {
  Sentry.withScope((scope) => {
    if (context) {
      scope.setExtras(context);
    }
    Sentry.captureException(error);
  });
}

// Usage
try {
  await riskyOperation();
} catch (error) {
  captureError(error, {
    userId: user.id,
    action: 'payment',
    amount: 99.99,
  });
  throw error;
}

Custom Error Boundary Logging

自定义错误边界日志

typescript
// src/components/error-boundary.tsx
'use client';

import { useEffect } from 'react';
import * as Sentry from '@sentry/nextjs';

export default function Error({
  error,
  reset,
}: {
  error: Error & { digest?: string };
  reset: () => void;
}) {
  useEffect(() => {
    // Log to error tracking
    Sentry.captureException(error);

    // Log to console with context
    console.error('Client Error:', {
      message: error.message,
      digest: error.digest,
      stack: error.stack,
      url: window.location.href,
      userAgent: navigator.userAgent,
    });
  }, [error]);

  return (
    <div>
      <h2>Something went wrong!</h2>
      <button onClick={reset}>Try again</button>
    </div>
  );
}
typescript
// src/components/error-boundary.tsx
'use client';

import { useEffect } from 'react';
import * as Sentry from '@sentry/nextjs';

export default function Error({
  error,
  reset,
}: {
  error: Error & { digest?: string };
  reset: () => void;
}) {
  useEffect(() => {
    // Log to error tracking
    Sentry.captureException(error);

    // Log to console with context
    console.error('Client Error:', {
      message: error.message,
      digest: error.digest,
      stack: error.stack,
      url: window.location.href,
      userAgent: navigator.userAgent,
    });
  }, [error]);

  return (
    <div>
      <h2>Something went wrong!</h2>
      <button onClick={reset}>Try again</button>
    </div>
  );
}

Log Analysis Queries

日志分析查询

Common Investigation Queries

常见排查查询

bash
undefined
bash
undefined

Find all errors for a specific user

Find all errors for a specific user

jq 'select(.userId == "usr_123" and .level == "error")' logs.json
jq 'select(.userId == "usr_123" and .level == "error")' logs.json

Find errors in the last hour

Find errors in the last hour

jq --arg time "$(date -d '1 hour ago' -Iseconds)"
'select(.timestamp > $time and .level == "error")' logs.json
jq --arg time "$(date -d '1 hour ago' -Iseconds)"
'select(.timestamp > $time and .level == "error")' logs.json

Group errors by message

Group errors by message

jq -s 'group_by(.error.message) | map({message: .[0].error.message, count: length}) | sort_by(-.count)' logs.json
jq -s 'group_by(.error.message) | map({message: .[0].error.message, count: length}) | sort_by(-.count)' logs.json

Find slow requests (>1 second)

Find slow requests (>1 second)

jq 'select(.duration > 1000)' logs.json
jq 'select(.duration > 1000)' logs.json

Trace a specific request

Trace a specific request

jq 'select(.requestId == "req_abc123")' logs.json | sort_by(.timestamp)
jq 'select(.requestId == "req_abc123")' logs.json | sort_by(.timestamp)

Find patterns in error messages

Find patterns in error messages

jq -r 'select(.level == "error") | .error.message' logs.json |
sort | uniq -c | sort -rn | head -20
undefined
jq -r 'select(.level == "error") | .error.message' logs.json |
sort | uniq -c | sort -rn | head -20
undefined

SQL-Style Log Analysis

SQL风格日志分析

sql
-- Using tools like clickhouse or DuckDB for log analysis

-- Error rate by hour
SELECT
  date_trunc('hour', timestamp) as hour,
  count(*) FILTER (WHERE level = 'error') as errors,
  count(*) as total,
  round(100.0 * count(*) FILTER (WHERE level = 'error') / count(*), 2) as error_rate
FROM logs
WHERE timestamp > now() - interval '24 hours'
GROUP BY 1
ORDER BY 1;

-- Top error messages
SELECT
  error_message,
  count(*) as occurrences,
  min(timestamp) as first_seen,
  max(timestamp) as last_seen
FROM logs
WHERE level = 'error'
  AND timestamp > now() - interval '24 hours'
GROUP BY error_message
ORDER BY occurrences DESC
LIMIT 10;

-- Slowest endpoints
SELECT
  path,
  count(*) as requests,
  avg(duration) as avg_duration,
  max(duration) as max_duration,
  percentile_cont(0.95) WITHIN GROUP (ORDER BY duration) as p95
FROM logs
WHERE timestamp > now() - interval '1 hour'
GROUP BY path
ORDER BY avg_duration DESC
LIMIT 10;
sql
-- Using tools like clickhouse or DuckDB for log analysis

-- Error rate by hour
SELECT
  date_trunc('hour', timestamp) as hour,
  count(*) FILTER (WHERE level = 'error') as errors,
  count(*) as total,
  round(100.0 * count(*) FILTER (WHERE level = 'error') / count(*), 2) as error_rate
FROM logs
WHERE timestamp > now() - interval '24 hours'
GROUP BY 1
ORDER BY 1;

-- Top error messages
SELECT
  error_message,
  count(*) as occurrences,
  min(timestamp) as first_seen,
  max(timestamp) as last_seen
FROM logs
WHERE level = 'error'
  AND timestamp > now() - interval '24 hours'
GROUP BY error_message
ORDER BY occurrences DESC
LIMIT 10;

-- Slowest endpoints
SELECT
  path,
  count(*) as requests,
  avg(duration) as avg_duration,
  max(duration) as max_duration,
  percentile_cont(0.95) WITHIN GROUP (ORDER BY duration) as p95
FROM logs
WHERE timestamp > now() - interval '1 hour'
GROUP BY path
ORDER BY avg_duration DESC
LIMIT 10;

Log Retention & Cleanup

日志保留与清理

Log Rotation Configuration

日志轮转配置

bash
undefined
bash
undefined

/etc/logrotate.d/myapp

/etc/logrotate.d/myapp

/var/log/myapp/*.log { daily rotate 14 compress delaycompress missingok notifempty create 0640 www-data www-data sharedscripts postrotate systemctl reload myapp endscript }
undefined
/var/log/myapp/*.log { daily rotate 14 compress delaycompress missingok notifempty create 0640 www-data www-data sharedscripts postrotate systemctl reload myapp endscript }
undefined

Automated Log Cleanup

自动化日志清理

typescript
// scripts/cleanup-logs.ts
import * as fs from 'fs';
import * as path from 'path';

const LOG_DIR = '/var/log/myapp';
const RETENTION_DAYS = 30;

function cleanupOldLogs() {
  const now = Date.now();
  const maxAge = RETENTION_DAYS * 24 * 60 * 60 * 1000;

  const files = fs.readdirSync(LOG_DIR);

  for (const file of files) {
    const filePath = path.join(LOG_DIR, file);
    const stats = fs.statSync(filePath);

    if (now - stats.mtime.getTime() > maxAge) {
      fs.unlinkSync(filePath);
      console.log(`Deleted: ${file}`);
    }
  }
}

cleanupOldLogs();
typescript
// scripts/cleanup-logs.ts
import * as fs from 'fs';
import * as path from 'path';

const LOG_DIR = '/var/log/myapp';
const RETENTION_DAYS = 30;

function cleanupOldLogs() {
  const now = Date.now();
  const maxAge = RETENTION_DAYS * 24 * 60 * 60 * 1000;

  const files = fs.readdirSync(LOG_DIR);

  for (const file of files) {
    const filePath = path.join(LOG_DIR, file);
    const stats = fs.statSync(filePath);

    if (now - stats.mtime.getTime() > maxAge) {
      fs.unlinkSync(filePath);
      console.log(`Deleted: ${file}`);
    }
  }
}

cleanupOldLogs();

Debugging Checklist

调试检查清单

When Investigating Issues

排查问题时

  • Identify the time window
  • Find related request IDs
  • Check for error spikes
  • Look at affected users/endpoints
  • Check for pattern changes
  • Review recent deployments
  • 确定时间窗口
  • 查找相关请求ID
  • 检查错误峰值
  • 查看受影响的用户/端点
  • 检查模式变化
  • 回顾最近的部署

Log Quality Check

日志质量检查

  • Structured JSON format
  • Appropriate log levels
  • Request IDs included
  • Sensitive data redacted
  • Stack traces for errors
  • Relevant context included
  • 结构化JSON格式
  • 合适的日志级别
  • 包含请求ID
  • 敏感数据已脱敏
  • 错误包含堆栈跟踪
  • 包含相关上下文

When to Use This Skill

何时使用本技能

Invoke this skill when:
  • Investigating production issues
  • Setting up logging infrastructure
  • Debugging application errors
  • Analyzing performance issues
  • Creating log analysis scripts
  • Setting up error tracking
  • Implementing request tracing
  • Cleaning up log data
在以下场景调用本技能:
  • 排查生产环境问题
  • 搭建日志基础设施
  • 调试应用错误
  • 分析性能问题
  • 创建日志分析脚本
  • 搭建错误追踪系统
  • 实现请求追踪
  • 清理日志数据