Honeycomb Production Investigation

The Core Analysis Loop

Investigation Workflow

Step 1: Orient

1. get_workspace_context

   → environments and datasets

2. get_slos

   → any SLOs in violation? (frames severity)

3. get_triggers

   → any alerts firing? (narrows scope)

4. find_queries

   → has anyone investigated this before?

Step 2: Characterize the Problem

• Latency spike: P99(duration_ms), HEATMAP(duration_ms) grouped by service or route
• Error surge: COUNT filtered on error=true, grouped by exception.message or service
• Unknown: COUNT grouped by service.name to find which service has anomalous volume

get_service_map

Step 3: BubbleUp to Find Differentiators

1. Run

   run_bubbleup

   on the query result, selecting the outlier region
2. BubbleUp compares outlier vs baseline distributions across all columns automatically
3. Look for fields where the distributions differ significantly

• Categorical fields (dimensions): A value overrepresented in outliers points to a cause (e.g.,

  deployment.version=v2.3.1

  is 90% of slow requests but only 20% of baseline)
• Numeric fields (measures): A shifted distribution shows correlated metrics (e.g.,

  db.query_duration

  is much higher in outliers)
• Typical root causes surfaced: deployment version, region, user cohort, specific endpoint, feature flag

Step 4: Drill Into Traces

1. Add BubbleUp findings as WHERE filters to narrow results
2. Pick a representative trace ID
3. Call

   get_trace

   to fetch the full trace

• Spans with disproportionate duration vs parent (the bottleneck)
• Sequential spans that could be parallelized (N+1 query patterns)
• Error spans — check span events for stack traces
• Gaps between child spans (missing instrumentation or idle wait)
• Service boundaries (where the trace crosses services)

Step 5: Verify Hypothesis

• Query WITH the suspected cause filtered in
• Query WITHOUT it (as a control)
• If the metrics diverge, you've found it

Step 6: Record Findings

create_board

• A text panel summarizing the root cause (Markdown)
• The key query run PKs that identified the problem
• Related SLOs if applicable

Investigation Patterns

Latency Spike

Error Surge

Deployment Regression

Dependency Failure

get_service_map

any.service.name

Stay on the Path

• "The cause is obvious, I can skip BubbleUp" — BubbleUp routinely surfaces causes that seem obvious in hindsight but weren't the first guess. It also catches secondary causes you'd miss entirely.
• "I already know it's a deployment issue" — verify with Step 5. Confirmation bias is strongest during incidents. Query with and without the suspected cause.
• "Traces confirmed it, no need to verify" — a single trace is an anecdote. The verification query proves the pattern holds across all traffic, not just one request.
• "This is a simple issue, the full workflow is overkill" — the workflow takes minutes; a wrong diagnosis during an incident costs hours.

When Results Are Empty or Unclear

• No results: Check field names with

  find_columns

  , expand time range, verify environment/dataset
• BubbleUp shows no signal: Try a different time selection, add filters to isolate the anomaly more clearly, or select a different calculation
• Trace missing spans: Sampling, instrumentation gaps, or cross-environment trace split

Additional Resources

Reference Files

• ${CLAUDE_PLUGIN_ROOT}/skills/production-investigation/references/investigation-playbooks.md

  — Step-by-step playbooks for latency spikes, error surges, deployment regressions, dependency failures, SLO budget burn, and health checks

• ${CLAUDE_PLUGIN_ROOT}/skills/production-investigation/references/bubbleup-guide.md

  — Detailed BubbleUp usage: selection types, time specifications, pagination, result interpretation

• ${CLAUDE_PLUGIN_ROOT}/skills/production-investigation/references/trace-exploration.md

  — Trace structure, get_trace parameters and view modes, waterfall analysis, span events and links

Cross-References

• For the conceptual foundations of the core analysis loop, see the observability-fundamentals skill
• For query construction patterns, see the query-patterns skill
• For SLO/trigger context during investigations, see the slos-and-triggers skill