System Debugging

Overview

Iron Rule

NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST

When to Use

• Test failures
• Production errors
• Unexpected behavior
• Performance issues
• Build failures
• Integration problems

• Under time pressure (urgency makes it easy to guess)
• "Just a quick fix" seems obvious
• You've already tried multiple fixes
• Previous fixes didn't work
• You don't fully understand the problem

• The problem seems simple (simple bugs also have root causes)
• You're in a hurry (hurry guarantees rework)
• Managers want an immediate fix (systematic approach is faster than chaos)

Four Phases

Phase 1: Root Cause Investigation

1. Read error messages carefully
   • Don't skip past errors or warnings
   • They often contain precise solutions
   • Read the full stack trace
   • Note line numbers, file paths, error codes
2. Reproduce consistently
   • Can you trigger it reliably?
   • What are the exact steps?
   • Does it happen every time?
   • If non-reproducible → collect more data, don't guess
3. Check recent changes
   • What change might have caused this?
   • Git diff, recent commits
   • New dependencies, configuration changes
   • Environment differences
4. Gather evidence in multi-component systems

For EACH component boundary:
  - Log what data enters component
  - Log what data exits component
  - Verify environment/config propagation
  - Check state at each layer

Run once to gather evidence showing WHERE it breaks
THEN analyze evidence to identify failing component
THEN investigate that specific component

# Layer 1: Workflow
echo "=== Secrets available in workflow: ==="
echo "IDENTITY: ${IDENTITY:+SET}${IDENTITY:-UNSET}"

# Layer 2: Build script
echo "=== Env vars in build script: ==="
env | grep IDENTITY || echo "IDENTITY not in environment"

# Layer 3: Signing script
echo "=== Keychain state: ==="
security list-keychains
security find-identity -v

# Layer 4: Actual signing
codesign --sign "$IDENTITY" --verbose=4 "$APP"

1. Trace data flow

root-cause-tracing.md

• Where does the bad value come from?
• What called this with the bad value?
• Keep tracing until you find the source
• Fix at the source, not the symptom

Phase 2: Pattern Analysis

1. Look for working examples
   • Find similar working code in the same codebase
   • What works that's similar to what's broken?
2. Compare to references
   • If implementing a pattern, read the reference implementation fully
   • Don't skim - read every line
   • Understand the pattern fully before applying
3. Identify differences
   • What's different between working and broken?
   • List every difference, no matter how small
   • Don't assume "that doesn't matter"
4. Understand dependencies
   • What other components does this require?
   • What settings, configurations, environment?
   • What assumptions does it make?

Phase 3: Hypothesis and Testing

1. Form a single hypothesis
   • State clearly: "I think X is the root cause because Y"
   • Write it down
   • Be specific, not vague
2. Minimal testing
   • Make the smallest possible change to test the hypothesis
   • One variable at a time
   • Don't fix multiple issues at once
3. Verify before proceeding
   • Did it work? Yes → Phase 4
   • Didn't work? Form a new hypothesis
   • Don't add more fixes on top
4. When you don't know
   • Say "I don't understand X"
   • Don't pretend to know
   • Ask for help
   • Research more

Phase 4: Implementation

1. Create a failing test case
   • Simplest reproduction possible
   • Automated test if possible
   • One-off test script if no framework
   • Must have this before fixing
   • Use

     superpowers:test-driven-development

     skill to write correct failing tests
2. Implement a single fix
   • Address the identified root cause
   • One change at a time
   • No "while I'm here" improvements
   • No bundled refactoring
3. Verify the fix
   • Do tests pass now?
   • No other tests broken?
   • Is the problem truly solved?
4. If the fix doesn't work
   • Stop
   • Count: How many fixes have you tried?
   • If < 3: Return to Phase 1, re-analyze with new info
   • If ≥ 3: Stop and question the architecture (Step 5 below)
   • Don't attempt fix #4 without architectural discussion
5. If 3+ fixes fail: Architectural Problem

• Each fix reveals new shared state/coupling/problems in different places
• Fix requires "massive refactoring" to implement
• Each fix creates new symptoms elsewhere

• Is this pattern fundamentally sound?
• Are we "sticking with it purely out of inertia"?
• Should we refactor the architecture or keep fixing symptoms?

Red Flags - Stop and Follow the Process

• "Quick fix now, investigate later"
• "Try changing X to see if it works"
• "Add multiple changes, run tests"
• "Skip testing, I'll verify manually"
• "Probably X, let me fix that"
• "I don't fully understand, but this might work"
• "The pattern says X, but I'll adapt it differently"
• "Here are the main issues: [list of uninvestigated fixes]"
• Propose solutions before tracing data flow
• "Just try one more fix" (after 2+ attempts)
• Each fix reveals new problems in different places

Signals from Your Human Partner That You're Doing It Wrong

• "Is that what's happening?" - You assumed without verifying
• "Would it tell us...?" - You should add evidence collection
• "Stop guessing" - You're proposing fixes without understanding
• "Ultrathink this" - Question fundamentals, not just symptoms
• "Are we stuck?" (frustration) - Your approach isn't working

Common Rationalizations

Quick Reference

When the Process Shows "No Root Cause"

1. You've completed the process
2. Document what you investigated
3. Implement appropriate handling (retries, timeouts, error messages)
4. Add monitoring/logging for future investigation

Companion Techniques

• root-cause-tracing.md

  - Trace errors backward through the call stack to find the original trigger

• defense-in-depth.md

  - Add multiple layers of validation after finding the root cause

• condition-based-waiting.md

  - Replace arbitrary timeouts with conditional polling

• superpowers:test-driven-development - For creating failing test cases (Phase 4, Step 1)
• superpowers:verify-before-complete - Verify fixes work before declaring success

Real-World Impact

• Systematic approach: 15-30 minutes to fix
• Random fix approach: 2-3 hours of whack-a-mole
• First-fix success rate: 95% vs 40%
• New bugs introduced: Near zero vs common