Systematic Debugging

Overview

Non-Negotiable Rule

No fix proposals allowed without root cause investigation

When to Use

• Test failures
• Production bugs
• Abnormal behavior
• Performance issues
• Build failures
• Integration problems

• Time constraints (emergencies are where guesswork fixes happen most often)
• Thinking "a small change will fix it"
• Having tried multiple fixes already
• The last fix didn't work
• You don't fully understand the problem

• The problem seems simple (simple bugs still have root causes)
• You're in a hurry (rushing leads to rework)
• A leader demands an immediate fix (systematic debugging is faster than repeated guesses)

Four Phases

Phase 1: Root Cause Investigation

1. Read error messages carefully
   • Don't skip errors or warnings
   • They often contain the solution directly
   • Read the full stack trace
   • Note line numbers, file paths, error codes
2. Stably reproduce the issue
   • Can you reliably trigger it?
   • What are the exact reproduction steps?
   • Does it reproduce every time?
   • If unable to reproduce → collect more data, don't guess
3. Check recent changes
   • What changes could have caused this issue?
   • git diff, recent commits
   • New dependencies, configuration changes
   • Environment differences
4. Collect evidence in multi-component systems
   When the system has multiple components (CI → Build → Signing, API → Service → Database):
   Before proposing a fix, add diagnostic instrumentation:

   For each component boundary:
     - Record data entering the component
     - Record data leaving the component
     - Verify environment/configuration propagation
     - Check state at each layer
   
   Execute once to collect evidence, identify where the break occurs
   Then analyze evidence to locate the faulty component
   Then conduct in-depth investigation on that component

   Example (multi-layer system):
   bash

   # Layer 1: Workflow
   echo "=== Secrets available in workflow: ==="
   echo "IDENTITY: ${IDENTITY:+SET}${IDENTITY:-UNSET}"
   
   # Layer 2: Build script
   echo "=== Env vars in build script: ==="
   env | grep IDENTITY || echo "IDENTITY not in environment"
   
   # Layer 3: Signing script
   echo "=== Keychain state: ==="
   security list-keychains
   security find-identity -v
   
   # Layer 4: Actual signing
   codesign --sign "$IDENTITY" --verbose=4 "$APP"

   This reveals: Which layer has the issue (secrets → workflow ✓, workflow → build ✗)
5. Trace data flow
   When errors occur deep in the call stack:
   See

   root-cause-tracing.md

   in this directory for complete reverse tracing techniques.
   Brief version:
   • Where did the incorrect value originate?
   • Who called this with the incorrect value?
   • Keep tracing upward until you find the source
   • Fix at the source, not at the symptom

Phase 2: Pattern Analysis

1. Find working examples
   • Look for similar working code in the same codebase
   • What working code resembles the problematic code?
2. Compare with reference implementations
   • If implementing a pattern, read the reference implementation fully
   • Don't skim — read line by line
   • Fully understand the pattern before applying it
3. Identify differences
   • What's different between the working code and the problematic code?
   • List every difference, no matter how small
   • Don't assume "that can't possibly matter"
4. Understand dependencies
   • What other components does this feature require?
   • What settings, configurations, environments are needed?
   • What implicit assumptions does it have?

Phase 3: Hypothesis & Verification

1. Form a single hypothesis
   • State clearly: "I believe X is the root cause because Y"
   • Write it down
   • Be specific, not vague
2. Minimal testing
   • Make the smallest change to verify the hypothesis
   • Change only one variable at a time
   • Don't fix multiple issues simultaneously
3. Verify before proceeding
   • Did it work? Yes → move to Phase 4
   • Didn't work? Form a new hypothesis
   • Don't stack more fixes on top
4. When you're unsure
   • Say "I don't understand X"
   • Don't pretend you know
   • Ask for help
   • Do more research

Phase 4: Implementation

1. Create a failing test case
   • Minimal reproduction
   • Use automated tests whenever possible
   • Write a one-time test script if no test framework exists
   • Must have a test before fixing
   • Use the

     superpowers:test-driven-development

     skill to write standard failing tests
2. Implement a single fix
   • Fix the identified root cause
   • Change only one thing at a time
   • Don't make "while I'm at it" optimizations
   • Don't bundle refactoring
3. Verify the fix
   • Does the test pass now?
   • Are other tests not broken?
   • Is the problem truly resolved?
4. If the fix doesn't work
   • Stop
   • Count: How many fixes have you attempted?
   • Fewer than 3: Return to Phase 1, re-analyze with new information
   • 3 or more: Stop and question the architecture (see Step 5 below)
   • Do not attempt a 4th fix without architectural discussion
5. If 3+ fixes fail: Question the architecture
   These patterns indicate architectural issues:
   • Each fix exposes new shared state/coupling/issues elsewhere
   • Fix requires "massive refactoring" to implement
   • Each fix creates new symptoms elsewhere
   Stop and question fundamental issues:
   • Is this pattern fundamentally reasonable?
   • Are we sticking to a wrong solution due to "momentum"?
   • Should we refactor the architecture or continue patching symptoms?
   Discuss with your partner before attempting more fixes
   This is not hypothesis failure — this is architectural failure.

Red Lines – Stop and Follow the Process

• "Just patch it temporarily, I'll investigate later"
• "Try changing X to see if it works"
• "Change multiple things at once and run tests"
• "Skip testing, I'll verify manually"
• "It's probably X, let me fix it"
• "I don't fully understand, but this should work"
• "The pattern says X, but I'll use it differently"
• "The main issues are: [listing fixes without investigation]"
• Proposing solutions without tracing data flow
• "Try one more fix" (after 2+ failed attempts)
• Each fix exposes new issues in different places

Signals from Your Partner – Indicating Your Approach is Wrong

• "Isn't it this way?" – You're making assumptions without verification
• "Can it tell us...?" – You should collect evidence first
• "Stop guessing" – You're proposing fixes without understanding
• "Think deeper" – Question fundamental issues, not just symptoms
• "Are we stuck?" (frustrated tone) – Your approach isn't working

Common Excuses

Quick Reference Table

When the Process Says "No Root Cause Found"

1. You've completed the process
2. Document what you investigated
3. Implement appropriate handling (retries, timeouts, error messages)
4. Add monitoring/logging for future investigation

Supporting Techniques

• root-cause-tracing.md

  - Trace bugs backward along the call stack to find the initial trigger point

• defense-in-depth.md

  - After finding the root cause, add validation at multiple levels

• condition-based-waiting.md

  - Replace hard-coded wait times with conditional polling

• superpowers:test-driven-development - For creating failing test cases (Phase 4, Step 1)
• superpowers:verification-before-completion - Verify the fix actually works before declaring success

Actual Results

• Systematic approach: 15-30 minutes to fix
• Random fix approach: 2-3 hours of trial and error
• First-fix success rate: 95% vs 40%
• New bugs introduced: Almost zero vs frequent occurrences