Systematic Debugging

NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST

Phase 1: Root Cause Investigation

1. Read Error Messages Carefully
   • Read stack traces completely
   • Note line numbers, file paths, error codes
   • Don't skip warnings
2. Reproduce Consistently
   • What are the exact steps?
   • If not reproducible → gather more data, don't guess
3. Check Recent Changes
   • Git diff, recent commits
   • New dependencies, config changes
   • Environmental differences
4. Gather Evidence in Multi-Component Systems
   WHEN system has multiple components (CI → build → signing, API → service → database):
   Add diagnostic instrumentation before proposing fixes:

   For EACH component boundary:
     - Log what data enters/exits component
     - Verify environment/config propagation
     - Check state at each layer
   
   Run once to gather evidence → analyze → identify failing component

   Example:
   bash

   # Layer 1: Workflow
   echo "=== Secrets available: ==="
   echo "IDENTITY: ${IDENTITY:+SET}${IDENTITY:-UNSET}"
   
   # Layer 2: Build script
   env | grep IDENTITY || echo "IDENTITY not in environment"
   
   # Layer 3: Signing
   security find-identity -v

5. Trace Data Flow
   See

   references/root-cause-tracing.md

   for backward tracing technique.
   Quick version: Where does bad value originate? Trace up call chain until you find the source. Fix at source.

Phase 2: Pattern Analysis

1. Find Working Examples - Similar working code in codebase
2. Compare Against References - Read reference implementations COMPLETELY, don't skim
3. Identify Differences - List every difference, don't assume "that can't matter"
4. Understand Dependencies - Components, config, environment, assumptions

Phase 3: Hypothesis and Testing

1. Form Single Hypothesis - "I think X is root cause because Y" - be specific
2. Test Minimally - SMALLEST possible change, one variable at a time
3. Verify - Worked → Phase 4. Didn't work → form NEW hypothesis, don't stack fixes
4. When You Don't Know - Say so. Don't pretend.

Phase 4: Implementation

1. Create Failing Test Case
   • Use the

     test-driven-development

     skill
   • MUST have before fixing
2. Implement Single Fix
   • ONE change at a time
   • No "while I'm here" improvements
3. Verify Fix
   • Test passes? Other tests still pass? Issue resolved?
4. If Fix Doesn't Work
   • Count attempts
   • If < 3: Return to Phase 1 with new information
   • If ≥ 3: Escalate (below)

Escalation: 3+ Failed Fixes

• Each fix reveals new problems elsewhere
• Fixes require massive refactoring
• Shared state/coupling keeps surfacing

• Is this pattern fundamentally sound?
• Are we continuing through inertia?
• Refactor architecture vs. continue fixing symptoms?

Red Flags → STOP and Return to Phase 1

• "Quick fix for now, investigate later"
• "Just try changing X"
• "I'll skip the test"
• "It's probably X"
• "Pattern says X but I'll adapt it differently"
• Proposing solutions before tracing data flow
• "One more fix" after 2+ failures

Human Signals You're Off Track

• "Is that not happening?" → You assumed without verifying
• "Will it show us...?" → You should have added evidence gathering
• "Stop guessing" → You're proposing fixes without understanding
• "Ultrathink this" → Question fundamentals
• Frustrated "We're stuck?" → Your approach isn't working

Supporting Techniques

references/

• root-cause-tracing.md

  - Trace bugs backward through call stack

• defense-in-depth.md

  - Add validation at multiple layers after finding root cause

• condition-based-waiting.md

  - Replace arbitrary timeouts with condition polling

• test-driven-development

  - Creating failing test case (Phase 4)

• verification-before-completion

  - Verify fix before claiming success