Test Gap Analysis

Identify discrepancies between requirements/features that should be tested and actual test coverage through systematic analysis of testing completeness, effectiveness, and risk coverage to improve software quality and reliability.

When to use me

Use this skill when:

Uncertainty exists about what parts of the system are adequately tested
New features have been added without corresponding test coverage
Testing resources are limited and need optimal allocation
Preparing for releases or deployments with confidence in test coverage
Auditing testing effectiveness for compliance or quality standards
Identifying high-risk areas with insufficient testing
Planning test automation or test improvement initiatives
Onboarding new team members to understand test coverage
Assessing technical debt in testing infrastructure
Validating that critical functionality has appropriate test protection

What I do

1. Requirements Testability Analysis

Analyze requirements for testability: Identify testable aspects of each requirement
Extract test conditions and scenarios from requirements documentation
Categorize requirements by test type: Unit, integration, system, acceptance, performance, security
Assess requirement clarity for testing: Ambiguous vs. testable requirements
Identify implicit requirements that should be tested but aren't documented

2. Test Coverage Analysis

Analyze existing test suites: Unit tests, integration tests, end-to-end tests, manual tests
Map tests to requirements: Determine which requirements have test coverage
Measure coverage metrics: Line coverage, branch coverage, requirement coverage, risk coverage
Identify test gaps: Requirements without tests, partial coverage, outdated tests
Assess test effectiveness: Test quality, flakiness, maintenance burden

3. Gap Identification & Classification

Missing test coverage: Requirements with no corresponding tests
Incomplete test coverage: Requirements with partial test coverage
Outdated tests: Tests that don't match current requirements
Ineffective tests: Tests that don't adequately validate requirements
Risk coverage gaps: High-risk areas with insufficient testing
Test type gaps: Missing test types (security, performance, accessibility)

4. Risk-Based Prioritization

Assess business impact of untested requirements
Evaluate technical risk of inadequate testing
Calculate test gap severity based on impact and likelihood
Prioritize test gaps for remediation planning
Estimate effort to address each test gap
Recommend test strategy based on risk profile

5. Test Improvement Recommendations

Generate specific test cases to address gaps
Recommend test types and approaches for each gap
Suggest test automation opportunities
Propose test infrastructure improvements
Design test data strategies for uncovered scenarios
Create test maintenance plans to prevent future gaps

Test Gap Types

1. Coverage Gaps

Missing Test Coverage: Requirements with no tests at all
Partial Test Coverage: Requirements with incomplete test coverage
Example: Authentication requirement tested for success but not failure cases

2. Type Gaps

Missing Test Types: Critical test types not represented (security, performance, etc.)
Inappropriate Test Types: Wrong test type for requirement (unit vs. integration)
Example: Performance-critical feature only has unit tests, no performance tests

3. Effectiveness Gaps

Ineffective Tests: Tests that don't adequately validate requirements
Flaky Tests: Unreliable tests that provide false confidence
Example: Test passes but doesn't actually validate the requirement

4. Maintenance Gaps

Outdated Tests: Tests that don't match current requirements
Unmaintained Tests: Tests that fail frequently but aren't fixed
Example: Tests for deprecated functionality still in test suite

5. Risk Coverage Gaps

High-Risk Untested: Critical functionality without adequate testing
Low-Risk Overtested: Non-critical functionality with excessive testing
Example: Payment processing with minimal tests but UI with extensive tests

Analysis Techniques

Requirements Test Extraction

python

def extract_testable_requirements(requirements_doc):
    """
    Extract testable conditions from requirements.
    """
    testable_items = []
    
    for req in requirements_doc:
        test_conditions = analyze_requirement_for_tests(req)
        
        if test_conditions:
            testable_items.append({
                'requirement_id': req['id'],
                'description': req['description'],
                'test_conditions': test_conditions,
                'test_types': determine_appropriate_test_types(req),
                'priority': calculate_test_priority(req),
                'risk_level': assess_requirement_risk(req)
            })
    
    return testable_items

def analyze_requirement_for_tests(requirement):
    """Extract testable conditions from a requirement."""
    test_conditions = []
    
    # Look for explicit testable statements
    if 'shall' in requirement['text'].lower():
        # Extract what the system shall do
        import re
        shall_pattern = r'shall\s+([^.!?]+)'
        matches = re.findall(shall_pattern, requirement['text'], re.IGNORECASE)
        test_conditions.extend(matches)
    
    # Look for acceptance criteria
    if 'acceptance_criteria' in requirement:
        for criterion in requirement['acceptance_criteria']:
            test_conditions.append(criterion)
    
    # Look for constraints
    if 'constraints' in requirement:
        for constraint in requirement['constraints']:
            test_conditions.append(f"Constraint: {constraint}")
    
    return test_conditions

Test Coverage Mapping

python

class TestCoverageAnalyzer:
    def __init__(self, requirements, tests):
        self.requirements = requirements
        self.tests = tests
        
    def analyze_coverage(self):
        """Analyze test coverage of requirements."""
        coverage_report = {
            'requirements_count': len(self.requirements),
            'tests_count': len(self.tests),
            'covered_requirements': [],
            'uncovered_requirements': [],
            'partial_coverage': [],
            'coverage_percentage': 0
        }
        
        for req in self.requirements:
            matching_tests = self.find_tests_for_requirement(req)
            
            if not matching_tests:
                coverage_report['uncovered_requirements'].append({
                    'requirement': req,
                    'gap_type': 'missing_coverage'
                })
            elif len(matching_tests) < self.get_expected_test_count(req):
                coverage_report['partial_coverage'].append({
                    'requirement': req,
                    'matching_tests': matching_tests,
                    'expected_count': self.get_expected_test_count(req),
                    'actual_count': len(matching_tests),
                    'gap_type': 'partial_coverage'
                })
            else:
                coverage_report['covered_requirements'].append({
                    'requirement': req,
                    'matching_tests': matching_tests
                })
        
        coverage_report['coverage_percentage'] = (
            len(coverage_report['covered_requirements']) / 
            len(self.requirements) * 100 if self.requirements else 0
        )
        
        return coverage_report
    
    def find_tests_for_requirement(self, requirement):
        """Find tests that cover a requirement."""
        matching_tests = []
        
        for test in self.tests:
            if self.test_covers_requirement(test, requirement):
                matching_tests.append(test)
        
        return matching_tests
    
    def test_covers_requirement(self, test, requirement):
        """Determine if a test covers a requirement."""
        # Check test name for requirement reference
        if requirement['id'] in test.get('name', ''):
            return True
        
        # Check test description for requirement reference
        if requirement['id'] in test.get('description', ''):
            return True
        
        # Check test tags for requirement reference
        if 'tags' in test and requirement['id'] in test['tags']:
            return True
        
        # Semantic analysis (simplified)
        if self.semantic_match(test.get('purpose', ''), requirement['description']):
            return True
        
        return False

Risk-Based Gap Prioritization

python

def prioritize_test_gaps(test_gaps, risk_assessment):
    """
    Prioritize test gaps based on risk and impact.
    """
    prioritized_gaps = []
    
    for gap in test_gaps:
        # Calculate priority score
        priority_score = calculate_priority_score(gap, risk_assessment)
        
        prioritized_gaps.append({
            **gap,
            'priority_score': priority_score,
            'priority_level': determine_priority_level(priority_score),
            'remediation_effort': estimate_remediation_effort(gap),
            'risk_exposure': calculate_risk_exposure(gap, risk_assessment)
        })
    
    # Sort by priority score (descending)
    prioritized_gaps.sort(key=lambda x: x['priority_score'], reverse=True)
    
    return prioritized_gaps

def calculate_priority_score(gap, risk_assessment):
    """Calculate priority score for a test gap."""
    weights = {
        'business_impact': 0.4,
        'technical_risk': 0.3,
        'user_impact': 0.2,
        'regulatory_requirement': 0.1
    }
    
    scores = {}
    
    # Business impact score
    if gap['requirement']['business_criticality'] == 'high':
        scores['business_impact'] = 1.0
    elif gap['requirement']['business_criticality'] == 'medium':
        scores['business_impact'] = 0.5
    else:
        scores['business_impact'] = 0.2
    
    # Technical risk score
    if gap['requirement']['technical_complexity'] == 'high':
        scores['technical_risk'] = 1.0
    elif gap['requirement']['technical_complexity'] == 'medium':
        scores['technical_risk'] = 0.5
    else:
        scores['technical_risk'] = 0.2
    
    # User impact score
    if gap['requirement']['user_exposure'] == 'high':
        scores['user_impact'] = 1.0
    elif gap['requirement']['user_exposure'] == 'medium':
        scores['user_impact'] = 0.5
    else:
        scores['user_impact'] = 0.2
    
    # Regulatory requirement score
    if gap['requirement'].get('regulatory_requirement', False):
        scores['regulatory_requirement'] = 1.0
    else:
        scores['regulatory_requirement'] = 0.0
    
    # Calculate weighted score
    weighted_score = sum(
        scores[factor] * weights[factor] 
        for factor in weights
    )
    
    return weighted_score

Examples

bash

# Analyze test coverage gaps
npm run test-gap-analysis:analyze -- --requirements specs/ --tests tests/ --output gaps.json

# Map tests to requirements
npm run test-gap-analysis:map -- --requirements requirements.md --test-suite tests/unit/ --output coverage-map.yaml

# Identify high-risk untested requirements
npm run test-gap-analysis:risk -- --requirements specs/ --tests tests/ --risk-factors risk-assessment.yaml --output risk-gaps.md

# Generate test recommendations
npm run test-gap-analysis:recommend -- --gaps gaps.json --output recommendations.md

# Calculate test coverage metrics
npm run test-gap-analysis:metrics -- --requirements specs/ --tests tests/ --output metrics.json

# Continuous test gap monitoring
npm run test-gap-analysis:monitor -- --watch --requirements specs/ --tests tests/ --alert-on-gap

Output format