reproducibility-audit

Original：🇺🇸 English
Translated

Audit a CS or AI research project for reproducibility across environment, data, code, configuration, logging, and documentation. Use this skill whenever the user wants to make experiments reproducible, prepare code for collaborators, debug environment drift, write a README, package a project for paper release, or ensure they can rerun results months later.

22installs
Sourcea-green-hand-jack/phd-skills
Added on

NPX Install

npx skill4agent add a-green-hand-jack/phd-skills reproducibility-audit

Tags

Translated version includes tags in frontmatter
reproducibility-auditai-researchenvironment-managementdocumentation-practicesexperiment-reproducibility

SKILL.md Content

View Translation Comparison →

Reproducibility Audit

Purpose

Help the user make a research project reproducible enough that another person, or their future self, can rerun it. This skill follows the handbook's reproducibility section: exact environments, data and code versioning, documentation, random seeds, parameter logging, and backups.
The output is an audit report plus a prioritized fix list.

When to Use

  • User is starting or cleaning a research codebase
  • User needs collaborator handoff
  • User cannot reproduce an old experiment
  • User is preparing for paper submission or public release
  • User is moving between local, server, HPC, Docker, or conda environments

Workflow

Stage 1: Identify the Project Context

Ask:
  • What repository or folder is being audited?
  • Is this local, remote server, HPC, or containerized?
  • Is the goal internal reproducibility, collaborator onboarding, or public release?
  • What result must be reproducible?
If the user grants filesystem access, inspect the repo structure before advising.

Stage 2: Audit the Three Pillars

Check:
Environment management
  • Conda/uv/pip environment file exists
  • Python/CUDA/framework versions are pinned where needed
  • Setup instructions are current
  • Dockerfile or container instructions exist if portability matters
Data and code versioning
  • Git tracks source and config, not generated clutter
  • Dataset source/version/preprocessing are documented
  • Large files have an intentional storage path
  • Checkpoints and outputs are named consistently
Documentation
  • README explains install, data setup, train/eval commands
  • Commands are copy-pasteable
  • Expected outputs and runtime are stated
  • Known caveats and hardware assumptions are explicit

Stage 3: Audit Experiment Hygiene

Check:
  • Random seeds
  • Config files
  • Hyperparameter logging
  • Commit hash logging
  • Metric files
  • Plot/table generation scripts
  • Failure logs
  • Result-to-paper traceability

Stage 4: Classify Issues

Use severity:
  • Blocker
    : current results cannot be rerun or verified
  • High
    : collaborators will likely fail to run it
  • Medium
    : results can run but are hard to inspect or compare
  • Low
    : polish or future-proofing
Prioritize fixes that protect irreplaceable results first.

Stage 5: Produce the Artifact

Save to 
~/phd-log/reproducibility/YYYY-MM-DD-[project].md
.
markdown
# Reproducibility Audit — [Project]

## Goal
[Internal / collaborator / release] reproducibility for [target result]

## Summary
- Overall status:
- Biggest risk:
- First fix:

## Environment
| Check | Status | Notes |
|---|---|---|

## Data and code versioning
| Check | Status | Notes |
|---|---|---|

## Documentation
| Check | Status | Notes |
|---|---|---|

## Experiment hygiene
| Check | Status | Notes |
|---|---|---|

## Issues
| Severity | Issue | Fix |
|---|---|---|

## Prioritized action plan
- [ ] Blocker:
- [ ] High:
- [ ] Medium:
- [ ] Low:

Tone

Be practical. A reproducibility audit should leave the user with 3-5 high-impact fixes, not a guilt-inducing wall of best practices.

What Not to Do

  • Do not demand Docker if conda or uv is enough for the collaboration context.
  • Do not recommend version pinning without explaining what needs to be pinned.
  • Do not ignore data provenance.
  • Do not treat README polish as more urgent than rerunnable experiments.