reproducibility-audit

Original：🇺🇸 English

Translated

Audit a CS or AI research project for reproducibility across environment, data, code, configuration, logging, and documentation. Use this skill whenever the user wants to make experiments reproducible, prepare code for collaborators, debug environment drift, write a README, package a project for paper release, or ensure they can rerun results months later.

2installs

Sourcea-green-hand-jack/phd-skills

Added on2026-04-25

NPX Install

npx skill4agent add a-green-hand-jack/phd-skills reproducibility-audit

SKILL.md Content

View Translation Comparison →

Reproducibility Audit

Purpose

Help the user make a research project reproducible enough that another person, or their future self, can rerun it. This skill follows the handbook's reproducibility section: exact environments, data and code versioning, documentation, random seeds, parameter logging, and backups.

The output is an audit report plus a prioritized fix list.

When to Use

User is starting or cleaning a research codebase
User needs collaborator handoff
User cannot reproduce an old experiment
User is preparing for paper submission or public release
User is moving between local, server, HPC, Docker, or conda environments

Workflow

Stage 1: Identify the Project Context

Ask:

What repository or folder is being audited?
Is this local, remote server, HPC, or containerized?
Is the goal internal reproducibility, collaborator onboarding, or public release?
What result must be reproducible?

If the user grants filesystem access, inspect the repo structure before advising.

Stage 2: Audit the Three Pillars

Check:

Environment management

Conda/uv/pip environment file exists
Python/CUDA/framework versions are pinned where needed
Setup instructions are current
Dockerfile or container instructions exist if portability matters

Data and code versioning

Git tracks source and config, not generated clutter
Dataset source/version/preprocessing are documented
Large files have an intentional storage path
Checkpoints and outputs are named consistently

Documentation

README explains install, data setup, train/eval commands
Commands are copy-pasteable
Expected outputs and runtime are stated
Known caveats and hardware assumptions are explicit

Stage 3: Audit Experiment Hygiene

Check:

Random seeds
Config files
Hyperparameter logging
Commit hash logging
Metric files
Plot/table generation scripts
Failure logs
Result-to-paper traceability

Stage 4: Classify Issues

Use severity:

```
Blocker
```
: current results cannot be rerun or verified
```
High
```
: collaborators will likely fail to run it
```
Medium
```
: results can run but are hard to inspect or compare
```
Low
```
: polish or future-proofing

Prioritize fixes that protect irreplaceable results first.

Stage 5: Produce the Artifact

Save to

~/phd-log/reproducibility/YYYY-MM-DD-[project].md

.

markdown

# Reproducibility Audit — [Project]

## Goal
[Internal / collaborator / release] reproducibility for [target result]

## Summary
- Overall status:
- Biggest risk:
- First fix:

## Environment
| Check | Status | Notes |
|---|---|---|

## Data and code versioning
| Check | Status | Notes |
|---|---|---|

## Documentation
| Check | Status | Notes |
|---|---|---|

## Experiment hygiene
| Check | Status | Notes |
|---|---|---|

## Issues
| Severity | Issue | Fix |
|---|---|---|

## Prioritized action plan
- [ ] Blocker:
- [ ] High:
- [ ] Medium:
- [ ] Low:

Tone

Be practical. A reproducibility audit should leave the user with 3-5 high-impact fixes, not a guilt-inducing wall of best practices.

What Not to Do

Do not demand Docker if conda or uv is enough for the collaboration context.
Do not recommend version pinning without explaining what needs to be pinned.
Do not ignore data provenance.
Do not treat README polish as more urgent than rerunnable experiments.