Run Experiment

Workflow

Step 1: Detect Environment

CLAUDE.md

• Local GPU: Look for local CUDA/MPS setup info
• Remote server: Look for SSH alias, conda env, code directory

CLAUDE.md

Step 2: Pre-flight Check

ssh <server> nvidia-smi --query-gpu=index,memory.used,memory.total --format=csv,noheader

nvidia-smi --query-gpu=index,memory.used,memory.total --format=csv,noheader
# or for Mac MPS:
python -c "import torch; print('MPS available:', torch.backends.mps.is_available())"

Step 3: Sync Code (Remote Only)

CLAUDE.md

code_sync

rsync

Option A: rsync (default)

rsync -avz --include='*.py' --exclude='*' <local_src>/ <server>:<remote_dst>/

Option B: git (when

code_sync: git

is set in CLAUDE.md)

# 1. Push from local
git add -A && git commit -m "sync: experiment deployment" && git push

# 2. Pull on server
ssh <server> "cd <remote_dst> && git pull"

Step 3.5: W&B Integration (when

wandb: true

in CLAUDE.md)

wandb

false

1. Check if wandb is already in the script — look for

   import wandb

   or

   wandb.init

   . If present, skip to Step 4.
2. If not present, add W&B logging to the training script:
   python

   import wandb
   wandb.init(project=WANDB_PROJECT, name=EXP_NAME, config={...hyperparams...})
   
   # Inside training loop:
   wandb.log({"train/loss": loss, "train/lr": lr, "step": step})
   
   # After eval:
   wandb.log({"eval/loss": eval_loss, "eval/ppl": ppl, "eval/accuracy": acc})
   
   # At end:
   wandb.finish()

3. Metrics to log (add whichever apply to the experiment):

   • train/loss

     — training loss per step

   • train/lr

     — learning rate

   • eval/loss

     ,

     eval/ppl

     ,

     eval/accuracy

     — eval metrics per epoch

   • gpu/memory_used

     — GPU memory (via

     torch.cuda.max_memory_allocated()

     )

   • speed/samples_per_sec

     — throughput
   • Any custom metrics the experiment already computes
4. Verify wandb login on the target machine:
   bash

   ssh <server> "wandb status"  # should show logged in
   # If not logged in:
   ssh <server> "wandb login <WANDB_API_KEY>"

The W&B project name and API key come from

CLAUDE.md

(see example below). The experiment name is auto-generated from the script name + timestamp.

Step 4: Deploy

Remote (via SSH + screen)

ssh <server> "screen -dmS <exp_name> bash -c '\
  eval \"\$(<conda_path>/conda shell.bash hook)\" && \
  conda activate <env> && \
  CUDA_VISIBLE_DEVICES=<gpu_id> python <script> <args> 2>&1 | tee <log_file>'"

Local

# Linux with CUDA
CUDA_VISIBLE_DEVICES=<gpu_id> python <script> <args> 2>&1 | tee <log_file>

# Mac with MPS (PyTorch uses MPS automatically)
python <script> <args> 2>&1 | tee <log_file>

run_in_background: true

Step 5: Verify Launch

ssh <server> "screen -ls"

Step 6: Feishu Notification (if configured)

~/.claude/feishu.json

• Send

  experiment_done

  notification: which experiments launched, which GPUs, estimated time
• If config absent or mode

  "off"

  : skip entirely (no-op)

Key Rules

• ALWAYS check GPU availability first — never blindly assign GPUs
• Each experiment gets its own screen session + GPU (remote) or background process (local)
• Use

  tee

  to save logs for later inspection
• Run deployment commands with

  run_in_background: true

  to keep conversation responsive
• Report back: which GPU, which screen/process, what command, estimated time
• If multiple experiments, launch them in parallel on different GPUs

CLAUDE.md Example

CLAUDE.md

## Remote Server
- SSH: `ssh my-gpu-server`
- GPU: 4x A100 (80GB each)
- Conda: `eval "$(/opt/conda/bin/conda shell.bash hook)" && conda activate research`
- Code dir: `/home/user/experiments/`
- code_sync: rsync          # default. Or set to "git" for git push/pull workflow
- wandb: false              # set to "true" to auto-add W&B logging to experiment scripts
- wandb_project: my-project # W&B project name (required if wandb: true)
- wandb_entity: my-team     # W&B team/user (optional, uses default if omitted)

## Local Environment
- Mac MPS / Linux CUDA
- Conda env: `ml` (Python 3.10 + PyTorch)

W&B setup: Run

wandb login

on your server once (or set

WANDB_API_KEY

env var). The skill reads project/entity from CLAUDE.md and adds

wandb.init()

+

wandb.log()

to your training scripts automatically. Dashboard:

https://wandb.ai/<entity>/<project>

.