Job Monitor

When to use

1. Auto-monitor — another skill (PTQ, evaluation, deployment) just submitted a job. Register the job and set up monitoring immediately.
2. User-initiated — user asks about a job status, possibly in a new conversation. Check the registry, identify the job, and report.

Job Registry

.claude/active_jobs.json

[
  {
    "type": "nel",
    "id": "<invocation_id or slurm_job_id>",
    "host": "<cluster_hostname>",
    "user": "<ssh_user>",
    "submitted": "YYYY-MM-DD HH:MM",
    "description": "<what this job does>",
    "last_status": "<last known status>"
  }
]

type

nel

slurm

launcher

On Job Submission

1. Add an entry to

   .claude/active_jobs.json

   . Create the file if it doesn't exist.
2. Set up a durable recurring cron (if one isn't already running) that polls all registered jobs every 15 minutes. The cron prompt should: read the registry, check each job, report state changes to the user, remove completed jobs, and delete itself when the registry is empty.

On Cron Fire / Status Check

1. Read the registry from

   .claude/active_jobs.json

2. Check each job using the appropriate method (see below)
3. Report only state changes — compare against

   last_status

   in registry
4. Update

   last_status

   in the registry
5. Remove completed jobs — any job in a terminal state (COMPLETED, FAILED, CANCELLED, KILLED)
6. If registry is empty — delete the recurring cron

How to Check Each Job Type

NEL jobs (

type: nel

)

• Check:

  nel status <id>

• On completion:

  nel info <id>

  to fetch results
• On failure:

  nel info <id> --logs

  then inspect server/client/SLURM logs via SSH

Launcher jobs (

type: launcher

)

• Check: Tail the launcher's background output file for key events
• Key events: experiment ID, SLURM job ID, container import, calibration progress, export path, final status
• On failure: Look for

  Traceback

  ,

  Error

  , or

  FAILED

  in the output

Raw SLURM jobs (

type: slurm

)

• Check:

  ssh <host> "squeue -j <id> -h -o '%T %M %R'"

  — if empty, job left the queue
• On completion:

  ssh <host> "sacct -j <id> --format=State,ExitCode,Elapsed -n"

• On failure: Check the job's output log file

Identifying Jobs (user-initiated, no ID given)

1. .claude/active_jobs.json

   — most reliable, has context

2. nel ls runs --since 1d

   — recent NEL runs

3. ssh <host> "squeue -u <user>"

   — active SLURM jobs

4. ls -lt tools/launcher/experiments/cicd/ | head -10

   — recent launcher experiments

Reporting Guidelines

• Report state changes proactively — PENDING → RUNNING, or job completes
• Aggregate multiple jobs — "2 of 4 completed (MMLU-Pro: 42.3%, GSM8K: 67.1%), 1 running, 1 pending"
• Summarize, don't echo — interpret events ("Calibration complete, exporting checkpoint") not raw logs
• On failure, diagnose immediately — check logs and report root cause without waiting for user to ask
• Minimize noise — don't report "still running" unless the user is actively asking