• Extracting existing evaluation tables from README content
• Importing benchmark scores from Artificial Analysis
• Running custom model evaluations with vLLM or accelerate backends (lighteval/inspect-ai)

• Model Cards: Updates model-index metadata for leaderboard integration
• Artificial Analysis: Direct API integration for benchmark imports
• Papers with Code: Compatible with their model-index specification
• Jobs: Run evaluations directly on Hugging Face Jobs with

  uv

  integration
• vLLM: Efficient GPU inference for custom model evaluation
• lighteval: HuggingFace's evaluation library with vLLM/accelerate backends
• inspect-ai: UK AI Safety Institute's evaluation framework

• huggingface_hub>=0.26.0
• markdown-it-py>=3.0.0
• python-dotenv>=1.2.1
• pyyaml>=6.0.3
• requests>=2.32.5
• re (built-in)

• inspect-ai>=0.3.0
• inspect-evals
• openai

• lighteval[accelerate,vllm]>=0.6.0
• vllm>=0.4.0
• torch>=2.0.0
• transformers>=4.40.0
• accelerate>=0.30.0

uv run

--create-pr

uv run scripts/evaluation_manager.py get-prs --repo-id "username/model-name"

1. DO NOT create a new PR - this creates duplicate work for maintainers
2. Warn the user that open PRs already exist
3. Show the user the existing PR URLs so they can review them
4. Only proceed if the user explicitly confirms they want to create another PR

All paths are relative to the directory containing this SKILL.md file. Before running any script, first

cd

to that directory or use the full path.

--help

uv run

uv run scripts/evaluation_manager.py --help
uv run scripts/evaluation_manager.py inspect-tables --help
uv run scripts/evaluation_manager.py extract-readme --help

1. get-prs

   → check for existing open PRs first

2. inspect-tables

   → find table numbers/columns

3. extract-readme --table N

   → prints YAML by default
4. add

   --apply

   (push) or

   --create-pr

   to write changes

• Inspect Tables: Use

  inspect-tables

  to see all tables in a README with structure, columns, and sample rows
• Parse Markdown Tables: Accurate parsing using markdown-it-py (ignores code blocks and examples)
• Table Selection: Use

  --table N

  to extract from a specific table (required when multiple tables exist)
• Format Detection: Recognize common formats (benchmarks as rows, columns, or comparison tables with multiple models)
• Column Matching: Automatically identify model columns/rows; prefer

  --model-column-index

  (index from inspect output). Use

  --model-name-override

  only with exact column header text.
• YAML Generation: Convert selected table to model-index YAML format
• Task Typing:

  --task-type

  sets the

  task.type

  field in model-index output (e.g.,

  text-generation

  ,

  summarization

  )

• API Integration: Fetch benchmark scores directly from Artificial Analysis
• Automatic Formatting: Convert API responses to model-index format
• Metadata Preservation: Maintain source attribution and URLs
• PR Creation: Automatically create pull requests with evaluation updates

• YAML Generation: Create properly formatted model-index entries
• Merge Support: Add evaluations to existing model cards without overwriting
• Validation: Ensure compliance with Papers with Code specification
• Batch Operations: Process multiple models efficiently

• Inspect-AI Integration: Run standard evaluations using the

  inspect-ai

  library
• UV Integration: Seamlessly run Python scripts with ephemeral dependencies on HF infrastructure
• Zero-Config: No Dockerfiles or Space management required
• Hardware Selection: Configure CPU or GPU hardware for the evaluation job
• Secure Execution: Handles API tokens safely via secrets passed through the CLI

• check the script path
• check uv is installed
• check gpu is available with

  nvidia-smi

• vLLM Backend: High-performance GPU inference (5-10x faster than standard HF methods)
• lighteval Framework: HuggingFace's evaluation library with Open LLM Leaderboard tasks
• inspect-ai Framework: UK AI Safety Institute's evaluation library
• Standalone or Jobs: Run locally or submit to HF Jobs infrastructure

• Preferred: use

  uv run

  (PEP 723 header auto-installs deps)
• Or install manually:

  pip install huggingface-hub markdown-it-py python-dotenv pyyaml requests

• Set

  HF_TOKEN

  environment variable with Write-access token
• For Artificial Analysis: Set

  AA_API_KEY

  environment variable

• .env

  is loaded automatically if

  python-dotenv

  is installed

**Via HF Jobs:**
```bash
hf jobs uv run scripts/lighteval_vllm_uv.py \
  --flavor a10g-small \
  --secrets HF_TOKEN=$HF_TOKEN \
  -- --model meta-llama/Llama-3.2-1B \
     --tasks "leaderboard|mmlu|5"

suite|task|num_fewshot

• leaderboard|mmlu|5

  - MMLU with 5-shot

• leaderboard|gsm8k|5

  - GSM8K with 5-shot

• lighteval|hellaswag|0

  - HellaSwag zero-shot

• leaderboard|arc_challenge|25

  - ARC-Challenge with 25-shot

suite|task|num_fewshot|0

0

• leaderboard

  - Open LLM Leaderboard tasks (MMLU, GSM8K, ARC, HellaSwag, etc.)

• lighteval

  - Additional lighteval tasks

• bigbench

  - BigBench tasks

• original

  - Original benchmark tasks

suite|task|num_fewshot

0

--tasks

• From file:

  leaderboard|mmlu|0

  → Use:

  leaderboard|mmlu|0

  (or change to

  5

  for 5-shot)
• From file:

  bigbench|abstract_narrative_understanding|0

  → Use:

  bigbench|abstract_narrative_understanding|0

• From file:

  lighteval|wmt14:hi-en|0

  → Use:

  lighteval|wmt14:hi-en|0

--tasks "leaderboard|mmlu|5,leaderboard|gsm8k|5"

**Via HF Jobs:**
```bash
hf jobs uv run scripts/inspect_vllm_uv.py \
  --flavor a10g-small \
  --secrets HF_TOKEN=$HF_TOKEN \
  -- --model meta-llama/Llama-3.2-1B \
     --task mmlu

• mmlu

  - Massive Multitask Language Understanding

• gsm8k

  - Grade School Math

• hellaswag

  - Common sense reasoning

• arc_challenge

  - AI2 Reasoning Challenge

• truthfulqa

  - TruthfulQA benchmark

• winogrande

  - Winograd Schema Challenge

• humaneval

  - Code generation

• Table Not Found: Script will report if no evaluation tables are detected
• Invalid Format: Clear error messages for malformed tables
• API Errors: Retry logic for transient Artificial Analysis API failures
• Token Issues: Validation before attempting updates
• Merge Conflicts: Preserves existing model-index entries when adding new ones
• Space Creation: Handles naming conflicts and hardware request failures gracefully

1. Check for existing PRs first: Run

   get-prs

   before creating any new PR to avoid duplicates
2. Always start with

   inspect-tables

   : See table structure and get the correct extraction command
3. Use

   --help

   for guidance: Run

   inspect-tables --help

   to see the complete workflow
4. Preview first: Default behavior prints YAML; review it before using

   --apply

   or

   --create-pr

5. Verify extracted values: Compare YAML output against the README table manually
6. Use

   --table N

   for multi-table READMEs: Required when multiple evaluation tables exist
7. Use

   --model-name-override

   for comparison tables: Copy the exact column header from

   inspect-tables

   output
8. Create PRs for Others: Use

   --create-pr

   when updating models you don't own
9. One model per repo: Only add the main model's results to model-index
10. No markdown in YAML names: The model name field in YAML should be plain text

• Removes markdown formatting (bold

  **

  , links

  []()

  )
• Normalizes names (lowercase, replace

  -

  and

  _

  with spaces)
• Compares token sets:

  "OLMo-3-32B"

  →

  {"olmo", "3", "32b"}

  matches

  "**Olmo 3 32B**"

  or

  "[Olmo-3-32B](...)

• Only extracts if tokens match exactly (handles different word orders and separators)
• Fails if no exact match found (rather than guessing from similar names)

• Finds the column header matching the model name
• Extracts scores from that column only

• Finds the row in the first column matching the model name
• Extracts all benchmark scores from that row only

• Solution: Check if README contains markdown tables with numeric scores

• Solution: The script will display available models. Use

  --model-name-override

  with the exact name from the list
• Example:

  --model-name-override "**Olmo 3-32B**"

• Solution: Set environment variable or add to .env file

• Solution: Ensure HF_TOKEN has write permissions for the repository

• Solution: Verify creator-slug and model-name match API values

• Solution: Add a payment method to your Hugging Face account to use non-CPU hardware

• Solution: Use a larger hardware flavor, reduce

  --gpu-memory-utilization

  , or use

  --tensor-parallel-size

  for multi-GPU

• Solution: Use

  --backend hf

  (inspect-ai) or

  --backend accelerate

  (lighteval) for HuggingFace Transformers

• Solution: Add

  --trust-remote-code

  flag for models with custom code (e.g., Phi-2, Qwen)

• Solution: Only use

  --use-chat-template

  for instruction-tuned models that include a chat template