Config Generation Progress:
- [ ] Step 0: Check workspace (if MODELOPT_WORKSPACE_ROOT is set)
- [ ] Step 1: Check if nel is installed and if user has existing config
- [ ] Step 2: Build the base config file
- [ ] Step 3: Configure model path and parameters
- [ ] Step 4: Fill in remaining missing values
- [ ] Step 5: Confirm tasks (iterative)
- [ ] Step 6: Advanced - Multi-node (Data Parallel)
- [ ] Step 7: Advanced - Interceptors
- [ ] Step 7.5: Check container registry auth (SLURM only)
- [ ] Step 8: Run the evaluation

nel

nel --version

pip install nemo-evaluator-launcher

???

recipes/tasks/

1. Read the task snippet(s) the user wants
2. Use

   recipes/examples/example_eval.yaml

   as the base config template
3. Replace the

   tasks:

   section with the selected snippet(s)
4. Do Step 3 (auto-detect model settings from checkpoint) and Step 4 (fill in

   ???

   values)
5. Proceed to Step 7.5/8

1. Execution:

• Local
• SLURM

1. Deployment:

• None (External)
• vLLM
• SGLang
• NIM
• TRT-LLM

1. Auto-export:

• None (auto-export disabled)
• MLflow
• wandb

1. Model type

• Base
• Chat
• Reasoning

1. Benchmarks: Allow for multiple choices in this question.
2. Standard LLM Benchmarks (like MMLU, IFEval, GSM8K, ...)
3. Code Evaluation (like HumanEval, MBPP, and LiveCodeBench)
4. Math & Reasoning (like AIME, GPQA, MATH-500, ...)
5. Safety & Security (like Garak and Safety Harness)
6. Multilingual (like MMATH, Global MMLU, MMLU-Prox)

Note: These categories come from NEL's

build-config

CLI. Always run

nel skills build-config --help

first to get the current options — they may differ from this list (e.g.,

chat_reasoning

instead of separate

chat

/

reasoning

,

general_knowledge

instead of

standard

). When the CLI's current options differ from this list, prefer the CLI's options.

nel skills build-config --execution <local|slurm> --deployment <none|vllm|sglang|nim|trtllm> --model_type <base|chat|reasoning> --benchmarks <standard|code|math_reasoning|safety|multilingual> [--export <none|mlflow|wandb>] [--output <OUTPUT>]

--output

• Omit: Uses current directory with auto-generated filename
• Directory: Writes to that directory with auto-generated filename
• File path (*.yaml): Writes to that specific file

• Checkpoint path (local directory — starts with

  /

  ,

  ./

  ,

  ../

  ,

  ~

  , or contains no

  /

  but exists on disk) → set

  deployment.checkpoint_path: <path>

  and

  deployment.hf_model_handle: null

• HF handle (e.g.,

  org/model-name

  — contains exactly one

  /

  and does not exist locally) → set

  deployment.hf_model_handle: <handle>

  and

  deployment.checkpoint_path: null

hf_quant_config.json

cat <checkpoint_path>/hf_quant_config.json 2>/dev/null

quantization.quant_algo

deployment.extra_args

quant_algo

FP8

--quantization modelopt

W4A8_AWQ

--quantization modelopt

NVFP4

NVFP4_AWQ

--quantization modelopt_fp4

--quantization modelopt

hf_quant_config.json

config.json

quantization_config

quant_method: "modelopt"

Note: Some models require additional env vars for deployment (e.g.,

VLLM_NVFP4_GEMM_BACKEND=marlin

for Nemotron Super). These are not in

hf_quant_config.json

— they are discovered during model card research below.

config.json

deployment.extra_args

cat <checkpoint_path>/config.json 2>/dev/null

config.json

max_position_embeddings

--max-model-len <value>

131072

--max-model-len 131072

auto_map

--trust-remote-code

--reasoning-parser

--reasoning-parser-plugin

--enable-auto-tool-choice --tool-call-parser <parser>

--mamba_ssm_cache_dtype float32

deployment.extra_args

--max-model-len 32768

config.json

references/quantization-benchmarks.md

references/model-card-research.md

• Find all remaining

  ???

  missing values in the config.
• Ask the user only for values that couldn't be auto-discovered from the model card (e.g., SLURM hostname, account, output directory, MLflow/wandb tracking URI). Don't propose any defaults here. Let the user give you the values in plain text.
• Ask the user if they want to change any other defaults e.g. execution partition or walltime (if running on SLURM) or add MLflow/wandb tags (if auto-export enabled).

1. Tell the user: "Run

   nel ls tasks

   to see all available tasks".
2. Ask if they want to add/remove tasks or add/remove/modify task-specific parameter overrides. To add per-task

   nemo_evaluator_config

   as specified by the user, e.g.:
   yaml

   tasks:
     - name: <task>
       nemo_evaluator_config:
         config:
           params:
             temperature: <value>
             max_new_tokens: <value>
             ...

3. Apply changes.
4. Show updated list and ask: "Is the task list final, or do you want to make more changes?"

• NeMo-Skills workaround (self-deployment only): If using

  nemo_skills.*

  tasks with self-deployment (vLLM/SGLang/NIM), add at top level:
  yaml

  target:
    api_endpoint:
      api_key_name: DUMMY_API_KEY

  For the None (External) deployment the

  api_key_name

  should be already defined. The

  DUMMY_API_KEY

  export is handled in Step 8.

references/multi-node.md

• Tell the user they should see: https://docs.nvidia.com/nemo/evaluator/latest/libraries/nemo-evaluator/interceptors/index.html .
• DON'T provide any general information about what interceptors typically do in API frameworks without reading the docs. If the user asks about interceptors, only then read the webpage to provide precise information.
• If the user asks you to configure some interceptor, then read the webpage of this interceptor and configure it according to the

  --overrides

  syntax but put the values in the YAML config under

  evaluation.nemo_evaluator_config.config.target.api_endpoint.adapter_config

  (NOT under

  target.api_endpoint.adapter_config

  ) instead of using CLI overrides. By defining

  interceptors

  list you'd override the full chain of interceptors which can have unintended consequences like disabling default interceptors. That's why use the fields specified in the

  CLI Configuration

  section after the

  --overrides

  keyword to configure interceptors in the YAML config.

• The docs may show incorrect parameter names for logging. Use

  max_logged_requests

  and

  max_logged_responses

  (NOT

  max_saved_*

  or

  max_*

  ).

vllm/vllm-openai:latest

lmsysorg/sglang:latest

nvcr.io/nvidia/tensorrt-llm/release:...

nvcr.io/nvidia/eval-factory/*:26.03

skills/common/slurm-setup.md

ssh <host> "grep -E '^\s*machine\s+' ~/.config/enroot/.credentials 2>/dev/null"

1. Check if the cluster has credentials for the default DockerHub image (see command above)
2. If DockerHub credentials exist → use the default image and submit
3. If DockerHub credentials are missing but can be added → add them (see

   slurm-setup.md

   section 6), then submit
4. If DockerHub credentials cannot be added → override

   deployment.image

   to the NGC alternative and submit:
   yaml

   deployment:
     image: nvcr.io/nvidia/vllm:<YY.MM>-py3  # check https://catalog.ngc.nvidia.com/orgs/nvidia/containers/vllm for latest tag

5. Do not retry more than once without fixing the auth issue

recipes/env.example

cp recipes/env.example .env

1. **Dry-run** (validates config without running):

   ```bash
   nel run --config <config_path> --dry-run

1. Test with limited samples (quick validation run):
   bash

   nel run --config <config_path> -o ++evaluation.nemo_evaluator_config.config.params.limit_samples=10

2. Re-run a single task (useful for debugging or re-testing after config changes):
   bash

   nel run --config <config_path> -t <task_name>

   Combine with

   -o

   for limited samples:

   nel run --config <config_path> -t <task_name> -o ++evaluation.nemo_evaluator_config.config.params.limit_samples=10

3. Full evaluation (production run):
   bash

   nel run --config <config_path>

nel

undefined