generate-judge-problems

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Generate judge v2 problems

生成 Judge v2 题目

Read these references before authoring any file:

references/problem-format.md — judge v2 directory layout and
```
problem.md
```
frontmatter schema.
references/problem-authoring.md — statement writing rules, difficulty levels, and machine-grading constraints.
references/example-problem.md — a complete minimal v2 example problem.

Generate v2 problems only: the statement file is exactly

problem.md

inside the problem directory. Never generate the legacy

<id>.problem.md

naming.

Confirm the inputs: learning objectives or the source material, target course, target programming language(s), number of problems, and difficulty range. Place each problem under
```
<courseId>/problems/<problemId>/
```
; never create a repository-global problem or reference another course's problem. Derive problem drafts (id, name, overview, objective, difficulty) from the inputs; when generating multiple problems, order them from easy to hard per the difficulty levels in references/problem-authoring.md.
Make the problem files reproducible from the repository. Preserve existing mise configuration and compatible pinned versions. If none of
```
mise.toml
```
,
```
.mise.toml
```
, or
```
.tool-versions
```
exists, create
```
.tool-versions
```
; otherwise keep the existing format. Add exact versions of every runtime, package manager, and command-line tool required to execute the tracked model answers, templates, judges, and debuggers. Declare their imported third-party libraries in each language's standard project manifest with exact direct versions when supported, and update its standard lockfile. Run
```
mise trust --yes
```
for a mise TOML config, run
```
mise install
```
, and then use the installed commands normally. Do not rely on global installations, ad hoc downloads, or temporary dependency environments, and do not add tools used only to author or validate the problem.
For each problem, write the primary model answer FIRST under
```
model_answers/<languageId>/
```
(e.g.
```
python/main.py
```
). Solving the problem yourself before writing the statement exposes ambiguities early. Use only concepts the learner has already studied; when no studied-concept list is given, derive reasonable assumptions from the learning objectives and state them in the final report.
Write
```
problem.md
```
(frontmatter + Japanese statement) per references/problem-format.md, following the statement rules in references/problem-authoring.md. Do NOT create
```
judge.ts
```
or
```
debug.ts
```
for a standard stdin/stdout problem — the judge server auto-generates both, and committed default-content copies are rejected by the validator. Write a custom
```
judge.ts
```
(with a matching
```
debug.ts
```
) only when the problem needs special judging (GUI, LLM, command-based, etc.); a genuinely custom
```
debug.ts
```
may also accompany a standard stdio problem on its own when the problem needs customized debugging.
Create test inputs under
```
test_cases/
```
: at least 1
```
example_*
```
case (shown to learners and mirrored as 入力例/出力例 in the statement; for an input-only problem the 出力例 shows the judged output file, see references/problem-format.md) and at least 1 hidden case, at least 4 in total. Make inputs diverse and include edge cases (minimum/maximum constraint values, boundary conditions such as 0 or negative numbers where allowed). All inputs must be distinct. A program that reads files instead of stdin takes them from
```
<id>.fin/
```
; a program that writes files is judged with
```
<id>.fout/
```
(see references/problem-format.md).

Produce every

.out

file and every file under

.fout/

by RUNNING the model answer locally — never hand-write expected outputs. For a plain stdin/stdout problem:

bash

cd <problemDir> && for f in test_cases/*.in; do python3 model_answers/python/main.py < "$f" > "${f%.in}.out"; done

When any case uses

.fin/

_shared.fin/

.fout/

, or omits

.in

, run each case in a scratch directory inside the problem directory (so the repository's pinned tools still apply), feeding

<id>.in

when it exists, capturing stdout, and replacing

<id>.fout/

with what the program wrote. Save the following as

<problemDir>/.tmp/expected.sh

(an indented heredoc would not terminate when pasted) and run

bash -euo pipefail <problemDir>/.tmp/expected.sh [<outputFile> ...]

bash

# The produced file paths are the arguments; stdout-only problems pass none.
cd <problemDir>
root=$PWD
# A scratch directory inside the problem keeps the pinned tools; it is removed on success and kept on failure.
mkdir -p .tmp && scratch=$(mktemp -d .tmp/expected.XXXXXX)
trap 'st=$?; if [ $st -eq 0 ]; then rm -rf "$scratch"; else echo "kept $scratch for inspection" >&2; fi; exit $st' EXIT
# Every test case id (the shared name of its entries), excluding the shared input directory.
for id in $(ls -1 test_cases | sed -E -n 's/\.(in|out|fin|fout)$//p' | sort -u | grep -vx _shared); do
  # Stage the case's working directory: shared inputs first, then the case's own inputs.
  w=$scratch/$id && mkdir -p "$w"
  if [ -d test_cases/_shared.fin ]; then cp -R test_cases/_shared.fin/. "$w"/; fi
  if [ -d "test_cases/$id.fin" ]; then cp -R "test_cases/$id.fin/." "$w"/; fi
  # Run the model answer with the case's stdin (if any) and capture its stdout.
  in=/dev/null && if [ -f "test_cases/$id.in" ]; then in="$root/test_cases/$id.in"; fi
  (cd "$w" && python3 "$root/model_answers/python/main.py" < "$in" > "$root/$scratch/$id.stdout")
  if [ -d "test_cases/$id.fout" ]; then
    # A file case: the produced file paths (the script arguments) are required and every file must exist
    # before .fout/ is refilled, which protects committed contents; .out is refreshed only when the case has one.
    [ $# -gt 0 ] || { echo "usage: expected.sh <outputFile> ... (test_cases/$id.fout/ needs the produced file paths)" >&2; exit 2; }
    for f in "$@"; do test -f "$w/$f"; done
    find "test_cases/$id.fout" -mindepth 1 -delete
    for f in "$@"; do mkdir -p "test_cases/$id.fout/$(dirname "$f")" && cp "$w/$f" "test_cases/$id.fout/$f"; done
    if [ -f "test_cases/$id.out" ]; then mv "$scratch/$id.stdout" "test_cases/$id.out"; fi
  else
    # A stdout case: the captured stdout is its expectation.
    mv "$scratch/$id.stdout" "test_cases/$id.out"
  fi
done

Pass every relative path the program writes as the script arguments; a stdout-only problem passes none (the loop refuses to replace a

.fout/

without arguments and verifies each file exists first, and

bash -euo pipefail

stops at the first failure whatever the interactive shell is); the recipe assumes every file case writes the same paths, so adapt it per case when they differ. Create

test_cases/<id>.fout/

beforehand for every case judged by files: the loop refills existing directories, writes

.out

for every other case, and refreshes an existing

.out

of a file case (create an empty

test_cases/<id>.out

beforehand when a file case's stdout is compared as well). Adapt the commands to the model answer's language. For a custom

judge.ts

, follow its own contract instead: keep the

.in

.fin/

and

_shared.fin/

entries it reads, and do not create

.out

when it does not compare stdout (e.g. it compares a file listed in

requiredOutputFilePaths

) — never print a marker such as the output file name just to have stdout.

Cross-check with a second model answer: write an independent model answer in another target language under
```
model_answers/
```
(for a single-language problem, write the second implementation in any language under
```
<problemDir>/.tmp/cross_check/
```
, judge it with the same command, and delete it afterwards instead of committing it), then judge every model answer with the
```
exercode-problem
```
CLI from the problem directory (the CLI ships with
```
@exercode/problem-utils
```
, which must be a declared repository dependency — see setup-exercode-course-repository):
bash
```
cd <problemDir> && bunx exercode-problem judge model_answers/<languageId>
```
Pass required harness parameters as the JSON second argument (e.g.
```
'{ "model": "<modelId>" }'
```
for an LLM judge). The run passes only when the command exits 0, prints at least one
```
TEST_CASE_RESULT
```
line, and every printed line has the accepted decision code
```
2000
```
(the comparison rule is described in references/problem-format.md). On any failure, diagnose the decision code and stderr and fix that cause — an output mismatch means the statement is ambiguous or an answer is wrong; regenerate the
```
.out
```
files only when the expected outputs change.
If you provide starter code under
```
templates/
```
, ensure it does NOT solve the problem: judge each template directory from the problem directory with
```
bunx exercode-problem judge templates/<languageId>
```
(judge
```
templates/_default
```
as
```
bunx exercode-problem judge templates/_default
```
, and files placed directly under
```
templates/
```
as
```
bunx exercode-problem judge templates
```
; in both cases pass the target language as
```
'{ "language": "<languageId>" }'
```
), confirm at least one test case is not accepted by an actually executed run — a
```
main file not found
```
result (decision code
```
1201
```
) means the language param matched no file in the template directory, so rerun with the language of the files it contains — and confirm no template directory contains, unchanged, every file a model answer is judged by (its source files for a standard problem, every file for a custom judge; shared helper modules may be identical).
If the statement constrains syntax (see the 制約 rules in references/problem-authoring.md), mirror the constraints in frontmatter
```
requiredRegExpsInCode
```
/
```
forbiddenRegExpsInCode
```
/
```
forbiddenTextsInCode
```
. Required patterns must match every model answer; forbidden patterns must match none, and must never match input-parsing helpers (
```
Scanner
```
,
```
map
```
,
```
int
```
,
```
split
```
, regexes). The patterns apply to submissions in every target language, so only use language-agnostic patterns; keep language-specific constraints (e.g. Python's
```
range
```
) in the statement prose instead.
Validate and fix until clean:
bash
```
bunx exercode-problem validate-problem <problemDir>...
```
Fix every reported error and rerun. Repeat until the command reports success. Address warnings too unless they are intentional; if you keep a warning, tell the user why in the final report.
Report the created problem directories with their difficulties, and any open decisions (e.g. constraint values you chose, intentionally kept warnings).

在编写任何文件前，请阅读以下参考资料：

references/problem-format.md — Judge v2 目录布局和
```
problem.md
```
frontmatter schema。
references/problem-authoring.md — 题面编写规则、难度级别和机器评分约束。
references/example-problem.md — 一个完整的最小 v2 示例题目。

仅生成 v2 题目：题面文件必须恰好是题目目录下的

problem.md

。绝不生成旧版的

<id>.problem.md

命名。

确认输入项：学习目标或源材料、目标课程、目标编程语言、题目数量和难度范围。将每道题放在
```
<courseId>/problems/<problemId>/
```
下；绝不创建仓库全局题目，也不要引用其他课程的题目。根据输入生成题目草稿（id、name、overview、objective、difficulty）；生成多道题时，按照 references/problem-authoring.md 中的难度级别从易到难排序。
确保题目文件可依据仓库复现。保留现有的 mise 配置和兼容的固定版本。如果
```
mise.toml
```
、
```
.mise.toml
```
或
```
.tool-versions
```
均不存在，则创建
```
.tool-versions
```
；否则保持现有格式。添加执行受版本控制的 model answer、模板、judge 和 debugger 所需的每种运行时、包管理器及命令行工具的精确版本。在每种语言的标准项目清单中声明其导入的第三方库；在支持的情况下使用精确的直接依赖版本，并更新该语言的标准 lockfile。对于 mise TOML 配置，运行
```
mise trust --yes
```
，然后运行
```
mise install
```
，之后正常使用已安装的命令。不要依赖全局安装、临时下载或临时依赖环境，也不要添加仅用于编写或验证题目的工具。
对于每道题，首先在
```
model_answers/<languageId>/
```
下编写主要 model answer（例如
```
python/main.py
```
）。先自行解题再编写题面，可以尽早发现歧义。仅使用学习者已经学过的概念；如果没有提供已学概念列表，则根据学习目标作出合理假设，并在最终报告中说明这些假设。
按照 references/problem-format.md 编写
```
problem.md
```
（frontmatter + 日语题面），并遵循 references/problem-authoring.md 中的题面规则。对于标准 stdin/stdout 题目，不要创建
```
judge.ts
```
或
```
debug.ts
```
——judge server 会自动生成二者，且验证器会拒绝提交默认内容的副本。仅当题目需要特殊评测时，才编写自定义
```
judge.ts
```
（并提供匹配的
```
debug.ts
```
），例如 GUI、LLM 或基于命令行的评测。如果标准 stdio 题目需要自定义调试，也可以单独提供真正自定义的
```
debug.ts
```
。
在
```
test_cases/
```
下创建测试输入：至少 1 个
```
example_*
```
用例（向学习者展示，并以
```
入力例/出力例
```
的形式镜像到题面中；对于仅输入型题目，
```
出力例
```
展示被评测的输出文件，参见 references/problem-format.md），以及至少 1 个隐藏用例，总数至少为 4。测试输入应多样化，并包含边界情况（约束值的最小值和最大值，以及在允许时包含 0 或负数等边界条件）。所有输入必须互不相同。读取文件而非 stdin 的程序从
```
<id>.fin/
```
获取输入；写出文件的程序使用
```
<id>.fout/
```
进行评测（参见 references/problem-format.md）。

必须通过在本地运行 model answer 来生成每个

.out

文件和

.fout/

下的每个文件——绝不手写预期输出。对于普通 stdin/stdout 题目：

bash

cd <problemDir> && for f in test_cases/*.in; do python3 model_answers/python/main.py < "$f" > "${f%.in}.out"; done

当任何用例使用

.fin/

、

_shared.fin/

或

.fout/

，或者省略

.in

时，请在题目目录内运行每个用例的临时工作目录（以便仓库的固定工具仍然生效），在存在

<id>.in

时将其作为 stdin 输入，捕获 stdout，并用程序写出的内容替换

<id>.fout/

。将以下内容保存为

<problemDir>/.tmp/expected.sh

（缩进的 heredoc 在粘贴时不会终止），并运行

bash -euo pipefail <problemDir>/.tmp/expected.sh [<outputFile> ...]

：

bash

# The produced file paths are the arguments; stdout-only problems pass none.
cd <problemDir>
root=$PWD
# A scratch directory inside the problem keeps the pinned tools; it is removed on success and kept on failure.
mkdir -p .tmp && scratch=$(mktemp -d .tmp/expected.XXXXXX)
trap 'st=$?; if [ $st -eq 0 ]; then rm -rf "$scratch"; else echo "kept $scratch for inspection" >&2; fi; exit $st' EXIT
# Every test case id (the shared name of its entries), excluding the shared input directory.
for id in $(ls -1 test_cases | sed -E -n 's/\.(in|out|fin|fout)$//p' | sort -u | grep -vx _shared); do
  # Stage the case's working directory: shared inputs first, then the case's own inputs.
  w=$scratch/$id && mkdir -p "$w"
  if [ -d test_cases/_shared.fin ]; then cp -R test_cases/_shared.fin/. "$w"/; fi
  if [ -d test_cases/$id.fin ]; then cp -R test_cases/$id.fin/. "$w"/; fi
  # Run the model answer with the case's stdin (if any) and capture its stdout.
  in=/dev/null && if [ -f "test_cases/$id.in" ]; then in="$root/test_cases/$id.in"; fi
  (cd "$w" && python3 "$root/model_answers/python/main.py" < "$in" > "$root/$scratch/$id.stdout")
  if [ -d "test_cases/$id.fout" ]; then
    # A file case: the produced file paths (the script arguments) are required and every file must exist
    # before .fout/ is refilled, which protects committed contents; .out is refreshed only when the case has one.
    [ $# -gt 0 ] || { echo "usage: expected.sh <outputFile> ... (test_cases/$id.fout/ needs the produced file paths)" >&2; exit 2; }
    for f in "$@"; do test -f "$w/$f"; done
    find "test_cases/$id.fout" -mindepth 1 -delete
    for f in "$@"; do mkdir -p "test_cases/$id.fout/$(dirname "$f")" && cp "$w/$f" "test_cases/$id.fout/$f"; done
    if [ -f "test_cases/$id.out" ]; then mv "$scratch/$id.stdout" "test_cases/$id.out"; fi
  else
    # A stdout case: the captured stdout is its expectation.
    mv "$scratch/$id.stdout" "test_cases/$id.out"
  fi
done

将程序写入的每个相对路径作为脚本参数传入；仅使用 stdout 的题目不传任何参数（循环会拒绝替换没有参数的

.fout/

，并在替换前验证每个文件都存在，而

bash -euo pipefail

无论交互 shell 如何，都会在首次失败时停止）；该方案假设每个文件用例写入相同的路径，因此当各用例路径不同时，需按用例调整命令。预先为每个按文件评测的用例创建

test_cases/<id>.fout/

：循环会重新填充已有目录，为其他每个用例写入

.out

，并刷新文件用例已有的

.out

（如果文件用例的 stdout 也要比较，请预先创建一个空的

test_cases/<id>.out

）。根据 model answer 的语言调整命令。对于自定义

judge.ts

，改为遵循其自身约定：保留它读取的

.in

、

.fin/

和

_shared.fin/

条目；如果它不比较 stdout（例如比较

requiredOutputFilePaths

中列出的文件），则不要创建

.out

——绝不要仅仅为了产生 stdout 而打印输出文件名之类的标记。

使用第二个 model answer 进行交叉核验：在
```
model_answers/
```
下用另一个目标语言编写独立的 model answer（对于单语言题目，在
```
<problemDir>/.tmp/cross_check/
```
下用任意语言编写第二个实现，使用相同命令进行评测，之后删除它而不提交），然后从题目目录运行
```
exercode-problem
```
CLI，评测每个 model answer（该 CLI 随
```
@exercode/problem-utils
```
提供，后者必须声明为仓库依赖——参见 setup-exercode-course-repository）：
bash
```
cd <problemDir> && bunx exercode-problem judge model_answers/<languageId>
```
将所需的 harness 参数作为第二个 JSON 参数传入（例如，LLM judge 使用
```
'{ "model": "<modelId>" }'
```
）。只有当命令以 0 退出、至少打印一行
```
TEST_CASE_RESULT
```
，且打印的每一行都包含可接受的判定码
```
2000
```
时，运行才通过（比较规则见 references/problem-format.md）。任何失败都应诊断判定码和 stderr 并修复根本原因——输出不匹配意味着题面存在歧义或答案有误；仅当预期输出发生变化时才重新生成
```
.out
```
文件。
如果提供了
```
templates/
```
下的起始代码，请确保它不能解决题目：从题目目录分别评测每个模板目录（评测
```
templates/_default
```
的命令为
```
bunx exercode-problem judge templates/_default
```
，直接放在
```
templates/
```
下的文件则使用
```
bunx exercode-problem judge templates
```
；在这两种情况下，都将目标语言作为
```
'{ "language": "<languageId>" }'
```
传入）。确认在实际执行的运行中至少有一个测试用例未被接受——出现
```
main file not found
```
结果（判定码
```
1201
```
）意味着 language 参数没有匹配模板目录中的任何文件，因此应改用模板目录内文件所用的语言重新运行。并确认任何模板目录都没有原样包含 model answer 评测所依据的每个文件（标准题目为其源文件，自定义 judge 为所有文件；共享辅助模块可以相同）。
如果题面约束语法（参见 references/problem-authoring.md 中的约束规则），则在 frontmatter
```
requiredRegExpsInCode
```
/
```
forbiddenRegExpsInCode
```
/
```
forbiddenTextsInCode
```
中镜像这些约束。必需模式必须与每个 model answer 匹配；禁止模式必须与任何内容都不匹配，并且绝不能匹配输入解析辅助代码（
```
Scanner
```
、
```
map
```
、
```
int
```
、
```
split
```
、regexes）。这些模式适用于所有目标语言的提交，因此只能使用与语言无关的模式；语言特定的约束（例如 Python 的
```
range
```
）应改为保留在题面正文中。
验证并修复，直至通过：
bash
```
bunx exercode-problem validate-problem <problemDir>...
```
修复所有报告的错误并重新运行。重复操作，直到命令报告成功。警告也应处理，除非它们是有意保留的；如果保留警告，请在最终报告中说明原因。
报告已创建的题目目录及其难度，以及任何待决事项（例如你选择的约束值、有意保留的警告）。