test -f ~/.config/jetty/token && echo "JETTY_OK" || echo "NO_TOKEN"

NO_TOKEN

"You'll need a Jetty account first. Run

/jetty-setup

to get started, then come back here."

JETTY_OK

What's a runbook?

	Skill	Workflow	Runbook
Format	Markdown (SKILL.md)	JSON (step configs)	Markdown (RUNBOOK.md)
Executed by	Coding agent	Jetty engine	Coding agent, calling workflows/APIs
Complexity	Single tool or short procedure	Fixed pipeline	Multi-phase process with judgment
Iteration	None — one-shot	None — runs to completion	Built-in: evaluate → refine → re-evaluate

A skill says "here's how to call the Jetty API." A runbook says "here's how to pull data, process it, evaluate the results, iterate until they're good enough, and produce a report — and here's how to know when you're done."

Let's build one.

• Header: "Task"
• Question: "What task do you want to automate? Describe it in a sentence or two — what goes in, what processing happens, and what comes out."
• Options:
  • "I'll describe it" / "Let me type a description" (user types in the text field)
  • "Show me examples first" / "Show example runbook tasks before I decide"

Example runbook tasks:

1. NL-to-SQL Regression — Pull failed queries from Langfuse, replay them against the NL-to-SQL API, execute on Snowflake, evaluate pass/fail, produce a regression report
2. PDF-to-Metadata Conversion — Extract metadata from academic PDFs, generate Croissant JSON-LD, validate against the schema, iterate on errors
3. Branded Social Graphics — Parse a text script, generate an AI image via Jetty workflow, compose HTML with text overlays, judge against a brand rubric, iterate
4. Clinical Training Content — Parse competency documents, generate training scenarios with rubric-scored quality, produce learning plans
5. Data Extraction Pipeline — Extract structured data from documents into multiple formats, validate schema compliance, produce quality report

• Header: "Evaluation"
• Question: "How will you know when the output is good enough?"
• Options:
  • "Programmatic checks" / "I can validate with code, a schema, tests, or an API (objective pass/fail)"
  • "Quality rubric" / "I need to score against multiple criteria (subjective quality on a 1-5 scale)"
  • "Help me decide" / "Not sure which fits my task"

Programmatic is right when:

• Your output is structured data (JSON, CSV, SQL)
• You can validate with a schema, test suite, or API call
• Pass/fail is objective — it either works or it doesn't
• Examples: schema validation, SQL execution, test suites, API response checks

Rubric is right when:

• Your output is creative or complex (text, images, reports, designs)
• Quality is subjective across multiple dimensions
• You need a numeric score to track improvement
• Examples: content quality, brand compliance, UX evaluation, report comprehensiveness

programmatic

rubric

• Header: "Location"
• Question: "Where should I create the RUNBOOK.md file?"
• Options:
  • "Here" / "Create ./RUNBOOK.md in the current directory"
  • "Custom path" / "Let me specify where to put it" (user types a path)

• Header: "Agent Runtime"
• Question: "Which agent will run this runbook on Jetty?"
• Options:
  • "Claude Code (Anthropic)" / "Uses claude-sonnet-4-6 — strong at reasoning and tool use. Requires an Anthropic API key."
  • "Codex (OpenAI)" / "Uses gpt-5.4 — strong at code generation. Requires an OpenAI API key."
  • "Gemini CLI (Google)" / "Uses gemini-3.1-pro-preview — free tier available. Requires a Google AI API key."

• Claude Code → agent:

  claude-code

  , model:

  claude-sonnet-4-6

• Codex → agent:

  codex

  , model:

  gpt-5.4

• Gemini CLI → agent:

  gemini-cli

  , model:

  gemini-3.1-pro-preview

• Header: "Sandbox"
• Question: "Will your runbook need a web browser (Playwright)?\nExamples: taking screenshots, web scraping, OAuth flows, testing web UIs"
• Options:
  • "Yes, I need a browser" / "Use prism-playwright snapshot (Python 3.12, uv, Playwright + Chromium pre-installed)"
  • "No browser needed" / "Use python312-uv snapshot (lighter, faster startup)"

• Programmatic: Read

  templates/programmatic.md

  from the skill's directory
• Rubric: Read

  templates/rubric.md

  from the skill's directory

find ~/.claude -path "*/create-runbook/templates/programmatic.md" 2>/dev/null | head -1

find . -path "*/create-runbook/templates/programmatic.md" 2>/dev/null | head -1

1. Title: Replace

   {Task Name}

   with a concise name derived from the task description
2. Objective: Write a 2-5 sentence objective based on what the user described — input, processing, output
3. Output manifest: Propose specific output files based on the task (replace

   {primary_output}

   with a real filename like

   results.csv

   ,

   output.json

   ,

   report.html

   , etc.)
4. Parameters: Propose parameters based on inputs mentioned in the task description. Always keep

   {{results_dir}}

   .
5. Agent/model/snapshot: Write the choices from Step 2d into the frontmatter fields
6. Steps 2-3: Rename and briefly describe the processing steps based on the task

{TODO: ...}

{How to fix it}

"I've created a runbook scaffold at

{path}

. It has the full structure with your task details filled in and placeholder markers where I need your input. Let's walk through each section."

• Header: "Objective"
• Question: "Here's the objective I drafted:\n\n{show the objective text}\n\nDoes this capture your task accurately?"
• Options:
  • "Looks good" / "Move on to the next section"
  • "Needs changes" / "Let me refine it" (user types corrections)

• Header: "Output Files"
• Question: "These are the files the runbook will produce:\n\n{list the files}\n\nDoes this look right?"
• Options:
  • "Looks good" / "This manifest is correct"
  • "Add a file" / "I need an additional output file"
  • "Change a file" / "One of these needs to be different"

validation_report.json

summary.md

• Header: "Parameters"
• Question: "These are the configurable inputs:\n\n{list parameters}\n\nAnything to add or change?"
• Options:
  • "Looks good" / "These parameters are sufficient"
  • "Add more" / "I need additional parameters" (user describes them)

• Header: "Dependencies"
• Question: "What does your runbook need beyond the base environment?"
• Options:
  • "Jetty workflows" / "I'll call Jetty workflows as sub-steps"
  • "External APIs" / "I call non-Jetty APIs (REST, GraphQL, etc.)"
  • "Python/Node packages" / "I need specific libraries installed"
  • "None" / "No special dependencies — just standard tools"

• Header: "Secrets"
• Question: "Does this runbook need any API keys, tokens, or other credentials?"
• Options:
  • "Yes" / "I need to declare secrets for API keys or credentials"
  • "No" / "No sensitive parameters needed"

• Logical name (e.g.,

  OPENAI_API_KEY

  )
• Collection environment variable name (usually same as logical name)
• Description
• Required or optional

secrets

secrets:
  OPENAI_API_KEY:
    env: OPENAI_API_KEY
    description: "OpenAI API key for embeddings"
    required: true

• Header: "Processing Steps"
• Question: "Here's the step sequence I'm proposing:\n\n{numbered list of steps}\n\nWant to adjust?"
• Options:
  • "Looks good" / "This sequence works"
  • "Add a step" / "I need an additional step"
  • "Change order" / "The steps need reordering"
  • "Remove a step" / "One of these isn't needed"

• Step name as header
• 2-3 sentence description of what to do
• Placeholder for API calls or code snippets:

  {TODO: add API call examples and expected response format}

• Placeholder for error handling:

  {TODO: add error handling for common failures}

• Header: "Pass/Fail Criteria"
• Question: "Define what PASS, PARTIAL, and FAIL mean for your outputs. What makes an output correct? What makes it partially correct? What's a failure?"
• Options:
  • "I'll define them" / "Let me describe each status" (user types)
  • "Use defaults" / "Keep the template defaults and I'll refine later"

• Header: "Rubric Criteria"
• Question: "What criteria matter for your output quality? Name 3-7 dimensions you'd score on a 1-5 scale.\n\nExamples: accuracy, completeness, clarity, brand compliance, technical correctness, creativity, formatting"
• Options:
  • "I'll list them" / "Let me name my criteria" (user types)
  • "Use 5 defaults" / "Start with generic criteria and I'll customize later"

• Header: "Rubric Details"
• Question: "For each criterion, briefly describe what 5 (excellent) and 1 (poor) look like. Or just list the criteria names and I'll draft reasonable descriptions.\n\n{list their criteria}"
• Options:
  • "I'll describe them" / "Let me define the scale for each" (user types)
  • "You draft them" / "Write reasonable descriptions and I'll review"

• Header: "Common Fixes"
• Question: "Do you know the typical failure modes for this task? If so, describe them and I'll build a fix table. If not, you can fill this in after your first few runs."
• Options:
  • "I know some" / "Let me describe common issues" (user types)
  • "Skip for now" / "I'll fill this in after running the runbook"

• Header: "Tips"
• Question: "Any domain-specific gotchas, API quirks, or hard-won lessons you want to capture? These help the agent avoid known pitfalls."
• Options:
  • "Yes" / "I have some tips to add" (user types)
  • "Skip" / "Nothing comes to mind — I'll add tips later"

• Header: "Test"
• Question: "Want me to do a dry run? I'll read through your runbook step by step and list what each step would do — flagging any missing credentials, unavailable APIs, or potential issues — without actually executing anything."
• Options:
  • "Yes, dry run" / "Walk through the plan without executing"
  • "Skip" / "I'll test it myself later"

1. List all parameters and whether they have values or need to be provided at runtime
2. For each step, describe what the agent would do:
   • Which APIs or services it would call
   • What data it would process
   • What files it would write
3. Flag potential issues:
   • Parameters without defaults that need values
   • External APIs or credentials referenced
   • Jetty workflows that need to exist
   • Packages that need to be installed
4. Estimate the rough scope (number of API calls, expected outputs)

{{results_dir}}

{results_dir}/plan.md

mkdir -p ./results

./results/plan.md

Your runbook is ready! Here's how to use it:

Run it locally: Open the runbook in a new conversation and tell the agent to follow it: "Follow the runbook in ./RUNBOOK.md. Use these parameters: results_dir=./results, {other params}..."

Run it on Jetty (recommended): Use the chat-completions endpoint with a

jetty

block — this is the single API call that configures everything: which agent runs it, which collection it belongs to, and what files to upload into the sandbox.

bash

curl -X POST "https://flows-api.jetty.io/v1/chat/completions" \
  -H "Authorization: Bearer $JETTY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "{model from frontmatter}",
    "messages": [
      {"role": "system", "content": "<contents of your RUNBOOK.md>"},
      {"role": "user", "content": "Execute the runbook."}
    ],
    "stream": true,
    "jetty": {
      "runbook": true,
      "collection": "{your-collection}",
      "task": "{task-name}",
      "agent": "{agent from frontmatter}",
      "file_paths": []
    }
  }'

The

jetty

block fields map directly to your runbook's frontmatter:

Frontmatter field	jetty block field	Purpose
agent	jetty.agent	Which agent CLI runs the runbook ( claude-code , codex , gemini-cli )
model	model (top-level)	Which LLM the agent uses
snapshot	(auto-selected)	Sandbox environment — set via collection config
—	jetty.collection	Namespace that holds your env vars and secrets
—	jetty.task	Task name for grouping trajectories
—	jetty.file_paths	Files to upload into the sandbox workspace

Or use

/jetty run runbook

to have the agent build this request for you interactively.

Iterate on the runbook: After your first few runs, come back and:

• Add entries to the Common Fixes table based on failures you observe
• Add Tips for gotchas the agent encountered
• Tighten evaluation criteria as your quality bar becomes clearer
• Bump the version when you make structural changes

Re-validate after changes: Run

/create-runbook

again on an existing RUNBOOK.md to re-validate it, or run the validation script from Step 5 manually.

• Always keep

  validation_report.json

  in the output manifest. This is the standardized machine-readable results filename across all Jetty runbooks. Never use

  scores.json

  ,

  results.json

  , or other variants.
• The

  {{results_dir}}

  parameter defaults to

  /app/results

  when running on Jetty and

  ./results

  when running locally.
• Bound iteration. Every iteration loop must specify a maximum round count (typically 3). Without bounds, the agent may loop indefinitely.
• Use imperative language in the output manifest and final checklist. Agents tend to wrap up early when they encounter errors — strong language like "Do NOT finish until all items pass" overrides this.
• Don't over-specify intermediate steps. The agent should have room to adapt. Specify what each step must produce, not every line of code.
• Don't mix evaluation patterns. Programmatic validation for structured output, rubric scoring for creative output. Don't rubric-score a JSON file or schema-validate a social graphic.