Senior Prompt Engineer

Overview

Phase 1: Requirements

Actions

1. Define the task objective clearly
2. Identify input/output format requirements
3. Determine quality criteria (accuracy, tone, format)
4. Assess edge cases and failure modes
5. Choose model and parameter constraints

STOP — Do NOT proceed to Phase 2 until:

• Task objective is stated in one sentence
• Input format and output format are defined
• Quality criteria are measurable
• Edge cases are listed
• Model selection is justified

Phase 2: Prompt Design

Actions

1. Draft system prompt with role, constraints, and format
2. Design few-shot examples (3-5 representative cases)
3. Add chain-of-thought scaffolding if reasoning is needed
4. Specify output structure (JSON, markdown, etc.)
5. Add error handling instructions

Prompt Architecture Layers

System Prompt Template

[Role] You are a [specific role] that [specific capability].

[Context] You have access to [tools/knowledge]. The user will provide [input type].

[Instructions]
1. First, [step 1]
2. Then, [step 2]
3. Finally, [step 3]

[Constraints]
- Always [requirement]
- Never [prohibition]
- If uncertain, [fallback behavior]

[Output Format]
Respond in the following format:
[format specification]

[Examples]
<example>
Input: [sample input]
Output: [sample output]
</example>

STOP — Do NOT proceed to Phase 3 until:

• All 7 layers are addressed (or intentionally omitted with rationale)
• Examples are representative and diverse
• Output format is unambiguous
• Constraints are specific (not vague)

Phase 3: Evaluation and Iteration

Actions

1. Create evaluation dataset (50+ examples minimum)
2. Define scoring rubric (automated + human metrics)
3. Run baseline evaluation
4. Iterate on prompt with targeted improvements
5. A/B test promising variants
6. Version and document the final prompt

STOP — Evaluation complete when:

• Evaluation dataset covers all input categories
• Metrics meet defined quality thresholds
• A/B test shows statistical significance (p < 0.05)
• Final prompt is versioned with metrics

Few-Shot Example Design

Selection Criteria Decision Table

Example Count Guidelines

Example Format

<example>
<input>
[Representative input]
</input>
<reasoning>
[Optional: show the thinking process]
</reasoning>
<output>
[Expected output in exact target format]
</output>
</example>

Chain-of-Thought Patterns

CoT Pattern Decision Table

<analysis>...</analysis>

<answer>...</answer>

When to Use CoT

Structured Output

Output Format Decision Table

JSON.parse()

JSON Output Example

Respond with a JSON object matching this schema:
{
  "classification": "positive" | "negative" | "neutral",
  "confidence": number between 0 and 1,
  "reasoning": "brief explanation",
  "key_phrases": ["array", "of", "phrases"]
}

Do not include any text outside the JSON object.

Temperature and Top-P Tuning

Rules

• Temperature 0 for tasks requiring consistency and correctness
• Higher temperature for creative tasks
• Top-P rarely needs tuning (keep at 0.9-1.0)
• Do not use both high temperature AND low top-p (contradictory)

Evaluation Metrics

Automated Metrics

Human Evaluation Dimensions

Evaluation Dataset Requirements

• Minimum 50 examples for statistical significance
• Cover all input categories proportionally
• Include edge cases (10-20% of dataset)
• Gold labels reviewed by 2+ evaluators
• Version-controlled alongside prompts

A/B Testing Process

1. Define hypothesis: "Prompt B will improve [metric] by [amount]"
2. Hold all variables constant except the prompt change
3. Run both variants on the same evaluation set
4. Calculate metric differences with confidence intervals
5. Require statistical significance (p < 0.05) before adopting

What to A/B Test

Prompt Versioning

Version File Format

id: classify-sentiment
version: 2.1
model: claude-sonnet-4-20250514
temperature: 0.0
created: 2025-03-01
author: team
changelog: "Added edge case examples for sarcasm detection"
metrics:
  accuracy: 0.94
  f1: 0.92
  eval_dataset: sentiment-eval-v3
system_prompt: |
  You are a sentiment classifier...
examples:
  - input: "..."
    output: "..."

Versioning Rules

• Semantic versioning: major.minor (major = behavior change, minor = refinement)
• Every version includes evaluation metrics
• Link to evaluation dataset version
• Document what changed and why
• Keep previous versions for rollback

Anti-Patterns / Common Mistakes

Integration Points

llm-as-judge

acceptance-testing

testing-strategy

senior-data-scientist

code-review

clean-code

Skill Type