Golden Dataset Curation

Curate high-quality documents for the golden dataset with multi-agent validation

Overview

This skill provides patterns and workflows for adding new documents to the golden dataset with thorough quality analysis. It complements

golden-dataset-management

which handles backup/restore.

When to use this skill:

Adding new documents to the golden dataset
Classifying content types and difficulty levels
Generating test queries for new documents
Running multi-agent quality analysis

Content Types

Type	Description	Quality Focus
`article`	Technical articles, blog posts	Depth, accuracy, actionability
`tutorial`	Step-by-step guides	Completeness, clarity, code quality
`research_paper`	Academic papers, whitepapers	Rigor, citations, methodology
`documentation`	API docs, reference materials	Accuracy, completeness, examples
`video_transcript`	Transcribed video content	Structure, coherence, key points
`code_repository`	README, code analysis	Code quality, documentation

Difficulty Levels

Level	Semantic Complexity	Expected Score	Characteristics
trivial	Direct keyword match	>0.85	Technical terms, exact phrases
easy	Common synonyms	>0.70	Well-known concepts, slight variations
medium	Paraphrased intent	>0.55	Conceptual queries, multi-topic
hard	Multi-hop reasoning	>0.40	Cross-domain, comparative analysis
adversarial	Edge cases	Graceful degradation	Robustness tests, off-domain

Quality Dimensions

Dimension	Weight	Perfect	Acceptable	Failing
Accuracy	0.25	0.95-1.0	0.70-0.94	<0.70
Coherence	0.20	0.90-1.0	0.60-0.89	<0.60
Depth	0.25	0.90-1.0	0.55-0.89	<0.55
Relevance	0.30	0.95-1.0	0.70-0.94	<0.70

Evaluation focuses:

Accuracy: Technical correctness, code validity, up-to-date info
Coherence: Logical structure, clear flow, consistent terminology
Depth: Comprehensive coverage, edge cases, appropriate detail
Relevance: Alignment with AI/ML, backend, frontend, DevOps domains

Multi-Agent Pipeline

INPUT: URL/Content
        |
        v
+------------------+
|   FETCH AGENT    |  Extract structure, detect type
+--------+---------+
         |
         v
+-----------------------------------------------+
|  PARALLEL ANALYSIS AGENTS                      |
|  Quality | Difficulty | Domain  | Query Gen   |
+-----------------------------------------------+
         |
         v
+------------------+
| CONSENSUS        |  Weighted score + confidence
| AGGREGATOR       |  -> include/review/exclude
+--------+---------+
         |
         v
+------------------+
|  USER APPROVAL   |  Show scores, confirm
+--------+---------+
         |
         v
OUTPUT: Curated document entry

Decision Thresholds

Quality Score	Confidence	Decision
>= 0.75	>= 0.70	include
>= 0.55	any	review
< 0.55	any	exclude

Quality Thresholds

yaml

# Recommended thresholds for golden dataset inclusion
minimum_quality_score: 0.70
minimum_confidence: 0.65
required_tags: 2          # At least 2 domain tags
required_queries: 3       # At least 3 test queries

Coverage Balance Guidelines

Maintain balanced coverage across:

Content types: Don't over-index on articles
Difficulty levels: Need trivial AND hard queries
Domains: Spread across AI/ML, backend, frontend, etc.

Duplicate Prevention Checklist

Before adding:

Check URL against existing
```
source_url_map.json
```
Run semantic similarity against existing document embeddings
Warn if >80% similar to existing document

Provenance Tracking

Always record:

Source URL (canonical)
Curation date
Agent scores (for audit trail)
Langfuse trace ID

Langfuse Integration

Trace Structure

python

trace = langfuse.trace(
    name="golden-dataset-curation",
    metadata={"source_url": url, "document_id": doc_id}
)

# Log individual dimension scores
trace.score(name="accuracy", value=0.85)
trace.score(name="coherence", value=0.90)
trace.score(name="depth", value=0.78)
trace.score(name="relevance", value=0.92)

# Final aggregated score
trace.score(name="quality_total", value=0.87)
trace.event(name="curation_decision", metadata={"decision": "include"})

Managed Prompts

Prompt Name	Purpose
`golden-content-classifier`	Classify content_type
`golden-difficulty-classifier`	Assign difficulty
`golden-domain-tagger`	Extract tags
`golden-query-generator`	Generate test queries

References

For detailed implementation patterns, see:

```
references/selection-criteria.md
```
- Content type classification, difficulty stratification, quality evaluation dimensions, and best practices
```
references/annotation-patterns.md
```
- Multi-agent pipeline architecture, agent specifications, consensus aggregation logic, and Langfuse integration

Related Skills

```
golden-dataset-management
```
- Backup/restore operations
```
golden-dataset-validation
```
- Validation rules and checks
```
langfuse-observability
```
- Tracing patterns
```
pgvector-search
```
- Duplicate detection

Version: 1.0.0 (December 2025) Issue: #599

Capability Details

content-classification

Keywords: content type, classification, document type, golden dataset Solves:

Classify document content types for golden dataset
Categorize entries by domain and purpose
Identify content requiring special handling

difficulty-stratification

Keywords: difficulty, stratification, complexity level, challenge rating Solves:

Assign difficulty levels to golden dataset entries
Ensure balanced difficulty distribution
Identify edge cases and challenging examples

quality-evaluation

Keywords: quality, evaluation, quality dimensions, quality criteria Solves:

Evaluate entry quality against defined criteria
Score entries on multiple quality dimensions
Identify entries needing improvement

multi-agent-analysis

Keywords: multi-agent, parallel analysis, consensus, agent evaluation Solves:

Run parallel agent evaluations on entries
Aggregate consensus from multiple analysts
Resolve disagreements in classifications

golden-dataset-curation

NPX Install

Tags

SKILL.md Content