Chunking Strategies

• Implementing document chunking for RAG
• Optimizing chunk size and overlap
• Comparing different chunking strategies
• Benchmarking chunking performance
• Processing different document types (markdown, code, PDFs)
• Evaluating retrieval quality with different chunk strategies

• scripts/chunk-fixed-size.py

  - Fixed-size chunking implementation

• scripts/chunk-semantic.py

  - Semantic chunking with paragraph preservation

• scripts/chunk-recursive.py

  - Recursive chunking for hierarchical documents

• scripts/benchmark-chunking.py

  - Benchmark and compare chunking strategies

• templates/chunking-config.yaml

  - Chunking configuration template

• templates/custom-splitter.py

  - Template for custom chunking logic

• examples/chunk-markdown.py

  - Markdown-specific chunking

• examples/chunk-code.py

  - Source code chunking

• examples/chunk-pdf.py

  - PDF document chunking

Chunking Strategy Overview

Strategy Selection Guide

• Best for: Uniform documents, simple content, consistent structure
• Pros: Fast, predictable, simple implementation
• Cons: May split semantic units, no context awareness
• Use when: Speed matters more than semantic coherence

• Best for: Natural language documents, articles, books
• Pros: Preserves semantic boundaries, better context
• Cons: Slower, variable chunk sizes
• Use when: Content has clear paragraph/section structure

• Best for: Hierarchical documents, technical docs, code
• Pros: Preserves structure, handles nested content
• Cons: Most complex, requires structure detection
• Use when: Documents have clear hierarchical organization

• Best for: Q&A pairs, chatbots, precise retrieval
• Pros: Natural boundaries, good for citations
• Cons: Small chunks may lack context
• Use when: Need precise attribution and citations

Implementation Scripts

1. Fixed-Size Chunking

scripts/chunk-fixed-size.py

python scripts/chunk-fixed-size.py \
  --input document.txt \
  --chunk-size 1000 \
  --overlap 200 \
  --output chunks.json

• chunk-size

  : Number of characters per chunk (default: 1000)

• overlap

  : Character overlap between chunks (default: 200)

• split-on

  : Split on sentences, words, or characters (default: sentences)

• Use 500-1000 character chunks for most RAG applications
• Set overlap to 10-20% of chunk size
• Split on sentences for better coherence

2. Semantic Chunking

scripts/chunk-semantic.py

python scripts/chunk-semantic.py \
  --input document.txt \
  --max-chunk-size 1500 \
  --output chunks.json

1. Detects natural boundaries (paragraphs, headings, line breaks)
2. Groups content while respecting max chunk size
3. Preserves semantic units (paragraphs stay together)
4. Adds context headers for nested sections

3. Recursive Chunking

scripts/chunk-recursive.py

python scripts/chunk-recursive.py \
  --input document.md \
  --chunk-size 1000 \
  --separators '["\\n## ", "\\n### ", "\\n\\n", "\\n", " "]' \
  --output chunks.json

1. Tries to split on first separator (e.g., ## headings)
2. If chunks still too large, recursively splits on next separator
3. Continues until all chunks are within size limit
4. Preserves hierarchical context

• Markdown:

  ["\\n## ", "\\n### ", "\\n\\n", "\\n", " "]

• Python:

  ["\\nclass ", "\\ndef ", "\\n\\n", "\\n", " "]

• General:

  ["\\n\\n", "\\n", ". ", " "]

4. Benchmark Chunking Strategies

scripts/benchmark-chunking.py

python scripts/benchmark-chunking.py \
  --input document.txt \
  --strategies fixed,semantic,recursive \
  --chunk-sizes 500,1000,1500 \
  --output benchmark-results.json

• Processing time: Speed of chunking
• Chunk count: Total chunks generated
• Chunk size variance: Consistency of chunk sizes
• Context preservation: Semantic unit integrity (scored)
• Retrieval quality: Simulated query performance

{
  "fixed-1000": {
    "time_ms": 45,
    "chunk_count": 127,
    "avg_size": 982,
    "size_variance": 12.3,
    "context_score": 0.72
  },
  "semantic-1000": {
    "time_ms": 156,
    "chunk_count": 114,
    "avg_size": 1087,
    "size_variance": 234.5,
    "context_score": 0.91
  }
}

Configuration Template

templates/chunking-config.yaml

chunking:
  # Global defaults
  default_strategy: semantic
  default_chunk_size: 1000
  default_overlap: 200

  # Strategy-specific configs
  strategies:
    fixed_size:
      chunk_size: 1000
      overlap: 200
      split_on: sentence  # sentence, word, character

    semantic:
      max_chunk_size: 1500
      min_chunk_size: 200
      preserve_paragraphs: true
      add_headers: true  # Include section headers

    recursive:
      chunk_size: 1000
      overlap: 100
      separators:
        markdown: ["\\n## ", "\\n### ", "\\n\\n", "\\n", " "]
        code: ["\\nclass ", "\\ndef ", "\\n\\n", "\\n", " "]
        text: ["\\n\\n", ". ", " "]

  # Document type mappings
  document_types:
    ".md": semantic
    ".py": recursive
    ".txt": fixed_size
    ".pdf": semantic

Custom Splitter Template

templates/custom-splitter.py

from typing import List, Dict
import re

class CustomChunker:
    def __init__(self, chunk_size: int = 1000, overlap: int = 200):
        self.chunk_size = chunk_size
        self.overlap = overlap

    def chunk(self, text: str, metadata: Dict = None) -> List[Dict]:
        """
        Implement custom chunking logic here.

        Returns:
            List of chunks with metadata:
            [
                {
                    "text": "chunk content",
                    "metadata": {
                        "chunk_id": 0,
                        "source": "document.txt",
                        "start_char": 0,
                        "end_char": 1000
                    }
                }
            ]
        """
        chunks = []

        # Your custom chunking logic here
        # Example: Split on custom pattern
        sections = self._split_sections(text)

        for i, section in enumerate(sections):
            chunks.append({
                "text": section,
                "metadata": {
                    "chunk_id": i,
                    "source": metadata.get("source", "unknown"),
                    "chunk_size": len(section)
                }
            })

        return chunks

    def _split_sections(self, text: str) -> List[str]:
        # Implement your splitting logic
        pass

Document-Specific Examples

Markdown Chunking

examples/chunk-markdown.py

• Preserves heading hierarchy
• Keeps code blocks together
• Maintains list structure
• Adds parent section context to chunks

python examples/chunk-markdown.py README.md --output readme-chunks.json

Code Chunking

examples/chunk-code.py

• Splits on class and function boundaries
• Preserves complete functions
• Includes docstrings with implementations
• Language-aware separator selection

python examples/chunk-code.py src/main.py --language python --output code-chunks.json

PDF Chunking

examples/chunk-pdf.py

• Extracts text from PDF
• Preserves page boundaries
• Maintains formatting clues
• Handles multi-column layouts

pypdf

pdfminer.six

python examples/chunk-pdf.py research-paper.pdf --strategy semantic --output pdf-chunks.json

Optimization Guidelines

Chunk Size Selection

benchmark-chunking.py

Overlap Strategies

• Prevents information loss at boundaries
• Improves retrieval of cross-boundary information
• Balances redundancy vs. coverage

• 10-15%: Minimal overlap for speed
• 15-20%: Standard overlap for most use cases
• 20-30%: High overlap for critical applications

Performance Optimization

# Use fixed-size for speed
python scripts/chunk-fixed-size.py --input large-doc.txt --chunk-size 1000

# Use semantic for better context
python scripts/chunk-semantic.py --input article.txt --max-chunk-size 1500

# Process multiple files
for file in documents/*.txt; do
  python scripts/chunk-semantic.py --input "$file" --output "chunks/$(basename $file .txt).json"
done

Evaluation Workflow

Step 1: Benchmark Strategies

python scripts/benchmark-chunking.py \
  --input sample-document.txt \
  --strategies fixed,semantic,recursive \
  --chunk-sizes 500,1000,1500

Step 2: Analyze Results

• Processing time (prefer < 100ms per document)
• Context preservation score (target > 0.85)
• Chunk size variance (lower is more predictable)

Step 3: A/B Test Retrieval

1. Chunk same corpus with different strategies
2. Run identical test queries against each
3. Measure precision@k and recall@k
4. Select strategy with best retrieval metrics

Step 4: Production Deployment

import yaml
from chunking_strategies import get_chunker

config = yaml.safe_load(open('chunking-config.yaml'))
chunker = get_chunker(config['chunking']['default_strategy'], config)

chunks = chunker.chunk(document_text)

Common Issues & Solutions

• Adjust

  chunk_size

  parameter
• Check document structure (may need different strategy)
• Verify separator selection for recursive chunking

• Increase overlap
• Switch to semantic chunking
• Add parent context to metadata

• Use fixed-size chunking for large batches
• Reduce overlap
• Process files in parallel

• Benchmark different strategies
• Increase chunk size
• Try hybrid approach (semantic + fixed fallback)

Dependencies

pip install tiktoken  # Token counting
pip install nltk      # Sentence splitting
pip install spacy     # Advanced NLP (optional)

pip install pypdf pdfminer.six

pip install pandas numpy scikit-learn

Best Practices Summary

1. Start with semantic chunking for most documents
2. Use recursive chunking for structured/hierarchical content
3. Benchmark on your actual data before production
4. Set overlap to 15-20% of chunk size
5. Include metadata (source, page, section) in chunks
6. Test retrieval quality, not just chunking speed
7. Use appropriate chunk size for your embedding model token limit
8. Document your chunking strategy choice and parameters