Text Summarization

Overview

When to Use

• Condensing long documents, reports, or article collections
• Building automated summary pipelines for content curation
• Comparing extractive vs abstractive approaches for a use case

• When full document understanding is needed (summarization loses detail)
• For structured data extraction (use NER or information extraction)

Algorithm

IRON LAW: Abstractive Summarization Can HALLUCINATE
Abstractive models may generate fluent text containing facts NOT in
the source. Always verify key claims in abstractive summaries against
the original document. For high-stakes use cases (legal, medical),
prefer extractive or use abstractive with factual consistency checking.

Phase 1: Input Validation

Phase 2: Core Algorithm

1. Split document into sentences
2. Build similarity graph (sentence nodes, cosine similarity edges)
3. Run PageRank on sentence graph
4. Select top-k sentences by rank, reorder by original position

1. Use pre-trained model (BART, T5, Pegasus)
2. Encode input document (handle length limits with chunking if needed)
3. Generate summary with beam search
4. Post-process: check for repetition, factual consistency

Phase 3: Verification

Phase 4: Output

Output Format

{
  "summary": "The company reported Q4 revenue of...",
  "method": "extractive_textrank",
  "metadata": {"input_words": 2000, "summary_words": 200, "compression_ratio": 0.10, "sentences_selected": 5}
}

Examples

Sample I/O

Edge Cases

Gotchas

• ROUGE ≠ quality: ROUGE measures n-gram overlap with references. A high-ROUGE summary can be incoherent, and a low-ROUGE summary can be excellent with different word choices.
• Input length limits: Transformer models have max token limits (512-4096). Long documents need chunking strategies (chunk-then-summarize or hierarchical summarization).
• Repetition: Abstractive models sometimes repeat phrases. Use repetition penalty during generation (no_repeat_ngram_size).
• Position bias: In news text, important information is front-loaded (inverted pyramid). Simple "take first N sentences" is a strong extractive baseline.
• Multi-document summarization: Summarizing multiple related documents requires handling redundancy and contradiction across sources.

References

• For TextRank/LexRank implementation details, see

  references/graph-based-extraction.md

• For factual consistency checking, see

  references/factual-consistency.md