Text Splitting Tool

Features

Usage Scenarios

• Process ultra-long text and split it into fragments that can be handled by agents.
• Perform text preprocessing to provide standardized input for subsequent content analysis, summarization, or evaluation tasks.
• Optimize text processing workflows to ensure the size of each processing unit is controllable and semantically complete.

Core Capabilities

• Semantic Integrity Priority: When splitting, try to avoid truncating sentences or paragraphs, and prioritize splitting at natural boundaries (such as periods, line breaks) to ensure the semantic integrity of each chunk.
• Preserve Original Format: Strive to retain the original format and structure of the text during splitting, such as Markdown, HTML tags, etc.
• Precise Chunk Size Control: Split strictly according to the specified size limit to ensure each text chunk does not exceed the preset maximum length.
• Multiple Splitting Strategies: Support multiple splitting strategies based on character count, token count, paragraph count, or specific delimiters.

Input Requirements

• Text Content: The original text to be split.
• Chunk Size Limit: The maximum length of each text chunk (it is recommended to provide character count or token count).
• Splitting Strategy (Optional): Specify the priority strategy for splitting, such as splitting by sentence, splitting by paragraph, splitting by specific delimiter, etc.

Output Format

[Text Splitting Report]

- Original Text Length: [Integer] characters/Token
- Target Chunk Size: [Integer] characters/Token
- Actual Number of Chunks: [Integer] chunks

### Split Results
- Chunk 1 (Length: [Integer] characters/Token): "[Content Preview...]"
- Chunk 2 (Length: [Integer] characters/Token): "[Content Preview...]"
- Chunk 3 (Length: [Integer] characters/Token): "[Content Preview...]"
...

Constraints

• The split results must strictly comply with the specified chunk size limit.
• Ensure maximum preservation of text semantic integrity during splitting.
• The output format must be structured, clearly displaying the content and length of each text chunk.
• Avoid introducing any additional information or explanations in the output, only provide split results.

Examples

{baseDir}/references/examples.md

• examples.md

  - Contains splitting examples of different lengths, different splitting strategies, and complex text structures.

Detailed Documentation

{baseDir}/references/examples.md

Version History