Web Download

Overview

node-list.txt

Workflow

1. Configure Parameters (Mandatory Before Starting)

Question: How many sub-agents to launch for parallel research?
Options:
- 1: Most conservative, suitable for resource-constrained scenarios
- 2: Default recommendation, balances efficiency and stability
- 3: Moderate, suitable for scenarios with many nodes
- Agent decides automatically: Intelligently adjust based on the number of nodes (max 3)

Question: What's the maximum number of Web Search attempts per node?
Options:
- 1: Quickly collect basic information
- 2: Default recommendation, balances coverage and efficiency
- 3: In-depth collection, suitable for important nodes

Question: What's the maximum number of Web Fetch attempts to read webpages per search result?
Options:
- 1: Only read the most relevant result
- 2: Read the top 2 relevant results
- 3: Default recommendation, fully covers search results

Question: What's the maximum number of webpages/documents to save per search result?
Options:
- 1: Only save the most relevant material
- 2: Save the top 2 relevant materials
- 3: Default recommendation, ensures material diversity

• Number of sub-agents: Max 2
• Search attempts per node: Max 2
• Web Fetch attempts per search: Max 3
• Save attempts per search: Max 3

2. Read Node List

node-list.txt

3. Parallel Research Strategy

• Strictly limit the number of sub-agents to the user-specified value
• Each sub-agent handles 1-2 nodes
• Distribute the node list evenly among sub-agents

Sub-agent 1: Node 1, Node 2, Node 3
Sub-agent 2: Node 4, Node 5, Node 6

4. Deep Retrieval Method (Strict Restrictions)

• Each node can have a maximum of user-configured Web Search attempts
• Prioritize using different keyword combinations to obtain diverse results
• Include both Chinese and English searches (within the limited attempts)

1st search: "{node name}"
2nd search: "{node name} principle tutorial" or "{node name} guide"

• A maximum of user-configured Web Fetch attempts per search result
• Prioritize official documents and authoritative sources
• Skip duplicate or low-quality URLs

• A maximum of user-configured webpages saved per search result
• Prioritize saving materials with high completeness and rich content

4. Material Collection and Saving

• Technical documents and official guides
• Academic papers and research reports
• Technical blogs and tutorials
• Practical cases and code examples

1. Create a

   materials/

   directory to store all materials
2. Use the web_reader tool to obtain complete webpage content
3. Save each material as an independent file, naming format:

   {node index}_{source identifier}.{ext}

4. Supported file formats:

   • .md

     - Markdown format content

   • .txt

     - Plain text content

   • .json

     - Structured data

5. Output Format

download.txt

Node 1 content: {node1_material1.md: source URL1}, {node1_material2.md: source URL2}
Node 2 content: {node2_material1.md: source URL1}, {node2_material2.md: source URL2}
...

• Use the format

  {serial number}_{brief description}.{extension}

• The serial number corresponds to the line number in node-list.txt
• The brief description reflects the material's topic

Scripts

scripts/parallel_fetch.py

• Concurrent downloading of multiple webpages
• Automatic retries for failed requests
• Progress display and error reporting

scripts/validate_sources.py

• Check the integrity of downloaded materials
• Verify URL accessibility
• Generate material quality reports

Examples

Example: Node Research (Default Configuration)

node-list.txt

Introduction to React Hooks
Docker Containerization Technology

Node 1: Introduction to React Hooks
- Search 1: "React Hooks introduction tutorial"
  - Fetch: Official documents, technical blogs (max 3 attempts)
  - Save: 3 most relevant materials
- Search 2: "React Hooks best practices"
  - Fetch: Articles related to best practices (max 3 attempts)
  - Save: 3 most relevant materials

download.txt

Introduction to React Hooks: {1_hooks_intro.md: https://react.dev/learn}, {1_hooks_guide.md: https://www.runoob.com/reactjs/react-hooks.html}, {1_hooks_best_practices.md: https://blog.logrocket.com/guide-to-react-hooks/}
Docker Containerization Technology: {2_docker_intro.md: https://docs.docker.com/get-started/}, {2_docker_tutorial.md: https://yeasy.gitbook.io/docker_practice/}

Example: Quick Collection (Low-Configuration Mode)

• Minimizes API calls
• Completes collection quickly
• Basic but sufficient materials

Materials Directory Structure

materials/
├── 1_hooks_intro.md
├── 1_hooks_guide.md
├── 1_hooks_best_practices.md
├── 2_docker_intro.md
├── 2_docker_tutorial.md
├── 3_microservices_patterns.md
└── 3_microservices_guide.md

Troubleshooting

Quality Standards

• At least 2-3 high-quality source materials
• Cover different perspectives (theory + practice)
• Priority ranking: Official documents > Authoritative tutorials > Technical blogs > Personal notes
• Time requirement: Prioritize materials from the last 1-2 years (for fields with rapid technological iteration)