FireCrawl Research

Overview

Enrich research documents by automatically searching and scraping web sources using the FireCrawl API. Extract research topics from markdown files and generate comprehensive research documents with source material.

When to Use This Skill

Use this skill when the user:

Says "Research this topic using FireCrawl"
Requests to enrich notes or documents with web sources
Wants to gather information about topics listed in a markdown file
Needs to search and scrape multiple topics systematically

How It Works

1. Topic Extraction

The script automatically extracts research topics from markdown files using two methods:

Method 1: Headers

markdown

## Spatial Reasoning in AI
### Computer Vision Applications

Both

Spatial Reasoning in AI

and

Computer Vision Applications

become research topics.

Method 2: Research Tags

markdown

- [research] Large Language Models for robotics
- [search] Theory of Mind in autonomous driving

Both tagged items become research topics.

2. Search and Scrape

For each topic:

Searches FireCrawl with the topic as query
Retrieves up to N results (default: 5)
Automatically scrapes full content from each result
Extracts markdown-formatted content (main content only)

3. Output Generation

Creates new markdown files in the specified output directory:

One file per topic
Filename:
```
{topic}_{timestamp}.md
```
Contains: title, date, sources count, full scraped content
Each source includes: title, URL, markdown content

Usage

Basic Usage

bash

python scripts/firecrawl_research.py research.md

Outputs to current directory.

Specify Output Directory

bash

python scripts/firecrawl_research.py research.md ./output

Creates files in

./output/

folder.

Limit Results Per Topic

bash

python scripts/firecrawl_research.py research.md ./output 3

Retrieves maximum 3 results per topic.

Configuration

API Key Setup

Copy
```
.env.example
```
to
```
.env
```
:
bash
```
cp .env.example .env
```

Add FireCrawl API key:

FIRECRAWL_API_KEY=fc-your-actual-api-key

The script automatically loads the API key from the skill's

.env

file.

Rate Limiting

The script includes automatic rate limiting for FireCrawl's free tier:

Free tier limit: 5 requests/minute
Built-in delay: 12 seconds between topics
Prevents API errors and credit exhaustion

When processing multiple topics, expect:

5 topics: ~1 minute
10 topics: ~2 minutes
20 topics: ~4 minutes

Workflow Example

User request: "Research these AI topics using FireCrawl"

Input file (
ai-research.md
):

markdown

# AI Research Topics

## Spatial Reasoning in Vision-Language Models

- [research] Embodied AI for robotics
- [research] Computer Use Agents

Command:

bash

python scripts/firecrawl_research.py ai-research.md ./research_output 5

Output:

research_output/
├── Spatial_Reasoning_in_Vision-Language_Models_20251122_140530.md
├── Embodied_AI_for_robotics_20251122_140542.md
└── Computer_Use_Agents_20251122_140554.md

Each file contains:

Topic title
Timestamp
Source count
Full scraped content from up to 5 sources
Source URLs

Common Patterns

Pattern 1: Quick Research

Extract topics from existing notes, research them, save to current folder:

bash

python scripts/firecrawl_research.py my-notes.md

Pattern 2: Organized Research

Create dedicated output folder for research results:

bash

python scripts/firecrawl_research.py topics.md ./research_results

Pattern 3: Deep Dive

Increase results per topic for comprehensive coverage:

bash

python scripts/firecrawl_research.py topics.md ./deep_research 10

Pattern 4: Obsidian Vault Integration

Direct output to vault's research folder:

bash

python scripts/firecrawl_research.py topics.md ~/Brains/brain/Research

Error Handling

"API key not found"

Create

.env

file in skill folder with

FIRECRAWL_API_KEY=...

"Rate limit exceeded"

Free tier: 5 req/min
Script has 12s delay built-in
If still hitting limit, reduce topics or wait between runs

"Insufficient credits"

Check FireCrawl account credits
Upgrade plan or wait for credit reset

"No topics found"

Add topics to markdown using:

```
## Header format
```
```
- [research] Topic format
```
```
- [search] Topic format
```

Script Details

Location:

scripts/firecrawl_research.py

Dependencies:

```
python-dotenv
```
- Environment variable management
```
requests
```
- HTTP requests to FireCrawl API

Install dependencies:

bash

pip install python-dotenv requests

FireCrawl Features Used:

```
/v1/search
```
endpoint - Search with automatic scraping
```
scrapeOptions.formats: ['markdown']
```
- Markdown output
```
scrapeOptions.onlyMainContent: true
```
- Filter noise

Academic Writing Templates

This skill includes templates for writing scientific papers in markdown format.

Available Templates

1. Pandoc Scholarly Paper (

assets/templates/pandoc-scholarly-paper.md

)

Standard academic paper format
Compatible with Pandoc converter
Supports citations via BibTeX
Exports to PDF, DOCX, HTML

2. MyST Scientific Paper (

assets/templates/myst-scientific-paper.md

)

MyST (Markedly Structured Text) format
Advanced cross-referencing
Professional scientific publishing
Multi-format export (PDF, LaTeX, DOCX)

Using Templates

Copy template to your project:

bash

cp assets/templates/pandoc-scholarly-paper.md my-paper.md
# or
cp assets/templates/myst-scientific-paper.md my-paper.md

Edit content:

Update YAML frontmatter (title, authors, affiliations)
Write your content in sections
Add citations using
```
[@AuthorYear]
```
(Pandoc) or
```
{cite}\
```
AuthorYear`` (MyST)

Convert to PDF/DOCX:

bash

python scripts/convert_academic.py my-paper.md pdf
python scripts/convert_academic.py my-paper.md docx
python scripts/convert_academic.py my-paper.md pdf --myst  # For MyST

Bibliography Generation

Convert FireCrawl research results into BibTeX bibliography entries:

bash

python scripts/generate_bibliography.py research_output/*.md -o references.bib

What it does:

Extracts URLs and titles from FireCrawl markdown files
Generates BibTeX
```
@misc
```
entries
Creates citation keys automatically
Adds access dates

Example workflow:

bash

# 1. Research topics
python scripts/firecrawl_research.py topics.md ./research

# 2. Generate bibliography
python scripts/generate_bibliography.py research/*.md -o refs.bib

# 3. Copy template
cp assets/templates/pandoc-scholarly-paper.md paper.md

# 4. Edit paper.md (add content, cite sources)

# 5. Convert to PDF
python scripts/convert_academic.py paper.md pdf

Citation Examples

Pandoc syntax:

markdown

Recent research [@Smith2024] shows...
Multiple studies [@Jones2023; @Brown2024] indicate...

MyST syntax:

markdown

Recent research {cite}`Smith2024` shows...
Multiple studies {cite}`Jones2023,Brown2024` indicate...

Example Bibliography File

An example bibliography is provided in

assets/references.bib

with common entry types:

Journal articles (
```
@article
```
)
Conference papers (
```
@inproceedings
```
)
Books (
```
@book
```
)
PhD theses (
```
@phdthesis
```
)
Web resources (
```
@misc
```
)
Preprints (
```
@article
```
with arXiv)

Tips

Organize topics hierarchically - Use
```
##
```
for main topics,
```
###
```
for subtopics
Use descriptive names - Topic text becomes filename, make it clear
Batch processing - Group related topics in one file for efficiency
Output organization - Create separate folders for different research projects
Content review - Results are truncated at 3000 chars/source for readability
Academic workflow - Use bibliography generator to cite research sources in papers
Template customization - Modify templates for your field's citation style

Limitations

No summarization - Returns raw scraped content, not summaries
No deduplication - Duplicate sources may appear across topics
No quality ranking - All results treated equally
New files only - Does not append to existing files
Free tier constraints - Rate limiting affects processing speed

firecrawl-research

NPX Install

Tags

SKILL.md Content

FireCrawl Research

Overview

When to Use This Skill

How It Works

1. Topic Extraction

2. Search and Scrape

3. Output Generation

Usage

Basic Usage

Specify Output Directory

Limit Results Per Topic

Configuration

API Key Setup

Rate Limiting

Workflow Example

Common Patterns

Pattern 1: Quick Research

Pattern 2: Organized Research

Pattern 3: Deep Dive

Pattern 4: Obsidian Vault Integration

Error Handling

"API key not found"

"Rate limit exceeded"

"Insufficient credits"

"No topics found"

Script Details

Academic Writing Templates

Available Templates

Using Templates

Bibliography Generation

Citation Examples

Example Bibliography File

Tips

Limitations