social-fetch

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

/social-fetch — Pull any social post by URL

/social-fetch — 通过URL抓取任意社交平台帖子

Normalized fetcher for social posts across platforms. Detects platform from URL, tries strategies in order, returns the same JSON shape regardless of source.

跨平台社交帖子标准化抓取工具。从URL中检测平台，按顺序尝试抓取策略，无论来源如何都返回统一格式的JSON数据。

Step 1 — Detect platform

步骤1 — 检测平台

URL pattern	Platform
`x.com/<user>/status/<id>` or `twitter.com/<user>/status/<id>`	x (Twitter)
`linkedin.com/posts/<slug>` or `linkedin.com/feed/update/urn:li:activity:<id>`	linkedin
`linkedin.com/in/<handle>` (profile, recent activity)	linkedin-profile
`instagram.com/p/<id>` or `instagram.com/reel/<id>`	instagram
`tiktok.com/@<user>/video/<id>`	tiktok
`bsky.app/profile/<handle>/post/<rkey>`	bluesky
`reddit.com/r/<sub>/comments/<id>/...`	reddit
`<mastodon-instance>/@<user>/<id>` (e.g. mastodon.social, hachyderm.io)	mastodon
`threads.net/@<user>/post/<id>`	threads
`news.ycombinator.com/item?id=<id>`	hn
`youtube.com/watch?v=<id>` or `youtu.be/<id>`	→ defer to `watch-video`

If the URL doesn't match any pattern, ask the user what platform it is.

URL模式	平台
`x.com/<user>/status/<id>` 或 `twitter.com/<user>/status/<id>`	X（Twitter）
`linkedin.com/posts/<slug>` 或 `linkedin.com/feed/update/urn:li:activity:<id>`	LinkedIn
`linkedin.com/in/<handle>` （个人主页、近期动态）	LinkedIn个人主页
`instagram.com/p/<id>` 或 `instagram.com/reel/<id>`	Instagram
`tiktok.com/@<user>/video/<id>`	TikTok
`bsky.app/profile/<handle>/post/<rkey>`	Bluesky
`reddit.com/r/<sub>/comments/<id>/...`	Reddit
`<mastodon-instance>/@<user>/<id>` （例如 mastodon.social、hachyderm.io）	Mastodon
`threads.net/@<user>/post/<id>`	Threads
`news.ycombinator.com/item?id=<id>`	HN（Hacker News）
`youtube.com/watch?v=<id>` 或 `youtu.be/<id>`	→ 转至 `watch-video` 工具

如果URL不匹配任何模式，请询问用户对应的平台。

Step 2 — Pick strategy chain

步骤2 — 选择策略链

Read

references/strategies.md

for the per-platform strategy chain. Each platform has 2–5 strategies tried in order.

Key principles:

Free strategies first (direct APIs, agent-browser)
Paid only as fallback (ScrapeCreators / Apify) — and only if the env key is set
Bluesky / Mastodon / HN / Reddit are free + reliable (public APIs)
X / LinkedIn / Instagram / TikTok / Threads need paid or scraping fallback for full data

查看

references/strategies.md

获取各平台的策略链。每个平台有2-5种按顺序尝试的策略。

核心原则：

优先使用免费策略（直接API、Agent浏览器）
仅将付费策略作为备选（ScrapeCreators / Apify）—— 且仅当设置了环境密钥时使用
Bluesky / Mastodon / HN / Reddit 免费且可靠（公开API）
X / LinkedIn / Instagram / TikTok / Threads 需要付费或抓取备选方案以获取完整数据

Step 3 — Execute strategy

步骤3 — 执行策略

For each strategy in the chain:

Try it
If success: parse → normalize → return
If failure (404, 402, auth wall, empty response): note the failure and try the next strategy

After exhausting the chain, return a clear error: which strategies were tried, why each failed, and what's needed to unlock (e.g., "Add

$SCRAPECREATORS_API_KEY

for X — see

references/auth-keys.md

").

针对策略链中的每个策略：

尝试执行
如果成功：解析 → 标准化 → 返回结果
如果失败（404、402、权限墙、空响应）：记录失败原因并尝试下一个策略

当所有策略尝试完毕后，返回清晰的错误信息：尝试过哪些策略、每个策略失败的原因，以及解锁所需的条件（例如："添加

$SCRAPECREATORS_API_KEY

以支持X平台 — 详见

references/auth-keys.md

"）。

Step 4 — Normalize output

步骤4 — 标准化输出

Return this shape regardless of platform (see

references/output-schema.md

for the full spec + platform-specific examples):

json

{
  "platform": "x",
  "url": "https://x.com/example/status/1234567890",
  "fetched_at": "2026-06-17T14:35:00Z",
  "raw_source": "scrapecreators",
  "author": {
    "handle": "@example",
    "name": "the user Ganim",
    "verified": true
  },
  "posted_at": "2026-06-17T16:53:00Z",
  "text": "The 80/20 of a useful AI second brain: ...",
  "media": [],
  "engagement": {
    "likes": 51,
    "reposts": 13,
    "replies": 9,
    "bookmarks": 7,
    "views": 32700
  },
  "is_thread": true,
  "thread": [],
  "replies": []
}

Fields with no equivalent on a platform (e.g.,

bookmarks

on Mastodon) get

null

, not

. Missing data is different from zero data.

无论平台如何，都返回以下格式（完整规范及平台特定示例请查看

references/output-schema.md

）：

json

{
  "platform": "x",
  "url": "https://x.com/example/status/1234567890",
  "fetched_at": "2026-06-17T14:35:00Z",
  "raw_source": "scrapecreators",
  "author": {
    "handle": "@example",
    "name": "the user Ganim",
    "verified": true
  },
  "posted_at": "2026-06-17T16:53:00Z",
  "text": "The 80/20 of a useful AI second brain: ...",
  "media": [],
  "engagement": {
    "likes": 51,
    "reposts": 13,
    "replies": 9,
    "bookmarks": 7,
    "views": 32700
  },
  "is_thread": true,
  "thread": [],
  "replies": []
}

平台中没有对应项的字段（例如Mastodon的

bookmarks

）将设为

null

而非

。缺失数据与零数据是不同的。

Step 5 — Optional enrichments

步骤5 — 可选增强功能

Based on flags / asks:

Flag	Behavior
`--with-replies`	Fetch top-level replies (1 hop). Costs extra API quota.
`--thread`	If the post is part of a thread by the same author, fetch the whole thread.
`--raw`	Include the raw API/scrape response in the output (for debugging)
`--media`	Download media files (images/videos) to `~/Documents/social-fetches/<platform>-<id>/`

Default: just the post itself, no replies, no media download (just URLs).

根据标记/需求执行：

标记	行为
`--with-replies`	获取顶层回复（仅一级）。会额外消耗API配额。
`--thread`	如果帖子属于同一作者的线程内容，抓取完整线程。
`--raw`	在输出中包含原始API/抓取响应（用于调试）
`--media`	将媒体文件（图片/视频）下载至 `~/Documents/social-fetches/<platform>-<id>/`

默认行为：仅抓取帖子本身，不包含回复，不下载媒体（仅返回URL）。

Step 6 — Cache (optional)

步骤6 — 缓存（可选）

~/Documents/social-fetches/_cache/

exists, cache successful fetches there by

{platform}-{id}.json

for 24h. Saves API quota when the same post is referenced repeatedly across skills.

Skip cache if

--no-cache

flag is set or for

--with-replies

--thread

(likely-stale).

如果

~/Documents/social-fetches/_cache/

存在，将成功抓取的内容以

{platform}-{id}.json

的格式缓存24小时。当同一帖子被多个技能重复引用时，可节省API配额。

若设置了

--no-cache

标记，或使用

--with-replies

--thread

（数据易过时），则跳过缓存。

Composes with

可组合的工具

```
deep-research
```
— cite specific posts in research briefs. When research surfaces a relevant tweet/post URL, fetch and include in the brief.
```
jab-hook
```
— pull recent posts from inspiration accounts for deeper format analysis (currently uses agent-browser inline; should call this skill instead).
```
business-brainstorm
```
— pull competitor / operator commentary as evidence during scoring.
```
second-brain
```
— capture a post into
```
raw/
```
with the
```
tweet-
```
/
```
bookmark-
```
prefix; the structured output makes for cleaner raw files than a screenshot or copy-paste.
```
watch-video
```
— for YouTube URLs (or any video — Loom, Vimeo, Riverside, MP4), route there instead.

```
deep-research
```
— 在研究简报中引用特定帖子。当研究发现相关推文/帖子URL时，抓取并纳入简报。
```
jab-hook
```
— 从灵感账号中抓取近期帖子以进行深度格式分析（当前使用内嵌Agent浏览器；应改为调用本工具）。
```
business-brainstorm
```
— 在评分过程中抓取竞品/运营者评论作为依据。
```
second-brain
```
— 将帖子以
```
tweet-
```
/
```
bookmark-
```
前缀存入
```
raw/
```
目录；结构化输出比截图或复制粘贴生成的原始文件更整洁。
```
watch-video
```
— 针对YouTube URL（或其他视频平台，如Loom、Vimeo、Riverside、MP4），转至该工具处理。

Known limits

已知限制

X: free strategies return tweet preview only (text, author, basic engagement). Full thread + replies need
```
$SCRAPECREATORS_API_KEY
```
or
```
$APIFY_API_TOKEN
```
.
LinkedIn: agent-browser works for profile recent-activity (after dismissing the modal). Specific post URLs (
```
linkedin.com/posts/...
```
) often need paid fallback.
Instagram / TikTok / Threads: heavy anti-bot. Paid fallback strongly recommended.
Bluesky / Mastodon / HN / Reddit: free + reliable.
Private / deleted posts: nothing helps. Try Wayback Machine for deleted content.

If a platform consistently fails on free strategies and the user uses it often, prompt to set up the paid key (see

references/auth-keys.md

X平台：免费策略仅返回推文预览（文本、作者、基础互动数据）。完整线程+回复需要
```
$SCRAPECREATORS_API_KEY
```
或
```
$APIFY_API_TOKEN
```
。
LinkedIn：Agent浏览器可处理个人主页近期动态（关闭弹窗后）。特定帖子URL（
```
linkedin.com/posts/...
```
）通常需要付费备选方案。
Instagram / TikTok / Threads：反机器人机制严格。强烈推荐使用付费备选方案。
Bluesky / Mastodon / HN / Reddit：免费且可靠。
私密/已删除帖子：无有效解决方案。可尝试使用Wayback Machine获取已删除内容。

如果某个平台在免费策略下持续失败，且用户经常使用该平台，提示用户设置付费密钥（详见

references/auth-keys.md

）。

Notes on quality

质量说明

Strategy chain, not single-source. Every platform has a fallback ladder (native oEmbed → agent-browser → SCS API → Apify). If one step fails, degrade gracefully to the next. Never fail hard on the first attempt.
Structured output over screenshots. Downstream skills (jab-hook, deep-research, second-brain) need JSON with author + text + engagement fields, not an image. Even when the underlying strategy is a screenshot, extract text before returning.
Cache aggressively, invalidate honestly. 24h TTL on
```
~/Documents/social-fetches/_cache/
```
prevents API burn when the same post is referenced across multiple skills in a session.
```
--with-replies
```
/
```
--thread
```
skip cache because replies age fast.
Respect paid-key economics. ScrapeCreators / Apify calls cost real money. Prompt before hitting paid strategies if the user hasn't confirmed they want depth. Free strategies first, always.
Media download is opt-in. Default is post text only;
```
--media
```
downloads images/videos. Silent media downloads eat disk quickly.
Private / deleted content is a hard stop. No strategy chain rescues private accounts or deleted posts. Suggest Wayback Machine for deleted content and stop.
Rate-limits are per-platform. X free strategies hit rate limits fast; LinkedIn agent-browser burns session fingerprints. Space out calls in loops or the workflow degrades to worse-than-manual.

策略链而非单一来源。每个平台都有备选梯次（原生oEmbed → Agent浏览器 → SCS API → Apify）。如果某一步失败，优雅降级至下一步。绝不因首次尝试失败而直接报错。
结构化输出优先于截图。下游工具（jab-hook、deep-research、second-brain）需要包含作者+文本+互动字段的JSON，而非图片。即使底层策略是截图，也要先提取文本再返回。
积极缓存，诚实失效。
```
~/Documents/social-fetches/_cache/
```
设置24小时TTL，避免在会话中同一帖子被多个工具引用时消耗过多API配额。
```
--with-replies
```
/
```
--thread
```
跳过缓存，因为回复数据更新快。
尊重付费密钥成本。ScrapeCreators / Apify调用会产生实际费用。如果用户未确认需要深度数据，在调用付费策略前需提示。始终优先使用免费策略。
媒体下载为可选功能。默认仅返回帖子文本；
```
--media
```
标记才会下载图片/视频。静默下载媒体会快速占用磁盘空间。
私密/已删除内容为硬限制。任何策略链都无法获取私密账号或已删除帖子的内容。建议使用Wayback Machine获取已删除内容，然后终止流程。
速率限制按平台划分。X平台免费策略容易触发速率限制；LinkedIn Agent浏览器会消耗会话指纹。循环调用时需间隔时间，否则工作流效率会低于手动操作。