CLI-EXPLORER — Complete Guide to Exploratory Adapter Development

This document teaches you (or an AI Agent) how to add commands for a new website to OpenCLI.
From scratch to release, it covers the full process of API discovery, solution selection, adapter writing, and testing & verification.

[!TIP] Just want to quickly generate a command for a specific page? Check CLI-ONESHOT.md (~150 lines, 4 steps to complete). This document applies to the full process of exploring a new site from scratch.

Must-read for AI Agent Developers: Explore with Browsers

[!CAUTION] You (AI Agent) must open the target website via a browser to explore!
Do not rely solely on the
opencli explore
command or static analysis to discover APIs.
You have browser tools available, you must actively use them to browse web pages, observe network requests, and simulate user interactions.

Why?

Many APIs are lazy-loaded (network requests are only triggered when the user clicks a certain button/tab). Deep data such as subtitles, comments, and follow lists will not appear in the Network panel when the page first loads. If you do not actively browse and interact with the page, you will never discover these APIs.

AI Agent Exploration Workflow (Must Follow)

Step	Tool	What to Do
0. Open browser	`browser_navigate`	Navigate to the target page
1. Observe page	`browser_snapshot`	Observe interactive elements (buttons/tabs/links)
2. First packet capture	`browser_network_requests`	Filter JSON API endpoints, record URL pattern
3. Simulate interaction	`browser_click` + `browser_wait_for`	Click buttons like "subtitles", "comments", "follow"
4. Second packet capture	`browser_network_requests`	Compare with step 2, find newly triggered APIs
5. Verify API	`browser_evaluate`	`fetch(url, {credentials:'include'})` test return structure
6. Write code	—	Write adapter based on confirmed API

Common Mistakes

❌ Wrong Practice	✅ Correct Practice
Only use `opencli explore` command, wait for results to come out automatically	Open the page with browser tools, browse actively
Directly `fetch(url)` in code without checking actual browser requests	Confirm API is available in browser first, then write code
Capture packets directly after page opens, expect all APIs to appear	Simulate click interactions (expand comments/switch tabs/load more)
Give up when encountering HTTP 200 with empty data	Check if Wbi signature or Cookie authentication is required
Fully rely on `__INITIAL_STATE__` to get all data	`__INITIAL_STATE__` only has first screen data, deep data needs API calls

Practical Success Case: Implement "Follow List" Adapter in 5 Minutes

The following is the complete process of actually discovering the Bilibili follow list API using the above workflow:

1. browser_navigate → https://space.bilibili.com/{uid}/fans/follow
2. browser_network_requests → Discovered:
   GET /x/relation/followings?vmid={uid}&pn=1&ps=24  →  [200]
   GET /x/relation/stat?vmid={uid}                    →  [200]
3. browser_evaluate → Verify API:
   fetch('/x/relation/followings?vmid=137702077&pn=1&ps=5', {credentials:'include'})
   → { code: 0, data: { total: 1342, list: [{mid, uname, sign, ...}] } }
4. Conclusion: Standard Cookie API, no Wbi signature required
5. Write following.ts → Build passes in one go

Key Decision Points:

Directly access the
```
fans/follow
```
page (not the homepage), the following API will be triggered as soon as the page loads
No
```
/wbi/
```
in the URL → no signature required → use
```
fetchJson
```
directly instead of
```
apiGet
```
API returns
```
code: 0
```
+ non-empty
```
list
```
→ Tier 2 Cookie strategy confirmed

Core Workflow

 ┌─────────────┐     ┌─────────────┐     ┌──────────────┐     ┌────────┐
 │ 1. API Discovery │ ──▶ │ 2. Select Strategy │ ──▶ │ 3. Write Adapter │ ──▶ │ 4. Test │
 └─────────────┘     └─────────────┘     └──────────────┘     └────────┘
   explore             cascade             YAML / TS            run + verify

Step 1: API Discovery

1a. Automated Discovery (Recommended)

OpenCLI has built-in Deep Explore, which automatically analyzes website network requests:

bash

opencli explore https://www.example.com --site mysite

Output to

.opencli/explore/mysite/

File	Content
`manifest.json`	Site metadata, framework detection (Vue2/3, React, Next.js, Pinia, Vuex)
`endpoints.json`	Discovered API endpoints, sorted by score, including URL pattern, method, response type
`capabilities.json`	Inferred functions ( `hot` , `search` , `feed` …), including confidence and recommended parameters
`auth.json`	Authentication method detection (Cookie/Header/no authentication), candidate strategy list

1b. Manual Packet Capture Verification

Explore's automatic analysis may not be perfect, use verbose mode for manual confirmation:

bash

# Open the target page in the browser, observe network requests
opencli explore https://www.example.com --site mysite -v

# Or directly use evaluate to test API
opencli bilibili hot -v   # View data flow of each step of the existing command pipeline

Key information to pay attention to in packet capture results:

URL pattern:
```
/api/v2/hot?limit=20
```
→ This is the endpoint you need to call
Method:
```
GET
```
/
```
POST
```
Request Headers: Cookie? Bearer? Custom signature headers (X-s, X-t)?
Response Body: JSON structure, especially the path where the data is located (
```
data.items
```
,
```
data.list
```
)

1c. Advanced API Discovery Heuristics

Before diving into complex packet capture interception, try the following methods in order of priority:

Suffix brute force method (
.json
): For complex sites like Reddit, just add
```
.json
```
after the URL (e.g.
```
/r/all.json
```
), you can get extremely clean REST data directly with
```
fetch
```
when carrying cookies (Tier 2 Cookie strategy is extremely fast). In addition, fully functional sites like Xueqiu can also use this pure API method to get data very simply, making it a golden benchmark for you to build simple YAML.
Global state lookup method (
__INITIAL_STATE__
): Many Server-Side Rendered (SSR) websites (such as Xiaohongshu, Bilibili) mount the full data of the homepage or detail page to the global window object. Instead of intercepting network requests, you can directly get the entire data tree via
```
page.evaluate('() => window.__INITIAL_STATE__')
```
.
Active interaction trigger method: Many deep APIs (such as video subtitles, replies under comments) are lazy-loaded. When you can't find data in static packet capture, try to actively click the corresponding button on the page (such as "CC", "Expand All") during the
```
evaluate
```
step or when manually setting breakpoints, to trigger hidden Network Fetch.
Framework detection and Store Action interception: If the site uses Vue + Pinia, you can use the
```
tap
```
step to call the action, letting the frontend framework complete the complex authentication signature encapsulation for you.
Low-level XHR/Fetch interception: Last resort, when all the above methods fail, use a TypeScript adapter for non-intrusive request capture.

1d. Framework Detection

Explore automatically detects frontend frameworks. If you need manual confirmation:

bash

# When the target website is already open
opencli evaluate "(()=>{
  const vue3 = !!document.querySelector('#app')?.__vue_app__;
  const vue2 = !!document.querySelector('#app')?.__vue__;
  const react = !!window.__REACT_DEVTOOLS_GLOBAL_HOOK__;
  const pinia = vue3 && !!document.querySelector('#app').__vue_app__.config.globalProperties.\$pinia;
  return JSON.stringify({vue3, vue2, react, pinia});
})()"

Sites using Vue + Pinia (such as Xiaohongshu) can bypass signatures directly via Store Action.

Step 2: Select Authentication Strategy

OpenCLI provides 5 levels of authentication strategies. Use the

cascade

command for automatic detection:

bash

opencli cascade https://api.example.com/hot

Strategy Decision Tree

Can you get data directly with fetch(url)?
  → ✅ Tier 1: public (public API, no browser required)
  → ❌ Can you get data with fetch(url, {credentials:'include'}) carrying cookies?
       → ✅ Tier 2: cookie (most common, fetch within evaluate step)
       → ❌ → Can you get data after adding Bearer / CSRF header?
              → ✅ Tier 3: header (e.g. Twitter ct0 + Bearer)
              → ❌ → Does the site have Pinia/Vuex Store?
                     → ✅ Tier 4: intercept (Store Action + XHR interception)
                     → ❌ Tier 5: ui (UI automation, last resort)

Comparison of Each Strategy


public
cookie
header
intercept
ui

Tier	Strategy	Speed	Complexity	Use Case	Example
1	`public`	⚡ ~1s	Simplest	Public API, no login required	Hacker News, V2EX
2	`cookie`	🔄 ~7s	Simple	Cookie authentication is sufficient	Bilibili, Zhihu, Reddit
3	`header`	🔄 ~7s	Medium	Requires CSRF token or Bearer	Twitter GraphQL
4	`intercept`	🔄 ~10s	Relatively high	Requests have complex signatures	Xiaohongshu (Pinia + XHR)
5	`ui`	🐌 ~15s+	Highest	No API, pure DOM parsing	Legacy websites

Step 2.5: Preparation (Before Writing Code)

Find Templates First: Start with the Most Similar Existing Adapter

Do not write from scratch. First check what adapters already exist for the same site:

bash

ls src/clis/<site>/    # Check what already exists
cat src/clis/<site>/feed.ts   # Read the most similar one

The most efficient way is to copy the most similar adapter, then modify 3 parts:

```
name
```
→ New command name
API URL → The endpoint you discovered in Step 1
Field mapping → Correspond to the fields of the new API

Platform SDK Cheat Sheet

Before writing a TS adapter, check if there are ready-made helper functions available for reuse for your target site:

Bilibili (

src/clis/bilibili/utils.ts

)

Function	Purpose	When to Use
`fetchJson(page, url)`	Fetch with cookie + JSON parsing	Normal Cookie-tier API
`apiGet(page, path, {signed, params})`	API call with Wbi signature	Interface with `/wbi/` in URL
`getSelfUid(page)`	Get UID of current logged in user	"My xxx" type commands
`resolveUid(page, input)`	Parse UID input by user (supports number/URL)	`--uid` parameter processing
`wbiSign(page, params)`	Low-level Wbi signature generation	Usually not used directly, already encapsulated in `apiGet`
`stripHtml(s)`	Remove HTML tags	Clean rich text fields

How to judge if
apiGet
is needed? Check the Network request URL:

Contains

/wbi/

w_rid=

→ Must use

apiGet(..., { signed: true })

No → Use
```
fetchJson
```
directly

Other sites (Twitter, Xiaohongshu, etc.) do not have dedicated SDKs for now, just use
page.evaluate
+
fetch
directly.

Step 3: Write Adapter

YAML vs TS? Check the Decision Tree First

Does your pipeline have an evaluate step (embedded JS code)?
  → ✅ Use TypeScript (src/clis/<site>/<name>.ts), automatically dynamically registered on save
  → ❌ Pure declarative (navigate + tap + map + limit)?
       → ✅ Use YAML (src/clis/<site>/<name>.yaml), automatically registered on save

Scenario	Choice	Example
Pure fetch/select/map/limit	YAML	`v2ex/hot.yaml` , `hackernews/top.yaml`
navigate + evaluate(fetch) + map	YAML (assess complexity)	`zhihu/hot.yaml`
navigate + tap + map	YAML ✅	`xiaohongshu/feed.yaml` , `xiaohongshu/notifications.yaml`
Has complex JS logic (Pinia state reading, conditional branches)	TS	`xiaohongshu/me.ts` , `bilibili/me.ts`
XHR interception + signature	TS	`xiaohongshu/search.ts`
GraphQL / pagination / Wbi signature	TS	`bilibili/search.ts` , `twitter/search.ts`

Rule of Thumb: If you find that there are more than 10 lines of JS embedded in YAML, it is more maintainable to switch to TS.

Common Pattern: Pagination API

Many APIs use

pn

(page number) +

ps

(page size) for pagination. Standard processing pattern:

typescript

args: [
  { name: 'page', type: 'int', required: false, default: 1, help: 'Page number' },
  { name: 'limit', type: 'int', required: false, default: 50, help: 'Page size (max 50)' },
],
func: async (page, kwargs) => {
  const pn = kwargs.page ?? 1;
  const ps = Math.min(kwargs.limit ?? 50, 50); // Respect API ps limit
  const payload = await fetchJson(page,
    `https://api.example.com/list?pn=${pn}&ps=${ps}`
  );
  return payload.data?.list || [];
},

The
ps
limit for most sites is 20~50. Exceeding it will be silently truncated or return an error.

Method A: YAML Pipeline (Declarative, Recommended)

File path:

src/clis/<site>/<name>.yaml

, automatically registered once placed.

Tier 1 — Public API Template

yaml

# src/clis/v2ex/hot.yaml
site: v2ex
name: hot
description: V2EX Hot Topics
domain: www.v2ex.com
strategy: public
browser: false

args:
  limit:
    type: int
    default: 20

pipeline:
  - fetch:
      url: https://www.v2ex.com/api/topics/hot.json

  - map:
      rank: ${{ index + 1 }}
      title: ${{ item.title }}
      replies: ${{ item.replies }}

  - limit: ${{ args.limit }}

columns: [rank, title, replies]

Tier 2 — Cookie Authentication Template (Most Used)

yaml

# src/clis/zhihu/hot.yaml
site: zhihu
name: hot
description: Zhihu Hot List
domain: www.zhihu.com

pipeline:
  - navigate: https://www.zhihu.com       # Load page first to establish session

  - evaluate: |                            # Send request in browser, automatically carries cookie
      (async () => {
        const res = await fetch('/api/v3/feed/topstory/hot-lists/total?limit=50', {
          credentials: 'include'
        });
        const d = await res.json();
        return (d?.data || []).map(item => {
          const t = item.target || {};
          return {
            title: t.title,
            heat: item.detail_text || '',
            answers: t.answer_count,
          };
        });
      })()

  - map:
      rank: ${{ index + 1 }}
      title: ${{ item.title }}
      heat: ${{ item.heat }}
      answers: ${{ item.answers }}

  - limit: ${{ args.limit }}

columns: [rank, title, heat, answers]

Key Point: The
fetch
inside the
evaluate
step runs in the browser page context, automatically carries
credentials: 'include'
, no need to handle cookies manually.

Advanced — With Search Parameters

yaml

# src/clis/zhihu/search.yaml
site: zhihu
name: search
description: Zhihu Search

args:
  query:
    type: str
    required: true
    positional: true
    description: Search query
  limit:
    type: int
    default: 10

pipeline:
  - navigate: https://www.zhihu.com

  - evaluate: |
      (async () => {
        const q = encodeURIComponent('${{ args.query }}');
        const res = await fetch('/api/v4/search_v3?q=' + q + '&t=general&limit=${{ args.limit }}', {
          credentials: 'include'
        });
        const d = await res.json();
        return (d?.data || [])
          .filter(item => item.type === 'search_result')
          .map(item => ({
            title: (item.object?.title || '').replace(/<[^>]+>/g, ''),
            type: item.object?.type || '',
            author: item.object?.author?.name || '',
            votes: item.object?.voteup_count || 0,
          }));
      })()

  - map:
      rank: ${{ index + 1 }}
      title: ${{ item.title }}
      type: ${{ item.type }}
      author: ${{ item.author }}
      votes: ${{ item.votes }}

  - limit: ${{ args.limit }}

columns: [rank, title, type, author, votes]

Tier 4 — Store Action Bridge (

tap

step, recommended for intercept strategy)

Suitable for Vue + Pinia/Vuex sites (such as Xiaohongshu), no need to write XHR interception code manually:

yaml

# src/clis/xiaohongshu/notifications.yaml
site: xiaohongshu
name: notifications
description: "Xiaohongshu Notifications"
domain: www.xiaohongshu.com
strategy: intercept
browser: true

args:
  type:
    type: str
    default: mentions
    description: "Notification type: mentions, likes, or connections"
  limit:
    type: int
    default: 20

columns: [rank, user, action, content, note, time]

pipeline:
  - navigate: https://www.xiaohongshu.com/notification
  - wait: 3
  - tap:
      store: notification       # Pinia store name
      action: getNotification   # Store action to call
      args:                     # Action arguments
        - ${{ args.type | default('mentions') }}
      capture: /you/            # URL pattern to capture response
      select: data.message_list # Extract sub-path from response
      timeout: 8
  - map:
      rank: ${{ index + 1 }}
      user: ${{ item.user_info.nickname }}
      action: ${{ item.title }}
      content: ${{ item.comment_info.content }}
  - limit: ${{ args.limit | default(20) }}

The
tap
step automatically completes: Inject fetch + XHR dual interception → Find Pinia/Vuex store → Call action → Capture response matching URL → Clean up interception.
If the store or action is not found, a
hint
will be returned listing all available store actions for easy debugging.

Tap Parameter	Required	Description
`store`	✅	Pinia store name (e.g. `feed` , `search` , `notification` )
`action`	✅	Store action method name
`capture`	✅	URL substring match (matches network request URL)
`args`	❌	Parameter array passed to action
`select`	❌	Path extracted from captured JSON (e.g. `data.items` )
`timeout`	❌	Timeout in seconds waiting for network response (default 5s)
`framework`	❌	`pinia` or `vuex` (auto detected by default)

Method B: TypeScript Adapter (Programmatic)

Suitable for scenarios that require embedded JS code to read Pinia state, XHR interception, GraphQL, pagination, complex data conversion, etc.

File path:

src/clis/<site>/<name>.ts

. The file will be dynamically scanned and registered at runtime (do not manually

import

index.ts

Tier 3 — Header Authentication (Twitter)

typescript

// src/clis/twitter/search.ts
import { cli, Strategy } from '../../registry.js';

cli({
  site: 'twitter',
  name: 'search',
  description: 'Search tweets',
  strategy: Strategy.HEADER,
  args: [{ name: 'query', required: true, positional: true }],
  columns: ['rank', 'author', 'text', 'likes'],
  func: async (page, kwargs) => {
    await page.goto('https://x.com');
    const data = await page.evaluate(`
      (async () => {
        // Extract CSRF token from Cookie
        const ct0 = document.cookie.split(';')
          .map(c => c.trim())
          .find(c => c.startsWith('ct0='))?.split('=')[1];
        if (!ct0) return { error: 'Not logged in' };

        const bearer = 'AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D...';
        const headers = {
          'Authorization': 'Bearer ' + decodeURIComponent(bearer),
          'X-Csrf-Token': ct0,
          'X-Twitter-Auth-Type': 'OAuth2Session',
        };

        const variables = JSON.stringify({ rawQuery: '${kwargs.query}', count: 20 });
        const url = '/i/api/graphql/xxx/SearchTimeline?variables=' + encodeURIComponent(variables);
        const res = await fetch(url, { headers, credentials: 'include' });
        return await res.json();
      })()
    `);
    // ... parse data
  },
});

Tier 4 — XHR/Fetch Dual Interception (Common pattern for Twitter/Xiaohongshu)

typescript

// src/clis/xiaohongshu/user.ts
import { cli, Strategy } from '../../registry.js';

cli({
  site: 'xiaohongshu',
  name: 'user',
  description: 'Get user notes',
  strategy: Strategy.INTERCEPT,
  args: [{ name: 'id', required: true }],
  columns: ['rank', 'title', 'likes', 'url'],
  func: async (page, kwargs) => {
    await page.goto(`https://www.xiaohongshu.com/user/profile/${kwargs.id}`);
    await page.wait(5);

    // Low-level XHR/Fetch interception: capture all requests containing 'v1/user/posted'
    await page.installInterceptor('v1/user/posted');

    // Trigger backend API: simulate human user scrolling to the bottom 2 times
    await page.autoScroll({ times: 2, delayMs: 2000 });

    // Extract all intercepted JSON response bodies
    const requests = await page.getInterceptedRequests();
    if (!requests || requests.length === 0) return [];

    let results = [];
    for (const req of requests) {
      if (req.data?.data?.notes) {
        for (const note of req.data.data.notes) {
           results.push({
             title: note.display_title || '',
             likes: note.interact_info?.liked_count || '0',
             url: `https://explore/${note.note_id || note.id}`
           });
        }
      }
    }

    return results.slice(0, 20).map((item, i) => ({
      rank: i + 1, ...item,
    }));
  },
});

Core idea of interception: Do not construct the signature yourself, but use
installInterceptor
to hijack the site's own
XMLHttpRequest
and
fetch
, let the site send the request, and we directly extract the parsed
response.json()
at the bottom layer.

Cascading requests (e.g. BVID→CID→subtitles) full template and key points can be found in the Advanced Mode: Cascading Requests section below.

Step 4: Testing

Build pass ≠ Functional normal.
npm run build
only validates TypeScript / YAML syntax, not runtime behavior.
Each new command is only completed after actually running and confirming the output is correct.

Required Checklist

bash

# 1. Build (confirm syntax is correct)
npm run build

# 2. Confirm command is registered
opencli list | grep mysite

# 3. Actually run the command (most important!)
opencli mysite hot --limit 3 -v        # verbose to view data flow of each step
opencli mysite hot --limit 3 -f json   # JSON output to confirm complete fields

tap Step Debugging (Exclusive for intercept strategy)

Do not guess store name / action name. First explore with evaluate, then write YAML.

Step 1: List all Pinia stores

After opening the target website in the browser:

bash

opencli evaluate "(() => {
  const app = document.querySelector('#app')?.__vue_app__;
  const pinia = app?.config?.globalProperties?.\$pinia;
  return [...pinia._s.keys()];
})()"
# Output: ["user", "feed", "search", "notification", ...]

Step 2: Check store action names

Intentionally write a wrong action name, tap will return all available actions:

⚠  tap: Action not found: wrongName on store notification
💡 Available: getNotification, replyComment, getNotificationCount, reset

Step 3: Confirm capture pattern with network requests

bash

# Open the target page in the browser, view network requests
# Find the URL feature of the target API (e.g. "/you/mentions", "homefeed")

Full Workflow

 ┌──────────────┐     ┌──────────────┐     ┌──────────────┐     ┌────────┐
 │ 1. Navigate  │ ──▶ │ 2. Explore store │ ──▶ │ 3. Write YAML   │ ──▶ │ 4. Test │
 │    to target page  │     │ name/action  │     │    tap step   │     │ Run verification │
 └──────────────┘     └──────────────┘     └──────────────┘     └────────┘

Verbose Mode & Output Verification

bash

opencli bilibili hot --limit 1 -v          # View data flow of each pipeline step
opencli mysite hot -f json | jq '.[0]'     # Confirm JSON can be parsed
opencli mysite hot -f csv > data.csv       # Confirm CSV can be imported

Step 5: Submit and Release

Put the file into

src/clis/<site>/

to register automatically (no manual import required for YAML or TS), then:

bash

opencli list | grep mysite                            # Confirm registration
git add src/clis/mysite/ && git commit -m "feat(mysite): add hot" && git push

Architecture Concept: OpenCLI has built-in Zero-Dependency jq data flow — all parsing is done in native JS within
evaluate
, the outer YAML uses
select
/
map
for extraction, no dependency on system
jq
binary.

Advanced Mode: Cascading Requests

When the target data requires multi-step API chained fetching (e.g.

BVID → CID → subtitle list → subtitle content

), you must use a TS adapter. YAML cannot handle this multi-step logic.

Template Code

typescript

import { cli, Strategy } from '../../registry.js';
import type { IPage } from '../../types.js';
import { apiGet } from './utils.js'; // Reuse platform SDK

cli({
  site: 'bilibili',
  name: 'subtitle',
  strategy: Strategy.COOKIE,
  args: [{ name: 'bvid', required: true }],
  columns: ['index', 'from', 'to', 'content'],
  func: async (page: IPage | null, kwargs: any) => {
    if (!page) throw new Error('Requires browser');

    // Step 1: Establish Session
    await page.goto(`https://www.bilibili.com/video/${kwargs.bvid}/`);

    // Step 2: Extract intermediate ID from page (__INITIAL_STATE__)
    const cid = await page.evaluate(`(async () => {
      return window.__INITIAL_STATE__?.videoData?.cid;
    })()`);
    if (!cid) throw new Error('Cannot extract CID');

    // Step 3: Call next-level API with intermediate ID (automatic Wbi signature)
    const payload = await apiGet(page, '/x/player/wbi/v2', {
      params: { bvid: kwargs.bvid, cid },
      signed: true, // ← Auto generate w_rid
    });

    // Step 4: Detect risk control degradation (null assertion)
    const subtitles = payload.data?.subtitle?.subtitles || [];
    const url = subtitles[0]?.subtitle_url;
    if (!url) throw new Error('subtitle_url is empty, suspected risk control degradation');

    // Step 5: Fetch final data (CDN JSON)
    const items = await page.evaluate(`(async () => {
      const res = await fetch(${JSON.stringify('https:' + url)});
      const json = await res.json();
      return { data: json.body || json };
    })()`);

    return items.data.map((item, idx) => ({ ... }));
  },
});

Key Points

Step	Notes
Extract intermediate ID	Prioritize getting from `__INITIAL_STATE__` to avoid extra API calls
Wbi signature	Bilibili `/wbi/` interfaces mandatory check `w_rid` , pure `fetch` will get 403
Null assertion	Even with HTTP 200, core fields may be empty strings (risk control degradation)
CDN URL	Usually starts with `//` , remember to add `https:`
`JSON.stringify`	Must use it to escape when splicing URL into evaluate to avoid injection

Common Pitfalls

Pitfall	Symptom	Solution
Missing `navigate`	evaluate throws `Target page context` error	Add `navigate:` step before evaluate
Nested field access	`${{ item.node?.title }}` does not work	Flatten data in evaluate, do not use optional chaining in templates
Missing `strategy: public`	Public API also starts browser, 7s → 1s	Add `strategy: public` + `browser: false` for public APIs
evaluate returns string	map step receives `""` instead of array	Pipeline has auto-parse, but it is recommended to reshape with `.map()` in evaluate
Search parameters are URL encoded	`${{ args.query }}` is double encoded by browser	Manually encode with `encodeURIComponent()` in evaluate
Cookie expired	Returns 401 / empty data	Log in to the target site again in the browser
Extension tab residual	Extra `chrome-extension://` tab in Chrome	Automatically cleaned; if residual, close manually
TS evaluate format	`() => {}` reports `result is not a function`	`page.evaluate()` in TS must use IIFE: `(async () => { ... })()`
Page asynchronous loading	evaluate gets empty data (store state not updated yet)	Use polling in evaluate to wait for data to appear, or increase `wait` time
YAML embedded large JS	Difficult debugging, string escaping issues	Commands with more than 10 lines of JS use TS adapter instead
Risk control interception (pseudo 200)	Core data in the obtained JSON is `""` (empty string)	Very easy to be misjudged. Must add assertion! If there is no core data, immediately request to upgrade authentication Tier and reconfigure Cookie
API not found	The results scored by the `explore` tool cannot get deep data	Click page buttons to trigger lazy loaded data, then combine with `getInterceptedRequests` to obtain

Automatically Generate Adapters with AI Agent

The fastest way is to let AI Agent complete the whole process:

bash

# One-click: Explore → Analyze → Synthesize → Register
opencli generate https://www.example.com --goal "hot"

# Or execute step by step:
opencli explore https://www.example.com --site mysite           # Discover API
opencli explore https://www.example.com --auto --click "字幕,CC"  # Simulate click to trigger lazy loaded API
opencli synthesize mysite                                        # Generate candidate YAML
opencli verify mysite/hot --smoke                                # Smoke test

The generated candidate YAML is saved in

.opencli/explore/mysite/candidates/

, which can be directly copied to

src/clis/mysite/

and fine-tuned.

Record Workflow

record

is a manual recording solution for pages that "cannot be automatically discovered with

explore

" (require login operations, complex interactions, in-SPA routing).

Working Principle

opencli record <url>
  → Open automation window and navigate to target URL
  → Inject fetch/XHR interceptor to all tabs (idempotent, can be injected repeatedly)
  → Poll every 2s: automatically inject when new tab is found, drain capture buffers of all tabs
  → Stop on timeout (default 60s) or Enter press
  → Analyze captured JSON requests: deduplicate → score → generate candidate YAML

Interceptor Features:

Patch both
```
window.fetch
```
and
```
XMLHttpRequest
```
at the same time
Only capture responses with
```
Content-Type: application/json
```
Filter responses with plain objects with less than 2 keys (avoid tracking/ping)
Cross-tab isolation: each tab has independent buffer, drained separately during polling
Idempotent injection: when injecting twice in the same tab, restore original functions first then re-patch, no lost captured data

Usage Steps

bash

# 1. Start recording (recommended to give enough operation time with --timeout)
opencli record "https://example.com/page" --timeout 120000

# 2. Operate the page normally in the pop-up automation window:
#    - Open lists, search, click items, switch tabs
#    - All operations that trigger network requests will be captured

# 3. Press Enter to stop after completing operations (or wait for timeout to stop automatically)

# 4. View results
cat .opencli/record/<site>/captured.json        # Raw capture
ls  .opencli/record/<site>/candidates/          # Candidate YAML

Page Types and Capture Expectations

Page Type	Expected Capture Volume	Description
List/Search Page	High (5~20+)	Each search/page turn triggers new requests
Detail Page (Read-only)	Low (1~5)	First screen data is returned at once, subsequent operations go through form/redirect
In-SPA route跳转	Medium	Route switching triggers new interfaces, but first screen requests are sent before injection
Pages requiring login	Depends on operation	Ensure Chrome is logged into the target site

Note: If the page sends most requests before navigation is completed (Server-Side Rendering / SSR hydration), the interceptor will miss these requests. Solution: After the page is loaded, manually trigger operations that generate new requests (search, page turn, switch tabs, expand collapsed items, etc.).

Candidate YAML → TS CLI Conversion

The generated candidate YAML is a starting point, usually needs to be converted to TypeScript (especially for internal systems such as tae):

Candidate YAML Structure (auto generated):

yaml

site: tae
name: getList          # Name inferred from URL path
strategy: cookie
browser: true
pipeline:
  - navigate: https://...
  - evaluate: |
      (async () => {
        const res = await fetch('/approval/getList.json?procInsId=...', { credentials: 'include' });
        const data = await res.json();
        return (data?.content?.operatorRecords || []).map(item => ({ ... }));
      })()

Convert to TS CLI (refer to the style of

src/clis/tae/add-expense.ts

typescript

import { cli, Strategy } from '../../registry.js';

cli({
  site: 'tae',
  name: 'get-approval',
  description: 'View reimbursement form approval process and operation records',
  domain: 'tae.alibaba-inc.com',
  strategy: Strategy.COOKIE,
  browser: true,
  args: [
    { name: 'proc_ins_id', type: 'string', required: true, positional: true, help: 'Process instance ID (procInsId)' },
  ],
  columns: ['step', 'operator', 'action', 'time'],
  func: async (page, kwargs) => {
    await page.goto('https://tae.alibaba-inc.com/expense/pc.html?_authType=SAML');
    await page.wait(2);
    const result = await page.evaluate(`(async () => {
      const res = await fetch('/approval/getList.json?taskId=&procInsId=${kwargs.proc_ins_id}', {
        credentials: 'include'
      });
      const data = await res.json();
      return data?.content?.operatorRecords || [];
    })()`);
    return (result as any[]).map((r, i) => ({
      step: i + 1,
      operator: r.operatorName || r.userId,
      action: r.operationType,
      time: r.operateTime,
    }));
  },
});

Conversion Key Points

Extract dynamic IDs in URL (
```
procInsId
```
,
```
taskId
```
, etc.) as
```
args
```
The real body structure in
```
captured.json
```
is used to determine the correct data path (e.g.
```
content.operatorRecords
```
)
Tae system uniformly uses
```
{ success, content, errorCode, errorMsg }
```
outer wrapper, data fetching should go through
```
content.*
```
Authentication method: cookie (
```
credentials: 'include'
```
), no extra header required
Put the file into
```
src/clis/<site>/
```
, no manual registration required, automatically discovered after
```
npm run build
```

Troubleshooting

Symptom	Cause	Solution
0 requests captured	Interceptor injection failed, or page has no JSON API	Check if daemon is running: `curl localhost:19825/status`
Low capture volume (1~3 requests)	Page is read-only detail page, first screen data was sent before injection	Manually trigger more requests (search/page turn), or use list page instead
0 candidate YAML	All captured JSON have no array structure	Directly view `captured.json` and write TS CLI manually
New tab is not intercepted	Tab is closed within polling interval	Shorten `--poll 500`
Data is not continuous when running record for the second time	Normal, each `record` starts a new automation window	No processing required

opencli-explorer

NPX Install

Tags

SKILL.md Content (Chinese)

CLI-EXPLORER — Complete Guide to Exploratory Adapter Development

Must-read for AI Agent Developers: Explore with Browsers

Why?

AI Agent Exploration Workflow (Must Follow)

Common Mistakes

Practical Success Case: Implement "Follow List" Adapter in 5 Minutes

Core Workflow

Step 1: API Discovery

1a. Automated Discovery (Recommended)

1b. Manual Packet Capture Verification

1c. Advanced API Discovery Heuristics

1d. Framework Detection

Step 2: Select Authentication Strategy

Strategy Decision Tree

Comparison of Each Strategy

Step 2.5: Preparation (Before Writing Code)

Find Templates First: Start with the Most Similar Existing Adapter

Platform SDK Cheat Sheet

Bilibili (src/clis/bilibili/utils.ts)

Step 3: Write Adapter

YAML vs TS? Check the Decision Tree First

Common Pattern: Pagination API

Method A: YAML Pipeline (Declarative, Recommended)

Tier 1 — Public API Template

Tier 2 — Cookie Authentication Template (Most Used)

Advanced — With Search Parameters

Tier 4 — Store Action Bridge (tap step, recommended for intercept strategy)

Method B: TypeScript Adapter (Programmatic)

Tier 3 — Header Authentication (Twitter)

Tier 4 — XHR/Fetch Dual Interception (Common pattern for Twitter/Xiaohongshu)

Step 4: Testing

Required Checklist

tap Step Debugging (Exclusive for intercept strategy)

Step 1: List all Pinia stores

Step 2: Check store action names

Step 3: Confirm capture pattern with network requests

Full Workflow

Verbose Mode & Output Verification

Step 5: Submit and Release

Advanced Mode: Cascading Requests

Template Code

Key Points

Common Pitfalls

Automatically Generate Adapters with AI Agent

Record Workflow

Working Principle

Usage Steps

Page Types and Capture Expectations

Candidate YAML → TS CLI Conversion

Conversion Key Points

Troubleshooting

Bilibili (
`src/clis/bilibili/utils.ts`
)

Tier 4 — Store Action Bridge (
`tap`
step, recommended for intercept strategy)