scripts/batch_download.mjs

SI confirmation gate — do this first. Before downloading any PDF, CAJ, HTML, XML, archive, or attachment, ask whether the user wants Supporting Information. An explicit request for SI counts as yes; an explicit request for正文 only counts as no. Otherwise ask once for the whole batch. Run the downloader with exactly one of

--si

or

--no-si

. Without either flag the script returns

si_confirmation_required

and does not create the output directory.

Main workflow. Normalize the DOI/title and identify language and publisher before routing. Chinese literature always uses CNKI. For English Elsevier, Springer Nature, and IEEE articles with usable provider credentials, try the publisher API first and do not require an OA determination after a successful API download. If that API attempt fails, automatically check legitimate OA sources. Other English publishers check OA first, then use the institutional Web Access route when OA is unavailable.

规范化 DOI/题名并识别语言、出版商
├─ 中文文献：直接走 CNKI
└─ 英文文献
   ├─ Elsevier / Springer Nature / IEEE，且已配置有效 Key
   │  ├─ 优先通过出版商 API 下载
   │  ├─ API 下载成功：结束，不强制判断 OA
   │  └─ API 下载失败：检查文章级 OA，再走 PMC / Unpaywall / 合法仓储
   └─ 其他出版商
      ├─ 检查文章级 OA
      └─ OA 不可用：走 Web Access 机构授权

Chinese literature is CNKI-only. A Chinese title,

zh

metadata language, explicit CNKI source URL, or

--route cnki

must use CNKI even if another OA copy appears to exist. Reuse the user's current Chrome library/CNKI login state and prefer configured

discovery.cnki_url

. Never export cookies or collect the institutional password.

Publisher API fallback. A valid API key does not guarantee full-text entitlement. When an Elsevier, Springer Nature, or IEEE API attempt returns no entitlement or no usable full text, automatically try legitimate OA sources first. Return

api_fallback_confirmation_required

and ask once whether to use Web Access only after both the publisher API and OA routes fail. Do not switch to institutional Web Access automatically.

Browser-state principle. Authorized downloads depend on the exact browser profile where the user is logged in. If a proxy, CDP session, or browser automation tool opens a fresh profile or a different browser with no login state, do not treat the failure as missing library permission. Switch to a control path that reuses the user's active browser session, or ask the user to authenticate in the controlled browser instance.

Format principle. PDF, HTML full text, and database-native formats such as CAJ are different deliverables. If the user asks for PDF only, require a real PDF link or

%PDF

response and report

no_authorized_pdf_found

/

pdf_fetch_failed

when none exists. Do not save CAJ, HTML, or a login page as if it were a PDF.

1. Collect a DOI, PMID, exact title, article URL, or a definite paper list, then normalize its metadata and publisher.
2. If it belongs to Elsevier, Springer Nature, or IEEE and usable provider credentials are configured, try that publisher API first. On success, record

   accessMode: publisher_api

   and

   oa_status: not_checked_api_first

   ; do not run OA resolution only to label the article.
3. If the publisher API fails, automatically search legitimate OA sources such as PMC, Unpaywall, publisher OA pages, arXiv, and other lawful repositories or clearly open PDF URLs. Preserve the failed API attempt in the manifest.
4. For all other English publishers, search those legitimate OA sources before Web Access.
5. For an exact title or an explicit OA-only request, prefer:
   bash

   node scripts/batch_download.mjs --title "<exact title>" --open-access --no-si --out "<project>"

   Use

   --pdf-url

   when the user supplies a known legitimate OA PDF URL.
6. Verify the downloaded file and record the source. Mark a successful PDF as

   open_access_downloaded

   .
7. If no lawful OA full text is found, mark

   oa_not_found

   . For a supported publisher whose API already failed, request confirmation before Web Access. For another publisher, continue to Web Access. If

   --route open_access

   was explicitly requested, stop after the OA result.

• Slider/drag challenges (including CNKI puzzle sliders): estimate the visible travel distance and simulate a gradual drag.
• ScienceDirect robot checks, managed Turnstile, and reCAPTCHA checkbox stages: try the visible checkbox once.
• Simple

  Continue

  ,

  Verify

  , or equivalent visible controls: click once, then re-check the page state.

1. The browser that holds the user's library/database login state is open on the user's machine.
2. The school configuration exists and is valid.
   • Run

     python3 scripts/configure_school.py show

     .
   • If missing, run

     python3 scripts/configure_school.py preset "<school name>"

     or guide the user through

     src/wizard.py

     .
3. The user has personally logged in to their institution/library route in that same browser, and can reach the library aggregation service, target database, or discovery entry.
4. The browser-control path can reuse that same logged-in browser profile.
   • For Chrome CDP, ask the user to open

     chrome://inspect/#remote-debugging

     and enable remote debugging for the current browser instance.
   • If CDP attaches to a stale browser, a temporary profile, or a different browser, use a browser-control channel that can reuse the user's active session instead of launching a new profile.
5. The environment can run Node.js 22+.
   • Try

     node --version

     .
   • If

     node

     is not on PATH in Codex Desktop, try

     %LOCALAPPDATA%\OpenAI\Codex\bin\node.exe

     .
6. The environment can run Python 3 for configuration and PDF text verification.
   • Try

     python3 --version

     .
   • Install Python helpers with

     pip install -r requirements.txt

     when needed.
7. The web-access CDP proxy is available or can be started.
   • Typical Claude Code path:

     %USERPROFILE%\.claude\skills\web-access-main\scripts\check-deps.mjs

     .
   • Typical shared agent path:

     %USERPROFILE%\.agents\skills\web-access-main\scripts\check-deps.mjs

     .
   • In Codex-only setups also check

     %USERPROFILE%\.codex\skills\web-access-main\scripts\check-deps.mjs

     .
8. The user has approved the target output folder.

$env:USERPROFILE\.claude\skills\nature-downloader

.codex\skills

.agents\skills

• pace requests appropriately for each provider and maintain the manifest throughout the batch
• attempt visible verification controls first; stop after at most two failed attempts, on institutional login expiry, or when an unusual/security-sensitive prompt appears

$node = "node"
if (-not (Get-Command node -ErrorAction SilentlyContinue)) {
  $node = "$env:LOCALAPPDATA\OpenAI\Codex\bin\node.exe"
}
$checkDepsCandidates = @(
  "$env:USERPROFILE\.claude\skills\web-access-main\scripts\check-deps.mjs",
  "$env:USERPROFILE\.agents\skills\web-access-main\scripts\check-deps.mjs",
  "$env:USERPROFILE\.codex\skills\web-access-main\scripts\check-deps.mjs"
)
$checkDeps = $checkDepsCandidates | Where-Object { Test-Path $_ } | Select-Object -First 1
if (-not $checkDeps) { throw "web-access-main/scripts/check-deps.mjs not found" }
& $node $checkDeps

Invoke-WebRequest -UseBasicParsing -Uri "http://127.0.0.1:3456/targets" -TimeoutSec 10

• Ask the user to confirm the remote debugging checkbox.
• Check

  %TEMP%\cdp-proxy.log

  .
• If targets appear but the database/library page is unauthenticated, suspect a stale CDP endpoint, wrong browser, or fresh browser profile before suspecting missing library permission.
• Do not attempt to read Chrome session files.

open_access_downloaded

--topic

--title "<exact title>" --open-access

--pdf-url

--topic "Attention Is All You Need"

--title "Attention Is All You Need" --open-access

1706.03762v7

webofscience.clarivate.cn

www.webofscience.com

*.webofknowledge.com

*.clarivate.com

1. Authenticate once: open Web of Science via the library aggregation / institutional entry. If Web of Science or another database shows authentication choices such as institutional login, Shibboleth/OpenAthens/CARSI, CAS/SSO, or IP login, use the route the user normally uses. If credentials, QR, CAPTCHA, SMS/OTP, or unclear consent appears, follow Institutional Authentication Handoff below.
2. Confirm you are on the authenticated Web of Science search page (institutional name visible, search box present).
3. Search the paper by DOI when available, otherwise by exact title:
   • Set the search field to

     DOI

     or

     Title

     , paste the value, run the search.
   • Read the results page with

     /eval

     and pick the record that matches title + year + authors.
4. Open the matching record and read it with

   /eval

   .
5. Click the full-text route, in this order of preference:

   • Free Full Text

     /

     Open Access

     if present
   • library resolver links:

     Find it at

     ,

     SFX

     ,

     OpenURL

     ,

     Full Text Links

     ,

     查看全文

     ,

     Full Text available via

     , database/provider names such as Ovid
   • publisher full-text link:

     View Full Text

     , the publisher name, or

     View PDF

   • The full-text link should inherit the institutional session, so the publisher often grants access without a second login. If a second institutional handoff appears, complete it once.
6. On the publisher page, find the PDF link (

   PDF

   ,

   View PDF

   ,

   Download PDF

   ,

   pdfft

   ,

   /doi/pdf/

   ) and save it with

   scripts/browser_pdf_downloader.mjs

   .
7. If the full-text resolver opens readable HTML full text but no valid PDF is exposed, save the HTML/text, mark

   full_text_html_available

   , and tell the user plainly: "已获取 HTML 全文，但当前授权路径没有可下载 PDF." Do not mislabel an HTML page as a PDF; if a PDF probe returns HTML, move it to diagnostics and explain that no valid PDF was downloaded.
8. If the resolver/provider explicitly says the institution has no entitlement, mark

   library_no_permission

   and tell the user: "当前图书馆资源没有该文献全文权限." Do not hide this behind

   failed_after_retry

   .
9. Do not download Supporting Information by default. Only fetch SI if the user explicitly asked; otherwise just note whether SI exists (see Supporting Information below).
10. Record the route taken (OA source or WoS → SFX/OpenURL/full-text provider → publisher/database) in the manifest.

no_full_text_link

library_no_permission

1. Prefer the library aggregation / CARSI entry before direct

   doi.org -> publisher

   navigation.
2. Process ScienceDirect and other sensitive publishers one article at a time.
3. Keep a visible audit trail in the manifest; do not open many publisher tabs in parallel.
4. Wait for each page to settle before looking for

   Download PDF

   ,

   View PDF

   , or

   PDF

   .
5. Reuse the same tab after the user completes a verification step instead of opening repeated new tabs.
6. Avoid retry loops. Use one attempt by default and no more than two attempts on the same tab before handing the page to the user.

1. First, attempt automatic verification via the built-in anti-bot module (

   scripts/lib/anti-bot.mjs

   ). The module tries: simple click challenges, ScienceDirect robot check, Cloudflare Turnstile, slider CAPTCHA (including CNKI Geetest-style), and reCAPTCHA checkbox.
2. If auto-verification succeeds, continue the download from the resolved page.
3. If auto-verification fails: a. Stop automated actions on that tab. b. Record the paper with status

   verification_auto_failed

   . Use

   sciencedirect_robot_check

   only when no automatic interaction was possible. c. Tell the user which paper and tab need manual attention. d. After the user says the verification is complete, continue from the same tab and try the visible article/PDF route once. e. If verification immediately reappears, mark

   do_not_auto_retry

   and move on.

publisher_verification.tsv

id	project	title	doi	year	venue	publisher	status	source_url	current_url	next_action	notes

next_action

user_complete_publisher_verification
retry_same_tab_after_user_confirms
try_aggregation_entry_route
try_authorized_oa_route
mark_do_not_auto_retry

1. Stop automated actions on that tab.
2. Record the paper in

   carsi_retry.tsv

   with status

   carsi_waiting_user

   .
3. Tell the user exactly which tab/page needs attention, for example: "This page is at your institution/database login. If the browser has already filled credentials, I can click the visible login/confirm button once with your authorization; otherwise please complete it in the browser." If a federation/WAYF page asks which institution to use, ask the user to pick their institution, or do it only when the choice is unambiguous and credential-free and the user authorized it.
4. Do not read, store, or request the password, QR result, OTP, SMS code, CAPTCHA, cookie, or local/session storage.
5. If the user explicitly authorizes clicking because credentials are already filled, click only the visible login/confirm/continue button once. Do not type into fields or inspect hidden credential values. For credential-free options such as "IP login", click only when the user authorizes that route or has just completed it manually.
6. If QR login, SMS/OTP, CAPTCHA, Cloudflare, or publisher bot verification appears, stop and let the user complete it manually.
7. After the login/confirm step completes, refresh or continue from the same tab.
8. Re-detect whether the page is now a publisher article page, a PDF viewer, or another institutional handoff.
9. If resolved, download and verify the PDF/SI, then update the manifest status to

   downloaded

   or

   downloaded_with_si

   .
10. If it loops back to the same institutional/database login after a completed user login, record

    failed_after_retry

    with the observed reason and move on.

403

401

$node = "$env:LOCALAPPDATA\OpenAI\Codex\bin\node.exe"
& $node "$env:USERPROFILE\.agents\skills\nature-downloader\scripts\browser_pdf_downloader.mjs" `
  --url "https://www.sciencedirect.com/science/article/pii/SXXXXXXXXXXXXXXXX/pdfft" `
  --out "D:\path\paper.pdf"

• Opens the URL in the user's controlled Chrome session unless

  --target

  is provided.
• Runs

  fetch(location.href, { credentials: "include" })

  inside the page.
• Transfers bytes in chunks through the local CDP proxy.
• Writes the binary file to disk.
• Verifies the

  %PDF

  signature by default.

--url <url>          PDF URL to open and save
--target <targetId>  Existing Chrome target/tab id to use
--out <path>         Output PDF path
--proxy <url>        CDP proxy URL, default http://127.0.0.1:3456
--close              Close the tab after download if the script opened it
--allow-non-pdf      Save even when content does not start with %PDF

--no-si

1. Open the article landing page, not only the PDF page.
2. Extract all links with text or href matching:

   • Supporting Information

   • Supplementary

   • Supplemental

   • /doi/suppl/

   • /suppl_file/

   • _si_

   • mmc1

     ,

     mmc2

     (Elsevier/ScienceDirect supplement pattern)
3. Download every PDF/DOCX/XLSX/video/data file that is clearly a legitimate supplement, using the browser context if needed.

--si

--topic

--title

--count 1

--si

• keep each paper in its own readable-title folder;
• place only the verified main PDF and clearly labelled SI files in that folder;
• preserve original attachment names when available;
• follow a supplementary landing page at most one level deep;
• exclude external repository links such as GitHub, Zenodo, Figshare, Dryad, and OSF;
• keep the main PDF and report

  si.status = not_found

  when no SI exists;
• report

  partial

  when some SI files fail without treating the main PDF as failed.

--open-access

--pdf-url

--dois

https://pubs.acs.org/doi/suppl/<DOI>/suppl_file/<journal-code>_si_001.pdf

$env:PYTHONUTF8='1'
python -X utf8 "$env:USERPROFILE\.claude\skills\nature-downloader\scripts\extract_pdf_text.py" `
  --pdf "D:\path\paper.pdf" `
  --pages 3

pdf

• File exists and size is plausible.
• First bytes are

  %PDF

  for PDF files.
• Page count is nonzero.
• Extracted text includes the article title, abstract, or supporting information title.
• For HTML full text, saved HTML/text includes the article title or DOI, and the user-facing reply states that no valid PDF was available.
• Save a small manifest with DOI, title, source URL, download date, and supplement status when doing more than one paper.

• First, attempt automatic verification via

  scripts/lib/anti-bot.mjs

  .
• If auto-verification succeeds, continue the download normally.
• If auto-verification fails, record

  verification_auto_failed

  or

  sciencedirect_robot_check

  .
• Ask the user to solve it in Chrome.
• Then continue once from the same now-open page.
• If the same challenge immediately reappears, mark

  do_not_auto_retry

  and move on.

Invoke-WebRequest

curl

• Use

  browser_pdf_downloader.mjs

  ; this is the normal institutional-access case.

• Do not ask for or accept institutional credentials in chat. Publisher API keys follow the separate save-on-receipt rule.
• Pause and ask the user to complete the verification in Chrome.
• Record

  publisher_verification_waiting_user

  in

  publisher_verification.tsv

  , or

  sciencedirect_robot_check

  for ScienceDirect.
• Continue only after the user says the browser step is complete.

• Do not ask for or accept institutional credentials in chat. Publisher API keys follow the separate save-on-receipt rule.
• If the user has explicitly authorized it and the browser has already filled credentials, click the visible login/confirm button once.
• Otherwise pause and ask the user to complete the login in the browser.
• Record

  carsi_waiting_user

  or

  carsi_resolved_retry_needed

  in

  carsi_retry.tsv

  as appropriate.

• Try the publisher's own

  Institutional login

  /

  机构登录

  / CARSI/Shibboleth/OpenAthens route and select the user's institution when authorized.
• Try the DOI on the publisher page once an institutional session exists.
• Check open-access copies only from legitimate sources.
• Record

  no_authorized_pdf_found

  rather than seeking unauthorized mirrors.

about:blank

• Treat it as a URL-fragment/encoding problem first, especially when the original URL contains

  #

  or

  #!

  .
• Reopen through

  scripts/cdp_open_url.mjs --url "<full URL>" --wait

  .
• Do not paste fragment-heavy URLs unquoted into shell commands or manually concatenate them into

  /new?url=...

  without URL encoding.

curl

• Use PowerShell

  Invoke-WebRequest

  for simple proxy checks.
• Prefer the bundled Node.js helper scripts for CDP proxy actions because Node's

  URLSearchParams

  preserves nested URL fragments correctly.

• Ask the user to re-authenticate through their institution/library route in the same browser, then reopen the publisher/database entry.

1. The exact institutional login, federation, proxy, WebVPN, or database hosts that appear in the address bar.
2. The base URL / link pattern of the library aggregation or database entries the user actually uses.
3. Whether a federation/WAYF/机构选择, IP-login, or database-login step appears, and whether the user authorizes selecting the unambiguous institution/login option.