Back to Details

google-maps-scraper

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Google Maps Scraper

Google Maps 抓取器

Turn a business type and a location into clean lead rows by driving the local engine's job API.
通过驱动本地引擎的任务 API，将商家类型和位置转换为精简的潜客数据行。

Mental model

思维模型

The engine (gosom/google-maps-scraper) runs as a local Docker container with a REST API at 
http://localhost:8080
. It has no auth, so it stays bound to localhost. A scrape is an async job: create it, poll until it is done, then download a CSV. One job can run many keywords. Each result has up to 34 raw fields.
引擎（gosom/google-maps-scraper）作为本地 Docker 容器运行，REST API 位于 
http://localhost:8080
。它不进行身份验证，因此仅绑定到 localhost。一次抓取是一个异步任务（async job）：先创建任务，然后轮询至完成，最后下载 CSV。一个 job 可以运行多个搜索词。每条结果最多包含 34 个原始字段。

Step 0: Check the engine is running

第 0 步：检查引擎是否正在运行

bash
curl -s http://localhost:8080/api/v1/jobs >/dev/null 2>&1 && echo UP || echo DOWN
If 
DOWN
: run 
docker compose up -d
from the kit root, wait ~10s, and retry. If Docker is not installed, point the user to 
SETUP.md
.
bash
curl -s http://localhost:8080/api/v1/jobs >/dev/null 2>&1 && echo UP || echo DOWN
如果输出为 
DOWN
：在工具包根目录运行 
docker compose up -d
，等待约 10 秒，然后重试。如果未安装 Docker，则让用户参阅 
SETUP.md
。

Step 1: Create a job (
POST /api/v1/jobs
)

第 1 步：创建任务（
POST /api/v1/jobs
）

Required fields. The API returns 
422
without them:
  • keywords
    : array of search strings. Bake the location into each term: 
    "plumbers in Denver CO"
    .
  • lat
    , 
    lon
    : strings, the city's coordinates: 
    "39.7392"
    , 
    "-104.9903"
    .
  • max_time
    : integer seconds, the job's wall-clock limit, e.g. 
    300
    . The API stores it as nanoseconds internally, but you always send seconds.
Recommended fields:
  • depth
    (default 10): how far to scroll, so roughly how many listings per keyword. Start at 
    5
    .
  • lang
    "en"
    , 
    zoom
    15
    (city level), 
    radius
    10000
    (meters), 
    fast_mode
    false
    .
  • email
    true
    : on by default in the kit.     Emails are the most valuable lead field. The engine visits each business website to find them, which is a little slower. Set 
    false
    only for a deliberately fast run without emails.
bash
curl -s -X POST http://localhost:8080/api/v1/jobs \
  -H "Content-Type: application/json" \
  -d '{"name":"job","keywords":["coffee shops in Austin TX"],"lang":"en","zoom":15,
       "lat":"30.2672","lon":"-97.7431","fast_mode":false,"radius":10000,
       "depth":5,"email":true,"max_time":300}'
必填字段。缺少这些字段时 API 会返回 
422
：
  • keywords
    ：搜索字符串数组。将位置融入每个词条：例如 
    "plumbers in Denver CO"
    。
  • lat
    、
    lon
    ：字符串，表示城市坐标：例如 
    "39.7392"
    、
    "-104.9903"
    。
  • max_time
    ：整数秒数，表示任务的实际运行时间上限，例如 
    300
    。API 内部以纳秒存储，但你始终应发送秒数。
推荐字段：
  • depth
    （默认 10）：滚动深度，因此大致对应每个搜索词返回的商家数量。建议从 
    5
    开始。
  • lang
    为 
    "en"
    ，
    zoom
    为 
    15
    （城市级别），
    radius
    为 
    10000
    （米），
    fast_mode
    为 
    false
    。
  • **
    email
    为 
    true
    ：在本工具包中默认开启。**邮箱是最有价值的潜客字段。引擎会访问每个商家的网站来查找邮箱，因此速度会稍慢。只有在明确需要不含邮箱的快速任务时，才将其设为 
    false
    。
bash
curl -s -X POST http://localhost:8080/api/v1/jobs \
  -H "Content-Type: application/json" \
  -d '{"name":"job","keywords":["coffee shops in Austin TX"],"lang":"en","zoom":15,
       "lat":"30.2672","lon":"-97.7431","fast_mode":false,"radius":10000,
       "depth":5,"email":true,"max_time":300}'

→ {"id":"<uuid>"} (HTTP 201; note: lowercase "id")

→ {"id":"<uuid>"} (HTTP 201; note: lowercase "id")


> **Handle two things on every scrape:**
> 1. **Emails: on by default** (`email:true`). Turn them off only if the user wants a fast run.
> 2. **Socials: ask first.** Before creating the job, ask the user once whether they also want
>    Instagram, Facebook and LinkedIn profiles (see "Social profiles" below). Never skip the question silently.

> **每次抓取都要处理两件事：**
> 1. **邮箱：默认开启**（`email:true`）。只有在用户要求快速任务时才将其关闭。
> 2. **社交账号：先询问。**创建 job 前，询问用户一次是否还需要 Instagram、Facebook 和 LinkedIn 资料（见下文“社交资料”）。绝不能默默跳过此问题。

Step 2: Poll until done (
GET /api/v1/jobs/{id}
)

第 2 步：轮询至任务完成（
GET /api/v1/jobs/{id}
）

The response field is 
"Status"
(capital S). It moves from 
working
to 
ok
(success) or 
failed
.
Harness rule: the Bash tool blocks foreground 
sleep
. Run the poll loop as a background Bash command (
run_in_background: true
) and read its output file when notified. Never poll with a foreground 
sleep
.
Background poll snippet. It parses the value safely: match up to the closing quote, do not anchor on 
$
.
bash
ID="<uuid>"
for i in $(seq 1 40); do
  S=$(curl -s "http://localhost:8080/api/v1/jobs/$ID" | grep -oE '"Status":"[^"]*"' | head -1 | cut -d'"' -f4)
  echo "status=$S"
  [ "$S" = ok ] && { echo DONE; break; }
  [ "$S" = failed ] && { echo FAILED; break; }
  sleep 15
done
响应字段为 
"Status"
（首字母大写）。其状态会从 
working
变为 
ok
（成功）或 
failed
。
Harness 规则：Bash 工具会阻止前台执行 
sleep
。应将轮询循环作为后台 Bash 命令运行（
run_in_background: true
），并在收到通知时读取其输出文件。绝不能使用前台 
sleep
进行轮询。
后台轮询代码片段。它会安全地解析状态值：匹配到闭合引号为止，不要以 
$
为锚点。
bash
ID="<uuid>"
for i in $(seq 1 40); do
  S=$(curl -s "http://localhost:8080/api/v1/jobs/$ID" | grep -oE '"Status":"[^"]*"' | head -1 | cut -d'"' -f4)
  echo "status=$S"
  [ "$S" = ok ] && { echo DONE; break; }
  [ "$S" = failed ] && { echo FAILED; break; }
  sleep 15
done

Step 3: Download and parse (
GET /api/v1/jobs/{id}/download
)

第 3 步：下载并解析（
GET /api/v1/jobs/{id}/download
）

bash
curl -s "http://localhost:8080/api/v1/jobs/$ID/download" -o results.csv
CSV columns (34): 
input_id, link, title, category, address, open_hours, popular_times, website, phone, plus_code, review_count, review_rating, reviews_per_rating, latitude, longitude, cid, status, descriptions, reviews_link, thumbnail, timezone, price_range, data_id, place_id, images, reservations, order_online, menu, owner, complete_address, about, user_reviews, user_reviews_extended, emails
.
bash
curl -s "http://localhost:8080/api/v1/jobs/$ID/download" -o results.csv
CSV 列（34 个）：
input_id, link, title, category, address, open_hours, popular_times, website, phone, plus_code, review_count, review_rating, reviews_per_rating, latitude, longitude, cid, status, descriptions, reviews_link, thumbnail, timezone, price_range, data_id, place_id, images, reservations, order_online, menu, owner, complete_address, about, user_reviews, user_reviews_extended, emails
。

Return only the 8 lead fields (default)

默认仅返回 8 个潜客字段

The raw CSV has 34 columns and most are noise for outreach. By default, return only the 8 lead fields, the data used to contact and qualify a lead, and drop everything else:
title
(name), 
phone
, 
emails
, 
website
, 
category
, 
address
, 
review_rating
, 
review_count
Drop by default. Show these only if the user explicitly asks: 
latitude
/
longitude
(no use for outreach), 
link
, 
plus_code
, 
cid
, 
data_id
, 
place_id
, 
open_hours
, 
popular_times
, 
reviews_per_rating
, 
reviews_link
, 
thumbnail
, 
images
, 
timezone
, 
price_range
, 
status
, 
input_id
, 
complete_address
, 
reservations
, 
order_online
, 
menu
, 
owner
, 
about
, 
descriptions
, 
user_reviews
, 
user_reviews_extended
.
scripts/scrape.py
already returns exactly the 8 lead fields. Use 
--full
to keep all columns, or 
--fields "a,b,c"
to choose your own. It saves a CSV file by default (
results-<id>.csv
); pass 
--json
for JSON. If you call the API directly, strip to the lead fields yourself before presenting. Never dump the full 34-column row at the user.
原始 CSV 有 34 列，其中大多数对于外联来说是噪声。默认仅返回用于联系和筛选潜客的 8 个潜客字段，并丢弃其他所有内容：
title
（名称）、
phone
、
emails
、
website
、
category
、
address
、
review_rating
、
review_count
**默认丢弃以下字段。**仅当用户明确要求时才展示：
latitude
/
longitude
（对外联没有用途）、
link
、
plus_code
、
cid
、
data_id
、
place_id
、
open_hours
、
popular_times
、
reviews_per_rating
、
reviews_link
、
thumbnail
、
images
、
timezone
、
price_range
、
status
、
input_id
、
complete_address
、
reservations
、
order_online
、
menu
、
owner
、
about
、
descriptions
、
user_reviews
、
user_reviews_extended
。
scripts/scrape.py
已精确返回这 8 个潜客字段。使用 
--full
可保留所有列，也可使用 
--fields "a,b,c"
自行选择字段。它默认保存 CSV 文件（
results-<id>.csv
）；传入 
--json
可输出 JSON。如果直接调用 API，必须在展示前自行精简为潜客字段。绝不能向用户直接输出完整的 34 列数据行。

Social profiles: always ask the user (Instagram, Facebook, LinkedIn)

社交资料：始终询问用户（Instagram、Facebook、LinkedIn）

Google Maps has no social links, so socials are an enrichment step: visit each business 
website
and regex out its Instagram, Facebook and LinkedIn URLs. Before scraping, ask the user once whether they want socials too, unless they already said. If yes, use the script: 
python3 scripts/scrape.py … --socials
.
  • Token cost. Tell the user this when they ask for socials: the extraction itself costs 0 LLM tokens, because it is plain HTTP and regex in the script. It adds ~40–50 tokens per business to your context only if you load the rows into chat, e.g. ~+2k tokens for 50 leads. That is negligible if you keep the file on disk and show a sample. Save the file and show a few rows.
  • Do not fetch each website yourself with WebFetch to find socials. That reads every page into your context and costs thousands of tokens. The script does it for free, so always prefer 
    --socials
    .
  • Socials are opt-in and slower (one HTTP fetch per business), and coverage is partial (~40–70%). Only businesses that link their socials on their site have them, and no website means no socials. Say so if the user expects 100%.
Shortcuts. Prefer these for common cases:
  • One keyword: 
    scripts/scrape.sh "<keyword>" <lat> <lon> [depth]
    (bash) creates, polls and downloads in one go.
  • Auto-geocode (no coordinates): 
    python3 scripts/scrape.py "<keyword>" --city "<City, ST>" [--depth N]
    resolves lat/lon via OpenStreetMap Nominatim. You can also geocode the city yourself and pass coordinates. Emails come back by default (
    --no-email
    skips them). Add 
    --socials
    if the user asked for socials.
  • Batch (many keywords, one job): 
    python3 scripts/scrape.py --keywords-file <file> --city "<City, ST>"
    . The API takes a 
    keywords
    array, so put all terms in a single job rather than firing many jobs. Use the manual curl flow only for custom job bodies, e.g. to set 
    proxies
    .
Whole-city runs: when the user says all, every or entire city, or a run returns about 120 rows (Google's per-search cap), use 
python3 scripts/scrape.py "<keyword>" --city "<City, ST>" --grid 3
. It splits the city into 3 x 3 tiles (9 jobs) and dedupes on 
place_id
. Tiles run one at a time, so a grid is slower and more likely to trip rate limits: warn once, don't block.
Deep email pass: on by default whenever emails are on. Rows with a website but no email get their contact and about pages checked, and the script prints the found rate. 
--no-deep-emails
skips it.
Google Maps 不提供社交链接，因此社交账号属于数据补全步骤：访问每个商家的 
website
，并通过 regex 提取其 Instagram、Facebook 和 LinkedIn URL。抓取前，除非用户已经说明，否则应询问用户一次是否也需要社交账号。如果需要，请使用脚本：
python3 scripts/scrape.py … --socials
。
  • **Token 成本。用户要求社交账号时应告知这一点：**提取过程本身消耗 0 LLM token，因为脚本仅使用普通 HTTP 和 regex。只有当你把各行加载到聊天上下文中时，每家企业才会给你的上下文增加约 40–50 token；例如 50 条潜客约增加 2k token。如果将文件保存在磁盘上并只展示样本，则这一开销可以忽略。保存文件并展示几行。
  • 不要自行使用 WebFetch 逐个获取网站来查找社交账号。这会把每个页面读入你的上下文，并消耗数千 token。脚本可以零成本完成此操作，因此始终优先使用 
    --socials
    。
  • 社交账号是可选项且速度较慢（每个商家需要发起一次 HTTP 请求），覆盖率不完整，约为 40–70%。只有在自己的网站上链接了社交账号的商家才会被找到；没有网站则无法获取社交账号。如果用户期待 100% 覆盖率，应向其说明这一点。
快捷方式。常见情况优先使用以下方式：
  • 单个搜索词：
    scripts/scrape.sh "<keyword>" <lat> <lon> [depth]
    （bash）可在一次操作中创建、轮询并下载任务。
  • 自动地理编码（无需坐标）：
    python3 scripts/scrape.py "<keyword>" --city "<City, ST>" [--depth N]
    通过 OpenStreetMap Nominatim 解析 lat/lon。你也可以自行对城市进行地理编码，然后传入坐标。 默认返回邮箱（
    --no-email
    可跳过）。如果用户需要社交账号，请添加 
    --socials
    。
  • 批量处理（多个搜索词，一个任务）：
    python3 scripts/scrape.py --keywords-file <file> --city "<City, ST>"
    。 API 接受 
    keywords
    数组，因此应将所有词条放入单个 job，而不是启动许多 job。 仅在需要自定义任务正文时使用手动 curl 流程，例如需要设置 
    proxies
    时。
**整座城市任务：**当用户表示“全部”“所有”或“整座城市”，或者任务返回约 120 行（Google 单次搜索上限）时，使用 
python3 scripts/scrape.py "<keyword>" --city "<City, ST>" --grid 3
。 它会将城市划分为 3 x 3 个网格（9 个 job），并根据 
place_id
去重。网格任务会逐个运行，因此速度更慢，也更容易触发限流：只警告一次，不要阻止执行。
**深度邮箱扫描：**只要启用邮箱，它默认就会运行。对于有网站但没有邮箱的行，脚本会检查其联系页面和 about 页面，并打印找到率。
--no-deep-emails
可跳过此步骤。

Other endpoints

其他端点

  • GET /api/v1/jobs
    lists jobs. 
    DELETE /api/v1/jobs/{id}
    deletes a job and frees disk.
  • Browser UI and OpenAPI docs: 
    http://localhost:8080
    and 
    http://localhost:8080/api/docs
    .
  • GET /api/v1/jobs
    列出任务。
    DELETE /api/v1/jobs/{id}
    删除任务并释放磁盘空间。
  • 浏览器 UI 和 OpenAPI 文档：
    http://localhost:8080
    和 
    http://localhost:8080/api/docs
    。

Best practices (from the upstream docs)

最佳实践（来自上游文档）

  • Depth: higher 
    depth
    returns more results but is slower and more likely to be blocked. Start at 5 and raise it as needed.
  • One job at a time locally. Many concurrent jobs without proxies lead to throttling by Google.
  • Email extraction (
    email:true
    )     visits each business website, so it is slower. It stays on by default because emails are the key lead field. Pass 
    --no-email
    (script) or 
    email:false
    (raw API) for a fast run.
  • fast_mode:true
     returns reduced data, up to ~21 results per query, faster. Use it for quick lookups.
  • Proxies: for large or repeated jobs, set 
    "proxies"
    (an array of 
    socks5://
    , 
    http://
    or 
    https://
    URLs, auth supported). The engine rotates them. This is the main defense against rate limits.
  • zoom
    and 
    radius
     set the search area around 
    lat
    /
    lon
    . Widen 
    radius
    if results are too few.
  • Extended reviews are available but heavy. Enable them only if the user asks for review text.
  • Depth：
    depth
    越高，返回的结果越多，但速度越慢，也更容易被阻止。建议从 5 开始，并根据需要逐步提高。
  • **本地一次只运行一个 job。**不使用 proxies 而并发运行许多 job 容易触发 Google 限流。
  • 邮箱提取（
    email:true
    ）会访问每个商家的网站，因此速度较慢。它默认开启    ，因为邮箱是关键潜客字段。传入 
    --no-email
    （脚本）或 
    email:false
    （原始 API）可执行快速任务。
  • fast_mode:true
     会返回精简数据，每次查询最多约 21 条结果，速度更快。适用于快速查询。
  • **Proxies：**对于大型或重复任务，请设置 
    "proxies"
    （一个由 
    socks5://
    、
    http://
    或 
    https://
    URL 组成的数组，支持身份验证）。引擎会自动轮换 proxies。
  • zoom
    和 
    radius
     用于设置 
    lat
    /
    lon
    周围的搜索区域。如果结果太少，可以扩大 
    radius
    。
  • 扩展评论可用，但开销较大。仅当用户明确要求评论文本时才启用。

Rate limits, blocks and proxies (read before large jobs)

限流、封锁与 proxies（执行大型任务前阅读）

The engine sends real requests to Google Maps. The upstream project's only formal note is a disclaimer:
"Please use this scraper responsibly and in accordance with applicable laws and regulations. Unauthorized scraping may violate terms of service."
There is no published hard threshold for blocks, so stay conservative:
  • Google may temporarily rate-limit your IP if you scrape too fast or too much. It clears in minutes to hours and does not affect the user's Google account, but jobs fail in the meantime.
  • Block signals: jobs returning 
    failed
    , empty or unusually short results, or a sudden drop in row count against an identical earlier run. When you see them, back off (pause, lower depth) or add proxies.
  • Concurrency and blocking (upstream): "Higher concurrency … can increase blocking or failures, especially without proxies. Start with the default for a first run." Reference throughput is about 120 places/min at 
    -c 8 -depth 1
    . Locally, run one job at a time and start at 
    depth 5
    .
When to add proxies (upstream: "For larger scraping jobs, proxies help avoid rate limiting"): large jobs, many keywords, repeated or scheduled runs, or after you see block signals.
  • Set the 
    "proxies"
    array in the job body. Types: 
    socks5
    , 
    socks5h
    , 
    http
    , 
    https
    .
  • Format: 
    protocol://user:pass@host:port
    (auth optional), e.g. 
    "proxies": ["socks5://user:pass@host:port", "http://host2:port2"]
    . The engine rotates them automatically.
引擎会向 Google Maps 发送真实请求。上游项目唯一明确的正式说明是一则免责声明：
“请负责任地使用本抓取器，并遵守适用的法律法规。未经授权的抓取可能违反服务条款。”
官方没有公布明确的封锁阈值，因此应保持谨慎：
  • 如果抓取过快或过多，Google 可能会临时限制你的 IP。限制通常会在几分钟至几小时内解除，并且不会影响用户的 Google 账号，但在此期间任务会失败。
  • 封锁信号：任务返回 
    failed
    、结果为空或异常少，或者与之前完全相同的任务相比行数突然下降。发现这些信号时，应    退避    （暂停、降低 depth）或添加 proxies。
  • 并发与封锁（上游说明）：*“更高的并发度……会增加封禁或失败的可能性，尤其是在不使用 proxies 时。首次运行请使用默认值。”*参考吞吐量约为在 
    -c 8 -depth 1
    时每分钟处理 120 个地点。在本地环境中，一次只运行一个 job，并从 
    depth 5
    开始。
何时添加 proxies（上游说明：“对于大型抓取任务，proxies 有助于避免限流。”）：大型任务、多个搜索词、重复或定时任务，或者已经发现封锁信号时。
  • 在任务正文中设置 
    "proxies"
    数组。支持的类型为：
    socks5
    、
    socks5h
    、
    http
    、
    https
    。
  • 格式为 
    protocol://user:pass@host:port
    （身份验证可选），例如 
    "proxies": ["socks5://user:pass@host:port", "http://host2:port2"]
    。引擎会自动轮换它们。

Safety and guardrails

安全与防护措施

  • Warn, don't block. When a request is large (high 
    depth
    , many keywords, repeated runs, a grid, or 
    email:true
    ), proceed with it, but first print ONE short warning about temporary IP rate limits and suggest proxies. Do not gate, force-stop or demand confirmation because a normal scrape is big. Refuse outright only for clearly abusive or illegal use (surveilling individuals, spam, harassment).
  • Never expose the API beyond 
    localhost
    without an auth proxy. Never print an API key if one exists.
  • Personal data: scraped emails and phones are personal data. If the user will store or contact them, remind them to follow GDPR, CCPA and CAN-SPAM (lawful basis, opt-outs, suppression). Do not help with spam or harassment.
  • Google's Terms of Service: scraping Google Maps is against them. Keep volume modest, treat output as leads to verify, and do not resell raw Google data. Refuse uses aimed at surveilling individuals.
  • Output hygiene: do not dump large CSVs into chat. Save the file to disk, summarize the counts and show a few sample rows. Dedupe by 
    place_id
    /
    cid
    before storing.
  • Disk: results pile up in the Docker volume. Offer to 
    DELETE
    old jobs from time to time.
  • **发出警告，不要阻止执行。**当请求规模较大时（高 
    depth
    、多个搜索词、重复运行、网格任务或 
    email:true
    ），仍应继续执行，但要先输出一条关于临时 IP 限流的简短警告，并建议使用 proxies。不要因为正常抓取任务规模较大而设置门槛、强制停止或要求确认。只有在用途明显具有滥用性或非法性时（监控个人、垃圾信息、骚扰），才应直接拒绝。
  • **绝不能在未配置身份验证 proxy 的情况下将 API 暴露到 localhost 之外。**如果存在 API key，绝不能将其打印出来。
  • **个人数据：**抓取到的邮箱和电话属于个人数据。如果用户将存储或联系这些数据，应提醒其遵守 GDPR、CCPA 和 CAN-SPAM（合法依据、退出机制、退订名单）。不要协助发送垃圾信息或进行骚扰。
  • **Google 服务条款：**抓取 Google Maps 违反其服务条款。应控制抓取量，将输出视为需要核验的潜客数据，并且不要转售原始 Google 数据。拒绝用于监控个人的请求。
  • **输出规范：**不要在聊天中直接输出大型 CSV。将文件保存到磁盘，汇总数量并展示少量样本行。存储前应按 
    place_id
    /
    cid
    去重。
  • **磁盘：**结果会持续堆积在 Docker volume 中。应定期向用户提供使用 
    DELETE
    清理旧任务的选项。

Troubleshooting

故障排查

  • 422 missing max time
    : add 
    max_time
    (seconds). 
    422 missing geo coordinates
    : add string 
    lat
    /
    lon
    .
  • Stuck on 
    working
    : lower 
    depth
    , raise 
    max_time
    , or the IP is throttled (add proxies or wait).
  • Empty CSV: the keyword is too narrow or the coordinates are wrong. Widen 
    radius
    or fix the coordinates.
  • Connection refused: the container is down. Run 
    docker compose up -d
    .
  • 422 missing max time
    ：添加 
    max_time
    （秒）。
    422 missing geo coordinates
    ：添加字符串形式的 
    lat
    /
    lon
    。
  • 一直停留在 
    working
    ：降低 
    depth
    、提高 
    max_time
    ，或者 IP 已被限流（添加 proxies 或等待）。
  • CSV 为空：搜索词过于狭窄或坐标错误。请扩大 
    radius
    或修正坐标。
  • Connection refused：容器未运行。执行 
    docker compose up -d
    。