Google Maps Scraper

Turn a business type and a location into clean lead rows by driving the local engine's job API.

Mental model

The engine (gosom/google-maps-scraper) runs as a local Docker container with a REST API at

http://localhost:8080

. It has no auth, so it stays bound to localhost. A scrape is an async job: create it, poll until it is done, then download a CSV. One job can run many keywords. Each result has up to 34 raw fields.

Step 0: Check the engine is running

bash

curl -s http://localhost:8080/api/v1/jobs >/dev/null 2>&1 && echo UP || echo DOWN

DOWN

: run

docker compose up -d

from the kit root, wait ~10s, and retry. If Docker is not installed, point the user to

SETUP.md

Step 1: Create a job (

POST /api/v1/jobs

)

Required fields. The API returns
422
without them:

```
keywords
```
: array of search strings. Bake the location into each term:
```
"plumbers in Denver CO"
```
.
```
lat
```
,
```
lon
```
: strings, the city's coordinates:
```
"39.7392"
```
,
```
"-104.9903"
```
.
```
max_time
```
: integer seconds, the job's wall-clock limit, e.g.
```
300
```
. The API stores it as nanoseconds internally, but you always send seconds.

Recommended fields:

```
depth
```
(default 10): how far to scroll, so roughly how many listings per keyword. Start at
```
5
```
.
```
lang
```
```
"en"
```
,
```
zoom
```
```
15
```
(city level),
```
radius
```
```
10000
```
(meters),
```
fast_mode
```
```
false
```
.
email

true
: on by default in the kit. Emails are the most valuable lead field. The engine visits each business website to find them, which is a little slower. Set
```
false
```
only for a deliberately fast run without emails.

bash

curl -s -X POST http://localhost:8080/api/v1/jobs \
  -H "Content-Type: application/json" \
  -d '{"name":"job","keywords":["coffee shops in Austin TX"],"lang":"en","zoom":15,
       "lat":"30.2672","lon":"-97.7431","fast_mode":false,"radius":10000,
       "depth":5,"email":true,"max_time":300}'
# → {"id":"<uuid>"}   (HTTP 201; note: lowercase "id")

Handle two things on every scrape:
Emails: on by default (
email:true
). Turn them off only if the user wants a fast run.
Socials: ask first. Before creating the job, ask the user once whether they also want Instagram, Facebook and LinkedIn profiles (see "Social profiles" below). Never skip the question silently.

Step 2: Poll until done (

GET /api/v1/jobs/{id}

)

The response field is

"Status"

(capital S). It moves from

working

ok

(success) or

failed

Harness rule: the Bash tool blocks foreground
sleep
. Run the poll loop as a background Bash command (
run_in_background: true
) and read its output file when notified. Never poll with a foreground
sleep
.

Background poll snippet. It parses the value safely: match up to the closing quote, do not anchor on

bash

ID="<uuid>"
for i in $(seq 1 40); do
  S=$(curl -s "http://localhost:8080/api/v1/jobs/$ID" | grep -oE '"Status":"[^"]*"' | head -1 | cut -d'"' -f4)
  echo "status=$S"
  [ "$S" = ok ] && { echo DONE; break; }
  [ "$S" = failed ] && { echo FAILED; break; }
  sleep 15
done

Step 3: Download and parse (

GET /api/v1/jobs/{id}/download

)

bash

curl -s "http://localhost:8080/api/v1/jobs/$ID/download" -o results.csv

CSV columns (34):

input_id, link, title, category, address, open_hours, popular_times, website, phone, plus_code, review_count, review_rating, reviews_per_rating, latitude, longitude, cid, status, descriptions, reviews_link, thumbnail, timezone, price_range, data_id, place_id, images, reservations, order_online, menu, owner, complete_address, about, user_reviews, user_reviews_extended, emails

Return only the 8 lead fields (default)

The raw CSV has 34 columns and most are noise for outreach. By default, return only the 8 lead fields, the data used to contact and qualify a lead, and drop everything else:

title

(name),

phone

emails

website

category

address

review_rating

review_count

Drop by default. Show these only if the user explicitly asks:

latitude

longitude

(no use for outreach),

link

plus_code

cid

data_id

place_id

open_hours

popular_times

reviews_per_rating

reviews_link

thumbnail

images

timezone

price_range

status

input_id

complete_address

reservations

order_online

menu

owner

about

descriptions

user_reviews

user_reviews_extended

scripts/scrape.py

already returns exactly the 8 lead fields. Use

--full

to keep all columns, or

--fields "a,b,c"

to choose your own. It saves a CSV file by default (

results-<id>.csv

); pass

--json

for JSON. If you call the API directly, strip to the lead fields yourself before presenting. Never dump the full 34-column row at the user.

Social profiles: always ask the user (Instagram, Facebook, LinkedIn)

Google Maps has no social links, so socials are an enrichment step: visit each business

website

and regex out its Instagram, Facebook and LinkedIn URLs. Before scraping, ask the user once whether they want socials too, unless they already said. If yes, use the script:

python3 scripts/scrape.py … --socials

Token cost. Tell the user this when they ask for socials: the extraction itself costs 0 LLM tokens, because it is plain HTTP and regex in the script. It adds ~40–50 tokens per business to your context only if you load the rows into chat, e.g. ~+2k tokens for 50 leads. That is negligible if you keep the file on disk and show a sample. Save the file and show a few rows.
Do not fetch each website yourself with WebFetch to find socials. That reads every page into your context and costs thousands of tokens. The script does it for free, so always prefer
```
--socials
```
.
Socials are opt-in and slower (one HTTP fetch per business), and coverage is partial (~40–70%). Only businesses that link their socials on their site have them, and no website means no socials. Say so if the user expects 100%.

Shortcuts. Prefer these for common cases:

One keyword:
```
scripts/scrape.sh "<keyword>" <lat> <lon> [depth]
```
(bash) creates, polls and downloads in one go.
Auto-geocode (no coordinates):
```
python3 scripts/scrape.py "<keyword>" --city "<City, ST>" [--depth N]
```
resolves lat/lon via OpenStreetMap Nominatim. You can also geocode the city yourself and pass coordinates. Emails come back by default (
```
--no-email
```
skips them). Add
```
--socials
```
if the user asked for socials.
Batch (many keywords, one job):
```
python3 scripts/scrape.py --keywords-file <file> --city "<City, ST>"
```
. The API takes a
```
keywords
```
array, so put all terms in a single job rather than firing many jobs. Use the manual curl flow only for custom job bodies, e.g. to set
```
proxies
```
.

Whole-city runs: when the user says all, every or entire city, or a run returns about 120 rows (Google's per-search cap), use

python3 scripts/scrape.py "<keyword>" --city "<City, ST>" --grid 3

. It splits the city into 3 x 3 tiles (9 jobs) and dedupes on

place_id

. Tiles run one at a time, so a grid is slower and more likely to trip rate limits: warn once, don't block.

Deep email pass: on by default whenever emails are on. Rows with a website but no email get their contact and about pages checked, and the script prints the found rate.

--no-deep-emails

skips it.

Other endpoints

GET /api/v1/jobs

lists jobs.

DELETE /api/v1/jobs/{id}

deletes a job and frees disk.

Browser UI and OpenAPI docs:

http://localhost:8080

and

http://localhost:8080/api/docs

Best practices (from the upstream docs)

Depth: higher
```
depth
```
returns more results but is slower and more likely to be blocked. Start at 5 and raise it as needed.
One job at a time locally. Many concurrent jobs without proxies lead to throttling by Google.
Email extraction (
email:true
) visits each business website, so it is slower. It stays on by default because emails are the key lead field. Pass
```
--no-email
```
(script) or
```
email:false
```
(raw API) for a fast run.
fast_mode:true
returns reduced data, up to ~21 results per query, faster. Use it for quick lookups.
Proxies: for large or repeated jobs, set
```
"proxies"
```
(an array of
```
socks5://
```
,
```
http://
```
or
```
https://
```
URLs, auth supported). The engine rotates them. This is the main defense against rate limits.
zoom
and
radius
set the search area around
```
lat
```
/
```
lon
```
. Widen
```
radius
```
if results are too few.
Extended reviews are available but heavy. Enable them only if the user asks for review text.

Rate limits, blocks and proxies (read before large jobs)

The engine sends real requests to Google Maps. The upstream project's only formal note is a disclaimer:

"Please use this scraper responsibly and in accordance with applicable laws and regulations. Unauthorized scraping may violate terms of service."

There is no published hard threshold for blocks, so stay conservative:

Google may temporarily rate-limit your IP if you scrape too fast or too much. It clears in minutes to hours and does not affect the user's Google account, but jobs fail in the meantime.
Block signals: jobs returning
```
failed
```
, empty or unusually short results, or a sudden drop in row count against an identical earlier run. When you see them, back off (pause, lower depth) or add proxies.
Concurrency and blocking (upstream): "Higher concurrency … can increase blocking or failures, especially without proxies. Start with the default for a first run." Reference throughput is about 120 places/min at
```
-c 8 -depth 1
```
. Locally, run one job at a time and start at
```
depth 5
```
.

When to add proxies (upstream: "For larger scraping jobs, proxies help avoid rate limiting"): large jobs, many keywords, repeated or scheduled runs, or after you see block signals.

Set the
```
"proxies"
```
array in the job body. Types:
```
socks5
```
,
```
socks5h
```
,
```
http
```
,
```
https
```
.

Format:

protocol://user:pass@host:port

(auth optional), e.g.

"proxies": ["socks5://user:pass@host:port", "http://host2:port2"]

. The engine rotates them automatically.

Safety and guardrails

Warn, don't block. When a request is large (high
```
depth
```
, many keywords, repeated runs, a grid, or
```
email:true
```
), proceed with it, but first print ONE short warning about temporary IP rate limits and suggest proxies. Do not gate, force-stop or demand confirmation because a normal scrape is big. Refuse outright only for clearly abusive or illegal use (surveilling individuals, spam, harassment).
Never expose the API beyond
```
localhost
```
without an auth proxy. Never print an API key if one exists.
Personal data: scraped emails and phones are personal data. If the user will store or contact them, remind them to follow GDPR, CCPA and CAN-SPAM (lawful basis, opt-outs, suppression). Do not help with spam or harassment.
Google's Terms of Service: scraping Google Maps is against them. Keep volume modest, treat output as leads to verify, and do not resell raw Google data. Refuse uses aimed at surveilling individuals.
Output hygiene: do not dump large CSVs into chat. Save the file to disk, summarize the counts and show a few sample rows. Dedupe by
```
place_id
```
/
```
cid
```
before storing.
Disk: results pile up in the Docker volume. Offer to
```
DELETE
```
old jobs from time to time.

Troubleshooting

422 missing max time

: add

max_time

(seconds).

422 missing geo coordinates

: add string

lat

lon

Stuck on
```
working
```
: lower
```
depth
```
, raise
```
max_time
```
, or the IP is throttled (add proxies or wait).
Empty CSV: the keyword is too narrow or the coordinates are wrong. Widen
```
radius
```
or fix the coordinates.
Connection refused: the container is down. Run
```
docker compose up -d
```
.

google-maps-scraper

NPX Install

Tags

SKILL.md Content

Google Maps Scraper

Mental model

Step 0: Check the engine is running

Step 1: Create a job (
`POST /api/v1/jobs`
)

Step 2: Poll until done (
`GET /api/v1/jobs/{id}`
)

Step 3: Download and parse (
`GET /api/v1/jobs/{id}/download`
)

Return only the 8 lead fields (default)

Social profiles: always ask the user (Instagram, Facebook, LinkedIn)

Other endpoints

Best practices (from the upstream docs)

Rate limits, blocks and proxies (read before large jobs)

Safety and guardrails

Troubleshooting

google-maps-scraper

NPX Install

Tags

SKILL.md Content

Google Maps Scraper

Mental model

Step 0: Check the engine is running

Step 1: Create a job (POST /api/v1/jobs)

Step 2: Poll until done (GET /api/v1/jobs/{id})

Step 3: Download and parse (GET /api/v1/jobs/{id}/download)

Return only the 8 lead fields (default)

Social profiles: always ask the user (Instagram, Facebook, LinkedIn)

Other endpoints

Best practices (from the upstream docs)

Rate limits, blocks and proxies (read before large jobs)

Safety and guardrails

Troubleshooting

Step 1: Create a job (
`POST /api/v1/jobs`
)

Step 2: Poll until done (
`GET /api/v1/jobs/{id}`
)

Step 3: Download and parse (
`GET /api/v1/jobs/{id}/download`
)