To scrape SERP data, collect search results for a chosen keyword and save the fields you need, such as position, page title and URL. SERP simply means search engine results page. For useful comparisons, keep the search location, language, device and date with the results too.
That data can help an SEO team track rankings, find competing pages or see whether a keyword returns guides, products or local businesses. Each job needs a slightly different table. Decide what you want to compare before collecting a large keyword list.
I built the Python example around SerpApi, a paid third-party service that fetches search results for you. It exports organic results to CSV, supports a list of queries and keeps failed jobs visible. You use your own account and allowance for live requests.
You can also export a saved response without making another request. Start with one keyword and location, inspect the rows, then add more searches once the output is useful for your report.
Which SERP Data Collection Method Should You Use?
Use a hosted service when the output matters more than the retrieval stack
I read SerpApi's Google documentation to check the request settings and response fields used in this example. I also reviewed DataForSEO as another option. I would compare the engines, result types and request modes your project needs, then inspect an output sample before deciding on price.
For a one-off export, I would also consider a hosted interface such as Apify's Google Search Results Scraper, especially if you prefer tool settings to maintaining code. Check inputs, output fields and usage terms before assuming two tools return equivalent data.
| Method | Useful starting point | Work you still own |
|---|---|---|
| Structured API | Repeated exports and reports | Query design, field selection and interpretation |
| Hosted scraper interface | A small job without building a client | Inputs, result checks and download settings |
| Scraper you run yourself | A justified need for retrieval control | Client, connections, extraction and maintenance |
| Search Console | Search activity for a property you can access | Property reporting, filters and interpretation |
I keep the last row separate because it is a different dataset. Search Console is useful for actual property impressions and clicks. It does not replace every competitor or arbitrary-query snapshot, and a scraped result is not evidence that anyone clicked that result.
Build your own collector for a defined reason
Running your own scraper can make sense when you need page behavior or fields a hosted service does not provide. Name that need before building it. You will also need to keep the scraper working when pages change.
For a small workload, I would include my own time in the cost comparison. Running browsers, repairing extraction and investigating failed pages can outweigh a modest request fee. For a larger workload, compare the complete cost of usable snapshots rather than an advertised price per request.
What Should You Collect From a Search Result Page?
Keep organic results separate from other result types
For a basic ranking table, collect query, position, title, destination URL and snippet. Keep search context alongside them: location, language, device and source time where available. Those details let you compare later observations without guessing what the earlier check represented.
Ads, local packs and shopping results need separate handling. Organic position two is a different measurement from local-pack position two. A combined spreadsheet needs a result-type column or separate tables rather than pretending everything is one ordered list.
For a competitor-content study, I would group result URLs by page type: product, category, guide, forum or tool. That can reveal what kind of page the query currently returns. It is an editorial clue, not an instruction to copy the titles or assume every result satisfies the same need.
Choose how deep the observation needs to go
Define the intended number of results collected before comparing runs. If one snapshot contains ten results and another contains twenty, an absent URL does not mean the same thing in both. Report "not found within the collected results" when that is the evidence.
Location in the query is also different from location in the request. "Dentist Boston" from a broad US setting and "dentist" requested from Boston are two distinct observations. Both can be useful, but they should not be collapsed into a single keyword history.
How Can You Scrape Search Engine Results With Python?
Set the key and the search settings and time
Install Requests with python -m pip install requests. Set SERPAPI_KEY to your own SerpApi token. The example uses a third-party Google-results service, with charges and allowance defined by your account; it is not a Google-owned API.
I kept country, language, device and location as separate command options, and saved that information with every output row. A request for an Austin desktop result and a Boston mobile result are different jobs even when the query text matches.
Save the collector and context-aware CSV export
Save this as serp_collect.py. It preserves the returned search context, source timestamp and source ID, with one row per organic result. A retrieval failure is marked failed. An empty organic list is recorded separately so it cannot masquerade as a normal ranking table.
"""Export organic SERP rows and the context needed for comparisons."""
import argparse, csv, json, os
from datetime import datetime, timezone
from pathlib import Path
import requests
FIELDS = ["status", "query", "location_requested", "location_used", "country", "language", "device",
"retrieved_at", "source_created_at", "source_id", "position", "title", "url", "snippet", "error"]
def export_rows(bundle):
payload = bundle.get("response", bundle)
params = payload.get("search_parameters", {})
metadata = payload.get("search_metadata", {})
base = {"query": params.get("q"), "location_requested": params.get("location_requested", params.get("location")),
"location_used": params.get("location_used"),
"country": params.get("gl"), "language": params.get("hl"), "device": params.get("device"),
"retrieved_at": bundle.get("_retrieved_at"), "source_created_at": metadata.get("created_at"),
"source_id": metadata.get("id")}
results = payload.get("organic_results")
if payload.get("error") or not isinstance(results, list):
return [{**base, "status": "failed", "error": "No valid organic result list; inspect raw JSON"}]
if not results:
return [{**base, "status": "no_organic_results"}]
rows = []
for result in results:
if not isinstance(result, dict):
raise ValueError("Unexpected organic result shape")
rows.append({**base, "status": "ok", "position": result.get("position"), "title": result.get("title"),
"url": result.get("link"), "snippet": result.get("snippet")})
return rows
def retrieve(query, args):
key = os.getenv("SERPAPI_KEY")
params = {"engine": "google", "q": query, "location": args.location, "gl": args.country,
"hl": args.language, "device": args.device, "api_key": key}
try:
response = requests.get("https://serpapi.com/search.json", params=params, timeout=90)
except requests.RequestException as error:
raise ValueError(type(error).__name__) from None
if not response.ok:
raise ValueError("API HTTP " + str(response.status_code))
try:
payload = response.json()
except ValueError:
raise ValueError("Non-JSON API response") from None
bundle = {"_retrieved_at": datetime.now(timezone.utc).isoformat(), "response": payload}
safe = json.dumps(bundle, ensure_ascii=False, indent=2).replace(key, "[redacted]")
folder = args.output.with_suffix(".raw")
folder.mkdir(parents=True, exist_ok=True)
(folder / (str(args.job_index) + ".json")).write_text(safe, encoding="utf-8")
return json.loads(safe)
def main():
parser = argparse.ArgumentParser()
source = parser.add_mutually_exclusive_group(required=True)
source.add_argument("--query")
source.add_argument("--queries", type=Path, help="UTF-8 file, one query per line")
source.add_argument("--json", type=Path)
parser.add_argument("--location")
parser.add_argument("--country", default="us")
parser.add_argument("--language", default="en")
parser.add_argument("--device", choices=["desktop", "mobile", "tablet"], default="desktop")
parser.add_argument("--output", type=Path, default=Path("serp.csv"))
args = parser.parse_args()
if args.json:
rows = export_rows(json.loads(args.json.read_text(encoding="utf-8")))
else:
if not os.getenv("SERPAPI_KEY") or not args.location:
parser.error("Set SERPAPI_KEY and supply --location")
queries = args.queries.read_text(encoding="utf-8-sig").splitlines() if args.queries else [args.query]
queries = list(dict.fromkeys(q.strip() for q in queries if q.strip()))
if not queries:
parser.error("The query file is empty")
rows = []
for args.job_index, query in enumerate(queries, 1):
try:
rows.extend(export_rows(retrieve(query, args)))
except ValueError as error:
rows.append({"status": "failed", "query": query, "location_requested": args.location,
"country": args.country, "language": args.language, "device": args.device,
"error": str(error)})
with args.output.open("w", encoding="utf-8-sig", newline="") as handle:
writer = csv.DictWriter(handle, fieldnames=FIELDS)
writer.writeheader()
writer.writerows(rows)
print(str(len(rows)) + " rows exported to " + str(args.output))
if any(row["status"] == "failed" for row in rows):
raise SystemExit(1)
if __name__ == "__main__":
main()
Run one query with explicit conditions:
python serp_collect.py --query "coffee shops" --location "Austin, Texas, United States" --country us --language en --device desktop --output austin-run1.csv
The live path creates a CSV and an austin-run1.raw directory of responses. Use a new output name for another collection period. This keeps earlier observations available for checking rather than overwriting the history you intend to compare.
Check an actual output row, not just the file's existence
Supply a real saved filename from that directory to export without another request:
python serp_collect.py --json austin-run1.raw/1.json --output austin-reexport.csv
I ran the exporter against the provider's complete public documentation example and got nine organic rows. The sample query was coffee, its source time was December 7, 2021, and its first organic result was Coffee - Wikipedia. I use that historical sample to check the exporter, not to report current rankings.
During that check, I found a field-mapping detail: the response used location_requested and location_used rather than a simple location field. The exporter handles those names and preserves the distinction. A program can successfully write a CSV while silently losing exactly the context needed to compare it later.
Why Can Google SERP Scraping Differ From Your Browser?
Compare request and returned context before rank
Check query spelling, country, location, language, device and source time. Then compare the same result type and depth. Organic position two is different from local-pack position two; a nearby business in a local widget should not be joined to an organic URL history as the same measurement.
| Column | Example from the provider's documented response | What it tells you |
|---|---|---|
query |
coffee |
The observed query, not a later spreadsheet label |
location_requested |
Austin, Texas, United States | The requested location input |
location_used |
Austin,Texas,United States | The returned location label |
country, language, device |
us, en, desktop |
Other conditions of that snapshot |
source_created_at |
December 7, 2021 | The sample's historical source time |
retrieved_at |
Blank for an unwrapped documentation sample | The program has not invented a live retrieval date |
The two location fields are useful evidence, but neither proves the complete state of a person's physical device. I checked Google's explanation of its location information, which draws on several sources. Network location, requested settings and browser state need not match across collection methods.
Diagnose a mismatch with the original response
When I read this local-ranking discussion, I noticed the distinction between a city inside the query and a generic query with city targeting. I would keep those separate when comparing tools. "Dentist Boston" and "dentist" from a Boston setting are not the same job.
If a URL looks absent, inspect the saved response. Was it returned under another result type, outside the collected depth, or omitted by the CSV mapping? A problem in export is different from a changed ranking, and both are different from a failed retrieval.
For a client report, state the search settings and time. Use "not found in this collected result set" when that is what you know. Do not convert one snapshot into a universal rank for every searcher, or assume every discrepancy requires another proxy.
PuraRoute for Search Results in Different Markets
For SEO teams running their own search-result collectors, PuraRoute is a proxy service to consider. It offers dynamic residential proxies with location choices, plus static residential proxies and static native proxies for longer use with an assigned IP. These options can support research across the countries and cities that matter to your website or clients.
Focus on the markets you want to compare
PuraRoute's available country, state and city choices are useful when a ranking project covers several locations. For example, a business may want to compare the pages appearing for its keywords in different US cities. Keep the keyword, language and device choice the same when that is the comparison you intend. The value is a more relevant regional check, with the location and date recorded beside the results.
Choose a plan around regular SEO checks
Dynamic residential proxies can suit separate recurring searches, while a static service can suit a tool that needs an assigned IP for longer. Begin with a short keyword list and inspect the pages and URLs returned. Check the running cost before adding more cities or keywords. Your own ranking project is the best test of which service and plan fit.
How Do You Collect More Queries Without Mixing the Results?
Run a query file for one explicit context
Create queries.txt with one query per line. The program removes exact duplicates and runs sequentially. Keep one country/location/device combination per command so the output is easy to inspect:
python serp_collect.py --queries queries.txt --location "Austin, Texas, United States" --country us --language en --device desktop --output austin-desktop-run1.csv
python serp_collect.py --queries queries.txt --location "Boston, Massachusetts, United States" --country us --language en --device mobile --output boston-mobile-run1.csv
With ten queries, those two commands represent twenty query-and-context jobs, not ten repeated observations of the same thing. Each job uses your own provider allowance. The example does not fetch additional result pages; check the provider's current pagination method if your project needs greater depth.
Keep failed rows visible when combining outputs. A failed Austin job should not inherit a successful Boston result or yesterday's saved response. The distinction matters more as the number of queries and contexts grows.
Match snapshot identity before calculating movement
For rank tracking, join query, market, device and result type first, then compare the URL's position within the same collected depth. Preserve source time and source ID so a re-exported response can be recognized as the same underlying snapshot.
For competitor research, group result URLs into page types: product, category, guide, forum or tool. If a query now returns several product pages where guides previously appeared, read the current pages before deciding the keyword's intent changed. The spreadsheet provides a useful prompt, not a proven cause.
Start with a sample small enough to inspect. A correct query-and-context definition makes later scheduling useful; a mistaken definition replicated across thousands of rows only produces a larger misleading report.
How Do You Turn SERP Data Into a Useful SEO Report?
Compare like-for-like searches before reporting a rank change
For a regular ranking report, group results by keyword, location, device and result type. Compare the same website URL within those groups. If one run is mobile in Boston and the next is desktop in Austin, a different position is not a clean measure of movement in one market.
| SEO task | What to look at in the export | Useful next action |
|---|---|---|
| Track your pages | The same URL and search conditions over time | Investigate a repeated change rather than a single surprising check |
| Find content competitors | The pages appearing for your target queries | Read the pages that actually compete for that intent |
| Plan content | Whether results are guides, product pages, categories or forums | Choose a page type that can answer the query well |
| Compare locations | The same query and device across the chosen markets | Check which differences matter to the business |
For content planning, do more than copy headings from the top pages. A query returning several comparison tools calls for a different answer from one returning step-by-step tutorials. Use the export to find those patterns, then read the pages before deciding what to write.
Keep missing results and failed jobs separate
I would not report a ranking loss from a failed request. I kept that job marked failed in the script. An explicit empty organic list gets its own status. A normal result set in which your URL does not appear is a third situation.
Open the saved response when a result looks wrong. Check whether the URL appeared under another result type or outside the depth you collected. If the value exists in the response but not the CSV, inspect the field mapping. If the request failed, check the provider's message and allowance.
I keep the source date with the result when rebuilding a report. Re-exporting an old file can fix the layout, but I would not label it a new ranking check. The spreadsheet may have changed while the underlying search evidence stayed the same.
Frequently Asked Questions About Scraping SERP Data
Does this code collect People Also Ask questions?
The example exports the organic-result list only. If you need People Also Ask, ads or local results, map the provider's documented fields into a separate table or result type. Their structure and meaning differ from organic positions, so silently mixing them into the same ordered list would be misleading.
Can I open the CSV in Excel?
Yes. The example writes a UTF-8 CSV with a byte-order marker to help spreadsheet applications recognize its encoding. Import URLs and identifiers as text where appropriate. Keep the original JSON if you need to recover nested information that the flat organic export does not include.
Will exporting saved JSON use another API credit?
The JSON mode reads the local file and performs no retrieval request. It is useful for checking an export or changing its format. It still represents the original source observation; the new CSV creation time should not be presented as a fresh search result or a new ranking measurement.
Does a result URL always point straight to the destination?
That depends on the source and returned field. Inspect the actual response rather than assuming every page link or redirect is the final destination. Use the provider's documented destination field for the export, and distinguish any unresolved redirect if your later analysis depends on the final domain.
Can I use the same code for another search engine?
The example is written for the documented Google response. Other engines can use different parameters or output fields. Check the provider's engine-specific documentation, then adapt and test the mapping. Changing only a label in the CSV does not turn a Google response into another engine's dataset.
