Robust API Collection: Pagination, Rate Limits, Failure Recovery

Collecting thousands of grocery store locations from Google Places API means handling: rate limits that throttle rapid requests, pagination that returns results in batches of 20, transient API errors, and the risk of losing hours of progress to an unexpected failure.

A straightforward script fails on multiple counts.

---

What Goes Wrong Without Robustness

A naive approach (loop through counties, query the API, append results to a list, save at the end) fails in predictable ways:

Rate limit exceeded: The script sends requests too quickly, receives a quota response, and stops without a recovery path.

Network timeout: A transient network issue interrupts a request. Without incremental saving, everything collected since the last checkpoint is lost.

Process interruption: A long-running collection stops. Without a durable cursor, restarting repeats completed queries and wastes API budget.

These aren't edge cases. They're the default outcome for any multi-hour API collection.

---

Rate Limiting

The first protection: never exceed API rate limits regardless of how fast the code runs.

@rate_limit(calls_per_second=5)
def search_places(query, location, radius):
    return gmaps.places(query=query, location=location, radius=radius)

A decorator pattern wraps the API call. If calls come too fast, the decorator sleeps until the rate limit window resets. The rest of the code doesn't need to know about rate limits; it just calls search_places() and the decorator handles pacing.

---

Checkpointing

The second protection: track progress so restarts don't mean starting over.

class CheckpointManager:
    def __init__(self, checkpoint_file):
        self.state = self._load()

    def mark_county_complete(self, county):
        completed = self.state.setdefault('completed_counties', [])
        if county not in completed:
            completed.append(county)
        self.save()

    def set_progress(self, county, page_token):
        self.state['current_county'] = county
        self.state['page_token'] = page_token
        self.save()

The checkpoint file records which counties are complete and where pagination left off. When the script restarts after a crash, it reads the checkpoint and resumes from the last saved position.

A restart test should deliberately interrupt pagination, reload the checkpoint, and verify that the resumed run neither skips nor duplicates a page. This page does not publish such a test artifact.

---

Incremental Saving

The third protection: save results continuously so crashes lose minimal work.

import json

def save_results_incremental(records, county, batch_num):
    filename = f'{county}_batch_{batch_num:03d}.json'
    with open(filename, 'w') as f:
        json.dump(records, f, sort_keys=True)

The deterministic county-and-batch filename makes a retry replace the same batch rather than create an additional batch. Saving each page promptly limits the unsaved interval, but an atomic temporary-file replacement and a post-write checksum are still needed to protect against interruption during the write itself.

---

Error Recovery

The fourth protection: retry transient errors instead of crashing.

from requests.exceptions import ConnectionError, Timeout
from tenacity import retry, retry_if_exception, stop_after_attempt, wait_exponential

RETRYABLE_STATUS_CODES = {429, 500, 502, 503, 504}

def is_transient_api_error(exc):
    response = getattr(exc, "response", None)
    status = getattr(response, "status_code", None)
    return isinstance(exc, (ConnectionError, Timeout)) \
        or status in RETRYABLE_STATUS_CODES

@retry(
    retry=retry_if_exception(is_transient_api_error),
    stop=stop_after_attempt(4),  # first call plus at most three retries
    wait=wait_exponential(multiplier=4, min=4, max=60),
    reraise=True,
)
def fetch_place_details(place_id):
    return gmaps.place(place_id, fields=[...])

This configuration makes at most four attempts: the first call and up to three retries, with waits of 4, 8, and 16 seconds before those retries. The predicate retries connection failures, timeouts, and HTTP 429 or selected 5xx responses; adapt it to the client library's documented exception types. Authentication and invalid-request errors should fail immediately. If the fourth attempt fails, the exception is re-raised and should be recorded as a permanent failure.

---

Withdrawn Execution Record

An earlier version listed county counts, completion times, a total record count, retry totals, checkpoint totals, and cost. The county counts summed to 5,903 rather than the stated total, and the completion chronology conflicted with the timeout narrative. No public run ledger resolves either inconsistency, so the numeric record has been removed.

A publishable execution record would include machine-readable per-request or per-page events, unique record identifiers, checkpoint transitions, retry outcomes, start and completion timestamps, a billing export, and assertions showing that county subtotals equal the final deduplicated total.

---

Division of Labor

Human responsibility: Which counties to query, what attributes to collect, acceptable cost bounds.

Agent responsibility: Rate limiting, checkpointing, incremental saving, retry logic, progress logging.

These are standard robustness patterns. An agent can draft the implementation, while the researcher defines quotas, checkpoint state, validation rules, retry limits, and the evidence a run must save. Any time-savings claim requires a measured comparison.

---

No Verified Cost Record

No public billing export or saved run output verifies a collection cost or retry allocation. The reusable lesson is to join request counts to the provider's contemporaneous price schedule and preserve the billing record with the run ledger.

Public Materials

Article only. The public analysis repository does not currently include a matching script, run log, saved output, or billing record for this collection.