Lead Scraping — Market Rotation & Budget Control

Overview

BlueClerk automatically finds and adds potential customers (leads) for you across 25 major US markets. Our system scrapes Google Places for contractors Monday through Friday at 6am UTC using a rotating market schedule that keeps monthly API costs under the $0 Google Places Enterprise free tier limit (1,000 calls/month). Each market is covered approximately 3 times per month via rotation.

How It Works

Market Rotation Strategy

• 25 US markets total - 16 Texas cities (Austin, Dallas, Houston, San Antonio, Fort Worth, Plano, Frisco, McKinney, Round Rock, Georgetown, Arlington, El Paso, Lubbock, Amarillo, Cedar Park, New Braunfels) + 9 top contractor-dense metros (Los Angeles, Phoenix, Atlanta, Miami, Tampa, Orlando, Denver, Las Vegas, Charlotte)
• 4 markets per run - Rotated by day using dayOfYear mod (total markets / markets per run)
• Daily Monday-Friday - Cron runs at 6am UTC (Mon-Fri only)
• 10 trades per run - Every trade queried every day (no rotation)
• 1 page per query - Fetches first page only (up to 20 results)
• 40 API calls per run - 4 markets × 10 trades × 1 page
• ~22 runs/month - Approximately 880 API calls/month (under 1,000 free cap)
• ~3x coverage per market - Each market appears in ~3 runs per month

Budget Protection

The system enforces a hard ceiling of 50 API calls per run to prevent overspending even if configuration drifts. This ensures a single run can never exceed the safe budget threshold.

Trade Coverage

All 10 trades are queried in every run:

• Plumbing
• Electrical
• HVAC
• Roofing
• Landscaping
• Painting
• Flooring
• Concrete
• Fencing
• General Contractor

Automatic Deduplication

Lead.placeId has a @unique constraint, so duplicate businesses across multiple market/trade queries are automatically rejected at insert time - ensuring clean data without manual dedup.

Enrichment

Hourly Email Enrichment

A separate cron runs hourly to find contact emails for leads:

• Improved website scraper - Extracts emails from contractor websites
• Apollo.io People Search - Finds contact info when website scraping fails
• Continuous processing - Enrichment runs independently of lead scraping

Cost History

Previous configuration: All 25 markets × 10 trades × 2 pages every run = ~5,500-7,150 calls/month at Enterprise rates = $158-$215/month after the universal $200/mo Maps credit was removed by Google on 2025-03-01.

Current configuration: 4 markets × 10 trades × 1 page = 40 calls/run × 22 runs/month = 880 calls/month < 1,000 free cap = ~$0/month.