Lead Scraping — Market Rotation & Budget Control

Overview

BlueClerk automatically finds and adds potential customers (leads) for you across 25 major US markets. Our system scrapes Google Places for contractors Monday through Friday at 6am UTC using a rotating market schedule that keeps monthly API costs under the $0 Google Places Enterprise free tier limit (1,000 calls/month). Each market is covered approximately 3 times per month via rotation. New leads are enriched with contact emails using Apollo.io People Search API.

How It Works

Market Rotation Strategy

• 25 US markets total - 16 Texas cities (Austin, Dallas, Houston, San Antonio, Fort Worth, Plano, Frisco, McKinney, Round Rock, Georgetown, Arlington, El Paso, Lubbock, Amarillo, Cedar Park, New Braunfels) + 9 top contractor-dense metros (Los Angeles, Phoenix, Atlanta, Miami, Tampa, Orlando, Denver, Las Vegas, Charlotte)
• 4 markets per run - Rotated by day using dayOfYear mod (total markets / markets per run)
• Daily Monday-Friday - Cron runs at 6am UTC (Mon-Fri only)
• 10 trades per run - Every trade queried every day (no rotation)
• 1 page per query - Fetches first page only (up to 20 results)
• 40 API calls per run - 4 markets × 10 trades × 1 page
• ~880 calls/month - 22 business days × 40 calls = under 1,000 free cap
• Budget cushion - Stays under $200/mo Google Places free tier with room for retries

What Gets Scraped

Every run covers all 10 trades in the rotated markets:

• HVAC - Heating and cooling contractors
• Plumbing - Residential and commercial plumbers
• Electrical - Licensed electricians
• Roofing - Roofing contractors
• General Contracting - GCs and remodelers
• Painting - Interior and exterior painters
• Flooring - Flooring installers
• Landscaping - Landscape contractors
• Concrete - Concrete and masonry
• Insulation - Insulation contractors

Daily Schedule Example

• Monday (Day 1): Austin, Dallas, Los Angeles, Phoenix
• Tuesday (Day 2): Houston, San Antonio, Atlanta, Miami
• Wednesday (Day 3): Fort Worth, Plano, Tampa, Orlando
• Thursday (Day 4): Frisco, McKinney, Denver, Las Vegas
• Friday (Day 5): Round Rock, Georgetown, Charlotte, Arlington
• Monday (Day 8): El Paso, Lubbock, Amarillo, Cedar Park
• Tuesday (Day 9): New Braunfels, Austin, Dallas, Los Angeles (rotation loops)

Each market appears roughly 3 times per month (22 business days ÷ 7 rotation cycles).

Lead Enrichment

Apollo.io People Search

After scraping, leads without contact emails are queued for enrichment:

• Hourly enrichment runs - Cron processes queue at :00 every hour
• 150 leads per run - Increased from 50 to reduce queue backlog faster
• Apollo People Search API - Finds verified contact emails using company name and website
• Automatic deduplication - Skips leads that already have emails
• Email verification - Only adds emails that Apollo marks as verified
• Website scraping fallback - If Apollo fails, tries to scrape email from company website

Enrichment Priority

Leads are enriched in order of creation (oldest first), ensuring recently scraped leads get emails within a few days.

What You Get

Lead Data

Each scraped lead includes:

• Company name - Business name from Google Places
• Address - Full street address, city, state
• Phone number - Primary business phone
• Website - Company website URL (when available)
• Trade - Primary trade category
• Source - Always "GOOGLE" for scraped leads
• Status - Starts as "new", changes to "contacted" after enrichment attempt

Lead Status

• New - Recently scraped, not yet contacted
• Contacted - Email found and enrichment complete
• Enriched - Has contact email (via Apollo or scraping)

Budget Impact

Cost Breakdown

• Google Places API - Free tier: 1,000 calls/month ($0)
  • Our usage: ~880 calls/month (40 per day × 22 business days)
  • Headroom: 120 calls for retries or manual scrapes
• Apollo.io People Search - $79/month for 1,000 credits
  • Our usage: ~3,300 enrichments/month (150/hour × 22 days)
  • Cost per lead email: ~$0.024

Total Monthly Cost

• Google Places: $0 (under free tier)
• Apollo.io: $79/month
• Total: $79/month for ~1,300-1,800 new leads with emails

Viewing Your Leads

Accessing Leads

1. Log in with an admin account
2. Navigate to Admin > Marketing > Leads
3. View all scraped leads with filters for status, trade, and city

Lead Filters

• Status - New, Contacted, Enriched
• Trade - Filter by contractor type
• City - See leads by market
• Source - Google (scraped) vs. Apollo (imported) vs. Web (form submissions)

Market Coverage

Why These Markets?

The 25 markets were chosen based on:

• Contractor density - High concentration of licensed contractors
• Population growth - Fast-growing cities with construction activity
• Geographic diversity - Coverage across US regions
• BlueClerk presence - Markets where we already have customers

Coverage Frequency

With 4 markets per run and 22 business days per month:

• Each market scraped ~3x/month (22 days ÷ 7 rotation cycles)
• All trades scraped every time a market is selected
• Consistent coverage across all 25 markets over the month

Questions

Q: Why not scrape every market every day? A: That would be 25 markets × 10 trades × 1 page = 250 API calls per day × 22 days = 5,500 calls/month, which exceeds the 1,000-call free tier and would cost ~$1,000/month. Rotation keeps us under budget while maintaining good coverage.

Q: Why 4 markets per run instead of 5 or 3? A: 4 markets gives us a clean 7-day rotation cycle (28 market-runs ÷ 4 markets = 7 cycles), ensuring even coverage across all 25 markets over the month with 40 API calls per day (well under the daily cap).

Q: What happens if a market doesn't return 20 results? A: We only fetch the first page (up to 20 results). If Google returns fewer, we get fewer leads - but we never fetch page 2 to stay under budget.

Q: How do I see which markets were scraped today? A: Check the lead source in Admin > Marketing > Leads and filter by "Created Today" - the city field shows which markets were hit.

Q: Can I manually trigger a scrape for a specific market? A: Not yet - scraping is fully automated via cron. Manual scrapes would use API calls and risk exceeding the free tier.