Load Test Monitoring

Overview

BlueClerk runs automated load tests every Sunday at 12am UTC to ensure the platform can handle real-world traffic patterns under stress. The test simulates 500 virtual users performing mixed read and write operations, then automatically cleans up all test data. Results are saved with detailed breakdowns of read vs. write performance, helping admins monitor system reliability.

How Load Testing Works

Test Configuration

• 500 virtual users - Simulates significant concurrent load
• Batched execution - 50 concurrent users per batch to avoid overwhelming the system
• Mixed operations - 90% read-heavy queries, 10% write operations (create mutations)
• 90-second timeout - Test stops automatically if it takes longer than expected
• Automatic cleanup - All test data is deleted after the test completes via POST /api/cron/load-test-cleanup

Read Operations Tested

• Ticket queries - Fetching ticket lists and details
• Job queries - Loading job schedules and status
• Invoice queries - Retrieving billing data
• Customer queries - Looking up customer information

Write Operations Tested

The load test now includes three critical write flows that historically break under load:

1. ticket.create - Full ticket creation with real propertyId, customer, and item data fetched from live data in setup(). Description is tagged "LOADTEST-ticket-..." for cleanup identification.
2. job.rescheduleJob - Picks a random non-completed job ID and moves it to a random future date/time. Job notes are tagged "LOADTEST reschedule" for cleanup.
3. ticket.update - Updates dueDate on an existing real ticket. Only touches the date field to preserve ticket descriptions (cleanup identifies tickets by description prefix).

Setup Phase

Before running write operations, the load test fetches fixtures per contractor session:

• Existing ticket to borrow propertyId, customerHomeownerId, and itemId
• Schedulable job IDs (non-completed jobs) for reschedule scenario
• Existing ticket IDs for edit-date scenario

This ensures write operations use real, valid data that mimics production usage patterns.

Test Data Cleanup

Automatic Deletion

After each load test completes, the cleanup endpoint (POST /api/cron/load-test-cleanup) runs automatically and deletes:

1. Jobs where ticketId belongs to a LOADTEST- ticket (deleted first because Job → Ticket has no onDelete cascade)
2. Tickets where description starts with "LOADTEST-" (cascades to TicketItem and TicketPhoto)
3. JobNotes with content starting "LOADTEST reschedule"
4. ContractorLeads with names starting "LOADTEST-"
5. Items with names starting "LOADTEST-"
6. CompanyLocations with names starting "LOADTEST-"

Safety Filters

Every deleted row MUST have a name or description starting with "LOADTEST-". This prefix is applied by the k6 script when creating rows, ensuring we can't accidentally delete real user data.

Viewing Results

Accessing Test Results

1. Log in with an admin account
2. Navigate to Admin > Operations > Health
3. Scroll to "Load Test Results" section
4. View latest test with pass/fail status and performance metrics

What's Included in Results

• Overall pass/fail - Whether the test completed successfully
• Total duration - How long the test took to run
• Read performance - Average response times for query operations
• Write performance - Average response times for create/update operations
• Error count - Number of failed requests
• Throughput - Requests per second

Questions

Q: Why run load tests weekly? A: Weekly tests catch performance regressions early, before they impact real users. Running on Sundays at midnight minimizes impact on production traffic.

Q: What happens if a load test fails? A: The system logs the failure with detailed error information. Admins receive notifications and can review the failure in the Health dashboard.

Q: Can I run a load test manually? A: Currently, load tests run automatically on schedule. Manual triggering is not available to prevent accidental load on production.

Q: How do write operations ensure data safety? A: All test data uses the "LOADTEST-" prefix, setup() fetches real fixture data to use as templates, and cleanup only deletes records matching the safety filter. Real user data is never touched.

Q: What if cleanup fails? A: The cleanup endpoint logs detailed results including counts of deleted records. Admins can review cleanup logs and manually trigger cleanup if needed.