Load Test Monitoring

Overview

BlueClerk runs automated load tests to ensure the platform can handle traffic under stress. When tests fail, Claude Haiku AI analyzes metrics to classify failures as auto-fixable (triggers automatic redeployment) or requiring human action (shows admin banner with diagnosis). Slack alerts differentiate between the two failure types.

How It Works

Automated Testing

• Load tests run on schedule to simulate user traffic
• Performance metrics tracked: Virtual users, response times (p95), error rates
• Pass/fail threshold determines system health

AI Diagnosis on Failure

When a load test fails:

1. Claude Haiku analyzes k6 metrics (virtual users, p95 response time, error rate)
2. Classifies the failure:
   • Auto-fixable: Transient issues like cold starts or temporary slowdowns
   • Action required: Persistent problems needing human investigation
3. Triggers response based on classification

Auto-Fix for Transient Issues

When AI detects an auto-fixable failure:

• Automatic Vercel redeploy is triggered
• Slack alert indicates auto-fix was applied
• No admin banner shown (handled automatically)
• Record updated with autoFixApplied field noting the action taken

Admin Alert for Persistent Issues

When AI detects an issue requiring human action:

• Admin banner appears at the top of all admin pages
• Shows diagnosis from Claude Haiku explaining the issue
• Includes test metrics: Virtual users, p95 response time, error rate
• Run date displayed for reference
• Slack alert indicates action is required
• Banner dismissible but will reappear until the underlying test passes

Viewing Load Test Results

System Health Page

1. Log in with an admin account
2. Navigate to /admin/operations/health
3. Scroll to Load Test Results section
4. View recent tests with pass/fail status, metrics, and AI diagnosis

Test Result Details

Each test record shows:

• Run date and time
• Duration of the test in seconds
• Virtual users simulated
• Average response time (ms)
• P95 response time (ms) - 95th percentile latency
• Error rate (percentage)
• Pass/fail status
• AI diagnosis (if failed)
• Action required flag (if human intervention needed)
• Auto-fix applied (if automatic remediation triggered)

Understanding Test Metrics

Virtual Users

Number of simulated concurrent users hitting the platform during the test. Higher numbers stress the system more.

P95 Response Time

The response time at the 95th percentile - meaning 95% of requests were faster than this value. A key indicator of user experience under load.

Error Rate

Percentage of requests that returned errors. Should be 0% for a passing test.

Tips

• Don't panic on one failure: AI diagnosis helps determine urgency
• Auto-fixed failures are normal: Cold starts and transient issues happen
• Action-required failures need attention: Check Slack alerts and diagnosis for next steps
• Review trends over time: Multiple failures may indicate a deeper issue

Questions

Q: How often do load tests run? A: Load tests run on a schedule (typically daily or after major deployments). Check with your platform admin for the exact schedule.

Q: What triggers an auto-fix? A: Claude Haiku analyzes metrics and classifies failures. If diagnosed as a transient issue (cold start, temporary slowdown), it triggers a Vercel redeploy automatically.

Q: Why do some failures show a banner and others don't? A: Only failures classified as "action required" show the admin banner. Auto-fixable failures are handled automatically without requiring admin attention.

Q: Can I dismiss the admin banner? A: Yes, but it will reappear on page load until the next load test passes and the issue is resolved.

Q: Where can I see the AI diagnosis? A: In the admin banner (for action-required failures), on the System Health page's Load Test Results section, and in Slack alerts.