Elevated timeout rate
Incident
Report for UserVoice
Postmortem
On Sunday, August 21, between 3:05AM and 3:28AM, and again between 6:44 and 6:52AM EDT, 40% of requests to our app failed.
Business Impact
- Admins and end users impacted by the downtime would have seen a 503 or 522 error when loading a UserVoice site.
- End users would have been able to open or load the widget, but would not have been able to submit a ticket or post an idea.
Root Cause
The partial outage was caused by database performance issues on our end that we are continuing to investigate.
What we are Doing to Prevent This
- Our Engineering Team is focused on optimizing our database, and improving our configurations, to prevent issues like this going forward, and provide the best performance possible.
If you have any questions, contact us at support@uservoice.com.
Claire Talbott
Support Manager
Resolved
The database cluster has been performing smoothly for the past 3 hours. We're resolving this incident now but we'll continue following up on database performance issues.
Monitoring
Overnight our master MySQL node began to experience performance issues and we switched traffic to another node. We believe the recent performance issues are related to syncing of other db nodes. Performance has stabilized and we will continue to investigate.
Investigating
We are currently experiencing database issues resulting in elevated timeouts for users. We are investigating.