500 errors in the web portal and admin console
Incident
Report for UserVoice
Postmortem
On Wednesday, May 16, 2018, from 12:01PM-12:31PM EDT, 14% of requests to UserVoice failed, resulting in 500 errors for admins and end users
Business Impact
Admins and end users impacted by the outage would have experienced the following:
- API timeouts
- Widget would not have loaded
- 500 error when loading or interacting with the web portal (forums or knowledge bases)
- 500 error when loading or interacting with the admin console
Emails would have been delayed during the outage, but no emails were lost.
Root Cause
- Increased load on our database due to a data cleanup process led to degraded performance.
What we are Doing to Prevent This
- Scaling back our data cleanup process to run at off-peak hours and on smaller sets of data.
- Modifying our auto-scaling capability to react more quickly under conditions of high load.
We apologize for the pain point this caused for you and your team. Any type of down time is something we take very seriously. If you have any questions or concerns, please, do not hesitate to reach out!
Claire Talbott
Support Manager
Resolved
This incident has been resolved.
Update
Our systems have recovered normal operation but we are still investigating the root cause of the incident.
Update
We are continuing to investigate this issue.
Investigating
We are currently investigating this issue.
This incident affected: Web Portal (subdomain), Admin Console, UserVoice API, Helpdesk API, Widgets, Email (incoming), and Email (delivery).