500 Errors on UserVoice Accounts
Incident Report for UserVoice
Postmortem

On May 22, 2018, from 10:04AM to 10:25AM EDT, UserVoice had a partial outage that resulted in slow response times or 500 errors for admins and end users.

Business Impact

Admins and end users could have experienced the following:

  • Widget, web portal or admin console being slow to load
  • 500 errors when trying to load the widget, web portal or admin console
  • Email delays (no emails were lost)

Root Cause

  • During a schema migration, a misconfiguration introduced an inconsistency into our database cluster. This resulted in our application being unable to handle all requests. A temporary fix was implemented at 10:25AM EDT, and a permanent fix was rolled out at on May 23, 2018 at 2:42AM EDT.

What We are Doing to Prevent This

  • Our engineers have implemented changes to our migration so this type of misconfiguration cannot occur again.

We apologize for the pain point this caused for you and your users using UserVoice. This is something our entire engineering team takes very seriously, and if you have any questions or concerns, please reach out and let us know.

Claire Talbott

Support Manager

Posted May 23, 2018 - 15:27 EDT

Resolved
This incident has been resolved.
Posted May 23, 2018 - 02:43 EDT
Monitoring
Hi everyone! I apologize for the delay here in giving an update. We have seen full service restored since 10:25AM EDT, but are continuing to monitor things as we work on identifying the root issue, and putting a permanent fix in place.
Posted May 22, 2018 - 13:30 EDT
Update
We are continuing to investigate this issue.
Posted May 22, 2018 - 10:25 EDT
Update
We are continuing to investigate this issue.
Posted May 22, 2018 - 10:23 EDT
Investigating
We are currently investigating reports of 500 errors and slowness in the application. We will keep you updated as we work to resolve this.
Posted May 22, 2018 - 10:23 EDT
This incident affected: Web Portal (subdomain), Admin Console, UserVoice API, Helpdesk API, Widgets, Search, and User Analytics.