503 errors in admin console
Incident Report for UserVoice
Postmortem

On August 5, 2019, from 5:49PM to 6:23PM EDT, the admin console and UserVoice API were down.

Business Impact

  • Admins saw 503 errors in the admin console, or noticed excessive slowness when loading parts of the application.
  • The UserVoice API returned errors.
  • Users on the web portal or with the widgets may have briefly seen errors or slowness at the start of the incident, but only for a few moments.

Root Cause

  • A configuration change during a routine Kubernetes Cluster upgrade caused an unexpected restart for key operational services. One of these services had trouble recovering, which caused the downtime.

What we are Doing to Prevent This

  • Our team identified and fixed the issue that caused the trouble during restart.
  • Our team also identified the setting that triggered the unplanned update, and how best to ensure that does not happen again.

We do apologize for the downtime. This interrupted your usage of UserVoice, and for that we truly are sorry.

If you have any questions or concerns, please, don’t hesitate to reach out!

Claire Talbott

Support Manager

claire.talbott@uservoice.com

Posted Aug 07, 2019 - 12:35 PDT

Resolved
We have pushed a fix for the issue and things are stable. Let us know at support@uservoice.com if you have any issues.
Posted Aug 05, 2019 - 18:14 PDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Aug 05, 2019 - 15:39 PDT
Investigating
We are currently investigating an issue that is causing 503 errors and loading issues in the admin console. We will keep you updated as we investigate further.
Posted Aug 05, 2019 - 14:59 PDT
This incident affected: Admin Console.