On 4/20/17 from 1:53PM to 2:02PM EDT, 87% of requests to UserVoice failed.
Business Impact
During the incident, the following parts of the app were affected, and failed to load or gave an error 82% of the time.
Email was delayed, but no email was lost.
Root Cause
Our engineering team was doing a data operation that was too aggressive. We immediately killed the operation, which resolved the issue.
What we are Doing to Prevent This
We’ve identified the problem with the job that caused the issue, and have implemented a fix. Our engineering team is working on better QA prior to operations like this to prevent something similar from occurring in the future.
We have also added tooling to improve our ability respond to misbehaving jobs.
Downtime for any reason is something we take seriously. We apologize for the disruption this caused for you and your team. If you have any questions, you can reach us at support@uservoice.com.
Claire Talbott Support Manager