On September 11, from 7:05 to 7:35 and again September 13, from 11:05AM to 12:05PM, UserVoice experienced degraded performance.

Business Impact

Admins interacting with the back end of UserVoice would have experienced the following:

504 Errors: Pages would have timed out, and admins would have seen 504 errors. Unresponsive: Trying to perform actions like merging ideas, or adding users, would have hung.

Only admins and other users of API v2 were impacted by this issue. Customer facing forums, knowledge bases, and widgets were not impacted by the incident.

Root Cause

• We received an unexpected pattern and volume of requests to one of our API endpoints. Due to a bug in our code, rate limits were not applied as expected.

What we are Doing to Prevent This

• We originally mitigated the issue, and implemented a complete fix to our rate limits 9/13. We also applied one additional tweak to make our rate limiting system more robust.
• We are continuing to identify improvements to our API as a whole to ensure max optimization for our customers and their use cases with our API.
• We have also identified process improvements to decrease time to resolution if a similar issue should arise.

If you have any additional questions in regards to this issue, please reach out to me directly at claire.talbott@uservoice.com.

Claire Talbott

Support Manager