On September 26 from 8:44AM to 8:58AM and on September 27 from 8:43 to 8:49AM EDT, a majority of users saw 503 errors when accessing their UserVoice admin console, widget or web portal.

Business Impact

• Admins received a 503 error when trying to access the admin console
• End users would have gotten an error when loading the widget or trying to use the web portal.
• Leading up to the incident, some customers also reported the app was unresponsive when trying to * perform ticket actions like “send” and “send and close.”

Root Cause

• At around 8:00 AM, a customer began a crawl of their users API endpoint. At a large scale, this is a very resource intensive endpoint. Previous optimizations to the database queries involved have become less effective over time, and combined with some coincidental heavy updates to user records, the crawling activity impacted database performance and caused a large spike in 503 errors.

What we are Doing to Prevent This

• We have reworked the database queries generated by this particular endpoint for a roughly 10x performance improvement.
• We have identified and eliminated some unnecessary sources of updates to user records that were exacerbating performance issues.
• We have also identified several other areas of improvement within our API and app, and will be implementing those within the next week.

Downtime of any kind is something we take very seriously. It affects your team’s ability to do their job. If you have any questions or concerns about this incident, please let me know.

Claire Talbott

Support Manager

claire.talbott@uservoice.com