Delays in processing incoming and outgoing mail
Incident Report for UserVoice
Postmortem

On Saturday, August 20th, we saw delays with incoming and outgoing email being delivered. The issue was resolved at 1:01PM EDT, and email delivery times returned to normal.

Business Impact

  • Incoming emails from end users, and outgoing replies by admins were delayed up to three hours in some cases.

  • No emails were lost

Root Cause

We had an inefficient worker job that when combined with an inefficient database query caused our worker queues to back up with redundant processes. This caused incoming and outgoing email to get backed up.

What we are Doing to Prevent This

  • We have optimized the job in question to avoid this kind of backup in the future.

  • We are dedicating more resources towards ongoing database performance improvements.

  • We are continuing to look for ways to optimize how emails are processed, to make it more efficient going forward.

If you have any questions, contact us at support@uservoice.com.

Claire Talbott

Support Manager

Posted Aug 24, 2016 - 10:54 EDT

Resolved
We're all caught up now, apologies for the turbulence!
Posted Aug 20, 2016 - 13:16 EDT
Monitoring
We've deployed a fix and are plowing through queues at full speed again. Incoming email has all been processed and all outgoing email should be delivered shortly. We'll post an update here when we are completely caught up.
Posted Aug 20, 2016 - 13:03 EDT
Identified
Our mail queues are currently backed up and users might experience delays in incoming and outgoing email. We're working on a fix to get things moving faster. No email has been lost.
Posted Aug 20, 2016 - 12:52 EDT