On December 14, 2017 from 12:54PM to 1:30PM EST, a bug with ticket fields prevented new tickets from being created with ticket fields via the API or updating of ticket fields on existing tickets in the admin console.

Business Impact

When an agent tried to change or update a ticket field in the admin console, it would have been stuck on saving and fail.

API integrations that created tickets and those tickets included ticket fields, would have gotten a 500 error.

Root Cause

We updated our MySQL to support a wider character set. However, during preliminary testing, we missed that ticket fields explicitly relied on the old version. When the update went live, the queries failed for ticket fields resulting in failures on creating, updating or saving ticket field values.

Our engineers rolled out a patch to enable ticket fields to use the new MySQL character set.

What we are Doing to Prevent This

We have identified the gap in our QA process, which allowed this bug in production. We have already implemented a fix to ensure this issue would not happen again if we were to do another update.

We also did additional testing to ensure no other parts of the app were impacted by this update.

We are sorry for the pain point this caused for you and your support agents. If you have any additional questions, please let us know!

Claire Talbott

Support Manager