At 07:42 UTC on December 15th, SheerID engineering updated a Database index to improve query performance. Initially, the change looked good until 10:30 UTC, when a particular type of API request used the new index in a way SheerID engineering did not intend. The database CPU spiked but recovered quickly, and the alert cleared. Then, at 14:34 UTC, SheerID saw a new batch of requests that caused another Database CPU spike, but this time, the system started to show a growing impact on our API latency. SheerID Engineering was alerted again and worked to identify the problematic query and fix the problematic index. At 18:30 UTC, the Database returned to typical CPU levels of ~20%, and API latency was restored to typical.