2021-03-31 - Outbound Calling Outage (Resolved)
Written by Marissa Orsini
Updated at April 22nd, 2023
Table of Contents
Event Description: Outbound calling failure
Event Start Time: 2021-03-31 10:21 EST
Event End Time: 2021-03-31 10:37 EST
RFO Issue Date: 2021-03-31
Affected Services
Voice Services (Outbound)
Event Summary
On March 31st, 2021, at approximately 10:21 AM EST, outbound calls failed to connect.
Event Timeline (All times 24-hour format, EST)
March 31st, 2021
- 10:21 We began receiving alerts that outbound calls were failing
- 10:22 Tier 3 began reviewing call traces and logs
- 10:30 Determined that our fraud and spam monitoring was blocking valid calls
- 10:35 Made routing changes to bypass errant blocking
- 10:38 Confirmed successful outbound calling
- 10:39 Discord server and tickets notified of issue resolution
Root Cause Analysis
The platform we use for monitoring and responding to fraud and spam events began blocking valid calls. This was due to a configuration change to stop alerting on some false positives.
Future Preventative Action
While the change was designed to lessen the veracity of the alerting platform, it was still against best practices to make the change during the day. Future routing changes on this platform, regardless of scope, will be done after hours.