Affected Services

Off-net outbound 10 & 11 digit calls across all cores. 

Inbound calls, emergency dialing, international calls and other services were not affected during this period.

Event Timeline (All times 24-hour format, EST)

May 25th, 2022

• 1025 - During the course of a support incident, a change to call routing on all cores was made. Initial testing showed inbound and outbound calls were processing successfully.    
• 1029 - First report of outbound call failures received, engineers began investigating the issue
• 1030 - A review of failed calls showed that the modified routing was sending outbound calls with an extra digit in the sender
• 1031 - Call route change was undone, further testing confirmed that outbound calling was successful
• 1034 - Notified partners and clients of resolution via discord
• 1047 - Notified partners and clients of resolution via email

Root Cause Analysis

The modified call route was intended to modify the display of outbound calling numbers in line with NANPA standards. This was to improve the behavior of forwarded calls. Unfortunately, the modification caused outbound calls to send out with an additional digit in the destination, causing outbound off-net calls to fail. 

Future Preventative Action

Our change and review processes have been augmented to include additional steps to prevent this oversight in the future, including improved testing methodology for implementations.