2022-01-06 Partial Inbound Call Failures (Resolved)
Written by Marissa Orsini
Updated at April 22nd, 2023
Table of Contents
Event Description: Core services restarted in SJE and ATL nodes
Event Start Time: 2021-11-29 12:12 PM EST
Event End Time: 2021-11-29 13:48 PM EST
RFO Issue Date: 2021-11-30
Affected Services
Inbound calling to some DIDs across all cores
Event Timeline (All times 24-hour format, EST)
January 6th, 2022
- 0837 - We began receiving reports of inbound calls failing.
- 0843 - Engineering assigned to begin investigating
- 0914 - Found all affected DIDs pointed to one SBC
- 0915 - Confirmed and began removing affected SBC from configured routes.
- 0959 - Verified inbound calling to all DIDs was successful.
Root Cause Analysis
The affected SBC had been planned for decommissioning in 2021 during a prior effort. It was supposed to be removed from all routes during that time. Unfortunately, some routes still had the SBC in their configuration. On 1/6/2022, licensing for the decomissioned SBC expired thus halting any calls using it.
Future Preventative Action
Our change and review processes have been augmented to include additional steps that will prevent this oversight in the future.