Affected Services

Inbound and outbound calling for some customers

Event Summary

We received reports of poor call quality from various customers across several ISPs.

Maintenance was performed in the Atlanta datacenter from 3:00 AM to 7:00 AM EST on 5/13/2021. We performed changes to the routing and switching infrastructure which required a restart. During this time, devices registered via SRV automatically re-routed to alternate data centers. All call services functioned normally from the alternate datacenters.

You can stay informed of this and all other issues by subscribing to notifications at https://status.oitvoip.com/

Root Cause Analysis

We found a culmination of issues that were affecting some customers depending on their route into our Atlanta data center. We believe the primary factor was a packet flood from one of our third-party vendors that utilized the affected link. This was exacerbated by a bug in our routing infrastructure, resulting in severe packet loss for any traffic over the affected link.  Other data centers were not affected due to the use of a different routing platform. 

We worked with the vendor to get their packet flood stopped. We also worked with our routing vendor to apply a fix for the buffer bug. Upon restart, we no longer experienced dropped packets over the affected link.

The maintenance actions were successful. All services returned to normal operation. The Atlanta data center was brought back online and remained nominal throughout the day. At this time we are confident that the incident has been resolved.

Future Preventative Action

We will continue with our plans of domain redistribution in order to reduce the potential impact of any issues in the Atlanta data center.  Updates will come in future notifications. If you have any questions please contact support.