Inbound Calls to Volta 6 & 7 Hosted Servers
Incident Report for TelcoSwitch Group
Resolved
This incident has been resolved.
Posted Mar 17, 2023 - 17:29 GMT
Monitoring
The root cause has been identified. We operate proprietary SIP Protection provided by a 3rd party across each of our stacks at Volta. For all intent and purpose this is a centrally managed firewall for SIP. This protection saw what it perceived as a scanning event to one server and blocked an important internal IP address as a result. In adding this IP to the blacklist it was dynamically distributed to all servers in the stack and our teams were notified per normal course of business, reviewing the addition within seconds. When they noted the presence of an internal IP it was added to the whitelist immediately. Unfortunately in haste the IP was added incorrectly by one digit. As a result, the incident took some time to circle back to origination, identify the incorrect entry and resolve. The resolution is permanent.
Posted Mar 17, 2023 - 14:52 GMT
Update
CC761 will be restored in full by 14:25. We continue to investigate the root cause that impacted Volta 6 & 7 inbound traffic.
Posted Mar 17, 2023 - 14:20 GMT
Identified
Traffic resumed to normal patterns as of 13:32. Incident commenced 12:31 to these stacks. We continue to investigate the root cause of this issue. CC761 remains impacted and resolution for this specific server is in hand.
Posted Mar 17, 2023 - 13:56 GMT
Update
The issue is isolated to a number of servers located in our Volta 6 & 7 stacks. We continue to investigate the issue and will update further in 15 minutes.
Posted Mar 17, 2023 - 13:38 GMT
Investigating
We are currently investigating reports of inbound calls failing. We are unaware at this stage if these reports are related to the current Gamma MSO. A further update will be provided in the next 15 minutes.
Posted Mar 17, 2023 - 13:26 GMT
This incident affected: Core Network.