Resolved -
This incident has been resolved.
Mar 17, 17:29 GMT
Monitoring -
The root cause has been identified. We operate proprietary SIP Protection provided by a 3rd party across each of our stacks at Volta. For all intent and purpose this is a centrally managed firewall for SIP. This protection saw what it perceived as a scanning event to one server and blocked an important internal IP address as a result. In adding this IP to the blacklist it was dynamically distributed to all servers in the stack and our teams were notified per normal course of business, reviewing the addition within seconds. When they noted the presence of an internal IP it was added to the whitelist immediately. Unfortunately in haste the IP was added incorrectly by one digit. As a result, the incident took some time to circle back to origination, identify the incorrect entry and resolve. The resolution is permanent.
Mar 17, 14:52 GMT
Update -
CC761 will be restored in full by 14:25. We continue to investigate the root cause that impacted Volta 6 & 7 inbound traffic.
Mar 17, 14:20 GMT
Identified -
Traffic resumed to normal patterns as of 13:32. Incident commenced 12:31 to these stacks. We continue to investigate the root cause of this issue. CC761 remains impacted and resolution for this specific server is in hand.
Mar 17, 13:56 GMT
Update -
The issue is isolated to a number of servers located in our Volta 6 & 7 stacks. We continue to investigate the issue and will update further in 15 minutes.
Mar 17, 13:38 GMT
Investigating -
We are currently investigating reports of inbound calls failing. We are unaware at this stage if these reports are related to the current Gamma MSO. A further update will be provided in the next 15 minutes.
Mar 17, 13:26 GMT