Intermittent Call Traffic - Inbound/Outbound
Incident Report for TelcoSwitch Group
Resolved
Following an extended enhanced Monitoring Period, this issue is now classified as ‘Resolved’.

We are currently finalising our detailed incident report, pending responses from various upstream providers involved. This will be distributed to affected customers in due course.

Thank you for your patience.
Posted Dec 20, 2022 - 09:31 GMT
Update
We expect the situation to be contained.
Posted Dec 15, 2022 - 17:38 GMT
Monitoring
This afternoon mitigating actions were taken by origin carriers via our upstream carrier partners to block malicious voice traffic coming into our network.

Following on from this, our internal senior technical service team has been working closely with the Senior Complex Investigations Team at our upstream carriers to proactively monitor traffic.

We expect the situation to be contained.
Posted Dec 15, 2022 - 17:36 GMT
Update
We have been collaborating with our upstream carriers' Fraud and Design Specialists in order to effectively reduce the amount of inbound malicious traffic and accurately identify the source of all robotically initiated calls.

We continue to monitor the situation closely as we push international origin carriers to block all inbound malicious traffic at source.

Our senior joint taskforce, which consists of experts from both our team and our colleagues at upstream carriers, are assessing the current status. We expect to have further clarification on the situation in due course.
Posted Dec 15, 2022 - 11:30 GMT
Update
Our internal technical teams have been working tirelessly throughout the night in collaboration with the Senior Complex Investigations Team from our upstream carriers. These teams have been closely coordinating with Fraud Specialists this morning to ensure the origin carriers take the required steps to block the malicious inbound call traffic as they have been instructed.

We expect a further update within 2 hours.
Posted Dec 15, 2022 - 08:32 GMT
Update
We continue to closely collaborate with our upstream providers to ensure that malicious inbound traffic traced to international origin carriers is blocked as soon as practically possible.

We will continue to provide further updates as these become available.
Posted Dec 14, 2022 - 22:41 GMT
Update
To mitigate the negative impact of malicious inbound call traffic on our network and customers, our upstream carriers continue to focus on blocking the identified sources of this traffic as much as possible through their dedicated Senior Complex Investigations Team.

Timings for resolution are impacted given the origin carriers of offending traffic are international.

We will continue to provide further updates as these become available.
Posted Dec 14, 2022 - 19:21 GMT
Update
We, along with our upstream providers, are currently still awaiting the identified origin carriers to block inbound malicious call traffic which is causing loss of voice service for users. Some of these origin carriers are not UK-based, which impacts the timing of action/resolution due to both time zones and regulatory differences across international territories.
Posted Dec 14, 2022 - 17:03 GMT
Update
We are actively working with the origin carriers who are responsible for blocking malicious inbound calls to prevent them from reaching our network and disrupting service availability for our users. We are awaiting updates from these carriers on the progress of their CLI blocking activity and are confident that they are doing everything in their power to address this issue as quickly as possible.

Current service loss affects Inbound services ONLY. All outbound services (including 999 and other emergency services) remain available and unaffected.

Separately, we are liaising with a number of regulatory bodies, including Ofcom and Comms Council UK, the National Cyber Security Centre, and the National Fraud & Cyber Crime Reporting Centre as a matter of procedure given the information we have compiled over the course of the last 24 hours.
Posted Dec 14, 2022 - 14:46 GMT
Update
As per the guidance from our upstream carrier, the origin carriers have initiated the process of blocking the offending caller line identification (CLI) numbers from which the influx of concurrent malicious calls are originating. The origin carriers will provide us with regular updates on their progress as they work towards resolving the issue.

For security reasons, certain technical details will be redacted from the updates that we receive and share in this status update.
Posted Dec 14, 2022 - 11:59 GMT
Update
After conducting a review involving the senior technical teams at our onward carriers, we can confirm that the majority of malicious inbound calls are from the same origin carriers. Upon conducting a thorough investigation, those origin carriers have determined the traffic is robotic in nature. In response to this finding, we are currently working closely with the origin carriers to take appropriate action to block the traffic and prevent further disruption.

The next update will be at 12:00.

Thank you for your patience as we continue to work to resolve this issue with our carrier partners.
Posted Dec 14, 2022 - 10:58 GMT
Update
As we continue to address the issue of malicious call traffic, we are currently implementing measures to ringfence numbers identified as most severely impacted by this problem, in order to reduce the overall load on our platform and mitigate the negative effects of the influx of incoming calls. We have identified sources of where the bad traffic is coming from and we are hoping the various upstream providers will be blocking the majority of that this morning.

Thank you for your patience as we continue to work to resolve this issue.
Posted Dec 14, 2022 - 07:38 GMT
Update
As we continue to address the issue of malicious call traffic, we are currently implementing measures to ringfence numbers identified as most severely impacted by this problem, in order to reduce the overall load on our platform and mitigate the negative effects of the influx of incoming calls.

Thank you for your patience as we continue to work to resolve this issue.
Posted Dec 13, 2022 - 17:36 GMT
Update
We are continuing to successfully scrub malicious call traffic to restore service to affected customers – this work will continue into the evening.

At this time, we cannot confirm when service will be restored for all the affected users but will provide accurate updates as soon as these are available.

We apologise for the continued inconvenience and thank you for your patience as our teams work to resolve this issue.
Posted Dec 13, 2022 - 16:31 GMT
Update
Malicious inbound calls flooding to specific tenants have impacted load across wider clusters of tenants.

We have been working closely with our upstream providers to identify and scrub malicious call traffic targeting specific customers and restore service to all affected services. We are making significant progress to redirect the malicious traffic.

At this time, we cannot confirm when service will be restored for all affected users but will provide accurate updates as soon as these are available.

The next scheduled update will be at 16:00 unless we have an interim update to share with you beforehand.
We apologise for the inconvenience we know this causes and thank you for your patience.
Posted Dec 13, 2022 - 15:08 GMT
Update
We are continuing to work with upstream providers feeding calls to us to identify any non-genuine fraudulent call traffic being received on our platform.
The current disruption is impacting 5% of UK-based end-users.
The next update will be at 15:00
We are extremely sorry for this disruption.
Posted Dec 13, 2022 - 14:06 GMT
Update
We are continuing to work with upstream providers feeding calls to us to identify any non-genuine call traffic being received on our platform.

The current disruption is impacting 5% of UK-based end-users.
The next update will be at 13:45
Posted Dec 13, 2022 - 13:00 GMT
Update
We are continuing to work with upstream providers feeding calls to us to identify any non-genuine call traffic being received on our platform.

The current disruption is impacting 7% of UK-based end-users.

The next update will be at 12:45
Posted Dec 13, 2022 - 11:47 GMT
Update
At approximately 13:30 yesterday afternoon (12/12/2022) we detected a significant and abrupt influx of inbound call traffic to our voice platform.

Overnight we have been working closely with upstream providers feeding calls to us to identify any non-genuine traffic, and this work has continued into this morning.

The current disruption impacts varying numbers of users depending on the intensity of inbound call traffic at any one time, but is approximately 5% of UK users.

As we continue to work on this issue we will be publishing hourly updates to inform you of progress. The next update will be at 11:45
Posted Dec 13, 2022 - 10:42 GMT
Update
We are currently working with our upstream providers to fix this issue.
Posted Dec 13, 2022 - 07:24 GMT
Update
We are continuing to work on a fix for this issue.
Posted Dec 12, 2022 - 21:51 GMT
Identified
We are experiencing a similar issue to earlier and are currently working to resolve.
Posted Dec 12, 2022 - 21:14 GMT
This incident affected: CallSwitch (Hosted Telephony Platform).