CSL IoT Service Outage Notification 20/06/2023
Incident Report for Legrand Care Cloud Service Status
Resolved
Update 14:00 (BST) 20/06/2023: All Tele2 SIM ranges have now been unblocked and we have seen a restoration of service. CSL and Tele2 will continue to monitor.
CSL will carry out a full investigation with Tele2 to understand the Root Cause of the issue and will share the findings and report.
Posted Jun 20, 2023 - 15:04 CEST
Update
Update 12:00 (BST) 20/06/2023: Tele2 are continuing to unblock SIM ranges, managing the load to avoid a signalling storm whilst prioritising critical (Telecare and Security) ranges. We are seeing some restores. Progress is slow to avoid a signalling storm, but we expect this to be completed by 13:30 BST. CSL and Tele2 will continue to monitor.
Posted Jun 20, 2023 - 13:21 CEST
Update
Update 11:00 (BST) 20/06/2023: Tele2 have started unblocking SIM ranges, managing the load to avoid a signalling storm whilst prioritising critical (Telecare and Security) ranges. We are starting to see some restores. CSL and Tele2 will continue to monitor.
Posted Jun 20, 2023 - 12:29 CEST
Update
Update 10:00 (BST) 20/06/2023: Tele2 and their signalling vendor have made a configuration change and early indications are that this route is stable. Tele2 will slowly manage the unblocking of SIM ranges to avoid a signalling storm whilst prioritising critical (Telecare and Security) ranges
Posted Jun 20, 2023 - 11:20 CEST
Update
Update 09:20 (BST) 20/06/2023: Tele2 and their signalling vendor has rerouted traffic via a different STP node, this has not had the desired effect and they are still seeing high levels of errors on the alternative STP node. Tele2 are investigating this further with their signalling vendor. Impact is for new sessions only, existing sessions will continue to work.
Posted Jun 20, 2023 - 10:24 CEST
Update
Update 08:00 (BST) 20/06/2023: Tele2 and their signalling vendor has rerouted traffic via a different STP node, they are monitoring the outcome of this re-route before unblocking signalling ranges starting with critical ranges (Telecare and Security). Impact is for new sessions only, existing sessions will continue to work.
Posted Jun 20, 2023 - 09:14 CEST
Update
Update 07:00 (BST) 20/06/2023: Tele2’s signalling vendor has completed the blocking of 2G and 3G SIM ranges in order to mitigate the load on STP nodes. This should lighten the existing load on the STP sufficiently, Tele2 and the vendor will monitor this before unblocking critical ranges (Telecare and Security).
Posted Jun 20, 2023 - 08:44 CEST
Update
Update 06:00 (BST) 20/06/2023: Tele2 are working with their Signalling Vendor to block all incoming IoT traffic to the Tele2 network from roaming partners. This is to avoid a suspected overload issue on the Signalling Transfer Point node (STP). At the moment, all new network registrations on 2/3G are being blocked to minimize the load on the network. This is impacting new attachments to the network, existing sessions will continue to work. Once all signalling traffic has been blocked this should lighten the existing load on the STP sufficiently that the critical ranges (Telecare and Security) can be unblocked.
Posted Jun 20, 2023 - 07:02 CEST
Update
Update 05:00 (BST) 20/06/2023: Tele2 are continuing to work to bring 2/3G traffic back online whilst implementing the playbook to avoid a signalling storm. Blocked IMSI ranges are being unblocked with Telecare and Security devices given priority, however we are seeing that the STP is not processing 2/3G registration attempts as it should be. Tele2 are working with their signalling vendors to understand the cause for this. CSL and Tele2 are continuing to work on restoring service.
Posted Jun 20, 2023 - 05:58 CEST
Monitoring
Ongoing Service areas affected: Tele2 SIMs
Outage
Start Date and Time (BST): 20/06/2023 - 01:00
End Date and Time (BST): 20/06/2023 -

We are aware of an issue affecting 2/3/4G, Voice, and SMS on Tele2 SIMS. CSL and Tele2 are investigating.

Update 02:00 (BST) 20/06/2023: Tele2 have identified and bypassed an affected node and we are seeing restores. CSL and Tele2 are continuing to monitor.

Update 03:00 (BST) 20/06/2023: Since the bypass of the affected node, 4G service has returned to normal values. To avoid a signalling storm when bringing 2/3G services back online, the Tele2 playbook to avoid a signalling storm will be implemented. This will involve throttling new 2/3G requests and gradually removing the block to manage incoming registration processes. As per agreement with Tele2, critical (Telecare and Security) devices will be blocked last and unblocked first.

Update 04:00 (BST) 20/06/2023: Tele2 are continuing to implement the playbook to avoid signalling storm and we are seeing some improvement to traffic. CSL and Tele2 are continuing to work on restoring service.

Next update at 05:00 (BST) or before.
Posted Jun 20, 2023 - 05:22 CEST
This incident affected: M2M (M2M Service).