Outage on 2nd october
Outage on 2nd october
At 15:15 BST on Wednesday 2nd October our monitoring system alerted us of a network outage that affected most of our services.
We had Openreach onsite at the time installing more fibre connections for redundancy to our datacentre when there was an outage. At first we thought this was related however we narrowed the issue down to SSE network to network interface with Virgin media of which we use for our fibre to datacentre connections and contacted them to report this.
They confirmed they had an issue with a cross-connection within a London datacentre and raised a support issue under their SLA. They resolved the issue at 17:30 however we still had some services down. This was down to our network system in our datacentre requiring a reboot to restart again. All services were resolved at 18:15.
We are addressing these issues and working on building a high availability network in our new datacentre and we can assure you we will carry on with the outstanding service we are known for. Openreach are currently installing new systems and fibre cables for higher availability. This work is due to be completed by the 13th of October. We have reviewed and made changes to our uplinks, network routes and SLAs since the last outage in August.
In the meantime, if you still have any concerns and would like to arrange a meeting or to visit our new datacentre please get in touch with us.
RFO - 02/10/2019 – Network outage – SSE NNI, London
Start time – 14:17 AM GMT
End time – between 17:15 and 18:30 PM GMT
SSE NNI between Virgin Media outage at a London Datacentre.
14:17 GMT – We became aware of network outage effecting DC and a few lease line customers. We had Openreach onsite at the time installing more fibre cables for more redundancy.
15:00 GMT – We had checked all cables and systems and the onsite work with Openreach was not the cause. We located the issue to SSE NNI in London
15:15 GMT – SSE NOC ticket and phone call raised and confirmed an issue.
15:30 GMT - Third Party engineers on-route to site to start their investigations.
17:00 GMT – SSE announced third party vendor awaiting access to data centre, loop testing has proved the issue onto the third party network.
17:30 GMT – SSE announced the issue had been resolved however we still had some services down.
18:15 GMT – We restarted network switches at datacentre and started to see traffic flowing again.