Outage report: 5th August 2019
At 8:48 BST on Monday the 5th of August our monitoring system alerted us of a network outage which affected most of our services.
We narrowed the issue down to one of our internet connection providers, Bandwidth Technologies, and contacted them to report this.
They confirmed they had an issue with a cross-connection within Telehouse and raised a support issue for replacement cable under their SLA. The cable was between SSE (another of our providers) and Bandwidth Technologies.
The delay for replacement of the cable was due to unknown ownership of the cable. This cable was an unknown cable, under no SLA or owner, and not registered with Telehouse. Therefore, it was not Telehouse at fault, nor was it under their SLA.
Bandwidth technologies visited Telehouse themselves to resolve the issue and Telehouse allowed a temporary cable to be installed to fix the issue.
An order has been placed for a permanent cross-connect cable to be installed under Telehouse SLA.
Fast2host has moved to a new datacentre and with partnerships and many company changes in the last 6 months. A new datacentre has been built and a whole new network put in place. The new datacentre has had network weak points which we are finding out. These are beyond our physical ownership but still need to be addressed with the companies involved. We will be locating these issues and fixing them asap.
We need to:
Review our Disaster Recovery Plan for the new datacentre.
Review our phone system and system updates, notifications.
Renew our internet connections, uplinks and routes.
Check all SLAs with any and all companies involved.
We take full responsibility for this issue and apologise sincerely. We fully understand the issue this has caused to our customers, and we are aware that for many of you the internet is your income and livelihood, and lost a full day of business. We have also had some great feedback from our long-standing customers which we appreciate. We will be addressing all these issues with some urgency.