(Note: At the time of the initial publication we have had no further details from our operator partner as to the nature of the issues they experienced during this incident. We have now received further information, and have updated accordingly)
All times are in UTC/GMT unless otherwise stated.
At approximately 13:21, Ziron monitoring detected an increasing error rate in response to lookup requests made to an operator that provides such services to Ziron. The on call engineer was paged.
A loss of connectivity to the operator’s IP network was also detected, and Ziron’s network team was engaged to investigate further - as well as an urgent ticket being raised with the operator’s service management centre. Initial investigation showed that VPN connectivity was the cause of the issue, and no fault could be found on the Ziron side.
At 13:41, a major incident was declared and the senior management team were paged. Status notifications were posted to all customers via the Ziron status page.
By 13:49, the operator had acknowledged they were experiencing a major network outage. An update was received at 14:25 advising that the problem was affecting all clients connecting to their network via VPN. A further update at 14:53 suggested that they had narrowed focus to a firewall.
Further updates followed from the operator at 15:38 and 16:06 advising that engineers were still investigating.
Ziron monitoring showed service was restored at 16:07. At 16:25 we advised customers that service had been restored and we would continue to monitor. Notification from the operator of service restoration followed at 16:44, and we closed this incident at 16:47.
In a root cause analysis provided on 20th November 2018, our operator partner advised that the outage was initially caused by the software crash of a primary VPN firewall device. Whilst the failover to the secondary VPN firewall device at a second site went to plan, a missing VLAN configuration on a layer-2 switch at the second site meant that an extended outage was caused. The operator has advised that they are planning to complete the introduction of a second set of VPN firewall devices by the end of this month - a project that was already underway before this outage.