At 21:17 we noticed a management card flap at our second router at NZS.

We are currently investigating into this as our BGP are loading out again.

Update 01:10:
We continue to see the management cards swap (both the active / standby) on this router at random intervals. Router 1 is carrying all the customer load at the moment while we debug this issue.

Update 10:30:
We have replaced the management cards of R2 and monitored the situation. After replacing the management cards, we have no longer seen any swapping between active/standby card.

We expect the issue is resolved however we still continue to monitor the situation during the day.

Due to a range of issues we are going to do an emergency reload of our router in EQX-AM7.

Internet Customers will experience a loss of internet as well as customers having only one BGP session. Customers having a redundant BGP session with us should not experience any issues.

We are aware it is the start of a workday for most but due to the on-going issues there is little choice left for us to do it at a very short timespan, we will try our best to keep the downtime to an absolute minimum.

Update 10:25
We have finished the reload and restored BGP sessions. The main cause for this emergency reload was the router no longer propagating new routes nor pruning old routes.

Around 12:00 we've noticed our connections from GSA towards Nedzone Steenbergen (NZS) and Bytesnet Rotterdam (BNR) went down.

Our darkfiber provider GTT has informed us of a fibercut between Amsterdam and Rotterdam, as both BNR and NZS makes use of this darkfiber they have had a loss of redundancy.

All traffic for NZS is re-routed over our darkfiber with EQX-AM7 / all traffic for BNR is re-routed over DBC.

Wave customers between NZS and GSA are currently down.

Update 1: 2021-09-04 10:23 UTC
Please be advised that the GTT Infra NOC detected an interruption occurring on our network between Amsterdam & Rotterdam at 09:18 UTC. Initial investigation has detected a potential fiber cut occurring. FLM teams have been engaged and are en route with an ETA of 90 mins to site.

Update 2: 2021-09-04 11:29 UTC
FLM teams report that they have now reached site & are beginning their investigations.
Once the OTDR has been shot crews will travel to the break point & ETTR will be made available following a damage assessment.

Update 3: 2021-09-04 12:23 UTC
Crews on site have shot the OTDR & discovered the location of the break to be occurring approx 450 meters outside of Amsterdam. Teams are en route to the break point now with an ETA on site set at roughly 60 minutes.

Update 4: 2021-09-04 14:05 UTC
Crews have now arrived at the break point & discovered the fiber impacted by 3rd party drilling activity on site. A damage assessment & repair plan is being formulated by the FLM crew & civils work will need to be undertaken to excavate the damaged cable. ETTR will become available once complete.

Update 5: 2021-09-04 15:15 UTC
Crews report that civils crews are now on site & are preparing to excavate the damaged fiber. Repair plan will include the introduction of two new handholds along the route of the fiber. ETTR will become available once civils work has been completed & the additional infrastructure deployed to provide a safe route around the ongoing drilling work.

Update 6: 2021-09-04 16:25 UTC
Crews report that excavation work on site is progressing well with splicing to begin at a tentative time of 18:30 UTC.

Update 7: 2021-09-04 17:42 UTC
Crews on site report that excavation work has now completed & they are prepping the fiber for splicing which will commence momentarily.

Final Update: 2021-09-04 18:56 UTC
FLM teams confirm that all fibers have now been spliced & the GTT NOC can confirm that all alarms have cleared with services restoring.
The actions taken by the team on site will constitute a permanent repair & no further disruption to service is anticipated.

We've became aware of multiple issues affecting a part of our Internet customers and are currently investigating.

Once we have an update it will be posted here.

Update 11:45: An engineer has been dispatched to replace a possibly faulty optic.

Update 14:50: An engineer from the external party is underway to troubleshoot the link.

Update 17:22:
At the moment our port is up again, we are going to request a formal RFO with Eurofiber since a downtime of 8 hours is not normal for the issue we were experiencing.

We've noticed some packet loss in NZS.

One of the optics in a bundle of multiple Hundred Gbit optics started giving issues without going down causing packet loss, because of it not going down the issue was hard to pin-point.

We have now disabled the troubling link and will schedule a debug session of this link without affecting production traffic.