Our wave service provider GTT reported that there is a possible fiber cut in the sea, disrupting our services between Global Switch Amsterdam and LINX.

All traffic towards LINX is now automatically rerouted over EQX-AM7.

Their reference: 4817514

Their current updates:
09-01-2020 at 3:30:
The damaged cable was found.
Repair works will start during good weather conditions.
Tentative ETR is 14th January.
Next update once every 24 hours or if any new changes appear.

13-01-2020 at 4:30:
All traffic has been restored and no further interruptions are expected.
If you continue to experience any issues with your dark fiber service, please check your RAMAN amplifiers first and in case no issue observed, feel free to contact our Support team on the mentioned phone numbers at the bottom of the email.

We can confirm that all traffic carried over our DWDM network is restored since 1:17 UTC.
We are pending confirmation from the on-site engineers that the customers' equipment on site is connected and powered one.
Once that is confirmed, dark fiber customers will be contacted for confirmation on the status of their service.

07-01-2020 at 1:19:
NMC managed to protect several more lines. Further migration options are reviewed. Repair ship has departed last night and if the weather allows, restoration work will commence this evening.

All possible migrations were applied.
GTT will review if other migration options are possible.
Next update will be available tomorrow.

Cut is confirmed to be 121 km away from Zandvoort side.
Colocation services will be shut down in order to operate normally with the cable.
Tier 3 engineers are reviewing options for traffic migration, depending on the available capacity in the network.
The whole cable is cut, so dark fiber service migration is not possible.
Repair ship dispatch is confirmed by the contractor. Departure expected later in the evening/tomorrow, depending on the completion of the spares loading.

06-01-2020 at 6:00:
Fault Description:Fiber Break
Leiston-Zandvoort

GTT is experiencing an outage on the subsea section of our transmission network between UK and Netherlands.

Initial Update:
Fibre break detected between Leiston and Zandvoort.
Our systems detect the cut at 91.4km from Leiston.
Engineers are sent to both locations for precise measurement and dispatch of the repair ship.
No ETA yet.

For transparency purposes we decided to publish this notification.

Currently we are having an outage on the rack b2.z5 in Databarn Amsterdam (DBA).

We've gotten a datacenter engineer on route to check on the issue at hand, once we have an update we'll update this post as well.

Update 18:02:
There was a server in this same rack which PSU caused a short circuit on the breaker of this PDU, after the engineer enabled the breaker the PDU came back online.

At 12:33 CET we noticed an issue on our VPLS ring through EQX-AM7.

During debugging it got very unstable to the point we have rebooted the entire stack.
We are still debugging as to what triggered the outage in the first place together with our vendor.

At the moment everything should be stable again but we continue to monitor the situation for the next couple of hours.

Update 13:30 CET:
Internet Customers may still experience issues with packetloss at this moment, our team is currently working around the clock on getting this solved as well.

Update 14:30 CET:
Also for Internet customers the issue should now be resolved.

Update 15:00 CET:
From investigation, the instability was present since the power outage of equinix AM7, during the black building test.

At 22:09 CET we noticed a complete outage on our Equinix Amsterdam 7 datacenter which appeared to be because of an outage during their annual blackbuilding test.

Equinix AM7 is a big connecting hub for Europe and the Netherlands and while our hosting datacenters did not have any issues but because this was such a big outage for a lot of party's involved at Equinix there will be many re-routes throughout the internet and a lot of packetloss happening or happened.

Currently we are experiencing packet loss in the datacenter Databarn Amsterdam.

We have found the culprit to be with one of the core ports and have disabled this port returning everything to stable.

As our suspicion is a faulty optic, we have planned in a replacement of the optic. This will cause no downtime.