At 12:33 CET we noticed an issue on our VPLS ring through EQX-AM7.

During debugging it got very unstable to the point we have rebooted the entire stack.
We are still debugging as to what triggered the outage in the first place together with our vendor.

At the moment everything should be stable again but we continue to monitor the situation for the next couple of hours.

Update 13:30 CET:
Internet Customers may still experience issues with packetloss at this moment, our team is currently working around the clock on getting this solved as well.

Update 14:30 CET:
Also for Internet customers the issue should now be resolved.

Update 15:00 CET:
From investigation, the instability was present since the power outage of equinix AM7, during the black building test.

At 22:09 CET we noticed a complete outage on our Equinix Amsterdam 7 datacenter which appeared to be because of an outage during their annual blackbuilding test.

Equinix AM7 is a big connecting hub for Europe and the Netherlands and while our hosting datacenters did not have any issues but because this was such a big outage for a lot of party's involved at Equinix there will be many re-routes throughout the internet and a lot of packetloss happening or happened.

Currently we are experiencing packet loss in the datacenter Databarn Amsterdam.

We have found the culprit to be with one of the core ports and have disabled this port returning everything to stable.

As our suspicion is a faulty optic, we have planned in a replacement of the optic. This will cause no downtime.

Currently we are experiencing an outage on the connection towards EQX-AM7 from DBC, carrying 400gbps of connectivity.

At this moment we are inspecting the issue and keep this post updated with our findings.

UPDATE 12-11-2019, 14:40:
Currently the connection is back online after 10 minutes. We are still investigating the root cause of signal loss. As the signal was lost, we suspect the issue at our dark fiber provider. We have contacted them to investigate at their end.

At 12:40 we noticed issues (packetloss) with communication between the routers and ring switches causing ebgp sessions to drop.

The issue seems to be with packets that were send/partially balanced over additional unused (disabled) ports, reset of the bridges on the ring-switches resolved the packetloss, BGP sessions are being re-established automatically.

As preventive measure NOC engineers will activate the currently unused (disabled) ports within the next 3 hours.