At 12:40 we noticed issues (packetloss) with communication between the routers and ring switches causing ebgp sessions to drop.

The issue seems to be with packets that were send/partially balanced over additional unused (disabled) ports, reset of the bridges on the ring-switches resolved the packetloss, BGP sessions are being re-established automatically.

As preventive measure NOC engineers will activate the currently unused (disabled) ports within the next 3 hours.

We have noticed an outage on the NLix Peering platform, Currently NL-ix is investigating the issue.

On their traffic graphs it is clearly visible an outage is visible https://www.nl-ix.net/network/traffic-stats/

Update 12:21:
We have received the following update from the NL-IX:

We are currently experiencing issues in our network. We are investigating the issue, it looks to be originating from our switch in Equinix FR4, but is causing packetlosss and traffic drops at multiple destinations in our network.

Update 14:40:
The following update was sent out by the NL-IX:
We found the port creating the issue on our switch in Equinix FR4, after shutting it down traffic is now restoring and the issue looks to be resolved.

At 16.00 CEST we received a notification from our monitoring system that temperatures were rising inside the datacenter, we immediately contacted the datacenter and ask for any issues with the cooling system.

They had an issue with the cooling system which has been repaired at this moment,

At 15:40 CEST a darkfiber between NZS (Nedzone Steenbergen) and EQX-AM7 (Equinix AM7) went down, carrying 200 Gbps traffic.

Currently all traffic is sent over the alternative path on our ring namely NZS - GSA.

We have contacted the darkfiber provider Eurofiber and are currently awaiting an exact issue description.

Update 19:20 CEST:
The issue in this case is a fiber break because of digging in the area, the break location has been found and engineers are currently on their way to the break location to asses and repair the broken fiber duct.

Update 2019-07-03 0:05 CEST:
The fiber has been repaired by our darkfiber provider and the darkfiber has been returned to full operation. At this time Nedzone is fully redundant again.

We have experienced a router linecard crash at 14:22, causing an outage on our network in Global Switch Amsterdam.

The linecard is back online and we have created a case at our vendor regarding to this crash.

Update 10-06-2019:
Vendor confirmed existing bug which is resolved with the next firmware. We are awaiting on the release of this firmware.