At 12:40 we noticed issues (packetloss) with communication between the routers and ring switches causing ebgp sessions to drop.

The issue seems to be with packets that were send/partially balanced over additional unused (disabled) ports, reset of the bridges on the ring-switches resolved the packetloss, BGP sessions are being re-established automatically.

As preventive measure NOC engineers will activate the currently unused (disabled) ports within the next 3 hours.

We have received a notification of a maintenance by one of our transit providers, GTT.

During their maintenance all GTT traffic will be automatically rerouted over other transit and peering connections.

Their maintenance reason is: "Improve network stability - software upgrade"

Their reference: 4598324

For transparency purposes we decided to publish this notification.

We have noticed an outage on the NLix Peering platform, Currently NL-ix is investigating the issue.

On their traffic graphs it is clearly visible an outage is visible https://www.nl-ix.net/network/traffic-stats/

Update 12:21:
We have received the following update from the NL-IX:

We are currently experiencing issues in our network. We are investigating the issue, it looks to be originating from our switch in Equinix FR4, but is causing packetlosss and traffic drops at multiple destinations in our network.

Update 14:40:
The following update was sent out by the NL-IX:
We found the port creating the issue on our switch in Equinix FR4, after shutting it down traffic is now restoring and the issue looks to be resolved.

Due to issues with iBGP we are forced to reload the routers during the following times:

Global Switch Amsterdam will be reloaded Thursday, 3 October 2019 at 0:00 AM CEST.
Equinix AM7 will be reloaded Saturday, 5 October 2019 at 0:00 AM CEST.

During the reload times, all traffic is redirected to the other datacenter and transit providers.

If you do not have a redundant BGP session, or have internet connectivity/transit at this location, you may notice full downtime during the reload.

Update 3 October 2019, 6:30 AM:
The maintenance is cancelled as it had no effect on Global Switch Amsterdam router.
However during the maintenance on Global Switch Amsterdam router, we did found the cruelpit on the AMSIX routeservers. Currently it is no longer present and the full routes are being send without any problems.

Update 4 October 2019, 8:30 AM:
The maintenance for AM7 is dropped as the issue was not resolved by reloading the router, but a conflicting peer on the AMSIX routeservers.

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance DE-CIX traffic will automatically reroute over peering and transit providers. We do not expect any issues.

Their maintenance reason is: "Cable works"

Maintenance location: Rees, Germany

Their reference: 4555588

For transparency purposes we decided to publish this notification.

Wave service provider which connects Equinix AM7 (EQX-AM7) towards DE-CIX is having a planned maintenance.

Their maintenance reason is "Emergency maintenance is needed because our provider detected a potential failure in their network therefore urgent action is required. These works will impact your service in the above mentioned service window."

KPN has given this maintenance reference numbers: 104439 and 104440

All DE-CIX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance LINX traffic will automatically reroute over other wave service providers. We do not expect any issues.

Their maintenance reason is: "HW Fix"

Their reference: 4469137

For transparency purposes we decided to publish this notification.

UPDATE 2019-09-04:
GTT has rescheduled this maintenance to 2019-09-07.

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance FRANCE-IX traffic will automatically reroute over other wave service providers. We do not expect any issues.

Their maintenance reason is: "Project description: NDIN14983 – reconstruction due to construction tunnel"

Their reference: 4458257

For transparency purposes we decided to publish this notification.

During this time we will do a preventive maintenance to ensure resilience of our platform between GSA and BNR.

The line between GSA and BNR will be temporary disabled during the maintenance, and the traffic will go over our DBC network.

The duration of the maintenance is approximately 1 hour, we do not expect any issues.

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance FRANCE-IX traffic will automatically reroute over peering and transit providers. We do not expect any issues.

Their maintenance reason is: "reconstruction due to construction of tunnel"

Maintenance location: Brussels

Their reference: 4451931

For transparency purposes we decided to publish this notification.

As the new firmware is running stable for the past weeks, we will be migrating back to the current platform to restore redundancy to IPv6.

The planning is as following:
NZS: 9 to 12 September 2019
DBA: 16 to 19 September 2019
DBC: 23 to 26 September 2019

The migration will take on these days between 6:30 and 9:00.

Per datacenter we planned out 4 days:
Day 1: Preparation work, no downtime expected.
Day 2: Moving transit/upstream providers, no downtime expected.
Day 3: Moving customer IPv6, you may notice downtime of a few seconds.
Day 4: Reserved day to complete the remainder of the work if maintenance cannot be completed within the first 3 days.

Update 2019-09-11 9:50:
Maintenance on NZS is completed.

Update 2019-09-17 15:00:
Maintenance on DBA is completed.

Update 2019-09-23 10:03:
Maintenance on DBC is completed early.

We have received a notification of a maintenance by one of our transit providers, Cogent.

During their maintenance traffic will automatically reroute over our Cogent connection at Equinix Amsterdam 5 and over other transit providers. We do not expect any issues.

Their maintenance reason is: "Linecard upgrade"

Their reference: EU5694

For transparency purposes we decided to publish this notification.

Friday 16th of August we will be upgrading our network capacity between Nedzone Steenbergen (NZS) and Globalswitch Amsterdam (GSA) to support further growth in the future.

We do not expect any issues as this is done up front by us preventative.

Update 2019-08-15:
Due to circumstances this upgrade has been rescheduled to 23th of August 2019.

Friday 9th of August we will be upgrading our network capacity between Nedzone Steenbergen (NZS) and Equinix AM7 (EQX-AM7) to support further growth in the future.

We do not expect any issues as this is done up front by us preventative.

Update 2019-08-09 14:00:
Due to a firmware mismatch between the chassis and the firmware of the blade this upgrade has been postponed to coming Tuesday 13th of August.

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance FRANCE-IX traffic will automatically reroute over other peering and transit providers. We do not expect any issues.

Their maintenance reason is: "Reroute"

Maintenance location: Nerville - France

Their reference: 4310408

For transparency purposes we decided to publish this notification.

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance DE-CIX traffic will automatically reroute over peering and transit providers. We do not expect any issues.

Their maintenance reason is: "Water infiltration and lots of broken trays (macrobending)"

Maintenance location: Germany

Their reference: 4292808

For transparency purposes we decided to publish this notification.

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance DE-CIX traffic will automatically reroute over other wave service providers. We do not expect any issues.

Their maintenance reason is: " Cable work "
Maintenance location: Germany

Their reference: 4302085

For transparency purposes we decided to publish this notification

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance FRANCE-IX traffic will automatically reroute over other wave service providers. We do not expect any issues.

Their maintenance reason is: " End contract , we must leave the existing HH on location "
Maintenance location: Roosendaal

Their reference: 4298643

For transparency purposes we decided to publish this notification

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance DE-CIX traffic will automatically reroute over other wave service providers. We do not expect any issues.

Their maintenance reason is: " Splice box replacement "
Maintenance location: Germany BB Ring2

Their reference: 4301833

For transparency purposes we decided to publish this notification

We have received reports of packetloss towards AS15542 (Delta).

We have denied the path over AMS-IX in the routing tables.

The preferred path is now going over Cogent.

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance DE-CIX traffic will automatically reroute over other wave service providers. We do not expect any issues.

Their maintenance reason is: " Cable maintenance "

Their reference: 4292504

For transparency purposes we decided to publish this notification.

UPDATE: this maintenance has been rescheduled by GTT.
Reschedule Reason: New date is provided by our local subcontractor. Sorry for inconvenience caused.

We have received reports of slow speeds towards AS41164 (GET Norway)

We have denied the path over Level 3 in the routing tables.

The best selected path is now going over NTT Communications (AS2914)

At 16.00 CEST we received a notification from our monitoring system that temperatures were rising inside the datacenter, we immediately contacted the datacenter and ask for any issues with the cooling system.

They had an issue with the cooling system which has been repaired at this moment,

We have received a notification of a maintenance by the Datacenter Global Switch Amsterdam (GSA) as they will be doing their "Annual Black Building test 2019".

The notification we received included the following:

The main purpose of this BBT is to test all full automatic functionality of all the emergency backup power systems, electrical distribution, and mechanical cooling systems.

- Before the works start all the technical installations will be checked. When there is a slight possibility for a failure or a deviant value (technically) the works won't start or will be aborted.
- The change level is assessed by GS OPS as a level 2 (Medium) as the supply by the main grid is shut down.

Their reference is: Change Control Form CCF2019-137

This is only an at risk notification, no outages are expected.

Wave service provider which connects Equinix AM7 (EQX-AM7) towards LINX is having a planned maintenance.

Their maintenance reason is "Maintenance needs to be performed on hardware in our network."

KPN has given this maintenance reference number: 102754

All LINX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

Wave service provider which connects Equinix AM7 (EQX-AM7) towards LINX is having a planned maintenance.

Their maintenance reason is "Maintenance needs to be performed on hardware in our network."

KPN has given this maintenance reference number: 102762

All LINX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance DE-CIX traffic will automatically reroute over other wave service providers. We do not expect any issues.

Their maintenance reason is: " faulty card replacement in our TX network "

Their reference: 4227382

For transparency purposes we decided to publish this notification.

At 15:40 CEST a darkfiber between NZS (Nedzone Steenbergen) and EQX-AM7 (Equinix AM7) went down, carrying 200 Gbps traffic.

Currently all traffic is sent over the alternative path on our ring namely NZS - GSA.

We have contacted the darkfiber provider Eurofiber and are currently awaiting an exact issue description.

Update 19:20 CEST:
The issue in this case is a fiber break because of digging in the area, the break location has been found and engineers are currently on their way to the break location to asses and repair the broken fiber duct.

Update 2019-07-03 0:05 CEST:
The fiber has been repaired by our darkfiber provider and the darkfiber has been returned to full operation. At this time Nedzone is fully redundant again.

We have received reports of packetloss towards AS21412 (Cgates).

We have denied the path over AMS-IX in the routing tables.

The preferred path is now going over Linx.

To avoid multiple mailings we have combined the maintenance into one post.

Between 15 July 2019 and 28 July 2019 we will be performing maintenances on our routers. During these times we will be upgrading the firmware patching several bugs that we have encountered in the past year. The full schedule for each firmware upgrade is as follow:

15 July 2019, 7:00 AM CEST - NZS R2 [Completed]
Affects: Routing Nedzone Datacenter, all routing will be done by R1. Packetloss may be observed when traffic will be moved over to Router 1.

16 July 2019, 7:00 AM CEST - DBA R2 [Completed]
Affects: Routing Databarn Amsterdam Datacenter, all routing will be done by R1. Packetloss may be observed when traffic will be moved over to Router 1.

17 July 2019, 7:00 AM CEST - DBC R2 [Completed]
Affects: Routing Databarn Capelle Datacenter, all routing will be done by R1. Packetloss may be observed when traffic will be moved over to Router 1.

21 July 2019, 1:00 AM CEST - EQX-AM7 [Completed]
Affects: Routing Equinix AM7, Internet customers terminated on this datacenter as well IP Transit/BGP. Traffic from other datacenters will be rerouted over GSA, Packet loss may be observed when traffic will be moved over to GSA. Note: If you do not have any redundancy on your connection in EQX-AM7, you may experience full downtime.

22 July 2019, 7:00 AM CEST - NZS R1 [Completed]
Affects: Routing Nedzone Datacenter, all routing will be done by R2. Packetloss may be observed when traffic will be moved over to Router 2.

23 July 2019, 7:00 AM CEST - DBA R1 [Completed]
Affects: Routing Databarn Amsterdam Datacenter, all routing will be done by R2. Packetloss may be observed when traffic will be moved over to Router 2.

24 July 2019, 7:00 AM CEST - DBC R1 [Completed]
Affects: Routing Databarn Capelle Datacenter, all routing will be done by R2. Packetloss may be observed when traffic will be moved over to Router 2.

28 July 2019, 1:00 AM CEST - GSA [Completed]
Affects: Routing Global Switch Amsterdam, Internet customers terminated on this datacenter as well IP Transit/BGP. Traffic from other datacenters will be rerouted over EQX-AM7, Packet loss may be observed when traffic will be moved over to EQX-AM7. Note: If you do not have any redundancy on your connection in GSA, you may experience full downtime.


We will update this post at each maintenance with the current status.

Update 2019-07-15, 7:45 AM CEST:
Maintenance and firmware upgrade of Router 2 of Nedzone is completed.

Update 2019-07-15, 8:00 AM CEST:
Maintenance and firmware upgrade of Router 2 of DBA is completed.

Update 2019-07-15, 8:00 AM CEST:
Maintenance and firmware upgrade of Router 2 of DBC is completed.

Update 2019-07-21, 4:00 AM CEST:
Maintenance and firmware upgrade of the Equinix-AM7 router is completed.

Update 2019-07-22, 8:00 AM CEST:
Maintenance and firmware upgrade of Router 1 of Nedzone is completed.

Update 2019-07-23, 8:00 AM CEST:
Maintenance and firmware upgrade of Router 1 of DBA is completed.

Update 2019-07-24, 8:00 AM CEST:
Maintenance and firmware upgrade of Router 1 of DBC is completed.

Update 2019-07-28, 4:10 AM CEST:
Maintenance and firmware upgrade of the Globalswitch router is completed.

We have received reports of packetloss towards AS48832 (Linkdotnet-Jordan).

We have denied the path over Seabone in the routing tables.

The preferred path is now going over GTT/Cogent.

Due to a software upgrade on the OnApp platform, we are rebooting the HVs during the coming two weeks.

We will performing live migrations to minimize downtime of virtual machines.

We have experienced a router linecard crash at 14:22, causing an outage on our network in Global Switch Amsterdam.

The linecard is back online and we have created a case at our vendor regarding to this crash.

Update 10-06-2019:
Vendor confirmed existing bug which is resolved with the next firmware. We are awaiting on the release of this firmware.

We have received reports of high latency towards AS5617 (Orange Polska).

We have denied the path over NTT in the routing tables.

The preferred path is now going over Cogent.

We have received reports of slow speeds towards AS7922 (Comcast Cable Communications, LLC (CCCS)).

We have changed the preferred path to Level 3 (AS3356).

We have received reports of packetloss towards AS132199 (Globe Telecom Inc., Philippines).

We have denied the path over Cogent and Seabone in the routing tables.

The best selected path is now going over Level 3.

We have received reports of packetloss towards AS34594 (Optima Telekom).

We have denied the path over the AMS-IX in the routing tables.

The best selected path is now going over Linx

Darkfiber provider Relined which connects Global Switch Amsterdam (GSA) towards Nikhef is having a planned maintenance. The connectivity between GSA and Nikhef will automatically reroute over our AM7 darkfiber during this maintenance.

Their maintenance reason is:
"Due to reconstruction work carried out by the local authorities. Relined is obliged to relocate a section of her network. This action will result in a temporary interruption of the service."

Their reference is: PA-361001

We have received reports of slow speeds towards Optimum Online (AS6128)

We have adjusted the path to go over Liberty Global as preferred route.

We received a maintenance notification from one of our server datacenters Databarn Capelle a/d IJssel (DBC) that they are having a pre-emptive power maintenance.

Their exact maintenance reason is:

General Electric / IMV Nederland will perform this maintenance, during this maintenance we will put both generators into operation.

As all our equipment will run on generators we expect no issues, this does not affect any server hosting customers.

On 27 May, 2019 we will be performing fiber maintenance at Global Switch Amsterdam.

The affected connections are:
- LINX
- AMSIX
- NLIX

Traffic towards these internet exchanges will be interrupted on Global Switch Amsterdam, and will be rerouted through Equinix AM7.

We do not expect any issues during this maintenance.

*This does not affect any hosting or transit customers*

Fiber to the business provider FORE Freedom is having a planned maintenance.

Their maintenance reason is:
We will be re-routing some cables and pipes as this is necessary to guarantee and improve the quality and reliability of our network.

Your connection may be interrupted during this maintenance.

This maintenance will take place on the night between 15 and 16 May and between the night of 16 and 17 May.

At 23:12 we received notification of the connection towards GSA (Global Switch Amsterdam) as well as in-house patch connections going down. After inspection we noticed this issue appears to be an in house issue at Nedzone.

At 23:54, we received notification the connection gone up again.

Currently we are still awaiting more information from Nedzone about the issue.

Update 28-04-2019 4:39:
The datacenter has updated us that the cause of this disruption was caused by a power cut. Our floor was unaffected but did cause equipment on the other floor, where our NZS - GSA Connection is connected on to go down.

At 15:25 we got warning alerts from our monitoring system that packetloss was noticed on multiple switches throughout the network in DBC.

After debugging we traced this back to one of the linecards in one of the coreswitches, as all ports on this card are setup redundantly we have rebooted this linecard.

After it went back online we first tested with a small number of racks to check the stability before enabling all the ports.

At this moment all ports have been enabled and stability has been resumed, we will keep monitoring this linecard and if the same issue occurs we will replace the linecard in question.

We have received reports of slow speeds towards AS577 (Bell Canada)

We have denied the path over Level 3 in the routing tables.

The best selected path is now going over GTT Communications (AS3257)

Darkfiber provider Relined which connects Global Switch Amsterdam (GSA) towards Nikhef is having a planned maintenance. The connectivity between GSA and Nikhef will automatically reroute over our AM7 darkfiber during this maintenance.

Their maintenance reason is:
"Due to reconstruction work carried out by the local authorities. Relined is obliged to relocate a section of her network. This action will result in a temporary interruption of the service."

Their reference is: PA-361001

Update 2019-04-24:
Darkfiber provider Relined has informed us that they have cancelled this maintenance, when they sent out a new notification we will send this as a new notification as well.

Wave service provider which connects Equinix AM7 (EQX-AM7) towards LINX is having a planned maintenance.

Their maintenance reason is "MAINTENANCE KPN, Fiber work, Preparation for move Lwst equipment."

KPN has given this maintenance reference numbers: 99536 and 99543

All LINX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

We have received reports of slow speeds towards AS6128 (Optimum Online)

We have denied the path over Corebackbone in the routing tables.

The best selected path is now going over Liberty Global (AS6830).

At 12:52 CEST we had a power outage in rack d4.z1 in Databarn Capelle, causing several servers and one switch to go down.

We sent one of our engineers over right away to assess the situation.
He has replaced a power distribution unit which seems to be defective.

Currently the rack is back online and all servers are operational again.

We have received reports of packet loss towards Hangzhou Alibaba Advertising Co.,Ltd (AS37963)

We have denied the path over NTT in the routing tables.

The best selected path is now going over GTT (AS3257).