Wave service provider which connects Global switch amsterdam (GSA) towards DECIX is having a planned maintenance.

Their maintenance reason is "Cable work near to your fibers. Maintenance is service threatening."

Their reference is: 4749799

All DECIX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

Wave service provider which connects Equinix AM7 (EQX-AM7) towards DECIX is having a planned maintenance.

Their maintenance reason is "Our provider informed us that, in order to guarantee the stability of their network, they need to conduct maintenance activities that will affect the above-mentioned service."

KPN has given this maintenance reference numbers: 107170 and 107171

All DECIX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

We have received reports of slow speeds towards Deutsche Telekom (AS3320)

We have denied the path over Core-backbone in the routing tables.

The best selected path is now going over NTT (AS2914), Seabone (AS6762) and Liberty Global (AS6830)

We have received a notification of a maintenance by one of our transit providers, NTT.

During their maintenance all NTT traffic will be automatically rerouted over our other NTT connection, other transits and peering connections.

Their maintenance reason is: "software upgrade"

Their reference: GIN-1-2202702512

For transparency purposes we decided to publish this notification.

Update 17:51:
NTT has sent out an update that the maintenance was rescheduled to 18th of December, we've updated the message accordingly.


Update 21:39:
NTT has sent out another update adjusting the time from 0:00 UTC - 0:04 UTC (1:00 AM CET to 1:04 AM CET) to 0:00 UTC 4:00 UTC (1 AM CET to 5 AM CET).

At 12:33 CET we noticed an issue on our VPLS ring through EQX-AM7.

During debugging it got very unstable to the point we have rebooted the entire stack.
We are still debugging as to what triggered the outage in the first place together with our vendor.

At the moment everything should be stable again but we continue to monitor the situation for the next couple of hours.

Update 13:30 CET:
Internet Customers may still experience issues with packetloss at this moment, our team is currently working around the clock on getting this solved as well.

Update 14:30 CET:
Also for Internet customers the issue should now be resolved.

Update 15:00 CET:
From investigation, the instability was present since the power outage of equinix AM7, during the black building test.

Wave service provider which connects Equinix AM7 (EQX-AM7) towards LINX is having a planned maintenance.

Their maintenance reason is "Maintenance needs to be performed on hardware in our network."

KPN has given this maintenance reference numbers: 106662 and 106664

All LINX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

At 22:09 CET we noticed a complete outage on our Equinix Amsterdam 7 datacenter which appeared to be because of an outage during their annual blackbuilding test.

Equinix AM7 is a big connecting hub for Europe and the Netherlands and while our hosting datacenters did not have any issues but because this was such a big outage for a lot of party's involved at Equinix there will be many re-routes throughout the internet and a lot of packetloss happening or happened.

Currently we are experiencing packet loss in the datacenter Databarn Amsterdam.

We have found the culprit to be with one of the core ports and have disabled this port returning everything to stable.

As our suspicion is a faulty optic, we have planned in a replacement of the optic. This will cause no downtime.

Darkfiber provider Relined which connects Equinix Amsterdam 7 (EQX-AM7) towards Nikhef is having a planned maintenance.

Their reason for this maintenance:
"Precautionary maintenance work. This work represents preventative maintenance work, Relined is obliged to carry out this work to maintain the quality of our fiber network."

Their references is: PA-150002.

At 4 December, 2019, between 10:00 and 14:00, we will have a maintenance, upgrading our ring capacity between NZS and EQX-AM7.

The maintenance will take about 10 minutes and traffic will be rerouted over GSA during this time.

We do not expect any issues during this time.

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance FRANCE-IX traffic will automatically reroute over other wave service providers. We do not expect any issues.

Their maintenance reason is: "Lift and shift"

Their reference: 4704965

For transparency purposes we decided to publish this notification.

Darkfiber provider Relined which connects Global Switch Amsterdam (GSA) towards Nikhef is having a planned maintenance.

Their reason for this maintenance:
"Precautionary maintenance work. This work represents preventative maintenance work, Relined is obliged to carry out this work to maintain the quality of our fiber network."

Their references is: PA-261001.

Update 2019-11-18:
This maintenance has been canceled by Relined, once they reschedule this maintenance we will send out a new notification.

Currently we are experiencing an outage on the connection towards EQX-AM7 from DBC, carrying 400gbps of connectivity.

At this moment we are inspecting the issue and keep this post updated with our findings.

UPDATE 12-11-2019, 14:40:
Currently the connection is back online after 10 minutes. We are still investigating the root cause of signal loss. As the signal was lost, we suspect the issue at our dark fiber provider. We have contacted them to investigate at their end.

We have received reports of slow speeds towards Lyse Tele AS/AltiBox (AS29695)

We have denied the path over Level 3 in the routing tables.

The best selected path is now going over NTT (AS2914)

Darkfiber provider Relined which connects Global Switch Amsterdam (GSA) towards Nikhef is having a planned maintenance.

They have mentioned that this is an at risk maintenance for this traject.

No maintenance reason was given.

Their references are: PA-121001, PA-336003

Darkfiber provider Relined which connects Equinix Amsterdam 7 (EQX-AM7) towards Nikhef is having a planned maintenance.
They have mentioned that this is an at risk maintenance for this traject.

No maintenance reason was given.

Their reference is: PA-294004

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance DE-CIX traffic will automatically reroute over peering and transit providers. We do not expect any issues.

Their maintenance reason is: "Water infiltration and lots of broken trays(macrobending)"

Maintenance location: Germany

Their reference: 4292808

For transparency purposes we decided to publish this notification.

Wave service provider which connects Equinix AM7 (EQX-AM7) towards LINX is having a planned maintenance.

Their maintenance reason is "Our provider informed us that, in order to guarantee the stability of their network, they need to conduct maintenance on the fiber network."

KPN has given this maintenance reference numbers: 105867 and 105847

All LINX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

We have received reports of slow speeds towards Smart Broadband, Inc. (AS10139)

We have denied the path over Seabone in the routing tables.

The best selected path is now going over Cogent (AS174)

We have received reports of slow speeds towards TPG Internet (AS7545)

We have denied the path over Seabone in the routing tables.

The best selected path is now going over Cogent (AS174)

At 12:40 we noticed issues (packetloss) with communication between the routers and ring switches causing ebgp sessions to drop.

The issue seems to be with packets that were send/partially balanced over additional unused (disabled) ports, reset of the bridges on the ring-switches resolved the packetloss, BGP sessions are being re-established automatically.

As preventive measure NOC engineers will activate the currently unused (disabled) ports within the next 3 hours.

We have received a notification of a maintenance by one of our transit providers, GTT.

During their maintenance all GTT traffic will be automatically rerouted over other transit and peering connections.

Their maintenance reason is: "Improve network stability - software upgrade"

Their reference: 4598324

For transparency purposes we decided to publish this notification.

We have noticed an outage on the NLix Peering platform, Currently NL-ix is investigating the issue.

On their traffic graphs it is clearly visible an outage is visible https://www.nl-ix.net/network/traffic-stats/

Update 12:21:
We have received the following update from the NL-IX:

We are currently experiencing issues in our network. We are investigating the issue, it looks to be originating from our switch in Equinix FR4, but is causing packetlosss and traffic drops at multiple destinations in our network.

Update 14:40:
The following update was sent out by the NL-IX:
We found the port creating the issue on our switch in Equinix FR4, after shutting it down traffic is now restoring and the issue looks to be resolved.

Due to issues with iBGP we are forced to reload the routers during the following times:

Global Switch Amsterdam will be reloaded Thursday, 3 October 2019 at 0:00 AM CEST.
Equinix AM7 will be reloaded Saturday, 5 October 2019 at 0:00 AM CEST.

During the reload times, all traffic is redirected to the other datacenter and transit providers.

If you do not have a redundant BGP session, or have internet connectivity/transit at this location, you may notice full downtime during the reload.

Update 3 October 2019, 6:30 AM:
The maintenance is cancelled as it had no effect on Global Switch Amsterdam router.
However during the maintenance on Global Switch Amsterdam router, we did found the cruelpit on the AMSIX routeservers. Currently it is no longer present and the full routes are being send without any problems.

Update 4 October 2019, 8:30 AM:
The maintenance for AM7 is dropped as the issue was not resolved by reloading the router, but a conflicting peer on the AMSIX routeservers.

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance DE-CIX traffic will automatically reroute over peering and transit providers. We do not expect any issues.

Their maintenance reason is: "Cable works"

Maintenance location: Rees, Germany

Their reference: 4555588

For transparency purposes we decided to publish this notification.

Wave service provider which connects Equinix AM7 (EQX-AM7) towards DE-CIX is having a planned maintenance.

Their maintenance reason is "Emergency maintenance is needed because our provider detected a potential failure in their network therefore urgent action is required. These works will impact your service in the above mentioned service window."

KPN has given this maintenance reference numbers: 104439 and 104440

All DE-CIX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance LINX traffic will automatically reroute over other wave service providers. We do not expect any issues.

Their maintenance reason is: "HW Fix"

Their reference: 4469137

For transparency purposes we decided to publish this notification.

UPDATE 2019-09-04:
GTT has rescheduled this maintenance to 2019-09-07.

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance FRANCE-IX traffic will automatically reroute over other wave service providers. We do not expect any issues.

Their maintenance reason is: "Project description: NDIN14983 – reconstruction due to construction tunnel"

Their reference: 4458257

For transparency purposes we decided to publish this notification.

During this time we will do a preventive maintenance to ensure resilience of our platform between GSA and BNR.

The line between GSA and BNR will be temporary disabled during the maintenance, and the traffic will go over our DBC network.

The duration of the maintenance is approximately 1 hour, we do not expect any issues.

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance FRANCE-IX traffic will automatically reroute over peering and transit providers. We do not expect any issues.

Their maintenance reason is: "reconstruction due to construction of tunnel"

Maintenance location: Brussels

Their reference: 4451931

For transparency purposes we decided to publish this notification.

As the new firmware is running stable for the past weeks, we will be migrating back to the current platform to restore redundancy to IPv6.

The planning is as following:
NZS: 9 to 12 September 2019
DBA: 16 to 19 September 2019
DBC: 23 to 26 September 2019

The migration will take on these days between 6:30 and 9:00.

Per datacenter we planned out 4 days:
Day 1: Preparation work, no downtime expected.
Day 2: Moving transit/upstream providers, no downtime expected.
Day 3: Moving customer IPv6, you may notice downtime of a few seconds.
Day 4: Reserved day to complete the remainder of the work if maintenance cannot be completed within the first 3 days.

Update 2019-09-11 9:50:
Maintenance on NZS is completed.

Update 2019-09-17 15:00:
Maintenance on DBA is completed.

Update 2019-09-23 10:03:
Maintenance on DBC is completed early.

We have received a notification of a maintenance by one of our transit providers, Cogent.

During their maintenance traffic will automatically reroute over our Cogent connection at Equinix Amsterdam 5 and over other transit providers. We do not expect any issues.

Their maintenance reason is: "Linecard upgrade"

Their reference: EU5694

For transparency purposes we decided to publish this notification.

Friday 16th of August we will be upgrading our network capacity between Nedzone Steenbergen (NZS) and Globalswitch Amsterdam (GSA) to support further growth in the future.

We do not expect any issues as this is done up front by us preventative.

Update 2019-08-15:
Due to circumstances this upgrade has been rescheduled to 23th of August 2019.

Friday 9th of August we will be upgrading our network capacity between Nedzone Steenbergen (NZS) and Equinix AM7 (EQX-AM7) to support further growth in the future.

We do not expect any issues as this is done up front by us preventative.

Update 2019-08-09 14:00:
Due to a firmware mismatch between the chassis and the firmware of the blade this upgrade has been postponed to coming Tuesday 13th of August.

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance FRANCE-IX traffic will automatically reroute over other peering and transit providers. We do not expect any issues.

Their maintenance reason is: "Reroute"

Maintenance location: Nerville - France

Their reference: 4310408

For transparency purposes we decided to publish this notification.

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance DE-CIX traffic will automatically reroute over peering and transit providers. We do not expect any issues.

Their maintenance reason is: "Water infiltration and lots of broken trays (macrobending)"

Maintenance location: Germany

Their reference: 4292808

For transparency purposes we decided to publish this notification.

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance DE-CIX traffic will automatically reroute over other wave service providers. We do not expect any issues.

Their maintenance reason is: " Cable work "
Maintenance location: Germany

Their reference: 4302085

For transparency purposes we decided to publish this notification

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance FRANCE-IX traffic will automatically reroute over other wave service providers. We do not expect any issues.

Their maintenance reason is: " End contract , we must leave the existing HH on location "
Maintenance location: Roosendaal

Their reference: 4298643

For transparency purposes we decided to publish this notification

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance DE-CIX traffic will automatically reroute over other wave service providers. We do not expect any issues.

Their maintenance reason is: " Splice box replacement "
Maintenance location: Germany BB Ring2

Their reference: 4301833

For transparency purposes we decided to publish this notification

We have received reports of packetloss towards AS15542 (Delta).

We have denied the path over AMS-IX in the routing tables.

The preferred path is now going over Cogent.

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance DE-CIX traffic will automatically reroute over other wave service providers. We do not expect any issues.

Their maintenance reason is: " Cable maintenance "

Their reference: 4292504

For transparency purposes we decided to publish this notification.

UPDATE: this maintenance has been rescheduled by GTT.
Reschedule Reason: New date is provided by our local subcontractor. Sorry for inconvenience caused.

We have received reports of slow speeds towards AS41164 (GET Norway)

We have denied the path over Level 3 in the routing tables.

The best selected path is now going over NTT Communications (AS2914)

At 16.00 CEST we received a notification from our monitoring system that temperatures were rising inside the datacenter, we immediately contacted the datacenter and ask for any issues with the cooling system.

They had an issue with the cooling system which has been repaired at this moment,

We have received a notification of a maintenance by the Datacenter Global Switch Amsterdam (GSA) as they will be doing their "Annual Black Building test 2019".

The notification we received included the following:

The main purpose of this BBT is to test all full automatic functionality of all the emergency backup power systems, electrical distribution, and mechanical cooling systems.

- Before the works start all the technical installations will be checked. When there is a slight possibility for a failure or a deviant value (technically) the works won't start or will be aborted.
- The change level is assessed by GS OPS as a level 2 (Medium) as the supply by the main grid is shut down.

Their reference is: Change Control Form CCF2019-137

This is only an at risk notification, no outages are expected.

Wave service provider which connects Equinix AM7 (EQX-AM7) towards LINX is having a planned maintenance.

Their maintenance reason is "Maintenance needs to be performed on hardware in our network."

KPN has given this maintenance reference number: 102754

All LINX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

Wave service provider which connects Equinix AM7 (EQX-AM7) towards LINX is having a planned maintenance.

Their maintenance reason is "Maintenance needs to be performed on hardware in our network."

KPN has given this maintenance reference number: 102762

All LINX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance DE-CIX traffic will automatically reroute over other wave service providers. We do not expect any issues.

Their maintenance reason is: " faulty card replacement in our TX network "

Their reference: 4227382

For transparency purposes we decided to publish this notification.

At 15:40 CEST a darkfiber between NZS (Nedzone Steenbergen) and EQX-AM7 (Equinix AM7) went down, carrying 200 Gbps traffic.

Currently all traffic is sent over the alternative path on our ring namely NZS - GSA.

We have contacted the darkfiber provider Eurofiber and are currently awaiting an exact issue description.

Update 19:20 CEST:
The issue in this case is a fiber break because of digging in the area, the break location has been found and engineers are currently on their way to the break location to asses and repair the broken fiber duct.

Update 2019-07-03 0:05 CEST:
The fiber has been repaired by our darkfiber provider and the darkfiber has been returned to full operation. At this time Nedzone is fully redundant again.

We have received reports of packetloss towards AS21412 (Cgates).

We have denied the path over AMS-IX in the routing tables.

The preferred path is now going over Linx.

To avoid multiple mailings we have combined the maintenance into one post.

Between 15 July 2019 and 28 July 2019 we will be performing maintenances on our routers. During these times we will be upgrading the firmware patching several bugs that we have encountered in the past year. The full schedule for each firmware upgrade is as follow:

15 July 2019, 7:00 AM CEST - NZS R2 [Completed]
Affects: Routing Nedzone Datacenter, all routing will be done by R1. Packetloss may be observed when traffic will be moved over to Router 1.

16 July 2019, 7:00 AM CEST - DBA R2 [Completed]
Affects: Routing Databarn Amsterdam Datacenter, all routing will be done by R1. Packetloss may be observed when traffic will be moved over to Router 1.

17 July 2019, 7:00 AM CEST - DBC R2 [Completed]
Affects: Routing Databarn Capelle Datacenter, all routing will be done by R1. Packetloss may be observed when traffic will be moved over to Router 1.

21 July 2019, 1:00 AM CEST - EQX-AM7 [Completed]
Affects: Routing Equinix AM7, Internet customers terminated on this datacenter as well IP Transit/BGP. Traffic from other datacenters will be rerouted over GSA, Packet loss may be observed when traffic will be moved over to GSA. Note: If you do not have any redundancy on your connection in EQX-AM7, you may experience full downtime.

22 July 2019, 7:00 AM CEST - NZS R1 [Completed]
Affects: Routing Nedzone Datacenter, all routing will be done by R2. Packetloss may be observed when traffic will be moved over to Router 2.

23 July 2019, 7:00 AM CEST - DBA R1 [Completed]
Affects: Routing Databarn Amsterdam Datacenter, all routing will be done by R2. Packetloss may be observed when traffic will be moved over to Router 2.

24 July 2019, 7:00 AM CEST - DBC R1 [Completed]
Affects: Routing Databarn Capelle Datacenter, all routing will be done by R2. Packetloss may be observed when traffic will be moved over to Router 2.

28 July 2019, 1:00 AM CEST - GSA [Completed]
Affects: Routing Global Switch Amsterdam, Internet customers terminated on this datacenter as well IP Transit/BGP. Traffic from other datacenters will be rerouted over EQX-AM7, Packet loss may be observed when traffic will be moved over to EQX-AM7. Note: If you do not have any redundancy on your connection in GSA, you may experience full downtime.


We will update this post at each maintenance with the current status.

Update 2019-07-15, 7:45 AM CEST:
Maintenance and firmware upgrade of Router 2 of Nedzone is completed.

Update 2019-07-15, 8:00 AM CEST:
Maintenance and firmware upgrade of Router 2 of DBA is completed.

Update 2019-07-15, 8:00 AM CEST:
Maintenance and firmware upgrade of Router 2 of DBC is completed.

Update 2019-07-21, 4:00 AM CEST:
Maintenance and firmware upgrade of the Equinix-AM7 router is completed.

Update 2019-07-22, 8:00 AM CEST:
Maintenance and firmware upgrade of Router 1 of Nedzone is completed.

Update 2019-07-23, 8:00 AM CEST:
Maintenance and firmware upgrade of Router 1 of DBA is completed.

Update 2019-07-24, 8:00 AM CEST:
Maintenance and firmware upgrade of Router 1 of DBC is completed.

Update 2019-07-28, 4:10 AM CEST:
Maintenance and firmware upgrade of the Globalswitch router is completed.