As the new firmware is running stable for the past weeks, we will be migrating back to the current platform to restore redundancy to IPv6.

The planning is as following:
NZS: 9 to 12 September 2019
DBA: 16 to 19 September 2019
DBC: 23 to 26 September 2019

The migration will take on these days between 6:30 and 9:00.

Per datacenter we planned out 4 days:
Day 1: Preparation work, no downtime expected.
Day 2: Moving transit/upstream providers, no downtime expected.
Day 3: Moving customer IPv6, you may notice downtime of a few seconds.
Day 4: Reserved day to complete the remainder of the work if maintenance cannot be completed within the first 3 days.

We have received a notification of a maintenance by one of our transit providers, Cogent.

During their maintenance traffic will automatically reroute over our Cogent connection at Equinix Amsterdam 5 and over other transit providers. We do not expect any issues.

Their maintenance reason is: "Linecard upgrade"

Their reference: EU5694

For transparency purposes we decided to publish this notification.

Friday 16th of August we will be upgrading our network capacity between Nedzone Steenbergen (NZS) and Globalswitch Amsterdam (GSA) to support further growth in the future.

We do not expect any issues as this is done up front by us preventative.

Update 2019-08-15:
Due to circumstances this upgrade has been rescheduled to 23th of August 2019.

Friday 9th of August we will be upgrading our network capacity between Nedzone Steenbergen (NZS) and Equinix AM7 (EQX-AM7) to support further growth in the future.

We do not expect any issues as this is done up front by us preventative.

Update 2019-08-09 14:00:
Due to a firmware mismatch between the chassis and the firmware of the blade this upgrade has been postponed to coming Tuesday 13th of August.

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance FRANCE-IX traffic will automatically reroute over other peering and transit providers. We do not expect any issues.

Their maintenance reason is: "Reroute"

Maintenance location: Nerville - France

Their reference: 4310408

For transparency purposes we decided to publish this notification.

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance DE-CIX traffic will automatically reroute over peering and transit providers. We do not expect any issues.

Their maintenance reason is: "Water infiltration and lots of broken trays (macrobending)"

Maintenance location: Germany

Their reference: 4292808

For transparency purposes we decided to publish this notification.

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance DE-CIX traffic will automatically reroute over other wave service providers. We do not expect any issues.

Their maintenance reason is: " Cable work "
Maintenance location: Germany

Their reference: 4302085

For transparency purposes we decided to publish this notification

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance FRANCE-IX traffic will automatically reroute over other wave service providers. We do not expect any issues.

Their maintenance reason is: " End contract , we must leave the existing HH on location "
Maintenance location: Roosendaal

Their reference: 4298643

For transparency purposes we decided to publish this notification

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance DE-CIX traffic will automatically reroute over other wave service providers. We do not expect any issues.

Their maintenance reason is: " Splice box replacement "
Maintenance location: Germany BB Ring2

Their reference: 4301833

For transparency purposes we decided to publish this notification

We have received reports of packetloss towards AS15542 (Delta).

We have denied the path over AMS-IX in the routing tables.

The preferred path is now going over Cogent.

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance DE-CIX traffic will automatically reroute over other wave service providers. We do not expect any issues.

Their maintenance reason is: " Cable maintenance "

Their reference: 4292504

For transparency purposes we decided to publish this notification.

UPDATE: this maintenance has been rescheduled by GTT.
Reschedule Reason: New date is provided by our local subcontractor. Sorry for inconvenience caused.

We have received reports of slow speeds towards AS41164 (GET Norway)

We have denied the path over Level 3 in the routing tables.

The best selected path is now going over NTT Communications (AS2914)

At 16.00 CEST we received a notification from our monitoring system that temperatures were rising inside the datacenter, we immediately contacted the datacenter and ask for any issues with the cooling system.

They had an issue with the cooling system which has been repaired at this moment,

We have received a notification of a maintenance by the Datacenter Global Switch Amsterdam (GSA) as they will be doing their "Annual Black Building test 2019".

The notification we received included the following:

The main purpose of this BBT is to test all full automatic functionality of all the emergency backup power systems, electrical distribution, and mechanical cooling systems.

- Before the works start all the technical installations will be checked. When there is a slight possibility for a failure or a deviant value (technically) the works won't start or will be aborted.
- The change level is assessed by GS OPS as a level 2 (Medium) as the supply by the main grid is shut down.

Their reference is: Change Control Form CCF2019-137

This is only an at risk notification, no outages are expected.

Wave service provider which connects Equinix AM7 (EQX-AM7) towards LINX is having a planned maintenance.

Their maintenance reason is "Maintenance needs to be performed on hardware in our network."

KPN has given this maintenance reference number: 102754

All LINX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

Wave service provider which connects Equinix AM7 (EQX-AM7) towards LINX is having a planned maintenance.

Their maintenance reason is "Maintenance needs to be performed on hardware in our network."

KPN has given this maintenance reference number: 102762

All LINX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance DE-CIX traffic will automatically reroute over other wave service providers. We do not expect any issues.

Their maintenance reason is: " faulty card replacement in our TX network "

Their reference: 4227382

For transparency purposes we decided to publish this notification.

At 15:40 CEST a darkfiber between NZS (Nedzone Steenbergen) and EQX-AM7 (Equinix AM7) went down, carrying 200 Gbps traffic.

Currently all traffic is sent over the alternative path on our ring namely NZS - GSA.

We have contacted the darkfiber provider Eurofiber and are currently awaiting an exact issue description.

Update 19:20 CEST:
The issue in this case is a fiber break because of digging in the area, the break location has been found and engineers are currently on their way to the break location to asses and repair the broken fiber duct.

Update 2019-07-03 0:05 CEST:
The fiber has been repaired by our darkfiber provider and the darkfiber has been returned to full operation. At this time Nedzone is fully redundant again.

We have received reports of packetloss towards AS21412 (Cgates).

We have denied the path over AMS-IX in the routing tables.

The preferred path is now going over Linx.

To avoid multiple mailings we have combined the maintenance into one post.

Between 15 July 2019 and 28 July 2019 we will be performing maintenances on our routers. During these times we will be upgrading the firmware patching several bugs that we have encountered in the past year. The full schedule for each firmware upgrade is as follow:

15 July 2019, 7:00 AM CEST - NZS R2 [Completed]
Affects: Routing Nedzone Datacenter, all routing will be done by R1. Packetloss may be observed when traffic will be moved over to Router 1.

16 July 2019, 7:00 AM CEST - DBA R2 [Completed]
Affects: Routing Databarn Amsterdam Datacenter, all routing will be done by R1. Packetloss may be observed when traffic will be moved over to Router 1.

17 July 2019, 7:00 AM CEST - DBC R2 [Completed]
Affects: Routing Databarn Capelle Datacenter, all routing will be done by R1. Packetloss may be observed when traffic will be moved over to Router 1.

21 July 2019, 1:00 AM CEST - EQX-AM7 [Completed]
Affects: Routing Equinix AM7, Internet customers terminated on this datacenter as well IP Transit/BGP. Traffic from other datacenters will be rerouted over GSA, Packet loss may be observed when traffic will be moved over to GSA. Note: If you do not have any redundancy on your connection in EQX-AM7, you may experience full downtime.

22 July 2019, 7:00 AM CEST - NZS R1 [Completed]
Affects: Routing Nedzone Datacenter, all routing will be done by R2. Packetloss may be observed when traffic will be moved over to Router 2.

23 July 2019, 7:00 AM CEST - DBA R1 [Completed]
Affects: Routing Databarn Amsterdam Datacenter, all routing will be done by R2. Packetloss may be observed when traffic will be moved over to Router 2.

24 July 2019, 7:00 AM CEST - DBC R1 [Completed]
Affects: Routing Databarn Capelle Datacenter, all routing will be done by R2. Packetloss may be observed when traffic will be moved over to Router 2.

28 July 2019, 1:00 AM CEST - GSA [Completed]
Affects: Routing Global Switch Amsterdam, Internet customers terminated on this datacenter as well IP Transit/BGP. Traffic from other datacenters will be rerouted over EQX-AM7, Packet loss may be observed when traffic will be moved over to EQX-AM7. Note: If you do not have any redundancy on your connection in GSA, you may experience full downtime.


We will update this post at each maintenance with the current status.

Update 2019-07-15, 7:45 AM CEST:
Maintenance and firmware upgrade of Router 2 of Nedzone is completed.

Update 2019-07-15, 8:00 AM CEST:
Maintenance and firmware upgrade of Router 2 of DBA is completed.

Update 2019-07-15, 8:00 AM CEST:
Maintenance and firmware upgrade of Router 2 of DBC is completed.

Update 2019-07-21, 4:00 AM CEST:
Maintenance and firmware upgrade of the Equinix-AM7 router is completed.

Update 2019-07-22, 8:00 AM CEST:
Maintenance and firmware upgrade of Router 1 of Nedzone is completed.

Update 2019-07-23, 8:00 AM CEST:
Maintenance and firmware upgrade of Router 1 of DBA is completed.

Update 2019-07-24, 8:00 AM CEST:
Maintenance and firmware upgrade of Router 1 of DBC is completed.

Update 2019-07-28, 4:10 AM CEST:
Maintenance and firmware upgrade of the Globalswitch router is completed.

We have received reports of packetloss towards AS48832 (Linkdotnet-Jordan).

We have denied the path over Seabone in the routing tables.

The preferred path is now going over GTT/Cogent.

Due to a software upgrade on the OnApp platform, we are rebooting the HVs during the coming two weeks.

We will performing live migrations to minimize downtime of virtual machines.

We have experienced a router linecard crash at 14:22, causing an outage on our network in Global Switch Amsterdam.

The linecard is back online and we have created a case at our vendor regarding to this crash.

Update 10-06-2019:
Vendor confirmed existing bug which is resolved with the next firmware. We are awaiting on the release of this firmware.

We have received reports of high latency towards AS5617 (Orange Polska).

We have denied the path over NTT in the routing tables.

The preferred path is now going over Cogent.

We have received reports of slow speeds towards AS7922 (Comcast Cable Communications, LLC (CCCS)).

We have changed the preferred path to Level 3 (AS3356).

We have received reports of packetloss towards AS132199 (Globe Telecom Inc., Philippines).

We have denied the path over Cogent and Seabone in the routing tables.

The best selected path is now going over Level 3.

We have received reports of packetloss towards AS34594 (Optima Telekom).

We have denied the path over the AMS-IX in the routing tables.

The best selected path is now going over Linx

Darkfiber provider Relined which connects Global Switch Amsterdam (GSA) towards Nikhef is having a planned maintenance. The connectivity between GSA and Nikhef will automatically reroute over our AM7 darkfiber during this maintenance.

Their maintenance reason is:
"Due to reconstruction work carried out by the local authorities. Relined is obliged to relocate a section of her network. This action will result in a temporary interruption of the service."

Their reference is: PA-361001

We have received reports of slow speeds towards Optimum Online (AS6128)

We have adjusted the path to go over Liberty Global as preferred route.

We received a maintenance notification from one of our server datacenters Databarn Capelle a/d IJssel (DBC) that they are having a pre-emptive power maintenance.

Their exact maintenance reason is:

General Electric / IMV Nederland will perform this maintenance, during this maintenance we will put both generators into operation.

As all our equipment will run on generators we expect no issues, this does not affect any server hosting customers.

On 27 May, 2019 we will be performing fiber maintenance at Global Switch Amsterdam.

The affected connections are:
- LINX
- AMSIX
- NLIX

Traffic towards these internet exchanges will be interrupted on Global Switch Amsterdam, and will be rerouted through Equinix AM7.

We do not expect any issues during this maintenance.

*This does not affect any hosting or transit customers*

Fiber to the business provider FORE Freedom is having a planned maintenance.

Their maintenance reason is:
We will be re-routing some cables and pipes as this is necessary to guarantee and improve the quality and reliability of our network.

Your connection may be interrupted during this maintenance.

This maintenance will take place on the night between 15 and 16 May and between the night of 16 and 17 May.

At 23:12 we received notification of the connection towards GSA (Global Switch Amsterdam) as well as in-house patch connections going down. After inspection we noticed this issue appears to be an in house issue at Nedzone.

At 23:54, we received notification the connection gone up again.

Currently we are still awaiting more information from Nedzone about the issue.

Update 28-04-2019 4:39:
The datacenter has updated us that the cause of this disruption was caused by a power cut. Our floor was unaffected but did cause equipment on the other floor, where our NZS - GSA Connection is connected on to go down.

At 15:25 we got warning alerts from our monitoring system that packetloss was noticed on multiple switches throughout the network in DBC.

After debugging we traced this back to one of the linecards in one of the coreswitches, as all ports on this card are setup redundantly we have rebooted this linecard.

After it went back online we first tested with a small number of racks to check the stability before enabling all the ports.

At this moment all ports have been enabled and stability has been resumed, we will keep monitoring this linecard and if the same issue occurs we will replace the linecard in question.

We have received reports of slow speeds towards AS577 (Bell Canada)

We have denied the path over Level 3 in the routing tables.

The best selected path is now going over GTT Communications (AS3257)

Darkfiber provider Relined which connects Global Switch Amsterdam (GSA) towards Nikhef is having a planned maintenance. The connectivity between GSA and Nikhef will automatically reroute over our AM7 darkfiber during this maintenance.

Their maintenance reason is:
"Due to reconstruction work carried out by the local authorities. Relined is obliged to relocate a section of her network. This action will result in a temporary interruption of the service."

Their reference is: PA-361001

Update 2019-04-24:
Darkfiber provider Relined has informed us that they have cancelled this maintenance, when they sent out a new notification we will send this as a new notification as well.

Wave service provider which connects Equinix AM7 (EQX-AM7) towards LINX is having a planned maintenance.

Their maintenance reason is "MAINTENANCE KPN, Fiber work, Preparation for move Lwst equipment."

KPN has given this maintenance reference numbers: 99536 and 99543

All LINX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

We have received reports of slow speeds towards AS6128 (Optimum Online)

We have denied the path over Corebackbone in the routing tables.

The best selected path is now going over Liberty Global (AS6830).

At 12:52 CEST we had a power outage in rack d4.z1 in Databarn Capelle, causing several servers and one switch to go down.

We sent one of our engineers over right away to assess the situation.
He has replaced a power distribution unit which seems to be defective.

Currently the rack is back online and all servers are operational again.

We have received reports of packet loss towards Hangzhou Alibaba Advertising Co.,Ltd (AS37963)

We have denied the path over NTT in the routing tables.

The best selected path is now going over GTT (AS3257).

Wave service provider which connects Global Switch Amsterdam (GSA) towards LINX is having a planned maintenance.

Their maintenance reason is " As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing hardware reconfiguration works on our transmission platform. This planned activity will take place in Netherlands. "

Interoute has given this maintenance reference number: 1-4885581691

All LINX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

Darkfiber provider Eurofiber which connects NZS (Nedzone Steenbergen) towards Bytesnet Rotterdam (BNR) is having a planned maintenance. The connectivity between NZS and BNR will automatically reroute over our EQX-AM7 darkfiber during this maintenance.

Their maintenance reason is:
Following last weeks optical issues we will be doing maintenance on this line to make sure issues like last week do not happen again.

At 01:00 CEST a darkfiber between DBC (Datbarn Capelle) and EQX-AM7 (Equinix AM7) went down, carrying 400 Gbps traffic.

Currently all traffic is sent over the alternative path on our ring namely DBC - GSA while we are investigating the cause into this.

Update 2019-04-13 03:32 CEST:
We've spoken to the darkfiber provider euNetworks and this is because of a maintenance which was not communicated to us.

Update 2019-04-14 09:00 CEST:
Currently the line is back online and our redundancy is operational again.

We have received reports of slow speeds towards AS4773 (M1 ISP)

We have denied the path over AMSIX in the routing tables.

The best selected path is now going over GTT (AS3257).

Our dark fiber provider has informed us about an urgent maintenance occurring today between Steenbergen and Rotterdam due to bad optical values.

They will be performing this during the day. The connection may be interrupted but as we are setup redundant, our other connection will take it over.

Their message is:
"Wij ervaren sinds gisterenmiddag een hogere demping op het glasvezeltraject tussen Steenbergen en Rotterdam. Aangezien de waardes sinds vanmorgen verder zijn verslechterd zijn wij genoodzaakt om met spoed metingen en reparatiewerkzaamheden te laten uitvoeren door de leverancier van deze glasvezels.
De kans is dus aanzienlijk dat het traject Steenbergen > Rotterdam in de loop van de dag voor enige tijd niet beschikbaar zal zijn. "

Translation:
"Since yesterday afternoon we have experienced higher values on the fiber optic route between Steenbergen and Rotterdam. Since the values have deteriorated further since this morning, we are forced to have measurements and repair work carried out urgently by the supplier of these glass fibers.
The chance is therefore considerable that the Steenbergen> Rotterdam route will not be available for some time during the course of the day."

Update 2019-04-09 15:05 CEST:
The maintenance done by our darkfiber provider has been completed and the line is fully functioning again.

Today at 10:47 CEST, our uplinks with LINX at GSA (Global Switch Amsterdam).

We have investigated the issue at our side but concluded that the issue is either at LINX side, or at our wave service provider.

We have contacted our wave service provider at 11:00 CEST, but have not yet received any updates.

Currently, all traffic towards LINX is routed through our other ports at EQX-AM7 (Amsterdam).

Once we have an update, we will update this post.

Update 2019-04-07 12:47 CEST:
The wave service provider found out the issue causing our link to be down:
It is connected to the subsea fiber break PI 1-4858348241 between Leiston – Zandvoort.

Update 2019-04-07 15:12 CEST:
Dear Customer, Please be informed that the fault is confirmed to be on 89.6km from our landing station in Netherlands. Currently the cable ship is on the port, preparing and loading all the spares needed for the restoration. No ETA or ETR. IMPORTANT: If you are a dark fiber customer, please SHUT DOWN your raman amplifiers. We need this step due to safety reasons.

Update 2019-04-07 16:28 CEST:
The ship is already prepared for the spares loading operation. The operation will start in 30 minutes. Once the ship is ready for sailing and have approval for it, we will provide tentative ETA to the fault location.

Update 2019-04-07 20:20 CEST:
Loading of the spares onto the ship is currently ongoing. In the meantime we are also waiting for approval in order for the ship to set sail to the fault location. As this is a time-consuming process it will take some time. Once we receive the approval we will share a tentative ETA to the fault location and possibly a tentative ETR.

Update 2019-04-08 1:42 CEST:
Loading of spares completed. Repair ship has left the port at 23:00 UTC.

Update 2019-04-08 10:50 CEST:
The Sovereign cable ship left port as planned at midnight last night. It is expected on the cable grounds at the fault location at approximately midnight tonight. Estimated date of repair completion 15/04/19. Next update will be available in 12 hours.

Update 2019-04-09 14:23 CEST:
As our lines are currently up, running on a backup line we do not experience any direct issues.
However, we do expect a switch over and keep this post updated for possible major incidents.
They have reported estimated date of repair completion 15/04/19.

Update 2019-04-15 07:30 CEST:
Please be informed that the cable fault was fixed already.
All traffic is recovered.

This notification is only for specific customers with equipment at our Equinix AM5 location.

Please be advised that Equinix AM5 will be performing preventive maintenance on the main low-voltage distribution panel B. There are no servers housed here.
All our own equipment in this location is powered redundantly and we should not notice this maintenance.

Notification from AM5:

Please be advised that Equinix will be performing preventive maintenance on the main low-voltage distribution panel B.

This maintenance involves switching the primary or redundant power supply(s) to multiple cabinets off for a time frame of 8 hours. After the maintenance is completed the power supply will be switched back on.

Wave service provider which connects Equinix AM7 (EQX-AM7) towards LINX is having a planned maintenance.

Their maintenance reason is "Relocating cables due to road construction works. Reconstruction of knooppunt De Nieuwe Meer."

KPN has given this maintenance reference numbers: 98825 and 98826

All LINX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

We have detected an issue on the active management module in the router of GSA (Global Switch Amsterdam).

At 2 April 2019, 13:00 CEST, we will be performing a switchover to the standby management as per recommendation from vendor to avoid a hard crash of the router.

After reviewing the configuration together with the vendor, all options are set to minimize impact of a switchover.

Wave service provider which connects Global Switch Amsterdam (GSA) towards FRANCEIX is having a planned maintenance.

Their maintenance reason is " As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing mandatory cable works on our fiberoptical network. This planned activity will take place in France."

Interoute has given this maintenance reference number: 1-4818939751

All FRANCEIX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

UPDATE 2019-04-22 10:00 CEST
Please be informed that this Planned Work has been postponed until further notice. As soon as we will have information about the new time window we will notify you.

UPDATE 2019-04-23 16:30 CEST
Reschedule