We have received reports of high latency towards AS12876 (Online.net).

We have denied the path over FranceIX in the routing tables.

The preferred path is now going over Cogent.

Wave service provider which connects Globalswitch Amsterdam (GSA) towards DECIX is having a planned maintenance.

Their maintenance reason is "Service traffic restoration to original path after subsea cable break"

GTT has given this maintenance reference number: 4844916

All DECIX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

Wave service provider which connects Globalswitch Amsterdam (GSA) towards DECIX is having a planned maintenance.

Their maintenance reason is "Restoring traffic back to subsea cable which were moved away during FC TT#(4817514)"

GTT has given this maintenance reference number: 4843174

All DECIX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

We have received a notification of a maintenance by one of our transit providers, CenturyLink.

During their maintenance all CenturyLink traffic will be automatically rerouted over our other transits and peering connections.

For transparency purposes we decided to publish this notification.

Slow speeds towards AS6327 (Shaw Communications Inc.)

We have received reports of slow speeds towards AS6327 (Shaw Communications Inc.).

We have denied the path over Cogent in the routing tables.

The preferred path is now going over GTT.

Our wave service provider GTT reported that there is a possible fiber cut in the sea, disrupting our services between Global Switch Amsterdam and LINX.

All traffic towards LINX is now automatically rerouted over EQX-AM7.

Their reference: 4817514

Their current updates:
09-01-2020 at 3:30:
The damaged cable was found.
Repair works will start during good weather conditions.
Tentative ETR is 14th January.
Next update once every 24 hours or if any new changes appear.

13-01-2020 at 4:30:
All traffic has been restored and no further interruptions are expected.
If you continue to experience any issues with your dark fiber service, please check your RAMAN amplifiers first and in case no issue observed, feel free to contact our Support team on the mentioned phone numbers at the bottom of the email.

We can confirm that all traffic carried over our DWDM network is restored since 1:17 UTC.
We are pending confirmation from the on-site engineers that the customers' equipment on site is connected and powered one.
Once that is confirmed, dark fiber customers will be contacted for confirmation on the status of their service.

07-01-2020 at 1:19:
NMC managed to protect several more lines. Further migration options are reviewed. Repair ship has departed last night and if the weather allows, restoration work will commence this evening.

All possible migrations were applied.
GTT will review if other migration options are possible.
Next update will be available tomorrow.

Cut is confirmed to be 121 km away from Zandvoort side.
Colocation services will be shut down in order to operate normally with the cable.
Tier 3 engineers are reviewing options for traffic migration, depending on the available capacity in the network.
The whole cable is cut, so dark fiber service migration is not possible.
Repair ship dispatch is confirmed by the contractor. Departure expected later in the evening/tomorrow, depending on the completion of the spares loading.

06-01-2020 at 6:00:
Fault Description:Fiber Break
Leiston-Zandvoort

GTT is experiencing an outage on the subsea section of our transmission network between UK and Netherlands.

Initial Update:
Fibre break detected between Leiston and Zandvoort.
Our systems detect the cut at 91.4km from Leiston.
Engineers are sent to both locations for precise measurement and dispatch of the repair ship.
No ETA yet.

For transparency purposes we decided to publish this notification.

Wave service provider which connects Globalswitch Amsterdam (GSA) towards DECIX is having a planned maintenance.

Their maintenance reason is "Cable works"

GTT has given this maintenance reference number: 4814175

All DECIX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

We have received a notification of a maintenance by one of our transit providers, NTT.

During their maintenance all NTT traffic will be automatically rerouted over our other NTT connection, other transits and peering connections.

Their maintenance reason is: "software upgrade"

Their reference: GIN-1-2243661620

For transparency purposes we decided to publish this notification.

We have received a notification of a maintenance by one of our transit providers, Centurylink.

During their maintenance all Centurylink traffic will be automatically rerouted over our other transits and peering connections.

For transparency purposes we decided to publish this notification.

Update 2020-01-06: Centurylink notified that this maintenance has been postponed.

Currently we are having an outage on the rack b2.z5 in Databarn Amsterdam (DBA).

We've gotten a datacenter engineer on route to check on the issue at hand, once we have an update we'll update this post as well.

Update 18:02:
There was a server in this same rack which PSU caused a short circuit on the breaker of this PDU, after the engineer enabled the breaker the PDU came back online.

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance FRANCE-IX traffic will automatically reroute over other wave service providers. We do not expect any issues.

Their maintenance reason is: "Cable migration"

Their reference: 4793343

For transparency purposes we decided to publish this notification.

We have received reports of packetloss towards AS58321 (Intergrid Group Pty Ltd).

We have denied the path over Cogent in the routing tables.

The preferred path is now going over GTT.

We have received reports of packetloss towards AS58321 (OXYLION-AS).

We have denied the path over AMS-IX in the routing tables.

The preferred path is now going over GTT.

We have received reports of slow speeds towards Eniig Fiber A/S (AS43557)

We have denied the path over NTT in the routing tables.

The best selected path is now going over Cogent

Wave service provider which connects Global switch amsterdam (GSA) towards DECIX is having a planned maintenance.

Their maintenance reason is "Cable work near to your fibers. Maintenance is service threatening."

Their reference is: 4749799

All DECIX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

6 December 2019 Update: The wave service provider cancelled this maintenance.

Wave service provider which connects Equinix AM7 (EQX-AM7) towards DECIX is having a planned maintenance.

Their maintenance reason is "Our provider informed us that, in order to guarantee the stability of their network, they need to conduct maintenance activities that will affect the above-mentioned service."

KPN has given this maintenance reference numbers: 107170 and 107171

All DECIX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

We have received reports of slow speeds towards Deutsche Telekom (AS3320)

We have denied the path over Core-backbone in the routing tables.

The best selected path is now going over NTT (AS2914), Seabone (AS6762) and Liberty Global (AS6830)

We have received a notification of a maintenance by one of our transit providers, NTT.

During their maintenance all NTT traffic will be automatically rerouted over our other NTT connection, other transits and peering connections.

Their maintenance reason is: "software upgrade"

Their reference: GIN-1-2202702512

For transparency purposes we decided to publish this notification.

Update 17:51:
NTT has sent out an update that the maintenance was rescheduled to 18th of December, we've updated the message accordingly.


Update 21:39:
NTT has sent out another update adjusting the time from 0:00 UTC - 0:04 UTC (1:00 AM CET to 1:04 AM CET) to 0:00 UTC 4:00 UTC (1 AM CET to 5 AM CET).

At 12:33 CET we noticed an issue on our VPLS ring through EQX-AM7.

During debugging it got very unstable to the point we have rebooted the entire stack.
We are still debugging as to what triggered the outage in the first place together with our vendor.

At the moment everything should be stable again but we continue to monitor the situation for the next couple of hours.

Update 13:30 CET:
Internet Customers may still experience issues with packetloss at this moment, our team is currently working around the clock on getting this solved as well.

Update 14:30 CET:
Also for Internet customers the issue should now be resolved.

Update 15:00 CET:
From investigation, the instability was present since the power outage of equinix AM7, during the black building test.

Wave service provider which connects Equinix AM7 (EQX-AM7) towards LINX is having a planned maintenance.

Their maintenance reason is "Maintenance needs to be performed on hardware in our network."

KPN has given this maintenance reference numbers: 106662 and 106664

All LINX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

At 22:09 CET we noticed a complete outage on our Equinix Amsterdam 7 datacenter which appeared to be because of an outage during their annual blackbuilding test.

Equinix AM7 is a big connecting hub for Europe and the Netherlands and while our hosting datacenters did not have any issues but because this was such a big outage for a lot of party's involved at Equinix there will be many re-routes throughout the internet and a lot of packetloss happening or happened.

Currently we are experiencing packet loss in the datacenter Databarn Amsterdam.

We have found the culprit to be with one of the core ports and have disabled this port returning everything to stable.

As our suspicion is a faulty optic, we have planned in a replacement of the optic. This will cause no downtime.

Darkfiber provider Relined which connects Equinix Amsterdam 7 (EQX-AM7) towards Nikhef is having a planned maintenance.

Their reason for this maintenance:
"Precautionary maintenance work. This work represents preventative maintenance work, Relined is obliged to carry out this work to maintain the quality of our fiber network."

Their references is: PA-150002.

At 4 December, 2019, between 10:00 and 14:00, we will have a maintenance, upgrading our ring capacity between NZS and EQX-AM7.

The maintenance will take about 10 minutes and traffic will be rerouted over GSA during this time.

We do not expect any issues during this time.

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance FRANCE-IX traffic will automatically reroute over other wave service providers. We do not expect any issues.

Their maintenance reason is: "Lift and shift"

Their reference: 4704965

For transparency purposes we decided to publish this notification.

6 December 2019:
An update was received: The maintenance is currently postponed.

Darkfiber provider Relined which connects Global Switch Amsterdam (GSA) towards Nikhef is having a planned maintenance.

Their reason for this maintenance:
"Precautionary maintenance work. This work represents preventative maintenance work, Relined is obliged to carry out this work to maintain the quality of our fiber network."

Their references is: PA-261001.

Update 2019-11-18:
This maintenance has been canceled by Relined, once they reschedule this maintenance we will send out a new notification.

Currently we are experiencing an outage on the connection towards EQX-AM7 from DBC, carrying 400gbps of connectivity.

At this moment we are inspecting the issue and keep this post updated with our findings.

UPDATE 12-11-2019, 14:40:
Currently the connection is back online after 10 minutes. We are still investigating the root cause of signal loss. As the signal was lost, we suspect the issue at our dark fiber provider. We have contacted them to investigate at their end.

We have received reports of slow speeds towards Lyse Tele AS/AltiBox (AS29695)

We have denied the path over Level 3 in the routing tables.

The best selected path is now going over NTT (AS2914)

Darkfiber provider Relined which connects Global Switch Amsterdam (GSA) towards Nikhef is having a planned maintenance.

They have mentioned that this is an at risk maintenance for this traject.

No maintenance reason was given.

Their references are: PA-121001, PA-336003

Darkfiber provider Relined which connects Equinix Amsterdam 7 (EQX-AM7) towards Nikhef is having a planned maintenance.
They have mentioned that this is an at risk maintenance for this traject.

No maintenance reason was given.

Their reference is: PA-294004

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance DE-CIX traffic will automatically reroute over peering and transit providers. We do not expect any issues.

Their maintenance reason is: "Water infiltration and lots of broken trays(macrobending)"

Maintenance location: Germany

Their reference: 4292808

For transparency purposes we decided to publish this notification.

Wave service provider which connects Equinix AM7 (EQX-AM7) towards LINX is having a planned maintenance.

Their maintenance reason is "Our provider informed us that, in order to guarantee the stability of their network, they need to conduct maintenance on the fiber network."

KPN has given this maintenance reference numbers: 105867 and 105847

All LINX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

We have received reports of slow speeds towards Smart Broadband, Inc. (AS10139)

We have denied the path over Seabone in the routing tables.

The best selected path is now going over Cogent (AS174)

We have received reports of slow speeds towards TPG Internet (AS7545)

We have denied the path over Seabone in the routing tables.

The best selected path is now going over Cogent (AS174)

At 12:40 we noticed issues (packetloss) with communication between the routers and ring switches causing ebgp sessions to drop.

The issue seems to be with packets that were send/partially balanced over additional unused (disabled) ports, reset of the bridges on the ring-switches resolved the packetloss, BGP sessions are being re-established automatically.

As preventive measure NOC engineers will activate the currently unused (disabled) ports within the next 3 hours.

We have received a notification of a maintenance by one of our transit providers, GTT.

During their maintenance all GTT traffic will be automatically rerouted over other transit and peering connections.

Their maintenance reason is: "Improve network stability - software upgrade"

Their reference: 4598324

For transparency purposes we decided to publish this notification.

We have noticed an outage on the NLix Peering platform, Currently NL-ix is investigating the issue.

On their traffic graphs it is clearly visible an outage is visible https://www.nl-ix.net/network/traffic-stats/

Update 12:21:
We have received the following update from the NL-IX:

We are currently experiencing issues in our network. We are investigating the issue, it looks to be originating from our switch in Equinix FR4, but is causing packetlosss and traffic drops at multiple destinations in our network.

Update 14:40:
The following update was sent out by the NL-IX:
We found the port creating the issue on our switch in Equinix FR4, after shutting it down traffic is now restoring and the issue looks to be resolved.

Due to issues with iBGP we are forced to reload the routers during the following times:

Global Switch Amsterdam will be reloaded Thursday, 3 October 2019 at 0:00 AM CEST.
Equinix AM7 will be reloaded Saturday, 5 October 2019 at 0:00 AM CEST.

During the reload times, all traffic is redirected to the other datacenter and transit providers.

If you do not have a redundant BGP session, or have internet connectivity/transit at this location, you may notice full downtime during the reload.

Update 3 October 2019, 6:30 AM:
The maintenance is cancelled as it had no effect on Global Switch Amsterdam router.
However during the maintenance on Global Switch Amsterdam router, we did found the cruelpit on the AMSIX routeservers. Currently it is no longer present and the full routes are being send without any problems.

Update 4 October 2019, 8:30 AM:
The maintenance for AM7 is dropped as the issue was not resolved by reloading the router, but a conflicting peer on the AMSIX routeservers.

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance DE-CIX traffic will automatically reroute over peering and transit providers. We do not expect any issues.

Their maintenance reason is: "Cable works"

Maintenance location: Rees, Germany

Their reference: 4555588

For transparency purposes we decided to publish this notification.

Wave service provider which connects Equinix AM7 (EQX-AM7) towards DE-CIX is having a planned maintenance.

Their maintenance reason is "Emergency maintenance is needed because our provider detected a potential failure in their network therefore urgent action is required. These works will impact your service in the above mentioned service window."

KPN has given this maintenance reference numbers: 104439 and 104440

All DE-CIX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance LINX traffic will automatically reroute over other wave service providers. We do not expect any issues.

Their maintenance reason is: "HW Fix"

Their reference: 4469137

For transparency purposes we decided to publish this notification.

UPDATE 2019-09-04:
GTT has rescheduled this maintenance to 2019-09-07.

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance FRANCE-IX traffic will automatically reroute over other wave service providers. We do not expect any issues.

Their maintenance reason is: "Project description: NDIN14983 – reconstruction due to construction tunnel"

Their reference: 4458257

For transparency purposes we decided to publish this notification.

During this time we will do a preventive maintenance to ensure resilience of our platform between GSA and BNR.

The line between GSA and BNR will be temporary disabled during the maintenance, and the traffic will go over our DBC network.

The duration of the maintenance is approximately 1 hour, we do not expect any issues.

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance FRANCE-IX traffic will automatically reroute over peering and transit providers. We do not expect any issues.

Their maintenance reason is: "reconstruction due to construction of tunnel"

Maintenance location: Brussels

Their reference: 4451931

For transparency purposes we decided to publish this notification.

As the new firmware is running stable for the past weeks, we will be migrating back to the current platform to restore redundancy to IPv6.

The planning is as following:
NZS: 9 to 12 September 2019
DBA: 16 to 19 September 2019
DBC: 23 to 26 September 2019

The migration will take on these days between 6:30 and 9:00.

Per datacenter we planned out 4 days:
Day 1: Preparation work, no downtime expected.
Day 2: Moving transit/upstream providers, no downtime expected.
Day 3: Moving customer IPv6, you may notice downtime of a few seconds.
Day 4: Reserved day to complete the remainder of the work if maintenance cannot be completed within the first 3 days.

Update 2019-09-11 9:50:
Maintenance on NZS is completed.

Update 2019-09-17 15:00:
Maintenance on DBA is completed.

Update 2019-09-23 10:03:
Maintenance on DBC is completed early.

We have received a notification of a maintenance by one of our transit providers, Cogent.

During their maintenance traffic will automatically reroute over our Cogent connection at Equinix Amsterdam 5 and over other transit providers. We do not expect any issues.

Their maintenance reason is: "Linecard upgrade"

Their reference: EU5694

For transparency purposes we decided to publish this notification.

Friday 16th of August we will be upgrading our network capacity between Nedzone Steenbergen (NZS) and Globalswitch Amsterdam (GSA) to support further growth in the future.

We do not expect any issues as this is done up front by us preventative.

Update 2019-08-15:
Due to circumstances this upgrade has been rescheduled to 23th of August 2019.

Friday 9th of August we will be upgrading our network capacity between Nedzone Steenbergen (NZS) and Equinix AM7 (EQX-AM7) to support further growth in the future.

We do not expect any issues as this is done up front by us preventative.

Update 2019-08-09 14:00:
Due to a firmware mismatch between the chassis and the firmware of the blade this upgrade has been postponed to coming Tuesday 13th of August.

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance FRANCE-IX traffic will automatically reroute over other peering and transit providers. We do not expect any issues.

Their maintenance reason is: "Reroute"

Maintenance location: Nerville - France

Their reference: 4310408

For transparency purposes we decided to publish this notification.

We have received a notification of a maintenance by one of our wave service providers, GTT.

During their maintenance DE-CIX traffic will automatically reroute over peering and transit providers. We do not expect any issues.

Their maintenance reason is: "Water infiltration and lots of broken trays (macrobending)"

Maintenance location: Germany

Their reference: 4292808

For transparency purposes we decided to publish this notification.