All Notifications

#689 Power maintenance Equinix-AM5

Posted: 2018-08-15 18:25

Start: 2018-09-10 07:30:00
End : 2018-09-10 09:00:00

Affects: Equipment located at Equinix-AM5

We received a maintenance notification from one of our infrastructure datacenters Equinix-AM5 that they are having their 3 yearly power maintenance.

Their exact maintenance reason is:
Please be advised that Equinix will temporarily switch off one of the redundant power supplies to some or all of your cabinets within AM5 for approximately 60 minutes. The remaining power supply to your cabinets will stay available including UPS and Emergency Generator backup.

We are performing 3 yearly maintenance activities to our main breakers.

Only one feed will be de-energized at to any one time to perform the maintenance.

As all our equipment is setup redundantly we expect no issues, this does not affect any server hosting customers.

their reference is 5-165605442492

#688 Slow transfer rate towards Google Fiber (AS16591)

Posted: 2018-08-14 09:49

Start: 2018-08-14 09:46:00
End : 2018-08-14 09:47:00

Affects: Affects: Traffic towards Google Fiber (AS16591)

We have received reports of slow transfer rates towards Google Fiber (AS16591)

We have denied the path over GTT in the routing tables.

Best selected path is now going over Level 3.

#687 Preparations core switch upgrade

Posted: 2018-08-03 12:34

Start: 2018-08-09 05:30:00
End : 2018-08-29 06:30:00

Affects: DBA Zone 4 and Zone 5

For future expansion to 40/100gbit towards the zones in DBA (Databarn Amsterdam), We will be performing preparations to Zone 4 and Zone 5 at 9 August 2018 and 14 August 2018.

We will be performing the following steps:

- At 9 August 2018, 5:30 AM CEST, we will preform preparations on the cores of Zone 4 and Zone 5 at IRF Stack level.
There should be no interruptions expected during this time.

- At 14 August 2018, 5:30 AM CEST, we will be moving physical ports to the new core infrastructure, supporting 40/100G connections between zones. (Postponed)
At this time, you may experience interruption of network connections for a brief moment (between 5 to 10 seconds).

- At 23 August 2018, 3:30 AM CEST, we will be upgrading the firmware of Zone 4 core switch and moving the physical ports to the new core infrastructure. At this time you may experience interruption of network connections for a brief moment (up to 1 minute). We will try to keep the downtime as low as possible.

- At 29 August 2018, 3:30 AM CEST, we will be upgrading the firmware of Zone 5 core switch and moving the physical ports to the new core infrastructure. At this time you may experience interruption of network connections for a brief moment (up to 1 minute). We will try to keep the downtime as low as possible.


Update 14-08-2018 9:44:
The maintenance of 14 August 2018 is postponed due to incompatibility issues with the optics on the older devices. We suspect a firmware upgrade is required.

Update 15-8-2018 9:27:
We have verified that the optics are only compatible with the latest firmware and we have planned in firmware upgrades for both Zone 4 (On 23 August 2018) and Zone 5 (On 29 August 2018).

#686 Transit provider GTT maintenance

Posted: 2018-07-30 13:47

Start: 2018-08-10 02:00:00
End : 2018-08-10 06:00:00

Affects: Routing GTT

We have received a notification of maintenance by one of our transit providers, GTT.

During their maintenance traffic will automatically reroute over our other transit providers. We do not expect any issues.

Their maintenance reason is "Software upgrade of core device in Amsterdam".

Their reference is: 2747726

For transparency purposes we decided to publish this notification.

#685 Darkfiber outage NZS - EQX-AM7

Posted: 2018-07-25 11:14

Start: 2018-07-25 10:30:00
End : 2018-07-25 13:25:00

Affects: Redundancy NZS

At 10:30 CEST a darkfiber between NZS (Nedzone Steenbergen) and EQX-AM7 (Equinix AM7) went down, carrying 200 Gbps traffic.

Currently all traffic is sent over the alternative path on our ring namely NZS - GSA.

We have contacted the darkfiber provider Eurofiber and were notified that currently one of the fiber paths is damaged and that crews are looking for the exact breaking point.

Update 13:25:

Currently the darkfiber is online again. The root cause of this issue is still not known, but as this line has been stable for the last two hours, we will be closing this case.

#683 IP transit provider Cogent Globalswitch Amsterdam

Posted: 2018-07-23 15:51

Start: 2018-07-27 06:00:00
End : 2018-07-27 07:00:00

Affects: Redundancy routing Cogent Globalswitch Amsterdam

We have received a notification of a maintenance by one of our transit providers, Cogent.

During their maintenance traffic will automatically reroute over our other Cogent connection as well as over other transit providers. We do not expect any issues.

Their maintenance reason is: "Service migration to a new linecard"

Their reference:EU5171

For transparency purposes we decided to publish this notification.

#680 IP transit provider NTT

Posted: 2018-07-19 15:53

Start: 2018-08-01 02:00:00
End : 2018-08-01 05:00:00

Affects: Redundancy routing NTT

We have received a notification of a maintenance by one of our transit providers, NTT.

During their maintenance traffic will automatically reroute over our other NTT connection as well as over other transit providers. We do not expect any issues.

Their maintenance reason is "We will be performing a software upgrade affecting the service(s) listed below.
Users connected to this device will experience latency/downtime as the router is reloaded and routing is reestablished."

For transparency purposes we decided to publish this notification.

#679 Darkfiber maintenance NZS - EQX-AM7

Posted: 2018-07-18 10:08

Start: 2018-07-19 04:00:00
End : 2018-07-19 06:00:00

Affects: Redundancy NZS

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Equinix AM7 (EQX-AM7) is having a planned maintenance. The connectivity between NZS and EQX-AM7 will automatically reroute over our GSA darkfiber during this maintenance.

This maintenance is done in follow up to the issues with the darkfiber between NZS - EQX-AM7 on 2018-06-29

Their reference: PA-876013

For transparancy purposes we decided to publish notification.

#678 High latency towards EXTREME LTD (AS39709)

Posted: 2018-07-16 09:54

Start: 2018-07-16 09:54:43
End : 2018-07-16 09:54:43

Affects: Traffic towards EXTREME LTD (AS39709)

We have received reports of high latency towards EXTREME LTD (AS39709)

We have denied the path over Core-Backbone in the routing tables.

Best selected path is now going over DE-CIX.

#677 IP transit provider Zayo

Posted: 2018-07-13 15:42

Start: 2018-07-27 00:01:00
End : 2018-07-27 02:00:00

Affects: Routing Zayo

We have received a notification of a maintenance by one of our transit providers, Zayo.

During their maintenance traffic will automatically reroute over other transit providers.

Their maintenance reason is "Zayo will perform circuit migration for network optimization".

Their reference is:TTN-0002548704

For transparency purposes we decided to publish this notification.

#676 Slow transfer rate towards Verizon (AS701)

Posted: 2018-07-12 10:38

Start: 2018-07-12 10:38:07
End : 2018-07-12 10:38:07

Affects: Traffic towards Verizon (AS701)

We have received reports of slow transfer rates towards Verizon (AS701)

We have denied the path over Cogent in the routing tables.

Best selected path is now going over NTT.

#675 Global Switch Amsterdam Black Building test

Posted: 2018-07-11 16:31

Start: 2018-10-09 04:00:00
End : 2018-10-09 21:00:00

Affects: Global Switch Amsterdam - at risk

We have received a notification of a maintenance by the Datacenter Global Switch Amsterdam (GSA) as they will be doing their "Annual Black Building test 2018".

They notification we received included the following:
We rescheduled the planned Annual Black Building (BBT) Test 2018 to October 9th which was initially planned to execute on Saturday the 3rd of November as mentioned in the Annual Maintenance Calendar 2018 (v1_1) and in the site letter.
On this day our Grid operator has to do maintenance on the Global Switch Amsterdam Main Feed supply (50KV) and we decided to combine this works with the Black Building test.

The main purpose of this BBT is to test all full automatic functionality of all the emergency backup power systems, electrical distribution, and mechanical cooling systems.

- Before the works start all the technical installations will be checked. When there is a slight possibility for a failure or a deviant value (technically) the works won't start or will be aborted.
- The change level is assessed by GS OPS as a level 2 (Medium) as the supply by the main grid is shut down.

Their reference is: Change Control Form 2018-188

This is only an at risk notification, no outages are expected.

#674 IP transit provider NTT

Posted: 2018-07-09 10:46

Start: 2018-07-19 02:00:00
End : 2018-07-19 05:00:00

Affects: Redundancy routing NTT

We have received a notification of a maintenance by one of our transit providers, NTT.

During their maintenance traffic will automatically reroute over our other NTT connection as well as over other transit providers. We do not expect any issues.

Their maintenance reason is "We will be performing a software upgrade affecting the service(s) listed below.
Users connected to this device will experience latency/downtime as the router is reloaded and routing is reestablished."

For transparancy purposes we decided to publish notification.

#673 Darkfiber outage NZS - EQX-AM7

Posted: 2018-06-29 12:44

Start: 2018-06-29 12:19:00
End : 2018-06-30 04:15:00

Affects: Redundancy NZS

At 12.19 CEST a darkfiber between NZS (Nedzone Steenbergen) and EQX-AM7 (Equinix AM7) went down, carrying 200 Gbps traffic.

Currently all traffic is send over the alternative path on our ring namely NZS - GSA.

We have contacted the darkfiber provider Eurofiber and were notified that currently one of the fiber paths is damaged and that crews are looking for the exact breaking point.

Update 17:11:
The location of the fiber damage has been found to be construction works in Etten-Leur, repair crews are underway to repair the fibers.

Update 20:38:
The damage to fibers and pipes is that severe that the decision has been made to place new pipes and put new fibers in them, while Eurofiber does not want to give out a solid ETA the estimate is that they will start placing new fibers at 22.00 CEST.

Update 2018-06-30 04.15:
The splicing seems to be finished and connection is re-established.

#672 Slow transfer rate towards AT&T Services (AS7018)

Posted: 2018-06-25 11:49

Start: 2018-06-25 11:49:41
End : 2018-06-25 11:49:41

Affects: Traffic towards AT&T Services (AS7018)

We have received reports of slow transfer rates towards AT&T Services (AS7018)

We have denied the path over Cogent in the routing tables.

Best selected path is now going over NTT.

#671 IP transit provider Telecom Italia Sparkle

Posted: 2018-06-22 08:44

Start: 2018-06-26 06:00:00
End : 2018-06-26 08:00:00

Affects: Redundancy routing Telecom Italia Sparkle

We have received a notification of a maintenance by one of our transit providers, Telecom Italia Sparkle.

During their maintenance traffic will automatically reroute over our other Telecom Italia Sparkle connection as well as over other transit providers. We do not expect any issues.

Their maintenance reason is "TIS will perform an Emergency Work. Upgrade Software on router JMX960A-AM_DRT01 in Amsterdam (HOL). *** CIRCUITS WILL BE SERVICE AFFECTING ***"

Their reference is: TIS IP-GL260618-01

For transparancy purposes we decided to publish notification.

#670 Slow transfer rate towards Orange Espagne SA (AS12479)

Posted: 2018-06-14 08:45

Start: 2018-06-14 08:45:36
End : 2018-06-14 08:45:36

Affects: Traffic towards Orange Espagne SA (AS12479)

We have received reports of slow transfer rates towards Orange Espagne SA (AS12479)

We have denied the path over Level3 in the routing tables.

Best selected path is now going over Cogent.

#669 Wave service maintenance LINX GSA

Posted: 2018-06-13 16:35

Start: 2018-07-23 00:00:00
End : 2018-07-23 08:00:00

Affects: Routing LINX via GSA

Wave service provider which connects GSA (Global Switch Amsterdam) towards the LINX peering exchange in London is having a planned maintenance.

As this is one port of the LAG towards LINX, only redundancy is affected.

Their maintenance reason is "Dear Customer, We are undertaking a huge endeavour in an effort to maintain highest possible standards of your services we will be carry out an 12km in-span overlay of the Concerto North cable system between England and the Netherlands. The objective is to increase the overall longevity of the cable and achieve an overall decrease of optical attenuation. This project has been scheduled with our partner operating submarine cable repair vessels to begin on 6th of August 2018 10:00:00 UTC and to finish before 20th of August 2018 20:00:00 UTC, where your service will be experiencing downtime for the full period. Please note that schedule is subject to change due to weather conditions and other incidents which the vessels might be prioritized to. YOUR SERVICE(s) will be MIGRATED to an alternative route via the EuroTunnel in order to avoid downtime for the full period of time, during the project. PLEASE NOTE that during this time your services might experience an increase in latency to some destinations. Also in the case where you purchase diverse services between UK and Europe there will be a risk of geographical overlap. As soon as the return migrations have been scheduled, you will receive the date notification stating when your service(s) will be normalized to its original design. The Interoute team will be making the best we can to minimize downtime and inconvenience to customers with the end goal of delivering the best quality of services. ".

Their reference is: 1-3898503942

For transparency purposes we decided to publish notification.

#668 Wave service maintenance LINX GSA

Posted: 2018-06-13 16:33

Start: 2018-07-21 00:00:00
End : 2018-07-21 08:00:00

Affects: Routing LINX via GSA

Wave service provider which connects GSA (Global Switch Amsterdam) towards the LINX peering exchange in London is having a planned maintenance.

As this is one port of the LAG towards LINX, only redundancy is affected.

Their maintenance reason is "Dear Customer, We are undertaking a huge endeavour in an effort to maintain highest possible standards of your services we will be carry out an 12km in-span overlay of the Concerto North cable system between England and the Netherlands. The objective is to increase the overall longevity of the cable and achieve an overall decrease of optical attenuation. This project has been scheduled with our partner operating submarine cable repair vessels to begin on 6th of August 2018 10:00:00 UTC and to finish before 20th of August 2018 20:00:00 UTC, where your service will be experiencing downtime for the full period. Please note that schedule is subject to change due to weather conditions and other incidents which the vessels might be prioritized to. YOUR SERVICE(s) will be MIGRATED to an alternative route via the EuroTunnel in order to avoid downtime for the full period of time, during the project. PLEASE NOTE that during this time your services might experience an increase in latency to some destinations. Also in the case where you purchase diverse services between UK and Europe there will be a risk of geographical overlap. As soon as the return migrations have been scheduled, you will receive the date notification stating when your service(s) will be normalized to its original design. The Interoute team will be making the best we can to minimize downtime and inconvenience to customers with the end goal of delivering the best quality of services.".

Their reference is: 1-3938453812

For transparency purposes we decided to publish notification.

#665 Ring VPLS Firmware Upgrades

Posted: 2018-06-08 07:56

Start: 2018-08-01 01:00:00
End : 2018-10-27 05:00:00

Affects: (See message for more information)

To improve our network quality and resolve minor bugs in the current firmware, we will be performing firmware upgrades at the following days and times:

Nikhef - 1 Augustus 2018 At 1 AM CEST (Completed)
Affected: Local connected customers. Some peering/transit traffic, which will be redirected over other connections.

EQX-AM5 - 3 Augustus 2018 At 1 AM CEST (Completed)
Affected: Local connected customers. Some peering/transit traffic, which will be redirected over other connections.

BNR - 6 Augustus 2018 At 1 AM CEST (Completed)
Affected: Internet Customers. Local connected customers. Some peering/transit traffic, which will be redirected over other connections.


EQX-AM7 - Rescheduled to: 22 September 2018 At 1 AM CEST
Affected: Internet Customers. Local connected customers. Some peering/transit traffic, which will be redirected over other connections.

NZS - Rescheduled to: 4 October 2018 At 1 AM CEST
Affected: Network connectivity NZS.

DBA - Rescheduled to: 11 October 2018 At 1 AM CEST
Affected: Network connectivity DBA.

DBC - Rescheduled to: 18 October 2018 At 1 AM CEST
Affected: Network connectivity DBC.

GSA - Rescheduled to: 27 October 2018 At 1 AM CEST
Affected: Internet Customers. Local connected customers. Some peering/transit traffic, which will be redirected over other connections.

Update 2018-08-03 01.00:
While upgrading the EQX-AM5 VPLS switches all appeared to have been going smooth, but after having both switches upgrades traffic became intermittent on both of leading to random packet loss, engineers are currently on-site debugging this issue.

Update 2018-08-03 03.51:
We have reverted the switch back to the previous firmware which restored all connectivity, we will send our debug logs to the vendor for investigation of why this failed.

Update 2018-08-03 09.16:
Due to a possible firmware bug, which caused the firmware upgrade of EQX-AM5 to cause intermittent packet loss we have decided to postpone major locations. We have not postponed BNR at this time as part of required debugging. As NIKHEF is completed and we are not experiencing any issues here, it is currently not planned in for a downgrade.

We will be awaiting our vendor for more details before we continue on with the firmware upgrades at our major locations. After our vendor have investigated the issue, we will plan in the new firmware upgrades and update this post.

Update 2018-08-09 11.23:
We will continue with the firmware upgrade of EQX-AM5 and debug the issue we are seeing with the vendor on Monday 13 August 2018. All traffic will be balanced over our other transit providers and peers. The firmware upgrade and debug sessions will be during the day.

We do not expect that the maintenance will cause disruptions as traffic will be offloaded before maintenance.

Update 2018-08-13 17.32:
We were able to debug and resolve the issues encountered previously. The firmware of EQX-AM5 has been updated. We will monitor this site closely the coming days. We will replan this maintenance window for the rest of the switches, and sent an update once available.

Update 2018-08-15 12.15:
We have re-scheduled the firmware upgrades for our Ring infrastructure, for more information on the date, please refer to the initial post above.

#664 Slow speeds towards China Telecom (AS4134)

Posted: 2018-06-03 21:34

Start: 2018-06-03 21:34:25
End : 2018-06-03 21:34:25

Affects: Traffic towards China Telecom (AS4134)

We have received reports of slow speeds towards China Telecom.

We have denied the path over Cogent in the routing tables.

Best selected path is now going over NTT.

#663 FTTB outage Stolwijk / Bergambacht / Nieuwerkerk aan de Ijssel

Posted: 2018-06-03 11:46

Start: 2018-06-03 10:22:00
End : 2018-06-03 13:31:00

Affects: Stolwijk / Bergambacht / Nieuwerkerk

*This does not affect any Hosting or Transit Customers and is only valid for Customers located in one of the affected areas.*

Due to a power outage in Waddinxveen the PoP of our transport provider Eurofiber is also without power.

Because of this our connections to Stolwijk / Bergambacht and Nieuwerkerk aan de Ijssel are currently not working.

According to Eurofiber the current ETA for solving the power disruption is 4 PM CEST.
They've also sent out engineers with a power generator to feed the PoP for which the ETA is set to 1 PM CEST.

Our apologies for the incovenience caused.

#662 Wave service maintenance LINX EQX-AM7

Posted: 2018-05-31 09:53

Start: 2018-06-27 00:00:00
End : 2018-06-27 07:00:00

Affects: Routing LINX

Wave service provider which connects EQX-AM7 (Equinix AM7 Amsterdam) towards the LINX peering exchange in London is having a planned maintenance.

Their maintenance reason is "To ensure an optimal service we need to perform Planned Maintenance activities that will affect one of your network connections."

In case of a full outage, all LINX routed traffic is automatically routed over our other LINX peering exchange connection. We therefore do not expect any issues.

Their reference is: 85684 and 85686

For transparency purposes we decided to publish notification.

Update from KPN 2018-06-26:
Please note due to administrative error, the maintenance did not took place but will take place tonight. Our apologies for the inconvenience.

#661 Wave service maintenance LINX EQX-AM7

Posted: 2018-05-30 16:43

Start: 2018-06-01 00:00:00
End : 2018-06-01 06:00:00

Affects: Routing LINX

Wave service provider which connects EQX-AM7 (Equinix AM7 Amsterdam) towards the LINX peering exchange in London is having a planned maintenance.

Their maintenance reason is "We would like to clean the fibers on this service. Our second line engineer informs us that there is to much attenuation on this service and would like to send an engineer to clean the fibers."

In case of a full outage, all LINX routed traffic is automatically routed over our other LINX peering exchange connection. We therefore do not expect any issues.

Their reference is: 132226

For transparency purposes we decided to publish notification.

#660 Wave service maintenance LINX GSA

Posted: 2018-05-29 09:01

Start: 2018-07-23 00:00:00
End : 2018-07-23 08:00:00

Affects: Traffic LINX

Wave service provider which connects GSA (Global Switch Amsterdam) towards the LINX peering exchange in London is having a planned maintenance.

All LINX routed traffic is automatically routed over our other LINX peering exchange connection. We therefore do not expect any issues.

Their maintenance reason is "As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing hardware upgrade works on our transmission platform. This planned activity will take place in Netherlands.".

Their reference is: 1-3898503942

For transparency purposes we decided to publish notification.

#656 Updates to our Privacy Policy

Posted: 2018-05-23 16:30

Start: 2018-05-25 00:00:00
End : 0000-00-00 00:00:00

Affects: N/A

On the 25th of May 2018, the European Union (EU) will bring into force the General Data Protection Regulation (GDPR). GDPR guarantees for its citizens many rights to their digital privacy. In order to be compliant with this regulation we are updating our privacy policy. This will provide even more transparency for our services.

The updated versions of our agreements, policies and guidelines are available at our legal page, which you can find at https://www.nforce.com/legal - will provide you with more insight over what we do with your personal data.

Your security, as well as your privacy, has always been important to us. We will continue to stay committed to protecting your data.

#655 Wave service maintenance LINX EQX-AM7

Posted: 2018-05-22 07:50

Start: 2018-05-29 05:00:00
End : 2018-05-29 07:00:00

Affects: Routing LINX

Wave service provider which connects EQX-AM7 (Equinix AM7 Amsterdam) towards the LINX peering exchange in London is having a planned maintenance.

In case of a full outage, all LINX routed traffic is automatically routed over our other LINX peering exchange connection. We therefore do not expect any issues.

Their maintenance reason is "Your service is protected but a switch-hit might occur twice. We need to switch the service back to the Ulysses cable.".

Their reference is: 85406

For transparency purposes we decided to publish notification.

#654 Packetloss towards Bani Networks LTD (AS134084)

Posted: 2018-05-20 18:52

Start: 2018-05-20 18:52:55
End : 2018-05-20 18:52:55

Affects: Traffic towards Bani Networks LTD (AS134084)

We have received reports of packet loss towards Bani Networks LTD.

We have denied the path over peer Bharti Airtel LTD (AS9498) in the routing tables.

Best selected path is now going over Cogent.

#653 Router firmware upgrades

Posted: 2018-05-17 12:19

Start: 2018-05-24 01:00:00
End : 2018-05-30 07:00:00

Affects: (See message for more information)

During these times we will be performing firmware upgrades of our routers, to solve a part of the issues we have been seeing in the past which is explained at our NOC post: https://noc.nforce.com/notifications/item/649

Our time schedule is as follows:

24 May 2018 at 1AM CEST we will be performing firmware upgrade on the router at Global Switch Amsterdam (GSA)
Affected: IP Transit customers, Redundancy network.

25 May 2018 at 11 AM CEST we will be performing firmware upgrade on the router at Equinix-AM5 (AM5)
Affected: Peer British Telecom

26 May 2018 at 1AM CEST we will be performing firmware upgrade on the router at Equinix-AM7 (AM7)
Affected: IP Transit customers, Internet customers, Redundancy network.

29 May 2018 at 1AM CEST we will be performing firmware upgrade on the router 2 at Nedzone (NZS).
Affected: Redundancy network NZS.

29 May 2018 at 5AM CEST we will be performing firmware upgrade on the router 2 at Databarn Capelle (DBC).
Affected: Redundancy network DBC.

30 May 2018 at 1AM CEST we will be performing firmware upgrade on the router 1 at Nedzone (NZS).
Affected: Redundancy network NZS.

30 May 2018 at 5AM CEST we will be performing firmware upgrade on the router 1 at Databarn Capelle (DBC).
Affected: Redundancy network DBC.

#652 Wave service maintenance FranceIX

Posted: 2018-05-16 22:58

Start: 2018-06-09 01:00:00
End : 2018-06-09 06:00:00

Affects: Routing FranceIX

Wave service provider which connects Globalswitch towards the FranceIX peering exchange in Paris is having a planned maintenance.

All FranceIX routed traffic is automatically routed over our other peering exchanges or IP transit connections.

Their maintenance reason is "As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing splice box replacement works on our fiberoptical network. This planned activity will take place in Belgium."

Their reference: 1-3787161641

For transparancy purposes we decided to publish this notification.


Update 2018-06-07 09.55:
Interoute moved the starttime one hour ahead.

#651 Wave service maintenance DECIX GSA

Posted: 2018-05-16 09:06

Start: 2018-08-05 00:00:00
End : 2018-08-05 06:00:00

Affects: Routing DE-CIX

Wave service provider which connects GlobalSwitch Amsterdam (GSA) towards Frankfurt is having a planned maintenance.

Their maintenance reason is " As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing mandatory cable works on our fiberoptical network. This planned activity will take place in Germany. "

Interoute has given this maintenance reference number:1-3794823251

All DE-CIX routed traffic is automatically routed over our other exchanges and IP transits.

For transparancy purposes we decided to publish this notification.


Update 2018-06-02:
Interoute has prosponed the maintenance to the 5th of August 2018.

#650 Wave service maintenance LINX GSA

Posted: 2018-05-15 08:11

Start: 2018-06-02 00:00:00
End : 2018-06-02 06:00:00

Affects: Routing LINX

Wave service provider which connects GSA (Global Switch Amsterdam) towards the LINX peering exchange in London is having a planned maintenance.

All LINX routed traffic is automatically routed over our other LINX peering exchange connection. We therefore do not expect any issues.

Their maintenance reason is "As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing hardware upgrade works on our transmission platform. This planned activity will take place in Netherlands.".

Their reference is: 1-3846472771

For transparency purposes we decided to publish notification.

#649 Router linecard crashes, vendor investigation (ongoing)

Posted: 2018-05-11 10:37

Start: 2018-04-01 00:00:00
End : 2018-07-06 00:00:00

Affects: Routing, primarely DBA

For transparency purposes we opened this case to keep you informed regarding an issue that causes crashing linecards in the routers.

During February and March we have been working with our vendor Brocade/Extreme to resolve packetloss on a 20x 10G linecard, which occurred when using more than 8 ports in a 160 Gbps lag (bundle). For this issue a work-around and fixes were implemented and till date this has been working. This was acceptable as these are the last linecards with 10G ports we use on our routing platform; they are marked/planned to be replaced by 100G linecards. All other routers and linecards in our hosting facilities DBC/NZS are already replaced with 100G models. The replacement for DBA was actually was planned to be done for last month, everything has been prepared and connect, however due to current new ongoing issue we have proponed this work.

Currently since ~1st of April we have noticed crashes of linecards and mgmt cards on Brocade/Extreme routers. First incidents were immediately reported to our vendor and a TAC case was created. During April we noticed 5 incidents at DBC, an incident at EQX-AM7. During April the TAC case has been escalated to the highest level possible with vendor. We are part of the critical customers program. During May the incidents increased further with less time apart between the incidents. So far we have counted 19 incidents on DBA during May. Our vendor has been made aware that failure to resolve forces us to break a 10-year partnership with them. Vendor has assured this case is handled with highest priority and all escalations possible are already done. It is currently in the hands of the engineers responsible for developing the code for these devices, if they cannot get this fixed then there is no one who can.

Our network is setup 100% redundantly in an active/active manner. Meaning redundant physical fiberpaths, redundant network chassis (routers and vpls) and also within these chassis we place everything redundant, thus double (or more) the amount of required psu's, redundant management cards and we spread the load of each LAG over multiple linecards. The latter (spreading the load of each LAG over multiple linecards) should have reduced the incidents to cause minimal issues (few millisecond). Unfortunately just before the linecard crashes, it seems to drop packets - if it were to crash immediately the impact would have been milliseconds, now the impact is seconds. As the traffic is coming in balanced over two routers, as well as going out balanced over two routers, the impact is reduced to roughly 1/4th connections. The incidents are recovered automatically without the need/presence/work from our NOC. This means that even though the incidents are there, at least it auto recovers when the card fails from the system and also auto-joins when it comes back. This means that no human interaction is required to recover during incidents.

This week we did patch the routers on DBA (based on the crash reports provided to Brocade), this seem to have resolved the management card crash. However the most pressing issue which is the linecard crashes are not yet (or not all of them) fixed.

So far it seems they have narrowed it down to their IPv6 code/routines. They are modifying the lab setup to start sending different packets in different scenario's to replicate the crashes, once they are able to replicate they can build a fix.

We hope that soon we can say that we are confident that the layer 3 routing platform in DBA is stable again, and everyone can enjoy the stability as they were used to from us before these incidents. Our apologies for the situation at hand, we can assure you that this situation is our highest priority - as well as with our vendor.


Update 2018-05-16 18.15:
We have implemented temporary partial fixes including an ACL blocking certain IPv6 packets at DBA. Since the the initial report of this post (2018-06-11) no more crashes were observed at DBA.

Last night at 00.50 there was a new crash at DBC R1, two linecards carrying 400 Gbps crashed within minutes apart. We applied the same ACL as applied on DBA, on all routers in our network.


Update 2018-07-06 17.23:
Since the end of May we have not seen any crashes anymore. Closing case.


Update 2018-07-16 10.40:
For archiving purposes the following defects were reported by NFOrce to vendor:
DEFECT000660088 [ifsr issue] (fixed in 6200b)
DEFECT000642455 [mgmt crash] (fixed in 6200b)
DEFECT000661713 [lp crash] (to be fixed in next rls)
DEFECT000661585 [lp crash]
DEFECT000661951 [lp crash]
DEFECT000661952 [lp crash]

#648 Transit provider GTT maintenance

Posted: 2018-05-10 06:50

Start: 2018-05-25 02:00:00
End : 2018-05-25 06:00:00

Affects: Routing GTT

We have received a notification of maintenance by one of our transit providers, GTT.

During their maintenance traffic will automatically reroute over our other transit providers. We do not expect any issues.

Their maintenance reason is " performing upgrades".

Their reference is: 2620079

For transparancy purposes we decided to publish notification.

#647 VPLS Outage EQX-AM7

Posted: 2018-05-06 23:32

Start: 2018-05-06 23:10:00
End : 2018-05-07 00:15:00

Affects: Fiber to the business Customers located on EQX-AM7, VPLS Customers EQX-AM7, Peering traffic R1 EQX-AM7

Currently there is an outage ongoing on the VPLS ring of EQX-AM7 where one of the switches in the stack rebooted itself.

The stack itself is setup redundantly where the standby took over it's place.

However due to a yet unknown reason a part of the VPLS ring went down even though the ports are still up.

We are currently investigating into the issue to find the cause and to resolve this and to collect data for our vendor.

Update 2018/05/07 00.05:
As even with trying different ways to get the remaining parts back online we decided to try a clean restart of the whole stack after collecting enough debug logs for our vendor.

After both switches were rebooted the remainder of the ports also came back online.

Because peering connections were still working on the EQX-AM7 router but had no way of being send over to the correct datacenters a higher latency or random packetloss may also have occurred.

At this moment the switches are stable again and we'll keep monitoring them while discussing this issue with our vendor.

#646 Firmware upgrade R1 DBA

Posted: 2018-05-06 15:13

Start: 2018-05-07 07:00:00
End : 2018-05-07 09:00:00

Affects: Traffic DBA

We planned in a maintenance for upgrading our router Firmware for R1 in DBA (Databarn Amsterdam) to the latest firmware.

We apologize for the short notice and unfortunately due to the urgency we cannot change the planned date.

The traffic will be rerouted through R2 of DBA (Databarn Amsterdam), if you have redundant BGP sessions/links, only one session will go down and the traffic will go through your other session/link.

For those who do not have redundant connection through this router, may experience an outage up to 1 hour.

This firmware fixes one of the open issues we have with our vendor regarding linecard crashes.

#645 Firmware upgrade R2 DBA

Posted: 2018-05-06 15:10

Start: 2018-05-08 00:00:00
End : 2018-05-08 03:00:00

Affects: Traffic DBA

We planned in a maintenance for upgrading our router Firmware for R2 in DBA (Databarn Amsterdam) to the latest firmware.

We apologize for the short notice and unfortunately due to the urgency we cannot change the planned date.

The traffic will be rerouted through R1 of DBA (Databarn Amsterdam), if you have redundant BGP sessions/links, only one session will go down and the traffic will go through your other session/link.

For those who do not have redundant connection through this router, may experience an outage up to 1 hour.

This firmware fixes one of the open issues we have with our vendor regarding linecard crashes.

Update 2018/05/07 00.43:
Due to the issues at EQX-AM7 this maintenance was postponed till 2018-05-08 at the same time.

#644 IP transit provider Core-Backbone

Posted: 2018-05-04 19:43

Start: 2018-05-15 00:00:00
End : 2018-05-15 06:00:00

Affects: Routing Core-Backbone

We have received a notification of a maintenance by one of our transit providers, Core-Backbone.

During their maintenance traffic will automatically reroute over our other transit providers. We do not expect any issues.

Their maintenance reason is "In the mentioned timeframe we will perform software upgrades at the following core-routers/core-switches:

ams10.core-backbone.com (Amsterdam1)
ams11.core-backbone.com (Amsterdam1)
ams12.core-backbone.com (Amsterdam1)
ams20.core-backbone.com (Amsterdam2)
prg10.core-backbone.com (Prague1)
prg11.core-backbone.com (Prague1)"

Their reference: 1988 and 1994

For transparancy purposes we decided to publish notification.

#643 Wave service maintenance LINX EQX-AM7

Posted: 2018-05-04 09:11

Start: 2018-05-18 00:00:00
End : 2018-05-18 07:00:00

Affects: Routing LINX

Wave service provider which connects Equinix AM7 towards the LINX peering exchange in London is having a planned maintenance.

All LINX routed traffic is automatically routed over our other LINX peering exchange connection. We therefore do not expect any issues.

Their maintenance reason is "We want to inform you about the upcoming maintenance on the sea cable between the UK and Europe from our partner.".

Their reference are: 84968, 84971

For transparancy purposes we decided to publish notification.

#642 Wave service maintenance LINX GSA

Posted: 2018-05-03 06:51

Start: 2018-05-16 00:00:00
End : 2018-05-16 06:00:00

Affects: Routing LINX GSA

Wave service provider which connects GSA (Global Switch Amsterdam) towards the LINX peering exchange in London is having a planned maintenance.

All LINX routed traffic is automatically routed over our other LINX peering exchange connection. We therefore do not expect any issues.

Their maintenance reason is " As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing software upgrade works on our transmission platform. This planned activity will take place in United Kingdom.".

Their reference is: 1-3773361961

For transparency purposes we decided to publish notification.

#641 Wave service maintenance LINX GSA

Posted: 2018-05-02 07:24

Start: 2018-05-14 00:00:00
End : 2018-05-14 06:00:00

Affects: Routing LINX

Wave service provider which connects GSA (Global Switch Amsterdam) towards the LINX peering exchange in London is having a planned maintenance.

All LINX routed traffic is automatically routed over our other LINX peering exchange connection. We therefore do not expect any issues.

Their maintenance reason is "As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing software upgrade works on our transmission platform. Infinera TIM software upgrade - Rollout of the new IQNOS 15.3.3 firmware to enable new features. This planned activity will take place in London, UK and Amsterdam , Netherlands".

Their reference is: 1-3825176901

For transparency purposes we decided to publish notification.

#640 Wave service maintenance DECIX EQX-AM7

Posted: 2018-05-01 07:40

Start: 2018-05-15 00:00:00
End : 2018-05-15 06:00:00

Affects: Routing DE-CIX

Wave service provider which connects Equinix AM7 towards the DECIX peering exchange is having a planned maintenance.

All DECIX routed traffic is automatically routed over our other DECIX peering exchange connection. We therefore do not expect any issues.

Their maintenance reason is "Fiber work".

Their reference is: 84646

For transparancy purposes we decided to publish notification.

#639 Wave service maintenance LINX EQX-AM7

Posted: 2018-05-01 07:38

Start: 2018-05-15 00:00:00
End : 2018-05-15 06:00:00

Affects: Routing LINX

Wave service provider which connects Equinix AM7 towards the LINX peering exchange in London is having a planned maintenance.

All LINX routed traffic is automatically routed over our other LINX peering exchange connection. We therefore do not expect any issues.

Their maintenance reason is "Fiber work".

Their reference is: 84649

For transparancy purposes we decided to publish notification.

#638 Packetloss traffic over NTT

Posted: 2018-04-28 21:10

Start: 2018-04-28 18:40:00
End : 2018-04-28 18:50:00

Affects: Routing NTT

We have received notification from NTT reporting issues with one of their routers causing packet loss for traffic through NTT.

Their notification: "r25.amstnl02.nl.bb experienced an issue where it's routing process crashed and did not gracefully fallback. This upstream device caused a brief outage while routing was reloaded and latency due to convergence. We are investigating with our vendor."

Their reference: VNOC-1-1698497805

For transparancy purposes we decided to publish this notification.

#637 Wave service maintenance DECIX GSA

Posted: 2018-04-25 18:54

Start: 2018-06-17 00:00:00
End : 2018-06-17 06:00:00

Affects: Routing DE-CIX

Wave service provider which connects GlobalSwitch Amsterdam (GSA) towards Frankfurt is having a planned maintenance.

Their maintenance reason is "As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing splice box replacement works on our fiberoptical network. This planned activity will take place in Netherlands. Please be advised your services will be affected as per table below between 16/Jun 22:00 UTC and 17/Jun 04:00 UTC. INTEROUTE would like to apologise for the inconvenience these works will cause to your business."

Interoute has given this maintenance reference number:1-3719422881.

All DE-CIX routed traffic is automatically routed over our other exchanges and IP transits.

For transparancy purposes we decided to publish this notification.

#636 Wave service maintenance DECIX GSA

Posted: 2018-04-25 18:48

Start: 2018-05-27 00:00:00
End : 2018-05-27 06:00:00

Affects: Routing DE-CIX

Wave service provider which connects GlobalSwitch Amsterdam (GSA) towards Frankfurt is having a planned maintenance.

Their maintenance reason is "As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing mandatory splicebox works on our fiberoptical network. The work is related with replacing of old splice box, which can create unexpected problems in the future. This planned activity will take place in Netherlands, Amsterdam. Please be advised your services will be affected as per table below between 26/05/2018 - 22:00 UTC and 27/05/2018 - 04:00 UTC . INTEROUTE would like to apologise for the inconvenience these works will cause to your business."

Interoute has given this maintenance reference number:1-3719372761.

All DE-CIX routed traffic is automatically routed over our other exchanges and IP transits.

For transparancy purposes we decided to publish this notification.

#635 Wave service maintenance FranceIX

Posted: 2018-04-25 18:39

Start: 2018-05-15 00:00:00
End : 2018-05-15 06:00:00

Affects: Routing FRANCEIX

Wave service provider which connects Globalswitch towards the FranceIX peering exchange in Paris is having a planned maintenance.

All FranceIX routed traffic is automatically routed over our other peering exchanges or IP transit connections.

Their maintenance reason is "As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing splice box replacement works on our fiberoptical network. This planned activity will take place in Belgium. Please be advised your services will be affected as per table below between 14/May 22:00 UTC and 15/May 04:00 UTC. INTEROUTE would like to apologise for the inconvenience these works will cause to your business."

Their reference: 1-3719293741

For transparancy purposes we decided to publish this notification.

#634 AMS-IX maintenance EQX-AM7

Posted: 2018-04-20 17:15

Start: 2018-04-24 00:00:00
End : 2018-04-25 04:00:00

Affects: Routing AMS-IX EQX-AM7

AMS-IX peering exchange is having a planned maintenance on the port directed towards Equinix-AM7.

Their maintenance reason is "AMS-IX engineers would like to continue the next phase of our maintenance started on 12th March 2018, previously detailed in the ticket:
"NOC24X7-12865 New Switch Migration (SLX-9850) At Equinix-AM7 Amsterdam (formerly Telecity-2)".
This operation was halted while we waited for the release of firmware for the new switch platform (SLX9850) with defect fixes.
After the new release came out and passed our testing process, we would like to continue with the following phase of the operation:
PHASE 3 - Failover 3xx group to the new 2xx group switches
Week 23rd - 26th April 2018
00:00 - 04:00 24th April 2018
- Move half of the member connections to new 2xx group switches ??
00:00 - 04:00 25th April 2018
- Move all member connections from the 3xx group to the new switches (2xx group)
- Leave all member connections running on the new switches to confirm stability"

All AMS-IX routed traffic is automatically routed over our other AMS-IX connection.

Because they do not mention to which half we belong we set the maintenance on both days.

#633 IP transit provider NTT

Posted: 2018-04-20 02:07

Start: 2018-05-03 02:00:00
End : 2018-05-03 05:00:00

Affects: Redundancy routing NTT

We have received a notification of a maintenance by one of our transit providers, NTT.

During their maintenance traffic will automatically reroute over our other NTT connection as well as over other transit providers. We do not expect any issues.

Their maintenance reason is "We will be performing a software upgrade affecting the service(s) listed below.
Users connected to this device will experience latency/downtime as the router is reloaded and routing is reestablished. "

For transparancy purposes we decided to publish notification.

Update 2018-05-02:
NTT has canceled this maintenance due to unforseen circumstances and will reschedule this maintenance to a later date.

#632 Wave service / Darkfiber maintenance

Posted: 2018-04-17 18:44

Start: 2018-05-04 00:00:00
End : 2018-05-04 06:00:00

Affects: Routing FRANCE-IX GSA / Redundancy NZS/BNR

Wave service provider which connects Globalswitch towards the FranceIX peering exchange in Paris is having a planned maintenance.

All FranceIX routed traffic is automatically routed over our other peering exchange or IP transit connections.

At the same time there will be a maintenance on the darkfiber connecting Globalswitch to Bytesnet Rotterdam (BNR) and Nedzone Steenbergen (NZS).

The traffic to BNR will automatically go over the darkfiber to DBC and traffic to NZS will automatically reroute over EQX-AM7.

Their maintenance reason is "As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing splice box replacement works on our fiberoptical network. This planned activity will take place in Netherlands. Please be advised your services will be affected as per table below between 03/May 22:00 UTC and 04/May 04:00 UTC. INTEROUTE would like to apologise for the inconvenience these works will cause to your business."

Their reference: 1-3719330071

For transparancy purposes we decided to publish this notification.