Darkfiber provider Relined which connects Global Switch Amsterdam (GSA) towards Nikhef is having a planned maintenance. The connectivity between GSA and Nikhef will automatically reroute over our AM7 darkfiber during this maintenance.

Their maintenance reason is:
"Due to reconstruction work carried out by the local authorities. Relined is obliged to relocate a section of her network. This action will result in a temporary interruption of the service."

Their reference is: PA-361001

Wave service provider which connects Equinix AM7 (EQX-AM7) towards LINX is having a planned maintenance.

Their maintenance reason is "MAINTENANCE KPN, Fiber work, Preparation for move Lwst equipment."

KPN has given this maintenance reference numbers: 99536 and 99543

All LINX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

We have received reports of slow speeds towards AS6128 (Optimum Online)

We have denied the path over Corebackbone in the routing tables.

The best selected path is now going over Liberty Global (AS6830).

At 12:52 CEST we had a power outage in rack d4.z1 in Databarn Capelle, causing several servers and one switch to go down.

We sent one of our engineers over right away to assess the situation.
He has replaced a power distribution unit which seems to be defective.

Currently the rack is back online and all servers are operational again.

We have received reports of packet loss towards Hangzhou Alibaba Advertising Co.,Ltd (AS37963)

We have denied the path over NTT in the routing tables.

The best selected path is now going over GTT (AS3257).

Wave service provider which connects Global Switch Amsterdam (GSA) towards LINX is having a planned maintenance.

Their maintenance reason is " As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing hardware reconfiguration works on our transmission platform. This planned activity will take place in Netherlands. "

Interoute has given this maintenance reference number: 1-4885581691

All LINX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

Darkfiber provider Eurofiber which connects NZS (Nedzone Steenbergen) towards Bytesnet Rotterdam (BNR) is having a planned maintenance. The connectivity between NZS and BNR will automatically reroute over our EQX-AM7 darkfiber during this maintenance.

Their maintenance reason is:
Following last weeks optical issues we will be doing maintenance on this line to make sure issues like last week do not happen again.

At 01:00 CEST a darkfiber between DBC (Datbarn Capelle) and EQX-AM7 (Equinix AM7) went down, carrying 400 Gbps traffic.

Currently all traffic is sent over the alternative path on our ring namely DBC - GSA while we are investigating the cause into this.

Update 2019-04-13 03:32 CEST:
We've spoken to the darkfiber provider euNetworks and this is because of a maintenance which was not communicated to us.

Update 2019-04-14 09:00 CEST:
Currently the line is back online and our redundancy is operational again.

We have received reports of slow speeds towards AS4773 (M1 ISP)

We have denied the path over AMSIX in the routing tables.

The best selected path is now going over GTT (AS3257).

Our dark fiber provider has informed us about an urgent maintenance occurring today between Steenbergen and Rotterdam due to bad optical values.

They will be performing this during the day. The connection may be interrupted but as we are setup redundant, our other connection will take it over.

Their message is:
"Wij ervaren sinds gisterenmiddag een hogere demping op het glasvezeltraject tussen Steenbergen en Rotterdam. Aangezien de waardes sinds vanmorgen verder zijn verslechterd zijn wij genoodzaakt om met spoed metingen en reparatiewerkzaamheden te laten uitvoeren door de leverancier van deze glasvezels.
De kans is dus aanzienlijk dat het traject Steenbergen > Rotterdam in de loop van de dag voor enige tijd niet beschikbaar zal zijn. "

Translation:
"Since yesterday afternoon we have experienced higher values on the fiber optic route between Steenbergen and Rotterdam. Since the values have deteriorated further since this morning, we are forced to have measurements and repair work carried out urgently by the supplier of these glass fibers.
The chance is therefore considerable that the Steenbergen> Rotterdam route will not be available for some time during the course of the day."

Update 2019-04-09 15:05 CEST:
The maintenance done by our darkfiber provider has been completed and the line is fully functioning again.

Today at 10:47 CEST, our uplinks with LINX at GSA (Global Switch Amsterdam).

We have investigated the issue at our side but concluded that the issue is either at LINX side, or at our wave service provider.

We have contacted our wave service provider at 11:00 CEST, but have not yet received any updates.

Currently, all traffic towards LINX is routed through our other ports at EQX-AM7 (Amsterdam).

Once we have an update, we will update this post.

Update 2019-04-07 12:47 CEST:
The wave service provider found out the issue causing our link to be down:
It is connected to the subsea fiber break PI 1-4858348241 between Leiston – Zandvoort.

Update 2019-04-07 15:12 CEST:
Dear Customer, Please be informed that the fault is confirmed to be on 89.6km from our landing station in Netherlands. Currently the cable ship is on the port, preparing and loading all the spares needed for the restoration. No ETA or ETR. IMPORTANT: If you are a dark fiber customer, please SHUT DOWN your raman amplifiers. We need this step due to safety reasons.

Update 2019-04-07 16:28 CEST:
The ship is already prepared for the spares loading operation. The operation will start in 30 minutes. Once the ship is ready for sailing and have approval for it, we will provide tentative ETA to the fault location.

Update 2019-04-07 20:20 CEST:
Loading of the spares onto the ship is currently ongoing. In the meantime we are also waiting for approval in order for the ship to set sail to the fault location. As this is a time-consuming process it will take some time. Once we receive the approval we will share a tentative ETA to the fault location and possibly a tentative ETR.

Update 2019-04-08 1:42 CEST:
Loading of spares completed. Repair ship has left the port at 23:00 UTC.

Update 2019-04-08 10:50 CEST:
The Sovereign cable ship left port as planned at midnight last night. It is expected on the cable grounds at the fault location at approximately midnight tonight. Estimated date of repair completion 15/04/19. Next update will be available in 12 hours.

Update 2019-04-09 14:23 CEST:
As our lines are currently up, running on a backup line we do not experience any direct issues.
However, we do expect a switch over and keep this post updated for possible major incidents.
They have reported estimated date of repair completion 15/04/19.

Update 2019-04-15 07:30 CEST:
Please be informed that the cable fault was fixed already.
All traffic is recovered.

This notification is only for specific customers with equipment at our Equinix AM5 location.

Please be advised that Equinix AM5 will be performing preventive maintenance on the main low-voltage distribution panel B. There are no servers housed here.
All our own equipment in this location is powered redundantly and we should not notice this maintenance.

Notification from AM5:

Please be advised that Equinix will be performing preventive maintenance on the main low-voltage distribution panel B.

This maintenance involves switching the primary or redundant power supply(s) to multiple cabinets off for a time frame of 8 hours. After the maintenance is completed the power supply will be switched back on.

Wave service provider which connects Equinix AM7 (EQX-AM7) towards LINX is having a planned maintenance.

Their maintenance reason is "Relocating cables due to road construction works. Reconstruction of knooppunt De Nieuwe Meer."

KPN has given this maintenance reference numbers: 98825 and 98826

All LINX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

We have detected an issue on the active management module in the router of GSA (Global Switch Amsterdam).

At 2 April 2019, 13:00 CEST, we will be performing a switchover to the standby management as per recommendation from vendor to avoid a hard crash of the router.

After reviewing the configuration together with the vendor, all options are set to minimize impact of a switchover.

Wave service provider which connects Global Switch Amsterdam (GSA) towards FRANCEIX is having a planned maintenance.

Their maintenance reason is " As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing mandatory cable works on our fiberoptical network. This planned activity will take place in France."

Interoute has given this maintenance reference number: 1-4818939751

All FRANCEIX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

UPDATE 2019-04-22 10:00 CEST
Please be informed that this Planned Work has been postponed until further notice. As soon as we will have information about the new time window we will notify you.

UPDATE 2019-04-23 16:30 CEST
Reschedule

We received a maintenance notification from one of our infrastructure datacenters Equinix-AM5 that they are having maintenance on generator C.

Their exact maintenance reason is:

Please be advised that Equinix will be performing preventive maintenance on generator C.

Only one generator will be maintained at a time, generator F will be placed back in service before maintenance commences on the next one.

No loss of resilience is expected with "N+1" redundancy maintained during this period and all cabinet supplies remaining UPS backed.

Additional there will also be performed a generator load test. This is a routine test of the generator which will be performed in a controlled manner,
during the test customer load will be transferred from Utility to Generator.

their reference is 5-186079286285

We received a maintenance notification from one of our infrastructure datacenters Equinix-AM5 that they are having maintenance on generator B.

Their exact maintenance reason is:

Please be advised that Equinix will be performing preventive maintenance on generator B.

Only one generator will be maintained at a time, generator F will be placed back in service before maintenance commences on the next one.

No loss of resilience is expected with "N+1" redundancy maintained during this period and all cabinet supplies remaining UPS backed.

Additional there will also be performed a generator load test. This is a routine test of the generator which will be performed in a controlled manner,
during the test customer load will be transferred from Utility to Generator.

their reference is 5-186080568035

Wave service provider which connects Global Switch Amsterdam (GSA) towards DECIX is having a planned maintenance.

Their maintenance reason is " As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing hardware upgrade works on our transmission platform. This planned activity will take place in Germany. "

Interoute has given this maintenance reference number: 1-4799234211

All DECIX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

We have received reports of slow speeds towards Bell Canada (AS577)

We have adjusted the path to go over Centurylink as preferred route.

We have received reports of packet loss towards Starhub (AS38861)

We have denied the path over AMSIX in the routing tables.

The best selected path is now going over Telstra Global (AS4637).

Wave service provider which connects Global Switch Amsterdam (GSA) towards DECIX is having a planned maintenance.

Their maintenance reason is " As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing cable rerouting works on our fiberoptical network. This planned activity will take place in Netherlands. "

Interoute has given this maintenance reference number: 1-4808878511

All DECIX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

UPDATE 2019-04-23 13:00 CEST
Hello, We would like to inform you that planned work has been cancelled.

Wave service provider which connects Global Switch Amsterdam (GSA) towards FRANCEIX is having a planned maintenance.

Their maintenance reason is " As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing cable rerouting works on our fiberoptical network. This planned activity will take place in Netherlands. "

Interoute has given this maintenance reference number: 1-4808634416

All FRANCEIX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

UPDATE 2019-04-04 09:00 CEST:
Reschedule

Darkfiber provider Interoute which connects Bytesnet Rotterdam (BNR) towards Globalswitch Amsterdam (GSA) is having a planned maintenance. The connectivity between BNR and GSA will automatically reroute over our DBC darkfiber and NZS will be rerouted automatically over AM7 during this maintenance.

Their maintenance reason is "As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing cable rerouting works on our fiberoptical network. This planned activity will take place in Netherlands".

Their reference: 1-4808625916

For transparency purposes we decided to publish notification.

UPDATE 2019-04-04 09:00 CEST:
Reschedule

We have received reports of packet loss towards Hetzner (AS24940)

We have denied the path over DE-CIX in the routing tables.

The best selected path is now going over NLIX.

We have received reports of slow speeds towards Orange Romania (AS8953)

We have denied the path over AMS-IX in the routing tables.

The best selected path is now going over LINX.

We have received reports of packet loss towards Globe Telecom Inc., Philippines (AS132199 )

We have denied the path over AMS-IX in the routing tables.

The best selected path is now going over Cogent.

After having contact with our vendor, one of the stack traces we have submitted seemed to be caused by a defect in the card.

At Thursday, 21 March 2019 8:00 AM CET, we will be replacing a line card in Router 2 of DBA.

During this time, traffic will be removed from the line card and balanced over Router 1 and the other line card of router 2. We do not expect any downtime.

We will be having a planned maintenance to migrate IPv6 to a different platform to ensure network stability. We will perform the maintenance on both DBC and NZS. DBA IPv6 is already moved to the new platform.

The move will consist out of 2 parts:

Monday, 11 March 2019 starting from 7:00 AM:
At this day, we will move customer IPv6 blocks to the new platform. During this time you may experience downtime on IPv6 up to 5 minutes. Customers using IPv6 BGP may notice that the second session will stay down after the maintenance. There are plans to bring back the second session later this year.

Tuesday, 12 March 2019 starting from 7:00 AM:
We will start the migration of transit and peering IPv6 to the new platform.
We do not expect any issues during this time.

Update, Tuesday, 12 March 2019 starting from 13:00:
We have successfully migrated the IPv6 to the new platform.

Wave service provider which connects Equinix AM7 (EQX-AM7) towards LINX is having a planned maintenance.

Their maintenance reason is "Maintenance needs to be performed on hardware in our network. These works will impact your service during the above-mentioned service window."

KPN has given this maintenance reference number: 97454

All LINX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

We have received reports of packet loss towards Converge ICT Solutions Inc. (AS17639)

We have denied the path over Cogent in the routing tables.

The best selected path is now going over NTT.

Currently we are experiencing an outage on our router 2 in DBC causing it to intermittently drop it's BGP neighbors causing some packetloss.

We are currently investigating the cause of this issue and will update this post accordingly.

Update 2019-02-25 17:28 CET:
The issue was followed back to the global access lists we use on our routers, we also noticed a much higher latency on the management of the routers. After removed the whole accesslist the latency returned to normal, after placing it back we saw it jump again in latency.

In the end we placed parts of it back while keeping an eye on the management layer, we will debug this more closely to see where the latency jump comes from when applying the entire access list.

Wave service provider which connects Global Switch Amsterdam (GSA) towards FRANCEIX is having a planned maintenance.

Their maintenance reason is " As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing mandatory splicebox works on our fiberoptical network. This planned activity will take place in France. "

Interoute has given this maintenance reference number: 1-4732551231

All FRANCEIX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

We have received reports of slow speeds towards Spectrum (AS20001)

We have denied the path over Cogent in the routing tables.

The best selected path is now going over Level3.

Currently we are experiencing an outage at our router in AM7 with the AMS-IX Peering platform.

Yesterday night AMS-IX performed a maintenance and we expect that this issue we are experiencing is caused by the maintenance.

AMS-IX is informed with additional details so they can investigate what happened.

As we have redundancy in place, our AMS-IX peering traffic is automatically rerouted over GSA. We do not expect any issues.

For transparency purposes we decided to publish this notification.

Update 2019-02-21, 11:28:
AMS-IX found out that the linecard they connected us to is defective and put us back on the primary path.

All traffic and BGP sessions of AMS-IX at AM7 are now restored.

We received a maintenance notification from one of our server datacenters Databarn Capelle a/d IJssel (DBC) that they are having a pre-emptive power maintenance.

Their exact maintenance reason is:

General Electric / IMV Nederland will perform this maintenance, during this maintenance we will put both generators into operation.

As all our equipment will run on generators we expect no issues, this does not affect any server hosting customers.

We have received a notification of a maintenance by one of our transit providers, GTT.

During their maintenance traffic will automatically reroute over other transit providers. We do not expect any issues.

Their maintenance reason is: " Performing upgrades; customer network grooming; optimization project "

Their reference: 3269891

For transparency purposes we decided to publish this notification.

At 6:35 CET we had a crash of the linecards in both routers R1 and R2 DBA causing also the BGP to our transit providers to flap.

After the cards came back online all routes also need to be relearned from all of our transit providers where packetloss or unreachabillity might have occured.

We've gathered logs for our vendor who is assisting us in debugging this issue.

Our sincere apologies for any issues caused by this outage.

At this moment the debugging process is still ongoing.

Update: 2019-02-21, 12:11:
While having close contact with our vendor, they have reported that the crash is happening within the IPv6 stack. They however were not able to resolve the issue at this moment.

We have looked for alternative options and did some tests. As this phase is completed, we would like to inform you that we are going to separate the IPv6 from our current routers away from IPv4. IPv6 will run independently.

Due to the recent crashes and our vendor yet to provide a patch against this issue, we will be performing the move of customer IPv6 to the new platform on DBA tomorrow (Friday, 22 Februari 2019) starting from 7:00. We expect to finish up around 12:00.

During this time, each IPv6 block can expect a downtime up to 5 minutes.

The IPv4 will be untouched and unaffected by this urgent maintenance.
There are no changes required at the customer side, please do note that when you do have IPv6 BGP sessions with us, 1 session will stay down until a later maintenance.


Update: 2019-02-22, 11:43:
We have finished the move of the customers IPv6 to the new platform.

At Monday, 25 Februari 2019, starting from 7:00, we will be moving the transit IPv6 sessions. We do not expect any downtimes and will be carefully moved. The process may take up until at latest 17:00.

As this is IPv6 only, IPv4 is not affected by the move.

Update: 2019-02-25, 14:42:
All transits and peering connectivity is moved and officially disconnected IPv6 from the old platform. IPv6 is now running independently.

For the status on the bug, we recommend you to keep an eye on the post: https://noc.nforce.com/notifications/item/749

Wave service provider which connects Global Switch Amsterdam (GSA) towards FRANCEIX is having a planned maintenance.

Their maintenance reason is " As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing mandatory cable works on our fiberoptical network. This planned activity will take place in France. "

Interoute has given this maintenance reference number: 1-4718224801

All FRANCEIX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

We have received reports of high latency towards Mob Servicos de Telecomunicacoes Ltda (AS28598)

We have denied the path over AMSIX in the routing tables.

The best selected path is now going over Cogent.

Wave service provider which connects Global Switch Amsterdam (GSA) towards DECIX is having a planned maintenance.

Their maintenance reason is " As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing works related to a cable permanent fix on our fiberoptical network. This planned activity will take place in Germany. "

Interoute has given this maintenance reference number: 1-4708343821

All DECIX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

Wave service provider which connects Equinix AM7 (EQX-AM7) towards LINX is having a planned maintenance.

Their maintenance reason is "Our provider informed us that, in order to guarantee the stability of their network, they need to conduct maintenance activities that will affect the above-mentioned service."

KPN has given this maintenance reference number: 96819

All LINX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

Darkfiber provider Relined which connects Equinix AM7 (EQX-AM7) towards Nikhef is having a planned maintenance. The connectivity between EQX-AM7 and Nikhef will automatically reroute over our GSA darkfiber during this maintenance.

Their maintenance reason is:
"Due to reconstruction work carried out by the local authorities. Relined is obliged to relocate a section of her network. This action will result in a temporary interruption of the service."

Their reference is: PA-008001

For transparency purposes we decided to publish notification.

At 22:40 CET, 1 Februari 2019 we have noticed a linecard crash on Router 1 of the datacenter Databarn Amsterdam.

As the linecard restored quickly after the crash there might have been interruptions of services due to the failover process.

We have made dumps of the crash logs and provided the information to the vendor for further investigation while we continue to monitor stability of the linecard.

Update 2019-02-02 8:35 CET:
We experienced another linecard crash at 2019-02-01 22:45 CET, which may briefly (~ 2 minutes) caused packetloss on your services in DBA.

Update 2019-02-07 12:46 CET:
We experienced another linecard crash last night at 3:13 AM CET, which may briefly (~ 2 minutes) caused packetloss on your services in DBA, this crashdump was added to the case with our vendor.

Update 2019-02-13 8:03 CET:
Around 18:28 CET, 12 Februari 2019, we have found our other linecard of our DBA router has crashed. We have submitted crashdump as well as other dumps to provide as much information as we can.

Update 2019-02-16 06:35 CET:
We experienced a linecard crash of both linecards in R1 and R2 of DBA. where also the BGP with our transits was dropped which needed to be re-converged after coming back online, this may have caused packetloss or unreachabillity.

Update 2019-02-18 04:52 CET:
We experienced linecard crashes both on R1 and R2 of DBC. As the stacktraces are exactly the same as the crashes of DBA, we expect this to be the same issue. Currently the vendor is still investigating the debugs.

Update 2019-04-18 14:00 CEST:
We have received an update from the vendor, having the issue found and solved. A new firmware will be released including the fix.
Once we have received the firmware, we will begin planning in firmware upgrades.

Wave service provider which connects Equinix AM7 (EQX-AM7) towards LINX is having a planned maintenance.

Their maintenance reason is "this maintenace is due to a faulty board, during the maintenace the estameted impact is 30 min"

KPN has given this maintenance reference number: 96442 and 96434

All LINX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

Update 2019-02-05 11:45:
KPN cancelled this maintenance.

Wave service provider which connects Equinix AM7 (EQX-AM7) towards LINX is having a planned maintenance.

Their maintenance reason is "MAINTENANCE KPN, Hardware"

KPN has given this maintenance reference number: 96339 and 96350

All LINX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

We have received reports of high latency towards Get AS (AS41164)

We have denied the path over Seabone in the routing tables.

The best selected path is now going over Centurylink.

Wave service provider which connects Global Switch Amsterdam (GSA) towards LINX is having a planned maintenance.

Their maintenance reason is "As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing hardware upgrade works on our transmission platform. This planned activity will take place in United Kingdom. "

Interoute has given this maintenance reference number: 1-4675538501

All LINX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

We have received reports of high latency towards Magyar Telekom plc (AS5483)

We have denied the path over GTT in the routing tables.

The best selected path is now going over Seabone.

Wave service provider which connects Global Switch Amsterdam (GSA) towards France-IX is having a planned maintenance.

Their maintenance reason is "As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing splice box replacement works on our fiberoptical network."

Interoute has given this maintenance reference number: 1-4660685311

All France-IX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

Wave service provider which connects Global Switch Amsterdam (GSA) towards DECIX is having a planned maintenance.

Their maintenance reason is "As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing splice box replacement works on our fiberoptical network."

Interoute has given this maintenance reference number: 1-4660685311

All DECIX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.