We have received a notification of a maintenance by one of our transit providers, Core-Backbone.

During their maintenance traffic will automatically reroute over our other transit providers. We do not expect any issues.

Their maintenance reason is "In the mentioned timeframe we will perform hardware upgrades at the following core-routers/core-switches:

ams10.core-backbone.com (Amsterdam1)
lon10.core-backbone.com (London1)
nbg40.core-backbone.com (Nuremberg4)
vie20.core-backbone.com (Vienna2)
vie10.core-backbone.com (Vienna1)
bam10.core-backbone.com (Bamberg1)
bam20.core-backbone.com (Bamberg2)".

Their reference number is: 1847

For transparancy purposes we decided to publish notification.

We have received reports of high latency and lag towards Saudi Telecom Company JSC.

We have removed AMSIX and Decix from the routing and the best route is going over Seabone.

Wave service maintenance
Routing LINX

Wave service provider which connects Equinix AM7 towards the LINX peering exchange in London is having a planned maintenance.

All LINX routed traffic is automatically routed over our other LINX peering exchange connection. We therefore do not expect any issues.

Their maintenance reason is "Please be informed that our provider need to perform fiber works.

Expected outage duration: 375 minutes ".

Their reference ID is 76757

For transparancy purposes we decided to publish notification.

At 00.10 CEST a darkfiber between DBC and GSA went down, carrying 200 Gbps traffic.

Currently all traffic is send over alternative paths on our ring, DBC - EQX-AM7 and DBC - NZS.

From DBC you might experience increased latency and lower throughput towards routes terminated at our GSA pop.

We have contacted the darkfiber provider euNetworks and requested a repair.

Update 11.50:
It was a planned maintenance of which we were not notified. This has been escalated to euNetworks to resolve.

At 03.10 the darkfiber was partially restored.

At 08.10 the darkfiber was full restored.

In order to support new features such as multiple power ports per server, we are currently migrating pdu port information.

If you are unable to operate the power (pdu port) of your server, please contact our support department. Our support department is able to manually migrate you with priority in order to restore this feature.

This morning after several messages of customers about packetloss with Seabone we started an investigation into these issues.

We found out the issue was with one of our BGP sessions with them and to resolve this while debugging we disabled this BGP session on all of the routers.

We shutdown the connections we had with them on this second connection and found out the issue lied in on of the ports, this issue has been solved now and the BGP sessions have been Established again.

We've tested with some of the IP's customers provided and do not see these issues currently.

Wave service provider which connects Equinix-AM7 towards Frankfurt is having a planned maintenance.

Their maintenance reason is "KPN International is always working to improve the service for our customers. To ensure an optimal service we need to perform Planned Maintenance activities that will affect one of your network connections. Details: Fiber relocation due to road construction works by the local authorities"

KPN International has given this maintenance reference number: 76409.

All DE-CIX routed traffic is automatically routed over our other exchanges and IP transits.

Wave service provider which connects Global Switch Amsterdam towards Paris is having a planned maintenance.

Their maintenance reason is "INTEROUTE has received notification that our supplier will perform Mandatory Maintenance on their network."

Interoute has given this maintenance reference number: 1-2763684963.

All FranceIX routed traffic is automatically routed over our other exchanges and IP transits.

Update 2017-09-22: The work was updated by INTERROUTE with the reason: "We would like to inform you these works have been cancelled by our external provider."

We have received reports of high latency towards IPKO.

We have removed Cogent from the routing and the best route is going over Seabone.

Currently there is a fiber cut on the darkfiber connecting Nedzone (NZS) to Equinix AM7 (EQX-AM7).

We noticed that the connection went down on a few occasions after each other and contacted Eurofiber about this issue.

Update 2017-09-08 13.12:
We have seen the network connection re-established and stabilized.

Update 2017-09-08 13.30:
We received the reason of outage from Eurofiber:

Due to non-affecting maintenance from Eurofiber, the engineer of Eurofiber accidentally broke a fiber.

We haven't seen any further interruptions on this connection however we will monitor the connection to ensure stable performance.

Due to redundancy, our service was not effected.

After a confirm that Etisalat has solved the issue, the previous route change has been removed

Wave service provider which connects Global Switch Amsterdam towards Paris is having a planned maintenance.

Their maintenance reason is "As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing mandatory cable works on our fiberoptical network. This planned activity will take place in France."

Interoute has given this maintenance reference number:1-2665689161.

All FranceIX routed traffic is automatically routed over our other exchanges and IP transits.

Update:
Interoute has reschedule this maintenance to a new date:
New Planned Work Start Date: 14/Oct/2017 21:00 UTC (14/Oct/2017 23:00 CEST)
New Planned Work End Date: 15/Oct/2017 05:00 UTC (15/Oct/2017 07:00 CEST)

We have received a notification of a maintenance by one of our transit providers, Seabone.

During their maintenance traffic will automatically reroute over our other Seabone connection as well as over other transit providers. We do not expect any issues.

Their maintenance reason is: "our engineers will carry out activities in Amsterdam (Holland): migration of fiber on new odf. This activity will cause a maximum downtime of 240 minutes.".
Reference: 1-914551833

For transparancy purposes we decided to publish notification.

We have received a notification of a maintenance by one of our transit providers, Seabone.

During their maintenance traffic will automatically reroute over our other Seabone connection as well as over other transit providers. We do not expect any issues.

Their maintenance reason is: "TIS has planned a Maintenance Work. Reconfiguration circuit due to network optimization. This cause an interruption of 30 min max for circuit".

Reference: 1-914450407

For transparancy purposes we decided to publish notification.

*** Please note that this outage only affects customers with a fiber to the business internet connection. ***

Currently there is a partial outage for internet access customers. The core ring topology carrying traffic from/to customer end-points is experiencing unknown resource issues. This causes mac-addresses to timeout and not immediately get properly (re)installed and exchanged over the ring topology.

We are currently working as hard as we can to solve this issue en get everyone back online.

Update 16.25:
Hardware resources restored on slot 2 (ring-sw2.eqx-am7).

Update 17.25:
Hardware resources restored on chassis (ring-sw2.eqx-am7).

Update 2017-08-22 09.00:
Diagnostic-information send to vendor for debugging.

We have received reports of slow speeds to Comcast.

We have removed NTT from the routing and the best route is going over Gtt.

Dear client,

Please note that this is merely an informational notification, no incidents in our network and hosting infrastructure were detected. However, we felt the need to post news on this outage regarding the power-grid directly adjecent to the DBA DC in Amsterdam. A midstation was affected over the Leander power network at 14.19 CEST.

At this moment we can confirm that the DBA datacenter in which we have presence did not show any sign of outage due to the midstation failure. Also none of our transit providers seem to be directly affected either. Neither do we see any decrease in bandwidth throughput.

We received notifications from the DBA hosting facility that they were running on backup power since the start of the incident 14.19 CEST and have shifted back to the power grid at 15.39 CEST with no incidents reported.

DBA staff has been present during the power-grid outage to monitor the situation throughout and will continue to do so in the coming hours. They confirm no services were affected and they report everything went as it should regarding the shift to and from UPS/Aggregate when at the given times.

We have received a notification of a maintenance by one of our transit providers, Seabone.

During their maintenance traffic will automatically reroute over our other Seabone connection as well as over other transit providers. We do not expect any issues.

Their maintenance reason is: "TIS has planned a Maintenance Work. Cable relocation in Amsterdam.".

For transparancy purposes we decided to publish notification.

During this timeframe there will be a routine maintenance on the darkfiber connecting GSA (Globalswitch Amsterdam) to EQX-AM5 (Equinix Amsterdam 5).

The traffic to EQX-AM5 will automatically reroute over our Equinix AM7 - Equinix AM5 darkfiber during this maintenance.

No outages or other issues are expected during this maintenance.

Third party reason: “Due to reconstruction work carried out by the local authorities. Relined is obliged to relocate a section of her network. This action
will result in a temporary interruption of the service.”

Relined has given this maintenance reference number: PA-476001

At 9.00 CEST the darkfiber going from DBC to Globalswitch Amsterdam (GSA) went down.

After having contact with the 3rd party maintaining the fiber a cable cut was discovered along the fiber in question, they are now sending engineers to the location of the cut.

At this moment we do not have an ETR available.

Currently all traffic is send over the other half of our ring between DBC - EQX-AM7, we do not expect issues.

Update 21:17:
Fibre crews are on-site and are currently trying to locate the joint which is buried in a new road scheme. There are construction crews either side of the joint location.

Update 22:00:
Fibre Crews are testing now on the closest joints to the fault. In parallel it is investigated if it is possible to move the route of the affected services to working spare fibres.

Update 00:14:
Due to the local conditions on site at the fault location further
digging works are necessary causing additional delay to the restoration.
We will keep you updated.

Update 9:00:
euNetworks Customer Care sees most of the alarms have cleared on our systems.
However, one Metro System remains down.
We have informed our local fibre provider to check these fibres with high priority to ensure fast restoration.
Please have your services checked and your ports reset.
In case of any problems please report back to us so we can ensure a fast resolution.

Update 15:00:
We have been informed that your service was affected by a fibre cut caused by civil works. There has been a delay in restoring your services as it was a partially fibre cut and it was very hard to locate the damage site. Due to the size of the construction site and to restore your services quicker we needed to move them to an alternate fiber pair. This has caused some extra delay as additional splicing was needed.

Our darkfiber running from DBC to GSA is now back to being fully functional again and the darkfiber ring now back to being redundant.

*** Please note that this maintenance only affects customers with a fiber to the business internet connection at their office(s). This third party maintenance is not related to, and thus will not affect, any hosting or IP transit services. ***

Brightfiber who operates the fiber infrastructure for FTTB customers in Vaandorp, Cornelisland and bijdorp is having a planned maintenance. This maintenance will start at 07.00 CEST and will end at 12.00 CEST.

We have received a notification of a maintenance by one of our transit providers, Zayo.

During their maintenance traffic will automatically reroute over our other transit providers. We do not expect any issues.

Their maintenance reason is "Zayo will perform scheduled software upgrade on mpr1.ams10.nl".

Zayo has given this maintenance reference number: TTN-0001831670.

For transparancy purposes we decided to publish notification.

During this timeframe there will be a routine maintenance on the darkfiber connecting DBA towards GSA (Globalswitch Amsterdam).

The connectivity between Globalswitch and DBA will automatically reroute over our Equinix AM7 darkfiber during this maintenance.

No outages or other issues are expected during this maintenance.

Third party reason: "As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing cable rerouting works on our fiberoptical network. This planned activity will take place in Netherlands."

Interoute has given this maintenance reference number: 1-2664446539.

Wave service provider which connects Global Switch Amsterdam towards Paris is having a planned maintenance.

Their maintenance reason is "As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing cable rerouting works on our fiberoptical network. This planned activity will take place in Netherlands."

Interoute has given this maintenance reference number: 1-2664427746.

All FranceIX routed traffic is automatically routed over our other exchanges and IP transits.

Wave service provider which connects Global Switch Amsterdam towards Paris is having a planned maintenance.

Their maintenance reason is "As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing hardware upgrade works on our transmission platform. This planned activity will take place in Belgium."

All FranceIX routed traffic is automatically routed over our other exchanges and IP transits.

Update 2017-07-26:
The planned worked announced for 2017-07-30 has been rescheduled by Interoute to 2017-07-29.

We have received a notification of a maintenance by one of our transit providers, Core-Backbone.

During their maintenance traffic will automatically reroute over our other transit providers. We do not expect any issues.

Their maintenance reason is:
In the mentioned timeframe we will perform hardware upgrades at the following core-routers/core-switches:

fra20.core-backbone.com (Frankfurt2)

For transparancy purposes we decided to publish notification.

We have received a notification of a maintenance by one of our transit providers, Core-Backbone.

During their maintenance traffic will automatically reroute over our other transit providers. We do not expect any issues.

Their maintenance reason is: "In the mentioned timeframe we will perform hardware upgrades at the following core-routers/core-switches:

nbg20.core-backbone.com (Nuremberg2)
muc10.core-backbone.com (Munich1)
muc20.core-backbone.com (Munich2)".

For transparancy purposes we decided to publish notification.

We have received a notification of a maintenance by one of our transit providers, Seabone.

During their maintenance traffic will automatically reroute over our other Seabone connection as well as over other transit providers. We do not expect any issues.

Their maintenance reason is: "We are forced to move our equipment in Amsterdam into a new room . The migration of the circuits will be carried out with the least possible interruption . Unfortunately the activity can be not postpone *** CIRCUITS WILL BE SERVICE AFFECTING ***".

For transparancy purposes we decided to publish notification.

Slow speeds reported to Etisalat.

We have denied the path over AMSIX in the routing tables.

Best selected path is now going over Cogent.

Wave service provider which connects Global Switch Amsterdam towards Paris is having a planned maintenance.

Their maintenance reason is "As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing hardware upgrade works on our transmission platform. This planned activity will take place in France."

All FranceIX routed traffic is automatically routed over our other exchanges and IP transits.

Wave service provider which connects Global Switch Amsterdam towards Paris is having a planned maintenance.

Their maintenance reason is "As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing hardware upgrade works on our transmission platform. This planned activity will take place in France."

All FranceIX routed traffic is automatically routed over our other exchanges and IP transits.

Wave service provider which connects Globalswitch Amsterdam towards the FranceIX peering exchange in Paris is having an outage.

Their outage reason was "Major Incident on Transmission Platform. Incident Type Fibre Break Interoute in Belgium".

All FranceIX traffic is automatically rerouted over other peerings and transits.

For transparancy purposes we decided to publish notification.


Update 20:10
Interoute resolved the fiber cut. Connectivity is restored.

Wave service provider which connects Global Switch Amsterdam towards Paris is having a planned maintenance.

Their maintenance reason is "As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing hardware upgrade works on our transmission platform. This planned activity will take place in France."

All FranceIX routed traffic is automatically routed over our other exchanges and IP transits.

Update 2017-07-07:
This maintenance has been canceled by Interoute, their cancellation reason is: "Please be informed that this Planned Work has been postponed until further notice. As soon as a new time window has been decided you will be notified. We apologize for the inconvenience and thank you for your understanding "

Thursday 6th of July 2017 at 01:00 until 02:00 we will have a maintenance on the router at Equinix AM7 (EQX-AM7), all traffic will be rerouted and should not be affected.

Transit Customers on EQX-AM7 without a redundant setup may experience outage at this moment.

All fiber to the business customers located on this router may experience an outage at this moment.

Due to the type of maintenance we will have to bring down the router for re-adjustments.

We strive to have the downtime as short as possible and expect the downtime on this location to be less than 1 hours.

Update 3:06:
The hardware maintenance on the router has been done, we are now establishing all BGP sessions and monitoring their progress.

Wave service provider which connects Globalswitch Amsterdam towards the LINX peering exchange in London is having a emergency planned maintenance.

As this is an at risk notification, we expect no issues.

Their maintenance reason is "Please be advised that Interoute will be performing a Service Threatening Planned Maintenance in order to test free fibers with risk of flaps. "

For transparancy purposes we decided to publish notification.

During this time frame there will be a maintenance on the dark fiber connecting NZS (Nedzone) towards EQX-AM7 (Equinix AM7).

The connectivity between NZS and EQX-AM7 will automatically reroute over our Global Switch Amsterdam dark fiber during this maintenance.

Third party reason: "Please be advised that we will perform an urgently required cable relocation, there will be 2 times a small interruption of 5 minutes at most".

A DDoS targeted towards DBC (Databarn Capelle) to a router IP caused one of the routers to massively drop traffic.

We are currently trying our best to defend against this attack and will start a further investigation in this issue.

Once we have more information available we will update this post.

Update 2017-06-11 1:17AM CEST:
As the router is becoming more stable at this time, we also have contacted the manufacturer of the routers as one of the defenses against these attacks did not work as intended.

Wave service provider which connects Global Switch Amsterdam towards Paris is having a planned maintenance.

Their maintenance reason is "in an effort to prevent an uncontrolled outage, we are forced to undertake an Emergency hardware upgrade works on our transmission platform. These works will be performed in France.".

All FranceIX routed traffic is automatically routed over our other exchanges and IP transits.

2017-06-08:
We have received an email from the wave service provider that the above maintenance has been cancelled.

At 13.10 CEST the darkfiber going from DBC to Equinix-AM7 (EQX-AM7) went down.

After having contact with the 3rd party maintaining the fiber a cable cut was discovered along the fiber in question, they are now sending engineers to the location of the cut.

Update 15:35:
Splicing crew has restored the fiber bundle and the darkfiber is now back online.

During this timeframe there will be a maintenance on the darkfiber connecting DBC towards Equinix-AM7 (EQX-AM7).

The connectivity between EQX-AM7 and DBC will automatically reroute over our GSA darkfiber during this maintenance.

We do not expect any issues during this maintenance.

Third party reason: "Please be advised that euNetworks vendor will perform a necessary cable relocation on the dates and times shown below".

We have received a notification of a maintenance by one of our transit providers, Seabone.

During their maintenance traffic will automatically reroute over our other Seabone connection as well as over other transit providers. We do not expect any issues.

Their maintenance reason is "Our provider in Amsterdam will implement planned activity Due to reconstruction work carried out by the local authorities our provider is obliged to relocate a section of her network. This action will result in a temporary interruption of the service.".

For transparancy purposes we decided to publish notification.

At 19:30 CEST one of the feeds located at the Globalswitch Amsterdam datacenter (GSA) shutdown because of a blown fuse, the reason was a defective router power module.

Because of the feed outage the amplifier going from GSA to Databarn Capelle (DBC) lost one of it power supplies, the device is connected redundantly but the backup power supply made the amplifier stop accepting signals.

We immediately discussed this with the vendor of the Amplifier and after a short debug where asked to proceed with the restart of the complete amplifier after which the connection came back online.

We will be discussing this further with the vendor.

We have received a notification of a maintenance by one of our transit providers, NTT.

During their maintenance traffic will automatically reroute over our other NTT connection as well as over other transit providers. We do not expect any issues.

Their maintenance reason is "Users connected to this device will experience latency/downtime as we will be performing hardware maintenance affecting the service(s) listed below".

For transparancy purposes we decided to publish notification.

"Router 1 at Global Switch Amsterdam (GSA) is experiencing CRC errors on one of the uplinks towards NL-IX.

We do not expect any issues as traffic will be rerouted over NL-IX at our router in Equinix-AM7 Amsterdam.

Once the engineer arrives at the datacenter we will directly start the maintenance. The maintenance should not take more than 30 minutes.

For transparency purposes we decided to publish notification."

Router 1 at GlobalSwitch Amsterdam (GSA) experienced a failover from the primary to secondary management controller. The failover itself happened at 11.19 and toke several seconds.

Several hundred BGP sessions flapped and are currently exchanging routes. During this you might see routes changing and rerouting.

Update 13:10: The routing table is fully loaded and the system appears to be stable.

There is a fiber cut on the darkfiber connecting Databarn Capelle (DBC) to Globalswitch Amsterdam (GSA).

Engineers are dispatched to the location of the probable cut of the fiber.

Because of the outage some higher latency or small packetloss can occur.


Update 2017-05-11 22.20:
The darkfiber has been re-established again using a new path, because this is a completely new fiber over a different path throughout the Netherlands the amplifiers need to be re-tuned for the new fiber, on the side of GSA this has been done now after which the fiber stabilized.

Engineers are currently on their way to DBC to do the same tuning there, during that tuning you will notice some packetloss as this can only be done live.

Once the tuning at DBC has been done we expect the darkfiber to be fully working again.


Update 2017-05-11 23.35:
The DBC side of the darkfiber has also been tuned and optimized after which the connection DBC-GSA is fully functional again now.

We also received word from euNetworks that they are having a maintenance on the darkfiber between DBC-EQX-AM7 between 0.00 and 09.00 CEST, we do not expect any issues during this maintenance.

After their maintenance both darkfibers should be fully functional again.


Update 2017-05-12 00.10:
During this incident the fiber was already relocated over the west-side of The Netherlands. Therefore the works planned for next week in case #474 is already finished and hereby cancelled.

Wave service provider which connects Globalswitch Amsterdam towards the FranceIX peering exchange in Paris is having an outage.

Their outage reason was "Major Incident on Transmission Platform. Incident Type Fibre Break Interoute in Belgium".

All FranceIX traffic is automatically rerouted over other peerings and transits.

For transparancy purposes we decided to publish notification.

Update 2017-05-11 15:01:00:
Engineers have repaired the fiber break in Belgium after which the connection came back online.

We have received a notification of a maintenance by one of our transit providers, NTT.

During their maintenance traffic will automatically reroute over our other NTT connection as well as over other transit providers. We do not expect any issues.

Their maintenance reason is "Users connected to this device will experience latency/downtime as the router is reloaded and routing is reestablished. We will be performing a software upgrade affecting the service(s) listed below.".

For transparancy purposes we decided to publish notification.

During this timeframe there will be a maintenance on the darkfiber connecting DBC towards Globalswitch Amsterdam (GSA).

The connectivity between GSA and DBC will automatically reroute over our Equinix AM7 darkfiber during this maintenance.

We do not expect any issues during this maintenance.

Update 2017-05-12 00.10
During the incident in #474 the fiber has already been relocated over another path throughout The Netherlands (west-side). Therefore this planned maintenance is hereby cancelled.

Wednesday 17th of May 2017 at 19:00 until 21:00 we will have a maintenance on the router at Global Switch Amsterdam (GSA), all traffic will be rerouted and should not be affected.

Transit Customers GSA without redundant setup may experience outage at this moment.

Due to the type of maintenance we will have to bring down the router for re-adjustments.

We strive to have the downtime as short as possible and expect the downtime on this location to be less than 1 hours.

Update:
We have re-scheduled the maintenance from 17th of May 2017 at 12:00 until 16:00, to 17th of May 2017 at 19:00 until 21:00 due to some adjustments in the maintenance.

Update 21:32:
The hardware maintenance on the router has been done, we are now establishing all BGP sessions and monitoring their progress.