#594 Wave service outage LINX

Posted: 2018-01-23 06:50

Start: 2018-01-23 03:45:00
End : 2018-01-23 07:45:00

Affects: Routing LINX EQX-AM7

At 3:45 AM we lost connectivity with one link at Equinix AM7 (Amsterdam) towards LINX. Our wave service provider have notified us about an outage within their network.

During the outage, all traffic towards LINX will be rerouted over our LINX connection at Global Switch Amsterdam.

The outage is not within NFOrce Network but with a 3rd party.

Update 5:26 AM: Our wave service provider updated us that the outage is classified as a major outage on their platform.

Update 5:46 AM: We received an update that their engineer will be onsite to investigate and resolve the issue.

Update 7:01 AM: We received an update from the wave service provider: We are facing power issues in Brentwood.

They have enabled a generator to restore the services temporary. Once the power is back they will switch back to normal power.

#593 Packetloss AMS-IX traffic

Posted: 2018-01-22 12:51

Start: 2018-01-22 11:30:00
End : 2018-01-22 11:45:00

Affects: AMS-IX traffic, mostly IP transit customers

At 11:30 AM, we have experienced a shift in traffic from AMS-IX to our other transit providers.

After investigating our network we did not find any issues within our network itself.

Final investigation results shows that one of the routers received the new route updates from AMS-IX later than our other router. This caused a shift that traffic moved around to other transit providers and packets to be dropped where the TTL (Time to Live) expired.

This issue only persisted for few minutes and while we have monitored the last hour we have not seen this issue return.

#581 Partial power outage Equinix AM5

Posted: 2018-01-02 01:30

Start: 2018-01-01 23:45:00
End : 2018-01-02 18:10:00

Affects: Power redundancy Equinix AM5

At 23.45 CET we got a notice from our monitoring system that one of the powerfeeds inside Equinix AM5 went down.

We've sent in a trouble ticket for Equinix AM5 staff to check and were notified Equinix had a power incident inside the datacenter.

The following was sent by Equinix staff:
INCIDENT SUMMARY: A power incident was detected on 01-JAN-2018 23:45 at AM5.


Equinix engineers are evaluating the situation and will provide more details as they become available.

The next update will be when a significant change to the situation occurs.

Our equipment is connected redundantly over 2 feeds, therefore we have only lost a redundancy in power, and do not expect any issues.

For transparency we decided to publish this notification.

Update 02:42:
Equinix engineers have identified a section of busbar that failed for UPS power feed C in Zone 1 of AM5.

The redundant feed remains operational.

Equinix engineers are still working to fully restore the faulty feed.

Affected customers are using the available functioning power feed.

We will keep you posted of the status as more information becomes available.

Thank you for your understanding and patience.

Update 07:00:
We are working on a temporary solution to connect the faulty busbar to provide redundant feed again.

We will keep you posted of the status as more information becomes available.

Thank you for your understanding and patience.

Update 11:30:
Spare parts are on their way to the IBX and will be installed immediately.

We will keep you posted of the status as more information becomes available.

Thank you for your understanding and patience.

Update 12:52:
All materials to create a temporary solution are now on site. We will start to install the temporary solution from 1400 CET onwards. The total work duration is estimated at 3 hours.

We will start to find the final solution and the root cause of this incident after the temporary solution is in place and operational. Our estimation at this moment is that this can take several weeks.

Update 18:10:
The feed that went down is now back up again and all our systems are redundantly connected again.

In the near future Equinix will look for a final solution but as our systems are up and running with redundant power we will close this case on our side.

#580 LINX Outage GSA

Posted: 2017-12-30 00:13

Start: 2017-12-29 22:42:00
End : 2017-12-30 16:31:00

Affects: LINX traffic GSA

At 22:42 we lost signal with LINX at our router in Global Switch Amsterdam (GSA), we have directly investigated the issue and suspect the issue to be with the wave service provider.

We have informed the wave service provider to inspect the issue and have been informed that this issue is located on their transmission platform in the United Kingdom.

As our setup is redundant, the traffic is automatically being rerouted over EQX-AM7.

Update 2017-12-30 22:25:
The break is located at a sub sea cable and a marine cable repair ship will be sent out at the 1st of January.

Therefore Interoute decided to migrate all connections physically to a backup path, which has been going on the entire day,currently our connection is back up again.

It's possible that the connection will be reverted once the primary path has been fixed.

#573 IP transit provider Core-Backbone

Posted: 2017-12-18 12:34

Start: 2017-12-22 00:00:00
End : 2017-12-22 06:00:00

Affects: Routing Core-Backbone

We have received a notification of a maintenance by one of our transit providers, Core-Backbone.

During their maintenance traffic will automatically reroute over our other transit providers. We do not expect any issues.

Their maintenance reason is: "In the mentioned timeframe we will perform hardware upgrades at the following core-routers/core-switches: (Amsterdam1)"

For transparancy purposes we decided to publish notification.