All Notifications

#605 DECIX maintenance EQX-AM7

Posted: 2018-02-22 15:11

Start: 2018-03-21 00:00:00
End : 2018-03-21 06:00:00

Affects: Routing DECIX EQX-AM7

DECIX peering exchange is having a planned maintenance on the port directed towards Equinix-AM7.

Their maintenance reason is "We will commence maintenance work to upgrade several of our nodes to improve redundancy. Therefore, we have to move your current connection. The work will cause a temporary downtime for about 5 minutes per port.

The migration is split into several sessions."

All DE-CIX routed traffic is automatically routed over our other DECIX connection.

#604 Power feed maintenance Equinix AM5

Posted: 2018-02-19 15:16

Start: 2018-03-27 07:30:00
End : 2018-03-27 15:30:00

Affects: Power redundancy Equinix AM5

This maintenance does not affect any hosting or ip transit services.

Following NOC maintenance #581 earlier this year Equinix will be placing the feed back on the main busbar after taking temporary measures in the earlier maintenance.

Below you can find the maintenance message which was sent to us:
Please be advised that Equinix and our approved contractor will be performing remedial works to migrate several sub-busbar sections back from there temporary source to the replaced main busbar which became defective as reported in incident AM5 - [5-123673165908].

During the migration, one (1) of your cabinet(s) power supplies will be temporary unavailable for approximately six (6) hours. The redundant power supply(s) remains available and UPS backed.


#603 Wave service maintenance FranceIX

Posted: 2018-02-19 14:19

Start: 2018-03-02 23:00:00
End : 2018-03-03 05:00:00

Affects: Routing FRANCEIX

Wave service provider which connects Globalswitch towards the FranceIX peering exchange in Paris is having a planned maintenance.

All FranceIX routed traffic is automatically routed over our other peering exchanges or IP transit connections.

Their maintenance reason is "As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing splice box replacement works on our fiberoptical network. This planned activity will take place in Belgium. Please be advised your services will be affected as per table below between 02/Mar 22:00 UTC and 03/Mar 04:00 UTC. INTEROUTE would like to apologise for the inconvenience these works will cause to your business."

Their reference: 1-3475945451

For transparancy purposes we decided to publish this notification.

#602 IP transit provider NTT

Posted: 2018-02-16 19:14

Start: 2018-03-02 02:00:00
End : 2018-03-02 05:00:00

Affects: Redundancy routing NTT

We have received a notification of a maintenance by one of our transit providers, NTT.

During their maintenance traffic will automatically reroute over our other NTT connection as well as over other transit providers. We do not expect any issues.

Their maintenance reason is "Users connected to this device will experience downtime during this maintenance window, We will be performing performing migrations affecting the service(s)".

For transparancy purposes we decided to publish notification.

#601 Wave service maintenance LINX

Posted: 2018-02-16 15:49

Start: 2018-03-10 23:00:00
End : 2018-03-11 05:00:00

Affects: Routing LINX GSA

Wave service provider which connects Globalswitch Amsterdam towards the LINX peering exchange in London is having a planned maintenance.

All LINX traffic is automatically routed over our other LINX peering exchange connection.

Their maintenance reason is "As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing ODF replacement on our fiberoptical network. This planned activity will take place in United Kingdom. Please be advised your services will be affected as per table below between 10/Mar 22:00 UTC and 11/Mar 04:00 UTC. INTEROUTE would like to apologise for the inconvenience these works will cause to your business."

Their reference: 1-3439647251

For transparancy purposes we decided to publish this notification.

#600 Wave service maintenance LINX

Posted: 2018-02-09 07:26

Start: 2018-03-02 00:01:00
End : 2018-03-02 06:30:00

Affects: Routing LINX

Wave service provider which connects Equinix AM7 towards the LINX peering exchange in London is having a planned maintenance.

All LINX routed traffic is automatically routed over our other LINX peering exchange connection. We therefore do not expect any issues.

Their maintenance reason is "Fiber optic works needs to take place in our network. These works will impact your service during the above mentioned service window.".

Their reference is: 81369

For transparancy purposes we decided to publish notification.

Update 2018-02-15 from carrier:
"By means of this e-mail I would like to inform you that the maintenance activities, with reference 81369, have been cancelled and will not be executed in the time window mentioned below. In case rescheduling is needed you will be informed accordingly."

#599 Fiber @ work maintenance

Posted: 2018-02-09 07:24

Start: 2018-03-07 00:00:00
End : 2018-03-07 06:00:00

Affects: Fiber @ Work customer at Kruisweg in Hoofddorp (NL)

Fiber provider which is delivering connectivity our POP at Kruisweg in Hoofddorp (NL) is having a planned maintenance.

This POP is only used for Fiber @ Work connectivity and does not affect any hosting or ip transit services.

Reason provided by provider:
To be able to offer future-proof services, Eurofiber will introduce an upgrade in software versions on a part of her network elements. The software upgrade will cause a brief disconnection (10-30 minutes).

Their reference: RFC-253002

#598 Fiber @ work maintenance

Posted: 2018-02-09 06:49

Start: 2018-03-06 00:00:00
End : 2018-03-06 06:00:00

Affects: Fiber @ Work customer at Kruisweg in Hoofddorp (NL)

Fiber provider which is delivering connectivity our POP at Kruisweg in Hoofddorp (NL) is having a planned maintenance.

This POP is only used for Fiber @ Work connectivity and does not affect any hosting or ip transit services.

Reason provided by provider:
To be able to offer future-proof services, Eurofiber will introduce an upgrade in software versions on a part of her network elements. The software upgrade will cause a brief disconnection (10-30 minutes).

Their reference: RFC-459002

#597 Wave service maintenance FranceIX

Posted: 2018-02-07 08:35

Start: 2018-02-14 00:00:00
End : 2018-02-14 09:00:00

Affects: Routing FranceIX GSA

Wave service provider which connects Globalswitch towards the FranceIX peering exchange in Paris is having a planned maintenance.

All FranceIX routed traffic is automatically routed over our other peering exchange or IP transit connections.

Their maintenance reason is " INTEROUTE has received notification that our supplier will perform cable segment replacement works on our fiberoptical network on their network. These works have been scheduled to take place between 13/Feb 23:00 UTC and 14/Feb 08:00 UTC. We hereby notify you that these works will cause your service(s) to be affected as per the table below. These works will be performed in Belgium. INTEROUTE would like to apologise for the inconvenience these works will cause to your business."


Their reference: 1-3421818778

For transparancy purposes we decided to publish this notification.

Update 2018-02-13 13.11 (from provider):
Dear Customer, Please be advised that your services will NOT be impacted by Planned Work 1-3421818778. We apologize for any inconvenience caused. Kind Regards, Network Operations.

#596 Upgrade capacity between BNR - DBC

Posted: 2018-01-31 15:31

Start: 2018-02-20 11:00:00
End : 2018-02-20 14:00:00

Affects: Traffic BNR towards DBC

At 20 February 2018, we will be performing an upgrade in our capacity from BNR (Bytesnet Rotterdam) towards DBC (Databarn Capelle aan den IJssel).

We will be installing new device that is making it possible to upgrade the capacity from 100 hundred Gbits to 200 hundred Gbits, at this time we will also increase the current capacity.

The traffic will be automatically rerouted over Globalswitch.

#595 Wave service maintenance FranceIX

Posted: 2018-01-30 18:36

Start: 2018-01-30 23:00:00
End : 2018-01-31 03:00:00

Affects: Routing FranceIX GSA

Wave service provider which connects Globalswitch towards the FranceIX peering exchange in Paris is having a planned maintenance.

All FranceIX routed traffic is automatically routed over our other peering exchange or IP transit connections.

Their maintenance reason is "Planned Work Description and Location: Belgium. INTEROUTE, in an effort to prevent an uncontrolled outage, are forced to undertake an Emergency hardware upgrade works on our transmission platform. These works will be performed between 30/Jan 22:00 UTC and 31/Jan 02:00 UTC."


Their reference: 1-3405834081

For transparancy purposes we decided to publish this notification.

#594 Wave service outage LINX

Posted: 2018-01-23 06:50

Start: 2018-01-23 03:45:00
End : 2018-01-23 07:45:00

Affects: Routing LINX EQX-AM7

At 3:45 AM we lost connectivity with one link at Equinix AM7 (Amsterdam) towards LINX. Our wave service provider have notified us about an outage within their network.

During the outage, all traffic towards LINX will be rerouted over our LINX connection at Global Switch Amsterdam.

The outage is not within NFOrce Network but with a 3rd party.

Update 5:26 AM: Our wave service provider updated us that the outage is classified as a major outage on their platform.

Update 5:46 AM: We received an update that their engineer will be onsite to investigate and resolve the issue.

Update 7:01 AM: We received an update from the wave service provider: We are facing power issues in Brentwood.

They have enabled a generator to restore the services temporary. Once the power is back they will switch back to normal power.

#593 Packetloss AMS-IX traffic

Posted: 2018-01-22 12:51

Start: 2018-01-22 11:30:00
End : 2018-01-22 11:45:00

Affects: AMS-IX traffic, mostly IP transit customers

At 11:30 AM, we have experienced a shift in traffic from AMS-IX to our other transit providers.

After investigating our network we did not find any issues within our network itself.

Final investigation results shows that one of the routers received the new route updates from AMS-IX later than our other router. This caused a shift that traffic moved around to other transit providers and packets to be dropped where the TTL (Time to Live) expired.

This issue only persisted for few minutes and while we have monitored the last hour we have not seen this issue return.

#592 Wave service maintenance LINX

Posted: 2018-01-22 09:34

Start: 2018-02-06 00:01:00
End : 2018-02-06 06:30:00

Affects: Routing LINX EQX-AM7

Wave service provider which connects Equinix AM7 (Amsterdam, Former Telecity 2) towards the LINX peering exchange in London is having a planned maintenance.

All LINX routed traffic is automatically routed over our other LINX peering exchange connection.

Their maintenance reason is "Fiber optic works needs to take place in our network"


Their reference: 80552

For transparancy purposes we decided to publish this notification.


2018-02-06 00:00
Maintenance is cancelled by KPN.

#591 Wave service maintenance LINX

Posted: 2018-01-17 19:11

Start: 2018-01-21 23:00:00
End : 2018-01-22 01:00:00

Affects: Routing LINX GSA

Wave service provider which connects Globalswitch towards the LINX peering exchange in London is having a planned maintenance.

All LINX routed traffic is automatically routed over our other LINX peering exchange connection.

Their maintenance reason is "As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing capacity optimization works on our transmission platform. This planned activity will take place in Netherlands."


Their reference: 1-3337691291

For transparancy purposes we decided to publish this notification.


Update:

Our Wave service provider LINX has reschedule his maintenance.
New Planned Work Start Date: 21/Jan/2018 23:00 CET
New Planned Work End Date: 22/Jan/2018 01:00 CET

#590 High latency towards Emirates Telecommunications Corporation (AS5384)

Posted: 2018-01-15 09:53

Start: 2018-01-15 09:53:24
End : 2018-01-15 09:53:24

Affects: Traffic towards Emirates Telecommunications Corporation (AS5384)

We have received reports of high latency Emirates Telecommunications Corporation (AS5384).

We have removed NTT from the routing, the best route is now Level3

#589 Packetloss towards Leaseweb Singapore (AS59253)

Posted: 2018-01-15 08:07

Start: 2018-01-15 08:07:13
End : 2018-01-15 08:07:13

Affects: Traffic towards Leaseweb Singapore

We have received reports of packet loss towards Leaseweb Singapore (AS59253).

We have removed NTT from the routing, the best route is now GTT

#588 Upgrade capacity between NZS - GSA

Posted: 2018-01-11 20:32

Start: 2018-01-15 09:00:00
End : 2018-01-15 18:00:00

Affects: Traffic NZS towards GSA

At 15 January 2018, we will be performing an upgrade in our capacity from NZS (Nedzone Steenbergen) towards GSA (Globalswitch Amsterdam).

We will be installing new hardware making it possible for us to upgrade the capacity in the future with several hundred Gbits, at this time we will also increase the current capacity.

The traffic will be automatically rerouted over EQX-AM7.

#587 Wave service maintenance LINX

Posted: 2018-01-11 19:32

Start: 2018-02-10 23:00:00
End : 2018-02-11 05:00:00

Affects: Routing LINX GSA

Wave service provider which connects Globalswitch towards the LINX peering exchange in London is having a planned maintenance.

All LINX routed traffic is automatically routed over our other LINX peering exchange connection.

Their maintenance reason is "As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing splice box replacement works on our fiberoptical network. This planned activity will take place in United Kingdom."


Their reference: 1-3076620991

For transparancy purposes we decided to publish this notification.

#586 Wave service maintenance DE-CIX

Posted: 2018-01-10 23:39

Start: 2018-01-26 00:01:00
End : 2018-01-26 06:30:00

Affects: Routing DE-CIX EQX-AM7

Wave service provider which connects Equinix-AM7 towards Frankfurt is having a planned maintenance.

Their maintenance reason is "Fiber optic works needs to take place in our network. These works will impact your service during the above mentioned service window."

KPN International has given this maintenance reference number: 80068.

All DE-CIX routed traffic is automatically routed over our other exchanges and IP transits.

#585 Route modification towards ASn 11888 (Television Internacional, S.A. de C.V.)

Posted: 2018-01-08 11:36

Start: 2018-01-04 13:35:00
End : 2018-01-04 13:35:00

Affects: Traffic towards ASn 11888 (Television Internacional, S.A. de C.V.)

We have received reports of high latency and lag towards Television Internacional, S.A. de C.V..

We have removed GTT from the routing.

#584 Route modification towards ASn 8151 (Uninet S.A. de C.V.)

Posted: 2018-01-08 11:35

Start: 2018-01-04 13:35:00
End : 2018-01-04 13:35:00

Affects: Traffic towards ASn 8151 (Uninet S.A. de C.V.)

We have received reports of high latency and lag towards Uninet S.A. de C.V..

We have removed GTT from the routing.

#583 Wave service maintenance LINX

Posted: 2018-01-04 20:17

Start: 2018-01-04 23:00:00
End : 2018-01-05 05:00:00

Affects: Routing LINX

Wave service maintenance
Routing LINX

Wave service provider which connects Globalswitch towards the LINX peering exchange in London is having a planned maintenance.

All LINX routed traffic is automatically routed over our other LINX peering exchange connection. We therefore do not expect any issues.

Their maintenance reason is "in an effort to prevent an uncontrolled outage, are forced to undertake an Emergency hardware upgrade works on our transmission platform."


Their reference: 1-3278875461

For transparancy purposes we decided to publish notification.

#582 IP transit provider Cogent

Posted: 2018-01-03 19:54

Start: 2018-01-13 00:00:00
End : 2018-01-13 07:00:00

Affects: Routing Cogent Global switch (GSA), Equinix AM5.

We have received a notification of a maintenance by one of our transit providers, Cogent.

During their maintenance the connections will be at risk.
We do not expect any issues.

Their maintenance reason is "The purpose of this work is to apply and activate software module upgrades on code running on the edge routers per the manufacturer's recommendations.".

For transparancy purposes we decided to publish notification.

#581 Partial power outage Equinix AM5

Posted: 2018-01-02 01:30

Start: 2018-01-01 23:45:00
End : 2018-01-02 18:10:00

Affects: Power redundancy Equinix AM5

At 23.45 CET we got a notice from our monitoring system that one of the powerfeeds inside Equinix AM5 went down.

We've sent in a trouble ticket for Equinix AM5 staff to check and were notified Equinix had a power incident inside the datacenter.

The following was sent by Equinix staff:
INCIDENT SUMMARY: A power incident was detected on 01-JAN-2018 23:45 at AM5.

INCIDENT DESCRIPTION:

Equinix engineers are evaluating the situation and will provide more details as they become available.

The next update will be when a significant change to the situation occurs.

Our equipment is connected redundantly over 2 feeds, therefore we have only lost a redundancy in power, and do not expect any issues.

For transparency we decided to publish this notification.

Update 02:42:
Equinix engineers have identified a section of busbar that failed for UPS power feed C in Zone 1 of AM5.

The redundant feed remains operational.

Equinix engineers are still working to fully restore the faulty feed.

Affected customers are using the available functioning power feed.

We will keep you posted of the status as more information becomes available.

Thank you for your understanding and patience.

Update 07:00:
We are working on a temporary solution to connect the faulty busbar to provide redundant feed again.

We will keep you posted of the status as more information becomes available.

Thank you for your understanding and patience.


Update 11:30:
Spare parts are on their way to the IBX and will be installed immediately.

We will keep you posted of the status as more information becomes available.

Thank you for your understanding and patience.


Update 12:52:
All materials to create a temporary solution are now on site. We will start to install the temporary solution from 1400 CET onwards. The total work duration is estimated at 3 hours.

We will start to find the final solution and the root cause of this incident after the temporary solution is in place and operational. Our estimation at this moment is that this can take several weeks.

Update 18:10:
The feed that went down is now back up again and all our systems are redundantly connected again.

In the near future Equinix will look for a final solution but as our systems are up and running with redundant power we will close this case on our side.

#580 LINX Outage GSA

Posted: 2017-12-30 00:13

Start: 2017-12-29 22:42:00
End : 2017-12-30 16:31:00

Affects: LINX traffic GSA

At 22:42 we lost signal with LINX at our router in Global Switch Amsterdam (GSA), we have directly investigated the issue and suspect the issue to be with the wave service provider.

We have informed the wave service provider to inspect the issue and have been informed that this issue is located on their transmission platform in the United Kingdom.

As our setup is redundant, the traffic is automatically being rerouted over EQX-AM7.

Update 2017-12-30 22:25:
The break is located at a sub sea cable and a marine cable repair ship will be sent out at the 1st of January.

Therefore Interoute decided to migrate all connections physically to a backup path, which has been going on the entire day,currently our connection is back up again.

It's possible that the connection will be reverted once the primary path has been fixed.

#579 Wave service maintenance

Posted: 2017-12-28 18:50

Start: 2018-02-24 23:00:00
End : 2018-02-25 05:00:00

Affects: Routing FranceIX

Wave service provider which connects Global Switch Amsterdam towards Paris is having a planned maintenance.

Their maintenance reason is "As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing cable rerouting works on our fiberoptical network. This planned activity will take place in Netherlands."

Interoute has given this maintenance reference number:1-3098650471.

All FranceIX routed traffic is automatically routed over our other exchanges and IP transits.

#577 IP transit provider Cogent

Posted: 2017-12-26 19:00

Start: 2017-12-28 00:00:00
End : 2017-12-28 02:00:00

Affects: Routing Cogent

We have received a notification of a maintenance by one of our transit providers, Cogent.

During their maintenance the connections will be at risk.
We do not expect any issues.

Their maintenance reason is "Replace defective power supply".

For transparancy purposes we decided to publish notification.

#576 Wave service maintenance

Posted: 2017-12-19 08:03

Start: 2018-01-12 23:00:00
End : 2018-01-13 05:00:00

Affects: Routing FranceIX

Wave service provider which connects Global Switch Amsterdam towards Paris is having a planned maintenance.

Their maintenance reason is "As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing works related to a cable permanent fix on our fiberoptical network. This planned activity will take place in Belgium."

Interoute has given this maintenance reference number:1-3127135061.

All FranceIX routed traffic is automatically routed over our other exchanges and IP transits.


Update 2018-01-12 15.23:
We received a new notification that this maintenance has been postponed until further notice.

#575 Wave service maintenance

Posted: 2017-12-19 08:00

Start: 2018-01-12 23:00:00
End : 2018-01-13 05:00:00

Affects: Routing FranceIX

Wave service provider which connects Global Switch Amsterdam towards Paris is having a planned maintenance.

Their maintenance reason is "As part of our commitment to continually improve the quality of service we provide to our customers, we wish to inform you INTEROUTE will be performing works related to a cable permanent fix on our fiberoptical network. This planned activity will take place in Netherlands."

Interoute has given this maintenance reference number:1-3121728861.

All FranceIX routed traffic is automatically routed over our other exchanges and IP transits.


Update 2018-01-12 15.23:
We received a new notification that this maintenance has been postponed until further notice.

#574 Upgrade switches BNR 100G

Posted: 2017-12-18 18:01

Start: 2018-01-03 23:00:00
End : 2018-01-04 05:00:00

Affects: IPT / Internet Access / VPLS Customers only

On the 3rd of January at 23.00 we will start with the replacement of the switches located in Bytesnet Rotterdam (BNR) with newer switches that support our 100G upgrades.

*This will affect all parties that have IP transit / Internet Customers and VPLS Customers located in BNR, hosting customers are not affected.*

During this time the current switches will go offline to make room for the newer switches.

#573 IP transit provider Core-Backbone

Posted: 2017-12-18 12:34

Start: 2017-12-22 00:00:00
End : 2017-12-22 06:00:00

Affects: Routing Core-Backbone

We have received a notification of a maintenance by one of our transit providers, Core-Backbone.

During their maintenance traffic will automatically reroute over our other transit providers. We do not expect any issues.

Their maintenance reason is: "In the mentioned timeframe we will perform hardware upgrades at the following core-routers/core-switches: ams12.core-backbone.com (Amsterdam1)"

For transparancy purposes we decided to publish notification.

#572 Reload R1 EQX-AM7

Posted: 2017-12-16 15:53

Start: 2017-12-20 00:30:00
End : 2017-12-20 04:00:00

Affects: Traffic EQX-AM7 / Internet Customers EQX-AM7

Based on the previous reload announced in maintenance #569 of R1 GSA our vendor has released an urgent patch which needs to be applied to the router at Equinix AM7 (EQX-AM7) as well which requires a reload of the router, this reload will be done starting Wednesday at 0:30 CET.

#571 Reload R1 GSA

Posted: 2017-12-16 15:48

Start: 2017-12-19 00:30:00
End : 2017-12-19 04:00:00

Affects: Traffic GSA

Based on the previous reload announced in maintenance #569 our vendor has released an urgent patch which requires a reload of the router, this reload will be done starting the Tuesday 19th at 0:30 CET.

#570 Route leaks from ASn 7713

Posted: 2017-12-16 12:27

Start: 2017-12-16 00:00:00
End : 2017-12-16 23:59:59

Affects: Abnormal routing

Last night a router from ASn 7713 (PT Telekomunikasi Indonesia Int) started leaking routes from networks such as ASn 3320 (Deutsche Telekom) and subroutes such as ASn 6830 (Aorta). The same we have seen them do for ASn 3257 (GTT). They leaked routes over

We noticed that in particular routes from Aorta were being re-announced from their infrastructure, causing high latency and unstable connectivity to Aorta networks. We have received some information from other routes that were affected, most likely this is causing random global issues.

We will keep monitoring this network the coming hours.

*** Please be aware that this issue/incident is outside of our scope and is a global issue on the internet, originated by ASn 7713. ***


2017-12-16 14.50:
We have received reports from global sources that ASn 7713 still has not resolved their route leaking. Therefore we have filtered out ASn 7713 on our network from all peering exchanges.

Notice from for example NL-IX:
"We observe that AS 7713 (PT Telekomunikasi Indonesia Int) is reannouncing plenty of prefixes from their upstreams (GTT, NTT, DTAG, KPN, and so on) via the NL-IX route servers, hijacking plenty of prefixes (16 dec at 2:30 CET) toward their NL-IX port in Marseille (well, not really hijacking prefixes in fact, but stealing the traffic)."

#569 Urgent reload R1 GSA

Posted: 2017-12-11 18:36

Start: 2017-12-12 00:30:00
End : 2017-12-12 04:00:00

Affects: Traffic R1 GSA

Coming night 12-12-2017 we will be reloading the router at Globalswitch Amsterdam (GSA).

This is due to a vendor bug and needs to be done to prevent any further issues.

#568 VPLS Outage BNR

Posted: 2017-12-10 05:08

Start: 2017-12-10 03:00:00
End : 2017-12-10 04:40:00

Affects: Traffic BNR

At 03.00 CET a part of our VPLS platform located in Bytesnet Rotterdam (BNR) stopped forwarding traffic to Globalswitch Amsterdam (GSA)

There was a planned maintenance done by interoute on ports connecting Bytesnet (BNR) to Globalswitch Amsterdam (GSA), when the ports went down no issues where present but when the ports came back up the link was established but the MPLS part didn't establish.

Engineers we're already monitoring the maintenance and shutdown the link between GSA and BNR.

The VPLS at BNR has 3 darkfibers, one to GSA, one to DBC and one to NZS.

However due to a cascade effect after the physical ports to GSA were shut again, the VPLS to DBC seemed to be online but stopped forwarding traffic as well.

The switches are setup in a redundant stack but have both been reloaded to restore all connectivity, logs have been collected and will be sent to the manufacturer for further debugging.

We will keep monitoring the switches for now to keep sure they show no issues.

#566 IP transit provider NTT

Posted: 2017-11-27 16:28

Start: 2017-11-28 02:00:00
End : 2017-11-28 04:00:00

Affects: Redundancy routing NTT

We have received a notification of a maintenance by one of our transit providers, NTT.

During their maintenance traffic will automatically reroute over our other NTT connection as well as over other transit providers. We do not expect any issues.

Their maintenance reason is "We will be performing an urgent hardware maintenance affecting the service(s) listed below".

For transparancy purposes we decided to publish notification.

#565 Darkfiber outage DBC - Equinix-AM7

Posted: 2017-11-25 03:06

Start: 2017-11-25 02:15:00
End : 2017-11-25 06:35:00

Affects: Redundancy DBC

At 02.15 CET a darkfiber between DBC and Equinix-AM7 (GSA) went down, carrying 400 Gbps traffic.

Currently all traffic is send over alternative paths on our ring, DBC - GSA and DBC - NZS.

From DBC you might experience increased latency and lower throughput towards routes terminated at our Equinix-AM7 pop.

We have contacted the darkfiber provider euNetworks and were notified that this is part of an emergency maintenance on their side, the end time of the maintence would be 06:00 CET

Update 05:41:
We were notified by euNetworks that when we first asked if there was any ongoing maintenance we were linked to the wrong maintenance case, the original maintenance our fibers runs over takes until 7.00 AM CET to complete.

Update 06:35:
Currently the darkfiber is back online and restored in service.

#564 Wave service maintenance

Posted: 2017-11-24 16:10

Start: 2017-12-09 23:00:00
End : 2017-12-10 05:00:00

Affects: Routing FranceIX

Wave service provider which connects Global Switch Amsterdam towards Paris is having a planned maintenance.

Their maintenance reason is "Please be advised that Interoute will be performing a Service Affecting Planned Maintenance in order to migrate a fiber optic cable on a different route near the city of Leiden, The Netherlands. Please note that we are forced to perform this work by the local city authorities or we are facing a risk of outage on our infrastructure due to upcoming construction works, which will affect our existing cable."

All FranceIX routed traffic is automatically routed over our other exchanges and IP transits.

Their reference: 1-3057438271

For transparancy purposes we decided to publish notification.

#563 Darkfiber maintenance

Posted: 2017-11-24 16:02

Start: 2017-12-09 23:00:00
End : 2017-12-10 05:00:00

Affects: Redundancy BNR

Darkfiber provider Interoute which connects Bytesnet Rotterdam (BNR) towards Globalswitch Amsterdam (GSA) is having a planned maintenance. The connectivity between BNR and GSA will automatically reroute over our DBC or NZS darkfiber during this maintenance.

Their maintenance reason is "Please be advised that Interoute will be performing a Service Affecting Planned Maintenance in order to migrate a fiber optic cable on a different route near the city of Leiden, The Netherlands. Please note that we are forced to perform this work by the local city authorities or we are facing a risk of outage on our infrastructure due to upcoming construction works, which will affect our existing cable.".

Their reference: 1-3057438271

For transparancy purposes we decided to publish notification.

#562 Wave service maintenance

Posted: 2017-11-21 16:47

Start: 2017-12-13 00:01:00
End : 2017-12-13 07:00:00

Affects: Routing LINX

Wave service provider which connects Equinix AM7 towards the LINX peering exchange in London is having a planned maintenance.

All LINX routed traffic is automatically routed over our other LINX peering exchange connection. We therefore do not expect any issues.

Their maintenance reason is "KPN will carry out a hardware upgrade in the network. This upgrade is essential, in order to provide our customers with an up to date network. This activity will impact your service during the above-mentioned service window.

Expected outage duration: 419 minutes".

Their reference number is 78862
For transparancy purposes we decided to publish notification.

Update 2017-12-13: The provider cancelled this maintenance.

#561 Relined Maintenance

Posted: 2017-11-15 16:55

Start: 2017-12-08 00:00:00
End : 2017-12-08 06:00:00

Affects: Redundancy Equinix AM5

During this timeframe there will be a routine maintenance on the darkfiber connecting Globalswitch Amsterdam (GSA) towards Equinix AM5 (EQX-AM5).

The connectivity between Globalswitch and Equinix AM5 will automatically reroute over our Equinix AM7 (EQX-AM7) darkfiber during this maintenance.

No outages or other issues are expected during this maintenance.

#560 Packetloss several vlans DBC

Posted: 2017-11-08 11:29

Start: 2017-11-08 08:45:00
End : 2017-11-08 11:15:00

Affects: DBC vlans 1018, 1069

At 08.45 we received notifications from our monitoring that for vlan 1018 and 1069 at the DBC facility there is packetloss. At 09.00 this automatically recovered.

We are currently investigating the cause.


Update 11.30:
During the timeframes 09.45 - 09.55 and 11.00 - 11.15 we have received the same reports.

Update 11.42:
Investigation is still ongoing, VRRP-E flaps due to duplicated mac-address. The issue seems to originate between the R1 <-> core-sw1.z1 and R2 <-> core-sw1.z1. We do not see any loops and our ports facing the routers are configured as route-only as well. Our debugging shows that it is most likely caused by a defective port which is not shutting itself down and still forwarding partial traffic. We are currently going through all 96x 10 Gbps ports connecting the core-sw1.z1 and the R1/R2 routers to locate the problematic port.

An engineer was already on-site since 09.00 for regular maintenance, that maintenance has been prosponed to later today and the engineer is working together with remote engineers to debug and resolve this issue.

The issue is not continuously ongoing, making the issue more difficult to identify. Hence we have enabled real-time monitoring to find the culprit.

We expect an update/resolve of the issue within approximately 1-2 hours.

Update 13.05:
We have found the problematic bridge and are currently moving away the vlans to debug this further. No incidents have occurred since 11.15.

Update 14.45:
The incident has been resolved, everything has been restored to primary paths. There we no more incidents since last incident which was at 11.15.

#558 Darkfiber outage DBC - GSA

Posted: 2017-11-08 00:26

Start: 2017-11-08 00:26:41
End : 2017-11-08 01:40:00

Affects: Redundancy DBC

At 00.12 CET a darkfiber between DBC and Globalswitch Amsterdam (GSA) went down, carrying 400 Gbps traffic.

Currently all traffic is send over alternative paths on our ring, DBC - EQX-AM7 and DBC - NZS.

From DBC you might experience increased latency and lower throughput towards routes terminated at our GSA pop.

We have contacted the darkfiber provider euNetworks and requested a repair.

Update 1:40:
At this moment the darkfiber is back online, we will keep monitoring the connection.

According to euNetworks this was due to an unscheduled emergency maintenance.

#557 Wave service maintenance

Posted: 2017-11-07 07:37

Start: 2017-11-09 00:01:00
End : 2017-11-09 06:00:00

Affects: Routing DECIX, Layer 2 DECIX

Wave service provider which connects Equinix AM7 Amsterdam towards Frankfurt is having a planned maintenance.

Their maintenance reason is "Urgent fiber re-location due to road construction works near Leiden, the Netherlands..".

Customers using specific layer 2 vlan transport services between Equinix AM7 towards DECIX will experience an outage during this window.

All DECIX routed traffic is automatically routed over our other DECIX port(s) at Globalswitch.

Their reference: 78426

For transparancy purposes we decided to publish notification.

#556 Upgrade capacity between DBC and AM7

Posted: 2017-11-03 14:52

Start: 2017-11-20 09:00:00
End : 2017-11-20 17:00:00

Affects: Traffic DBC towards AM7

At 20 November 2017, we will be preforming an upgrade in our capacity from DBC (Databarn Capelle) towards Equinix AM7 (Amsterdam).

We will be installing new hardware making it us possible to double the current capacity available, at this time we also increase the capacity.

The traffic will be automatically rerouted over GSA.

Update 15.20:
We are currently in the process of testing the new links, while traffic is send over GSA higher latency may occur due to the balancing between GSA and EQX-AM7.

Update 16.30:
The new links are now active, the connection between DBC and EQX-AM7 has been restored and is now operating at 400G capacity.

#555 Darkfiber outage DBC - EQX-AM7

Posted: 2017-11-01 13:49

Start: 2017-11-01 14:08:00
End : 2017-11-01 14:17:00

Affects: Redundancy DBC

At 14.08 CEST a darkfiber between DBC and Equinix-AM7 (EQX-AM7) went down, carrying 200 Gbps traffic.

Currently all traffic is send over alternative paths on our ring, DBC - GSA and DBC - NZS.

From DBC you might experience increased latency and lower throughput towards routes terminated at our EQX-AM7 pop.

We have contacted the darkfiber provider euNetworks and requested a repair.

At 14.17 the darkfiber recovered, therefore this has not been sent though email.

#554 VPLS outage EQX-AM7

Posted: 2017-10-31 05:30

Start: 2017-10-31 02:30:00
End : 2017-10-31 05:55:00

Affects: VPLS & IPT EQX-AM7

At 02.30 we received error notifications from ring-sw2 at EQX-AM7. Traffic passing through this location (VPLS & IPT) are partially affected.

We have gather debugging reports for our vendor and currently in the process of debugging.

Update 05.45:
ISSU with failover upgrade/reload did not resolve the issue. We will now be reloading the chassis as whole.

Update 05.55:
Ring-sw2 is reloaded and mpls is being established. Running tests and restoring traffic.

Update 06.20:
Traffic has been restroing since 05.55. We will reach out to the vendor with tech logs for debugging further. Incident itself is closed for now.

#553 Internet access maintenance

Posted: 2017-10-31 01:40

Start: 2017-11-08 06:00:00
End : 2017-11-08 08:00:00

Affects: Internet access customers at Kruisweg in Hoofddorp (NL)

Fiber provider which is delivering connectivity our POP at Kruisweg in Hoofddorp (NL) is having a planned maintenance.

This POP is only used for Fiber @ Work connectivity and does not affect any hosting or ip transit services.

Reason provided by provider:
Power maintenance is being done on the cabinet our switch is placed in.