Status

[#991] Darkfiber outage between DBA and GSA

Posted: 2022-07-06 10:44

Start: 2022-07-06 09:00:00
End : 2022-07-06 14:39:00

Affects: Redundancy DBA

Currently our darkfiber between Databarn Amsterdam (DBA) and Globalswitch Amsterdam (GSA) is down.

We've sent in a ticket with our darkfiber provider for more information and will update accordingly.

All traffic for GSA is re-routed over our darkfiber with EQX-AM7.

Update 14:39:
Currently the darkfiber is back online, while we haven't been given the reason as to why by our provider yet, they mentioned an RFO will follow.

[#983] Major AMS-IX issues

Posted: 2022-04-06 11:00

Start: 2022-04-06 10:40:00
End : 2022-04-06 11:30:00

Affects: Traffic to/from AMS-IX

Currently AMS-IX is having major issues as their graph bottomed out and we also started noticing issues with BGP sessions flapping every 20 seconds or so.

Until we get an explanation and the OK from the AMS-IX team we've shut all our BGP sessions with AMS-IX.

We will update this post with more information later on.

Update 11:30:
We received the following from the AMS-IX:

We are experiencing an increased amount of BUM traffic on the platform as well as several down BGP sessions. We are investigating the problem, and we will update you as soon as possible.

Update 14:00:
While we are still awaiting an official RFO we've enabled the sessions again after it has been stable for the last couple of hours.

[#982] Rack down in NZS (A-2-12)

Posted: 2022-03-06 12:05

Start: 2022-03-06 11:36:00
End : 2022-03-06 13:00:00

Affects: Rack A-2-12 NZS

As of 11:36 rack A-2-12 in Nedzone Steenbergen (NZS) has gone down.

Our engineers are currently investigating the issue.

Our sincere apologies for this inconvenience.

When we have more information on the exact issue we'll update the noc post here.

Update 13:04:
As of 12:38 most servers should have began coming back online, the root cause has been found and solved. If you still have any issues with your server please let us know by replying to your ticket you already had with us or by sending a message to support@nforce.com.

[#969] Router 2 NZS MGMT Failover

Posted: 2021-09-29 21:53

Start: 2021-09-29 21:17:00
End : 2021-09-30 08:30:00

Affects: Traffic NZS

At 21:17 we noticed a management card flap at our second router at NZS.

We are currently investigating into this as our BGP are loading out again.

Update 01:10:
We continue to see the management cards swap (both the active / standby) on this router at random intervals. Router 1 is carrying all the customer load at the moment while we debug this issue.

Update 10:30:
We have replaced the management cards of R2 and monitored the situation. After replacing the management cards, we have no longer seen any swapping between active/standby card.

We expect the issue is resolved however we still continue to monitor the situation during the day.

[#968] Emergency reload R1 EQX-AM7

Posted: 2021-09-14 09:01

Start: 2021-09-14 09:00:00
End : 2021-09-14 10:25:00

Affects: FTTB/FTTH Customer, IP transit Customers

Due to a range of issues we are going to do an emergency reload of our router in EQX-AM7.

Internet Customers will experience a loss of internet as well as customers having only one BGP session. Customers having a redundant BGP session with us should not experience any issues.

We are aware it is the start of a workday for most but due to the on-going issues there is little choice left for us to do it at a very short timespan, we will try our best to keep the downtime to an absolute minimum.

Update 10:25
We have finished the reload and restored BGP sessions. The main cause for this emergency reload was the router no longer propagating new routes nor pruning old routes.