We noticed one of the power distribution units (PDU) namely PDU1 not functioning as it should anymore. Therefore under emergency maintenance this PDU will be replaced, customers located on this PDU will have their servers rebooted.

As the switch is also located on this PDU customers in this rack will notice a short downtime in network.

Currently our darkfiber between Databarn Amsterdam (DBA) and Globalswitch Amsterdam (GSA) is down.

We've sent in a ticket with our darkfiber provider for more information and will update accordingly.

All traffic for GSA is re-routed over our darkfiber with EQX-AM7.

Update 14:39:
Currently the darkfiber is back online, while we haven't been given the reason as to why by our provider yet, they mentioned an RFO will follow.

As currently there are multiple public resolvers who are made specially for this purpose and we see ours being used less and less we have decided to retire our public DNS resolvers.

At the 1st of August we will be shutting down the resolvers, any customers then still using the resolvers won't be able to resolve any outside names.

Below you can find the most popular DNS resolvers which can be used instead of ours:

Google:
8.8.8.8
8.8.4.4

Cloudflare:
1.1.1.1
1.0.0.1

Quad9:
9.9.9.9
149.112.112.112

As we do not limit which resolvers you use, you are free to pick any other you want as well.

If you have any questions or are not sure if you are using ours you can send us an email at support@nforce.com.

Darkfiber provider Exainfra which connects Bytesnet Rotterdam (BNR) towards GlobalSwitch Amsterdam (GSA) is having a planned maintenance.

Their reason for this maintenance:

"Cable Reroute between Luchtvaartstraat and Zuidasdok Amsterdam due to construction works"

The connectivity between NZS and GSA will automatically reroute over our ring network during this maintenance.

Customers having waves towards GSA will see some downtime.

At the 18th of May between 9.00 AM - 12 PM CEST we will be upgrading the firmware of the switches located in rack B-2-1 in Nedzone.

During this timeframe there will be multiple interruptions of a few minutes while the switch is reloading into the new firmware, we will keep this as short as possible.

Update 2022-05-18 09:57:
The switch has been firmware upgraded, and is fully reachable again.
In case you still have issues please send an email to support@nforce.com so we can check on your issue.

Darkfiber provider Exainfra which connects Bytesnet Rotterdam (BNR) towards GlobalSwitch Amsterdam (GSA) is having a planned maintenance.

Their reason for this maintenance:
"Emergency repair of damaged cable"

The connectivity between NZS and GSA will automatically reroute over our ring network during this maintenance.

Customers having waves towards GSA will see some downtime.

Darkfiber provider Relined which connects GlobalSwitch Amsterdam (GSA) towards Equinix AM5 (EQX-AM5) is having a planned maintenance.

Their reason for this maintenance:

"Due to reconstruction work carried out by the local authorities. Relined is obliged to relocate a section of her network. This action will result in a temporary interruption of the service."

Their references is: PA-060002.

Darkfiber provider Relined which connects GlobalSwitch Amsterdam (GSA) towards Nikhef is having a planned maintenance.

Their reason for this maintenance:

"Due to reconstruction work carried out by the local authorities. Relined is obliged to relocate a section of her network. This action will result in a temporary interruption of the service."

Their references is: PA-718001.

Currently AMS-IX is having major issues as their graph bottomed out and we also started noticing issues with BGP sessions flapping every 20 seconds or so.

Until we get an explanation and the OK from the AMS-IX team we've shut all our BGP sessions with AMS-IX.

We will update this post with more information later on.

Update 11:30:
We received the following from the AMS-IX:

We are experiencing an increased amount of BUM traffic on the platform as well as several down BGP sessions. We are investigating the problem, and we will update you as soon as possible.

Update 14:00:
While we are still awaiting an official RFO we've enabled the sessions again after it has been stable for the last couple of hours.

As of 11:36 rack A-2-12 in Nedzone Steenbergen (NZS) has gone down.

Our engineers are currently investigating the issue.

Our sincere apologies for this inconvenience.

When we have more information on the exact issue we'll update the noc post here.

Update 13:04:
As of 12:38 most servers should have began coming back online, the root cause has been found and solved. If you still have any issues with your server please let us know by replying to your ticket you already had with us or by sending a message to support@nforce.com.

The fiber provider which connects Databarn Capelle (DBC) towards GlobalSwitch Amsterdam (GSA) is having a planned maintenance. The connectivity between DBC and GSA will automatically reroute over our ring network during this maintenance.

Please note this is not a duplicate of Post #975, this is separate maintenance affecting the same fiber route.

The fiber provider which connects Nedzone Steenbergen (NZS) towards Globalswitch Amsterdam (GSA) is having a planned maintenance. The connectivity between NZS and GSA will automatically reroute over our ring network during this maintenance.

Customers having waves towards GSA will see some downtime.

Because of maintenance related tasks and to debug an issue we will be reloading our DBA R2 router on the 24th of February at 10:00 AM CET.

Because of maintenance related tasks and to debug an issue we will be reloading our DBA R1 router on the 22th of February at 10:00 AM CET.

The fiber provider which connects Nedzone Steenbergen (NZS) towards Equinix Amsterdam 7 (EQX-AM7) is having a planned maintenance. The connectivity between NZS and EQX-AM7 will automatically reroute over our ring network during this maintenance.

Customers having waves towards EQX-AM7 will see some downtime.

The fiber provider which connects Databarn Capelle (DBC) towards Equinix Amsterdam (EQX-AM7) is having a planned maintenance. The connectivity between DBC and EQX-AM7 will automatically reroute over our ring network during this maintenance.

The fiber provider which connects Databarn Capelle (DBC) towards GlobalSwitch Amsterdam (GSA) is having a planned maintenance. The connectivity between DBC and GSA will automatically reroute over our ring network during this maintenance.

EDIT: Notification resent due to maintenance only affecting DBC connectivity, not DBA.

Because of some maintenance related tasks we will be reloading our GSA router on the 4th of February at 1:00 AM CET.

Our peering traffic will automatically flow over our EQX-AM7 router.

Internet and Transit customers only having a connection to this router will experience some downtime during the reload.

During the night 3 to 4 november, at 1 AM, Dataplace will have a maintenance on the dark fiber.

The line will be interrupted between 1 AM and 2 AM. This only affects traffic towards GSA, and will automatically pass through AM7 via our ring network when the maintenance starts.

For customers with own wave services, this path will go down for up to an hour and should be fully restored after the maintenance is complete.

As the emergency maintenance last week showed success, we going to schedule reloads of the routers of DBC as well as DBA to apply new settings to IPv6.

9Th of November at 7:30 CET:
We will reload R1 of DBA.

10Th of November at 7:30 CET:
We will reload R2 of DBC.

11Th of November at 7:30 CET:
We will reload R1 of DBC.

As IPv6 continues to give issues, we are hereby reloading router 2 to apply new settings to combat the ongoing issues of IPv6.

We will make sure that the traffic falls back on R1 before the reload.

Update 9:21:
As the router R2 is back in production, we also will be reloading R1 to bring these settings in effect.

Update 10:18:
Both routers are reloaded with new settings and confirmed that the issue is resolved.

We got a notice today from the datacenter that the power company Liander is doing an emergency maintenance tomorrow to repair a 10Kv ring in Amsterdam.

Initially this was planned for 2022 but they moved it way back to 20th of October to be able to guarantee a good working electricity grid.

The datacenter themselves were also only informed about this today.

We asked for some assurances and they assured us that no outages are expected as they have UPSsses, generators and have a company bring an extra generator as well.

We will be monitoring the situation closely tomorrow.

At 21:17 we noticed a management card flap at our second router at NZS.

We are currently investigating into this as our BGP are loading out again.

Update 01:10:
We continue to see the management cards swap (both the active / standby) on this router at random intervals. Router 1 is carrying all the customer load at the moment while we debug this issue.

Update 10:30:
We have replaced the management cards of R2 and monitored the situation. After replacing the management cards, we have no longer seen any swapping between active/standby card.

We expect the issue is resolved however we still continue to monitor the situation during the day.

Due to a range of issues we are going to do an emergency reload of our router in EQX-AM7.

Internet Customers will experience a loss of internet as well as customers having only one BGP session. Customers having a redundant BGP session with us should not experience any issues.

We are aware it is the start of a workday for most but due to the on-going issues there is little choice left for us to do it at a very short timespan, we will try our best to keep the downtime to an absolute minimum.

Update 10:25
We have finished the reload and restored BGP sessions. The main cause for this emergency reload was the router no longer propagating new routes nor pruning old routes.

Around 12:00 we've noticed our connections from GSA towards Nedzone Steenbergen (NZS) and Bytesnet Rotterdam (BNR) went down.

Our darkfiber provider GTT has informed us of a fibercut between Amsterdam and Rotterdam, as both BNR and NZS makes use of this darkfiber they have had a loss of redundancy.

All traffic for NZS is re-routed over our darkfiber with EQX-AM7 / all traffic for BNR is re-routed over DBC.

Wave customers between NZS and GSA are currently down.

Update 1: 2021-09-04 10:23 UTC
Please be advised that the GTT Infra NOC detected an interruption occurring on our network between Amsterdam & Rotterdam at 09:18 UTC. Initial investigation has detected a potential fiber cut occurring. FLM teams have been engaged and are en route with an ETA of 90 mins to site.

Update 2: 2021-09-04 11:29 UTC
FLM teams report that they have now reached site & are beginning their investigations.
Once the OTDR has been shot crews will travel to the break point & ETTR will be made available following a damage assessment.

Update 3: 2021-09-04 12:23 UTC
Crews on site have shot the OTDR & discovered the location of the break to be occurring approx 450 meters outside of Amsterdam. Teams are en route to the break point now with an ETA on site set at roughly 60 minutes.

Update 4: 2021-09-04 14:05 UTC
Crews have now arrived at the break point & discovered the fiber impacted by 3rd party drilling activity on site. A damage assessment & repair plan is being formulated by the FLM crew & civils work will need to be undertaken to excavate the damaged cable. ETTR will become available once complete.

Update 5: 2021-09-04 15:15 UTC
Crews report that civils crews are now on site & are preparing to excavate the damaged fiber. Repair plan will include the introduction of two new handholds along the route of the fiber. ETTR will become available once civils work has been completed & the additional infrastructure deployed to provide a safe route around the ongoing drilling work.

Update 6: 2021-09-04 16:25 UTC
Crews report that excavation work on site is progressing well with splicing to begin at a tentative time of 18:30 UTC.

Update 7: 2021-09-04 17:42 UTC
Crews on site report that excavation work has now completed & they are prepping the fiber for splicing which will commence momentarily.

Final Update: 2021-09-04 18:56 UTC
FLM teams confirm that all fibers have now been spliced & the GTT NOC can confirm that all alarms have cleared with services restoring.
The actions taken by the team on site will constitute a permanent repair & no further disruption to service is anticipated.

We've became aware of multiple issues affecting a part of our Internet customers and are currently investigating.

Once we have an update it will be posted here.

Update 11:45: An engineer has been dispatched to replace a possibly faulty optic.

Update 14:50: An engineer from the external party is underway to troubleshoot the link.

Update 17:22:
At the moment our port is up again, we are going to request a formal RFO with Eurofiber since a downtime of 8 hours is not normal for the issue we were experiencing.

We've noticed some packet loss in NZS.

One of the optics in a bundle of multiple Hundred Gbit optics started giving issues without going down causing packet loss, because of it not going down the issue was hard to pin-point.

We have now disabled the troubling link and will schedule a debug session of this link without affecting production traffic.

We have planned an upgrade of network capacity towards TI Sparkle.

During this maintenance all TI Sparkle traffic will be automatically rerouted over our other transits and peering connections.

Today we had packet loss in our datacenter Nedzone Steenbergen.

After investigation we have noticed a hardware defect in one of the linecards.
The card was only using limited memory space for the routes installed (1M vs 2M).

The issue was not clear at first as everything was looking correctly including optical values. After further investigation we see that the card is not functioning correctly because of the above reason and have currently disabled the ports affected until we have a fix at hand.

Currently we are seeing a DDOS against our DNS resolvers, as many servers are using these you may be having issues as well.

While we are currently focusing on solving these issues a solution to get it solved instantly would be to set any of the resolvers from one of the following:

Google 8.8.8.8,8.8.4.4
OpenDNS 208.67.222.222,208.67.220.220
Cloudflare 1.1.1.1

Update 15:35:
Currently DDOS is mitigated, we will keep monitoring the situation.

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Globalswitch Amsterdam (GSA) is having a planned maintenance. The connectivity between NZS and GSA will automatically reroute over our EQX-AM7 darkfiber during this maintenance.

Customers having their own waves from NZS to GSA will experience an outage during this time.

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Equinix AM7 (EQX-AM7) is having a planned maintenance. The connectivity between NZS and EQX-AM7 will automatically reroute over our GSA darkfiber during this maintenance.

Customers having their own waves from NZS to EQX-AM7 will experience an outage during this time.

Darkfiber provider GTT which connects Databarn Amsterdam (DBA) towards Equinix AM7 (EQX-AM7) is having a planned maintenance. The connectivity between DBA and EQX-AM7 will automatically reroute over our GSA darkfiber during this maintenance.

We are currently investigating an issue with the power distribution at DBC. The power started to flap at 12:50 and went completely down at 13:00.

We have contacted the datacenter for more information on the issue and are waiting for a response.

Update 13:29:
There is an outage reported by Stedin, we have received notification of an issue in a 100kv substation. More information can be found at https://www.stedin.net/storing-en-onderhoud/overzicht .

Update 17:17:

We received a message from the datacenter:

"We just had a major impact in our power supply due to a problem with Stedin.

The power was cut of at 12:34 and returns after several seconds. The next problem that occurred was a low voltages on 1 feed

https://www.stedin.net/storing-en-onderhoud/storingshistorie

The ups stabilised the drops as it kept on coming and going. For safety measurements we tried to bypass the UPS and put the full load on our generators. But since the power did not fully dropped it declined to bypass. Due to an error the power was cut of for several minutes. "

As of 08:43 a rack in Databarn Amsterdam (DBA) has gone down.

Our engineers are currently investigating the issue.

Our sincere apologies for this inconvenience.

When we have more information on the exact issue we'll update the noc post here.

UPDATE 11:55: Switch has been replaced and connectivity should be restored. If your servers' are still offline, please make a support ticket.

We are experiencing since 23:30 an outage of the cooling systems in Databarn Amsterdam (DBA).

Currently, engineers of the datacenter are resolving the issue.

Systems may affect shutdown due to overheating.

We are chasing them for an ETR and solution.

Update 00:10:
Currently a backup system has been started to sent as much cold air into the rooms as possible while the vendor is debugging the primary cooling system.

Update 00:40:
Currently we see a drop in temperatures throughout all dc rooms, since all hot air needs to be replaced by cold air it might take time before it returns to previous temperatures.

Update 00:55:
At the moment the primary and backup cooling is running to cool off the datacenter as quicky as possible and we've seen temperatures dropping throughout all rooms back to the normal values.

In case you have one or more servers that went into a thermal shutdown please let us know by emailing support@nforce.com so we can follow up for you.

Our sincere apologies for this inconvenience.

When we have more information on the exact issue with the primary cooling we'll update the noc post here.

We have received a notification of a maintenance by one of our transit providers, NTT.

During their maintenance all NTT traffic will be automatically rerouted over our other transits and peering connections.

Their maintenance reason: Services migration

For transparency purposes we decided to publish this notification.

We have received a notification of a maintenance by one of our transit providers, NTT.

During their maintenance all NTT traffic will be automatically rerouted over our other transits and peering connections.

Their maintenance reason: Services migration

For transparency purposes we decided to publish this notification.

We've gotten reports of packet loss in DBC.

It appears an optic has failed between our core-switch and one of our routers, causing packet loss.

We have properly disabled the troubling link and will schedule a replacement in the oncoming week.

We have received reports of packetloss towards Facebook (AS32934)

We have removed the path over LINX in the routing tables.

The best selected path is now over Cogent

Because of some maintenance related tasks we will be reloading our GSA router on the 28th of February at 0:30.

Our peering traffic will automatically flow over our EQX-AM7 router.

Internet and Transit customers only having a connection to this router will experience some downtime during the reload.

We have noticed a big outage on one of the dutch Internet Exchanges named NL-ix.

On their traffic graphs it is clearly visible an outage is visible https://www.nl-ix.net/network/traffic-stats/

The issue started at 13:15 and is resolved around 13:40.

We will be monitor the situation and if the issue returns, we will temporary disable our sessions with NL-ix.

Update sent by the NLix 15:07:
Investigation about cause is still ongoing, but is seems this is related to the general issue going on in Nikhef.

We have received reports of high latency towards Global Connect (AS2116)

We have preferred the path over NTT in the routing tables.

The previous best selected path was over Lumen

Wave service provider which connects Equinix AM7 (EQX-AM7) towards LINX is having a planned maintenance.

Their maintenance reason is "Maintenance on our Fibre infrastructure platform related to Cable Splicing."

GTT has given this maintenance reference number: 5639371

All LINX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Equinix AM7 (EQX-AM7) is having a planned maintenance. The connectivity between NZS and EQX-AM7 will automatically reroute over our GSA darkfiber during this maintenance.

Customers having their own waves from NZS to EQX-AM7 will experience an outage during this time.

Wave service provider which connects Global Switch Amsterdam (GSA) towards FranceIX is having a planned maintenance.

Their maintenance reason is "Maintenance on our Transmission platform related to Hardware Replacement."

GTT has given this maintenance reference number: 5669155

All FranceIX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

Around 9:00 we've noticed our connection from EQX-AM7 towards DECIX went down.

Wave service provider which connects Equinix AM7 (EQX-AM7) towards DECIX is investigating the issue.

Update 14:00: The 3rd Party Engineer´s performed OTDR measurements and showing an interruption between Cologne and Dusseldorf.
Fibre team on the way to break location , no ETA/ETR yet.

Final update from our wave service provider: After concluding our investigation, the GTT optical NOC can report back that we observed outage on our DWDM backhaul between Dusseldorf and Linz through which your circuit traverses. Further case was opened with the underlying fibre provider.Our partner immediately started their investigation and dispatched techs to Dusseldorf to troubleshoot the fiber. They performed OTDR test and identified possible fiber cut. Our partner dispatched the engineer and found a fiber cut due to construction work at Bergisch Gladbacher Str., 51063 Cologne. Our partner splices the fiber which restored the services. Restoration work delay due to our partner need to installed approximately 400ft of new cable with 6 Tubes needs to spliced to restore connectivity. Our partner confirmed this is permanent fixed.

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Globalswitch Amsterdam (GSA) is having a planned maintenance. The connectivity between NZS and GSA will automatically reroute over our EQX-AM7 darkfiber during this maintenance.

Customers having their own waves from NZS to GSA will experience an outage during this time.

Update 2020-11-13:
This maintenance has been postponed by Eurofiber for now, once a new date is available we'll create a new post.

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Globalswitch Amsterdam (GSA) is having a planned maintenance. The connectivity between NZS and GSA will automatically reroute over our EQX-AM7 darkfiber during this maintenance.

Customers having their own waves from NZS to GSA will experience an outage during this time.

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Equinix AM7 (EQX-AM7) is having a planned maintenance. The connectivity between NZS and EQX-AM7 will automatically reroute over our GSA darkfiber during this maintenance.

Customers having their own waves from NZS to EQX-AM7 will experience an outage during this time.