Maintenance Details:
Following the latest issues with AMS-IX, we have noticed some BGP issues originating in EQX-AM7 Router, to ensure redundancy, stability and reliability of our network, we will be conducting essential maintenance on our router located in EQX-AM7.

Expected Impact:
During the maintenance window, directly connected customers may experience a temporary loss of internet connectivity. This impact will primarily affect customers with a single BGP session. However, customers with redundant BGP sessions should not encounter any service disruptions.

Why This Time?
We have carefully selected this time slot to minimize the impact on our customers. It falls within our lowest network traffic period, ensuring that the maintenance is performed with the least possible disruption.

We apologize for any inconvenience this may cause and appreciate your understanding as we work to enhance the reliability and performance of our network infrastructure.

If you have any questions or require further assistance, please do not hesitate to contact our NOC Team

Due to prolonged issues at AMS-IX as of 19:05 and onward, we've turned off the peering sessions with direct peers as well as the AMS-IX route servers to prevent further packetloss.

We are monitoring the situation closely at AMS-IX Statistics and will re-enable BGP sessions over AMS-IX once the issue appears resolved.

AMS-IX Notification:
AMS-IX Outage on Amsterdam Peering Platform

We've found an issue with both optics leading to our rackswitch in Rack B-2-13, in Nedzone Steenbergen.

After replacing optics and fibers, as well as cleaning them, there are still errors occurring.

We're currently preparing a new switch to seamlessly move servers onto.

Please expect an update here once it is done.

Update 23:26 - Switch has been replaced and network traffic is restored. If you experience any issues, please contact us through Ticketing.

We've identified traffic issue with the AMS-IX port located at GlobalSwitch. To ensure the stability of our network and to perform necessary diagnostic checks, we've taken the decision to temporarily disable this port.

Our NOC team is currently investigating the root cause of the problem. We are aiming to conduct all required checks and, if deemed safe, plan to return the AMS-IX port to its operational state by tomorrow night.

We do not anticipate any capacity issues as a result of this action. Our network has been designed with redundancies in place to handle such situations, and we expect no interruption to the normal flow of data.

Update: We've troubleshooted the issue and determined a reload of the linecard could remedy this issue. The connection to AMS-IX has been restored and peering session operate nominally.

As of October 16th, 21:00 our connection to LINX LON1 has gone down.

Our fiber provider's teams in both NL and the UK are currently investigating the issue.

We'll update this NOC post with updates as they come in. We see the link is operational as of October 27th 6:35 and we'll monitor if it stays stable. Peering will resume automatically.

Current updates so far:

Update 34 2023-10-28 16:28:15 : Final update: EXA can confirm all final checks have been completed on remaining customer services. No further interruptions are expected.
Update 33 2023-10-27 12:14:36 : EXA team are seeing alarms clear and we are confirming status with the remaining customers. Please contact EXA NOC if you are still seeing an impact on your service. Final update will be sent once all services are confirmed restored.
Update 32 2023-10-27 10:08:36 : EXA teams are on our locations in UK and Netherlands and are currently working to complete the remaining patches to reconnect all fibres. Customers should start to see services recover in the next 30-60 minutes. Further update will be provided in the next 60 minute
Update 31 2023-10-27 07:52:48 : EXA has confirmed that testing has been completed and we will begin connecting fibres within the next 90 minutes. EXA will request all customers to verify connectivity has restored and will send a further update once all services have been confirmed.
Update 30 2023-10-26 19:14:03 : EXA's marine repair contractor has confirmed that the final splicing is progressing as planned. Testing of the final splice is expected to be completed on the morning of the 27th of October. EXA will then begin to contact customers to confirm status once the repairs are completed. Next Update: 27th October 09:00 GMT
Update 29 2023-10-26 09:55:11 : EXA's marine repair contractor has confirmed that the repair vessel is continuing repair operations on the final splice. Once this is completed, the cable will need to be fully tested to verify that the repair is successful and that all fibres have been repaired. Repair operations are expected to be completed in the early hours of Friday the 27th of October. Next Update: 26th October 18:00 GMT
Update 28 2023-10-25 15:00:14 : EXA's marine repair contractor has confirmed that the initial splice has been completed successfully and they are now proceeding with retrieving the UK end of the cable. The repair vessel will then commence the final splice phase of the repair operations. Repair operations are expected to be completed in the early hours of Friday the 27th of October. Further updates will be provided as the repair progresses. Next Update: 26th October 10:00 GMT
Update 27 2023-10-24 19:19:25 : EXAs marine repair contractor has advised that the repair vessel is continuing work on the initial splicing on the Netherlands side of the cable. The repair vessel expects to complete testing of the initial splice tomorrow morning. Once completed successfully, the vessel will transit to the UK side of the cable to perform splicing on that side. Next update: 25th October 18:00 GMT
Update 26 2023-10-24 10:10:47 : EXA's marine repair contractor has confirmed that initial splice operations by the repair vessel are continuing on the Netherlands side of the cable and are expected to be completed within the next 24 hours. Once this splice is completed and successfully tested, the repair vessel will then transit to commence splicing operations on the UK side of the cable. As with all submarine repairs, the progress of repair operations is dependent on suitable weather conditions and the repair vessel is closely monitoring the forecast for the coming days. Next Update: 24th October 18:00 GMT
Update 25 2023-10-23 17:32:05 : EXA's marine repair contractor has confirmed that they have successfully recovered the cable at the UK side of the fault. The cable has been buoyed off and the repair vessel has commenced working to retrieve the Netherlands side of the cable. Once the cable has been retrieved from the seabed and have the cable on board, the repair vessel team will commence preparations for the initial splice. As with all submarine repairs, the progress of repair operations is dependent on suitable weather conditions and the repair vessel is closely monitoring the forecast for tonight and the coming days. Next Update: 24th October 10:00 GMT
Update 24 2023-10-22 18:25:52 : EXAs marine repair contractor has advised that they were able to successfully launch the ROV to retrieve the Netherlands side of the submarine cable. Repair operations are progressing with good weather conditions today and good weather is forecasted for the next 24 hours. Next update: 23rd October 18:00 GMT
Update 23 2023-10-22 09:36:01 : Update 23 2023-10-22 09:35GMT: EXAs marine repair contractor has advised that weather conditions overnight were unsafe to proceed with repair operations. They have confirmed that weather conditions have improved this morning and the repair vessel is preparing to commence repair operations. Next update: 22nd October 18:00 GMT
Update 22 2023-10-21 18:35:27 : EXAs marine repair contractor has advised that the repair vessel remains on location at the cable grounds and monitoring weather forecast for suitable conditions to commence repair operations. Next Update: 22nd October 10:00 GMT
Update 21 2023-10-21 12:07:33 : EXA's marine repair contractor has advised that the vessel remains on location at the cable grounds and continuing to monitor the weather for suitable conditions to commence the repair operations. Based on the current weather forecast, the repair vessel is expecting to commence repair operations on the 22nd of October. Next Update: 21st October 18:00 GMT
Update 20 2023-10-20 17:35:04 : EXA's marine repair contractor has advised that the vessel remains on location at the cable grounds and continuing to monitor the weather for suitable conditions to commence the repair operations. Based on the current weather forecast, the repair vessel is expecting to commence repair operations on the 22nd of October. Next Update: 21st October 10:00 GMT
Update 19 2023-10-20 10:09:46 : EXA's marine repair contractor has confirmed that the repair vessel is due to arrive at the cable grounds shortly. The repair vessel will monitor weather conditions closely for a suitable window to commence the repair operations. The current weather forecast is expecting stormy conditions for the next few days and the repair vessel expects that they will be able to begin operations in the early hours of Sunday 22nd October. As with all submarine cable repairs, the estimated time to be completed is dependent on several factors, including suitable weather conditions. The estimated completion date is October the 25th . Next Update: 18:00 GMT 20th October
Update 18 2023-10-19 17:51:07 : EXA's marine repair contractor has confirmed that the repair vessel has departed from port earlier today and is now in transit to the cable grounds. The repair vessel is expected to reach the cable grounds late morning on the 20th of October. The estimated time for completion of the repair work is the 25th of October. Next Update: 20th October 10:00 GMT
Update 17 2023-10-19 10:28:44 : EXA's marine repair contractor has advised that the repair vessel's departure from port has been delayed due to storm conditions in the UK in the last 24 hours. The conditions were deemed unsafe for departure and the repair vessel remains on standby awaiting suitable conditions to leave port. As with all submarine cable repairs, the estimated time to be completed is dependent on several factors, including suitable weather conditions. The estimated completion date is early morning on October the 25th . Next Update: 18:00 GMT 19th October
Update 16 2023-10-18 20:03:28 : EXA marine repair contractor has advised that loading has been completed and the vessel will depart for the cable grounds in the coming hours. The vessel will then transit to the cable grounds to begin the repair process in the coming days. The current ETR is the 24th of October. Next update: 10:00 AM GMT 19th October
Update 15 2023-10-18 10:18:24 : EXA's marine repair contractor has advised that loading of the repair vessel was continuing this morning. The repair vessel is expected to depart from the harbour within the next 24 hours and begin transit to the cable grounds. The completion date for the repair is currently 24th of October 2023. We will continue to provide regular updates as the repair process progresses. Next Update: 18:00 GMT 18th October
Update 14 2023-10-17 18:49:37 : EXA's marine repair contractor has confirmed that the repair vessel has been mobilized. The repair vessel will load the necessary equipment on board before departing from dock and transiting towards the cable grounds. A tentative completion date of the 24th of October has been provided. As with all submarine cable repairs, the timeline for completion of the repair will be dependent on several factors including weather and sea conditions and may be subject to change. More updates to follow. Next Update: 10:00 AM GMT 18th October
Update 13 2023-10-17 10:16:53 : OTDR testing has been completed and we can confirm that the fault has been identified on our submarine cable approximately 121km from Netherlands towards the UK. This is a Force Majeure event and EXA have engaged with our marine repair contractor to mobilise a repair vessel. As with all submarine cable repairs, the timeline for completion will be dependent on several factors including weather and sea conditions. At this time an ETR cannot be provided. Next Update: 18:00 GMT 17th October
Update 12 2023-10-17 08:56:10 : Please be advised that our team is continuing with further testing and our senior engineers will review the results once obtained. The incident is escalated to the highest level within the senior leadership and are pending further feedback. More updates to follow. Next update approximately 60 minutes (Or earlier pending feedback)
Update 11 2023-10-17 07:15:15 : Engineers after OTDR the fiber have identified that no fibers are free and the spares are also affected by the outage. Unfortunately rolling of fibers is unavailable. Our team is working closely with field engineers at this time and will advise on further information from the field at the earliest. In the interim our senior technical team is exploring all possibilities to expedite the restoration and will advise on further updates as soon as possible. Please note this is escalated to the highest level in EXA and our team will continue to follow up with the teams for further information. There is no ETR at this time. More updates are to follow. Next update approximately 60 - 120 minutes (Or earlier)
Update 10 2023-10-17 05:42:17 : Dear customers, EXA has received the OTDR test results and is currently analysing these results to form a POA. An ETR is unknown at present. More updates will follow.
Update 9 2023-10-17 04:19:03 : Dear customers, Our field team are on site in Zandvoort and will begin to test the fibres shortly. We should receive these results shortly for analysis. More updates to follow
Update 8 2023-10-17 03:04:10 : Dear customers, Our technician in Leiston has performed the necessary OTDR. We are waiting for our tech in Zandvoort to perform his OTDR upon arrival at the site. EXA require both tests to isolate the fault across the backbone. More updates to follow.
Update 7 2023-10-17 02:02:04 : Dear customers, Our field tech in Leiston is currently performing the OTDR test on the affected fibres which are showing hard down. We have been given an ETA of 3 hours from our field team in the Netherlands to perform an additional OTDR test from the site in Zandvoort. We will keep you posted with further updates.
Update 6 2023-10-17 00:59:09 : Dear customers, Please be advised that our field tech has arrived on site in Leiston and will begin to OTDR the fibres. More updates to follow.
Update 5 2023-10-17 00:05:45 : Dear customers, Our field team are due on site in the UK shortly based on the prior ETA given to us. We are following up with the field team to confirm the ETA before OTDR testing commences. More updates to follow.
Update 4 2023-10-16 23:01:53 : Dear customers, As advised prior the ETA provided to us by our field team remains at an hour away. We will keep you posted with further updates regarding this fault. More updates to follow.
Update 3 2023-10-16 22:02:50 : Dear customers, As advised in the prior update, we have engaged the relevant field teams to investigate this outage on either side in the UK and Netherlands. We have been given an ETA of 2 hours from our UK counterparts to perform an OTDR test. More updates will follow.
Update 2 2023-10-16 21:08:57 : Dear customers, We're dispatching our field teams on the site to further investigate the issue and waiting for the ETA More updates to follow
Initial Update 1 2023-10-16 19:43:50 : Dear customers, Please be advised we are experiencing an outage between Leiston, UK to Zandvoort, Netherlands. We have contacted the relevant teams to investigate the ongoing fault. More updates to follow.

Because of soft errors in the memory of one of our linecards we will be replacing a linecard in our NZS R2 router on the 28th of October at 11:00 PM CET.

Update: 2023-10-28 22:30: Final update

The Linecard has been replaced successfully and working without any issues.

We have experienced no downtime, this is due to our already redundant network.

As of 18:56 Cloud VPS services have gone down.

Our engineers are currently investigating the issue.

Our sincere apologies for this inconvenience.

When we have more information on the exact issue we'll update the NOC post here.

Update 20:09:
We've found a DDoS that was targeting an IP and blocked it. Traffic is recovered now and situation looks stable. We'll keep monitoring for now.

Update 20:52:
We confirmed the problem to be resolved as of initial fix at 20:00.

Dear Valued Customer,

We regret to inform you that our Anti-DDoS services are currently suspended due to an unexpected hardware failure in our Distributed Denial of Service (DDoS) mitigation equipment. Unfortunately, the issue could not be resolved immediately, but we are actively collaborating with the vendor to identify a prompt solution. In the meantime, we have already initiated a migration plan to adopt a new solution, and we will provide further updates later this month.

Rest assured, our Network Operations Center (NOC) team is fully dedicated to supporting you during this period. They will be available to assist with special cases, such as implementing custom Access Control List (ACL) rules or finding quick resolutions for any DDoS incidents you might encounter.

We apologize for any inconvenience this may cause and appreciate your understanding and patience during this situation.

**This message pertains to customers who have an active DDoS protection plan**

Darkfiber provider ExaInfra which connects Databarn Amsterdam (DBA) towards Equinix AM7 (EQX-AM7) is having a planned maintenance. The connectivity between DBA and EQX-AM7 will automatically reroute over our GSA darkfiber during this maintenance.

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Equinix-AM7 is having a planned maintenance.

The connectivity between NZS and EQX-AM7 will automatically reroute over our ring.

Customers having wave service will be down during this time.

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Equinix-AM7 is having a planned maintenance.

The connectivity between NZS and EQX-AM7 will automatically reroute over our ring.

Customers having wave service will be down during this time.

Dear Valued Customers,

We wanted to update you on the removal of external access to our ticketing system.
As we have noticed that usage with our ticketing system is mostly operated via emails and with the following considerations in mind:

- Outdated Code-base: The system relies on an outdated code base that is no longer considered trustworthy.

- Unsupported Operating System: The ticketing system requires an outdated operating system that is no longer supported, leaving it vulnerable to security risks.

- Integration with Customer Portal: Our goal is to replace the ticketing system with an integrated view system for tickets within the customer portal. This streamlined approach will enhance user experience and improve ticket management efficiency.

For the reasons mentioned above we have decided to take immediate action to remove all access to the system.

History Access - While we are working to integrate a view system for tickets in our customer portal https://ssc.nforce.com if you have a specific request you may direct your request to our administration department and we may permit temporary access to view history.

Darkfiber provider EuNetworks which connects Databarn Capelle (DBC) towards Globalswitch Amsterdam (GSA) is having a planned maintenance.

The connectivity between DBC and GSA will automatically reroute over our ring.

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Equinix-AM7 is having a planned maintenance.

The connectivity between NZS and EQX-AM7 will automatically reroute over our ring.

Customers having wave service will be down during this time.

In finishing up our migrations we still need to migrate the IP addresses currently originating in DBC which means there will be an interruption in your traffic, we will do our best to keep this as short as possible which should not last longer then 30 minutes per IP block, in most case this will be much shorter.

If you currently have an IP in any of the following IP ranges, this will have an effect on your service.

Do keep in mind that even though the maintenance lists it for a week this will not mean you will have a weeks worth of downtime, it means that in this week at a moment you will have some downtime.

46.166.184.0/24
46.166.188.0/24
46.166.189.0/24
46.166.191.0/24
46.166.136.0/24
46.166.137.0/24
46.166.138.0/24
46.166.140.0/24
46.166.143.0/24
46.166.146.0/24
46.166.147.0/24
46.166.150.0/24

We will update this post once ranges have been moved.

In finishing up our migrations we still need to migrate the IP addresses currently originating in DBC which means there will be an interruption in your traffic, we will do our best to keep this as short as possible.

If you currently have an IP in any of the following IP ranges, this will have an effect on your service.

46.166.139.0/24
46.166.142.0/24
46.166.148.0/24
46.166.151.0/24
46.166.187.0/24

We will update this post once ranges have been moved.

Update 12:44 CET:
All ranges have been moved to the correct routers now.

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Equinix-AM7 is having a planned maintenance.

The connectivity between NZS and EQX-AM7 will automatically reroute over our ring.

Customers having wave service will be down during this time.

Darkfiber provider Relined which connects GlobalSwitch Amsterdam (GSA) towards Nikhef is having a planned maintenance.

Their reason for this maintenance:

"Relined performs maintenance work to maintain the availability and quality of its fiber optic network."

Their references is: PA-948002.

Darkfiber provider Relined which connects GlobalSwitch Amsterdam (GSA) towards Equinix AM5 (EQX-AM5) is having a planned maintenance.

Their reason for this maintenance:

"Relined performs maintenance work to maintain the availability and quality of its fiber optic network."

Their references is: PA-948002.

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Equinix-AM7 is having a planned maintenance.

The connectivity between NZS and EQX-AM7 will automatically reroute over our ring.

Customers having wave service will be down during this time.

Update 2023-01-31:
Due to unforeseen circumstances this will be postponed, once the new date has been known we will update this post.

Update 2023-02-03:
This maintenance has been rescheduled to 7 February

Update 2023-02-07:
During the migration of the links, one of the channels kept flapping, after a debugging session it was rebuild back to the old situation.

On the 15th of February we'll debug this further including separate cards and other channels.

Currently we have a switch outage in BNR (Bytesnet Rotterdam), we are sending out an engineer to investigate the situation.

Update 09:12:
SWe were notified by the datacenter that this was a power issue on one of the feeds in the datacenter which has now been solved.

Currently our darkfiber between Nedzone Steenbergen (NZS) and Equinix Amsterdam 7 (EQX-AM7) is down.

We've sent in a ticket with our darkfiber provider for more information, they have it currently under investigation.

All traffic for EQX-AM7 is re-routed over our darkfiber with GSA.

Direct wave customers will experience an outage for the duration of this issue.

Update 1:14:
Currently the connection is back online. We haven't had an RFO from our dark fiber provider yet and will monitor the situation.

We noticed one of the power distribution units (PDU) namely PDU2 not functioning as it should anymore. Therefore under emergency maintenance this PDU will be replaced, customers located on this PDU will have their servers rebooted.

As the switch is also located on this PDU customers in this rack will notice a short downtime in network.

Customer located in this rack will be informed by e-mail.

We are aware that currently customer having a connection over Levelfour have no connection to our network.

We are currently investigating into this matter.

Update 13.00:
Levelfour has debugged the issue and provided a fix for us, all connections are currently working again.

Due to BGP issues we are going to do maintenance of our router in EQX-AM7.

During the maintenance, Internet Customers will experience a loss of internet as well as customers having only one BGP session. Customers having a redundant BGP session with us should not experience any issues.

We've chosen this time to minimally impact our customers and be during our time of lowest network traffic.

We noticed one of the power distribution units (PDU) namely PDU1 not functioning as it should anymore. Therefore under emergency maintenance this PDU will be replaced, customers located on this PDU will have their servers rebooted.

As the switch is also located on this PDU customers in this rack will notice a short downtime in network.

Currently our darkfiber between Databarn Amsterdam (DBA) and Globalswitch Amsterdam (GSA) is down.

We've sent in a ticket with our darkfiber provider for more information and will update accordingly.

All traffic for GSA is re-routed over our darkfiber with EQX-AM7.

Update 14:39:
Currently the darkfiber is back online, while we haven't been given the reason as to why by our provider yet, they mentioned an RFO will follow.

As currently there are multiple public resolvers who are made specially for this purpose and we see ours being used less and less we have decided to retire our public DNS resolvers.

At the 1st of August we will be shutting down the resolvers, any customers then still using the resolvers won't be able to resolve any outside names.

Below you can find the most popular DNS resolvers which can be used instead of ours:

Google:
8.8.8.8
8.8.4.4

Cloudflare:
1.1.1.1
1.0.0.1

Quad9:
9.9.9.9
149.112.112.112

As we do not limit which resolvers you use, you are free to pick any other you want as well.

If you have any questions or are not sure if you are using ours you can send us an email at support@nforce.com.

Darkfiber provider Exainfra which connects Bytesnet Rotterdam (BNR) towards GlobalSwitch Amsterdam (GSA) is having a planned maintenance.

Their reason for this maintenance:

"Cable Reroute between Luchtvaartstraat and Zuidasdok Amsterdam due to construction works"

The connectivity between NZS and GSA will automatically reroute over our ring network during this maintenance.

Customers having waves towards GSA will see some downtime.

At the 18th of May between 9.00 AM - 12 PM CEST we will be upgrading the firmware of the switches located in rack B-2-1 in Nedzone.

During this timeframe there will be multiple interruptions of a few minutes while the switch is reloading into the new firmware, we will keep this as short as possible.

Update 2022-05-18 09:57:
The switch has been firmware upgraded, and is fully reachable again.
In case you still have issues please send an email to support@nforce.com so we can check on your issue.

Darkfiber provider Exainfra which connects Bytesnet Rotterdam (BNR) towards GlobalSwitch Amsterdam (GSA) is having a planned maintenance.

Their reason for this maintenance:
"Emergency repair of damaged cable"

The connectivity between NZS and GSA will automatically reroute over our ring network during this maintenance.

Customers having waves towards GSA will see some downtime.

Darkfiber provider Relined which connects GlobalSwitch Amsterdam (GSA) towards Equinix AM5 (EQX-AM5) is having a planned maintenance.

Their reason for this maintenance:

"Due to reconstruction work carried out by the local authorities. Relined is obliged to relocate a section of her network. This action will result in a temporary interruption of the service."

Their references is: PA-060002.

Darkfiber provider Relined which connects GlobalSwitch Amsterdam (GSA) towards Nikhef is having a planned maintenance.

Their reason for this maintenance:

"Due to reconstruction work carried out by the local authorities. Relined is obliged to relocate a section of her network. This action will result in a temporary interruption of the service."

Their references is: PA-718001.

Currently AMS-IX is having major issues as their graph bottomed out and we also started noticing issues with BGP sessions flapping every 20 seconds or so.

Until we get an explanation and the OK from the AMS-IX team we've shut all our BGP sessions with AMS-IX.

We will update this post with more information later on.

Update 11:30:
We received the following from the AMS-IX:

We are experiencing an increased amount of BUM traffic on the platform as well as several down BGP sessions. We are investigating the problem, and we will update you as soon as possible.

Update 14:00:
While we are still awaiting an official RFO we've enabled the sessions again after it has been stable for the last couple of hours.

As of 11:36 rack A-2-12 in Nedzone Steenbergen (NZS) has gone down.

Our engineers are currently investigating the issue.

Our sincere apologies for this inconvenience.

When we have more information on the exact issue we'll update the noc post here.

Update 13:04:
As of 12:38 most servers should have began coming back online, the root cause has been found and solved. If you still have any issues with your server please let us know by replying to your ticket you already had with us or by sending a message to support@nforce.com.

The fiber provider which connects Databarn Capelle (DBC) towards GlobalSwitch Amsterdam (GSA) is having a planned maintenance. The connectivity between DBC and GSA will automatically reroute over our ring network during this maintenance.

Please note this is not a duplicate of Post #975, this is separate maintenance affecting the same fiber route.

The fiber provider which connects Nedzone Steenbergen (NZS) towards Globalswitch Amsterdam (GSA) is having a planned maintenance. The connectivity between NZS and GSA will automatically reroute over our ring network during this maintenance.

Customers having waves towards GSA will see some downtime.

Because of maintenance related tasks and to debug an issue we will be reloading our DBA R2 router on the 24th of February at 10:00 AM CET.

Because of maintenance related tasks and to debug an issue we will be reloading our DBA R1 router on the 22th of February at 10:00 AM CET.

The fiber provider which connects Nedzone Steenbergen (NZS) towards Equinix Amsterdam 7 (EQX-AM7) is having a planned maintenance. The connectivity between NZS and EQX-AM7 will automatically reroute over our ring network during this maintenance.

Customers having waves towards EQX-AM7 will see some downtime.

The fiber provider which connects Databarn Capelle (DBC) towards Equinix Amsterdam (EQX-AM7) is having a planned maintenance. The connectivity between DBC and EQX-AM7 will automatically reroute over our ring network during this maintenance.

The fiber provider which connects Databarn Capelle (DBC) towards GlobalSwitch Amsterdam (GSA) is having a planned maintenance. The connectivity between DBC and GSA will automatically reroute over our ring network during this maintenance.

EDIT: Notification resent due to maintenance only affecting DBC connectivity, not DBA.

Because of some maintenance related tasks we will be reloading our GSA router on the 4th of February at 1:00 AM CET.

Our peering traffic will automatically flow over our EQX-AM7 router.

Internet and Transit customers only having a connection to this router will experience some downtime during the reload.

During the night 3 to 4 november, at 1 AM, Dataplace will have a maintenance on the dark fiber.

The line will be interrupted between 1 AM and 2 AM. This only affects traffic towards GSA, and will automatically pass through AM7 via our ring network when the maintenance starts.

For customers with own wave services, this path will go down for up to an hour and should be fully restored after the maintenance is complete.

As the emergency maintenance last week showed success, we going to schedule reloads of the routers of DBC as well as DBA to apply new settings to IPv6.

9Th of November at 7:30 CET:
We will reload R1 of DBA.

10Th of November at 7:30 CET:
We will reload R2 of DBC.

11Th of November at 7:30 CET:
We will reload R1 of DBC.

As IPv6 continues to give issues, we are hereby reloading router 2 to apply new settings to combat the ongoing issues of IPv6.

We will make sure that the traffic falls back on R1 before the reload.

Update 9:21:
As the router R2 is back in production, we also will be reloading R1 to bring these settings in effect.

Update 10:18:
Both routers are reloaded with new settings and confirmed that the issue is resolved.

We got a notice today from the datacenter that the power company Liander is doing an emergency maintenance tomorrow to repair a 10Kv ring in Amsterdam.

Initially this was planned for 2022 but they moved it way back to 20th of October to be able to guarantee a good working electricity grid.

The datacenter themselves were also only informed about this today.

We asked for some assurances and they assured us that no outages are expected as they have UPSsses, generators and have a company bring an extra generator as well.

We will be monitoring the situation closely tomorrow.

At 21:17 we noticed a management card flap at our second router at NZS.

We are currently investigating into this as our BGP are loading out again.

Update 01:10:
We continue to see the management cards swap (both the active / standby) on this router at random intervals. Router 1 is carrying all the customer load at the moment while we debug this issue.

Update 10:30:
We have replaced the management cards of R2 and monitored the situation. After replacing the management cards, we have no longer seen any swapping between active/standby card.

We expect the issue is resolved however we still continue to monitor the situation during the day.

Due to a range of issues we are going to do an emergency reload of our router in EQX-AM7.

Internet Customers will experience a loss of internet as well as customers having only one BGP session. Customers having a redundant BGP session with us should not experience any issues.

We are aware it is the start of a workday for most but due to the on-going issues there is little choice left for us to do it at a very short timespan, we will try our best to keep the downtime to an absolute minimum.

Update 10:25
We have finished the reload and restored BGP sessions. The main cause for this emergency reload was the router no longer propagating new routes nor pruning old routes.

Around 12:00 we've noticed our connections from GSA towards Nedzone Steenbergen (NZS) and Bytesnet Rotterdam (BNR) went down.

Our darkfiber provider GTT has informed us of a fibercut between Amsterdam and Rotterdam, as both BNR and NZS makes use of this darkfiber they have had a loss of redundancy.

All traffic for NZS is re-routed over our darkfiber with EQX-AM7 / all traffic for BNR is re-routed over DBC.

Wave customers between NZS and GSA are currently down.

Update 1: 2021-09-04 10:23 UTC
Please be advised that the GTT Infra NOC detected an interruption occurring on our network between Amsterdam & Rotterdam at 09:18 UTC. Initial investigation has detected a potential fiber cut occurring. FLM teams have been engaged and are en route with an ETA of 90 mins to site.

Update 2: 2021-09-04 11:29 UTC
FLM teams report that they have now reached site & are beginning their investigations.
Once the OTDR has been shot crews will travel to the break point & ETTR will be made available following a damage assessment.

Update 3: 2021-09-04 12:23 UTC
Crews on site have shot the OTDR & discovered the location of the break to be occurring approx 450 meters outside of Amsterdam. Teams are en route to the break point now with an ETA on site set at roughly 60 minutes.

Update 4: 2021-09-04 14:05 UTC
Crews have now arrived at the break point & discovered the fiber impacted by 3rd party drilling activity on site. A damage assessment & repair plan is being formulated by the FLM crew & civils work will need to be undertaken to excavate the damaged cable. ETTR will become available once complete.

Update 5: 2021-09-04 15:15 UTC
Crews report that civils crews are now on site & are preparing to excavate the damaged fiber. Repair plan will include the introduction of two new handholds along the route of the fiber. ETTR will become available once civils work has been completed & the additional infrastructure deployed to provide a safe route around the ongoing drilling work.

Update 6: 2021-09-04 16:25 UTC
Crews report that excavation work on site is progressing well with splicing to begin at a tentative time of 18:30 UTC.

Update 7: 2021-09-04 17:42 UTC
Crews on site report that excavation work has now completed & they are prepping the fiber for splicing which will commence momentarily.

Final Update: 2021-09-04 18:56 UTC
FLM teams confirm that all fibers have now been spliced & the GTT NOC can confirm that all alarms have cleared with services restoring.
The actions taken by the team on site will constitute a permanent repair & no further disruption to service is anticipated.