Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Equinix-AM7 is having a planned maintenance.

"This work is necessary to continue guaranteeing the high quality and reliability of our network.
We will make sure to minimize the duration of any impact on your connection(s)."

The connectivity between NZS and EQX-AM7 will automatically reroute over our ring.

Customers having wave service will be down during this time.

Our interconnect provider LevelFour has announced planned maintenance. Their reason is stated below:

"To improve the network, the connection is transferred to a new switch.
Interruption length: within 5 min."

During this maintenance we expect some downtime to occur for customers connected through LevelFour.

Due to power issues, several of our websites went down. Some emails sent to our ticketing system may have been delayed.

We are sorry for this inconvenience and aim to fully restore power, as we are currently running on a single feed.

No customer equipment has been affected by this.

Currently we have a switch outage in Rack D-2-11 in NZS (Nedzone Steenbergen), we have an engineer on-site to investigate the situation.

We're currently replacing the faulty switch.

Update: The switch is replaced and connectivity is restored.

Our datacenter provider Bytesnet has announced planned maintenance on one of the power feeds in Bytesnet. Their reason is stated below:

"In order to continue to guarantee proper operation, the circuit breakers are maintained every 5 years in accordance with supplier instructions. The maintenance of the A-feed is planned for this year."

During this maintenance any equipment in BNR that only has power from the A-feed would be powered off during these maintenance works.

Our transit provider Colt has announced planned maintenance on their routers. Their reason is stated below:

"Work is required to complete the code upgrades on network devices to the Colt domain to ensure resilience, scalability and continued integrity of the network."

During this maintenance all Lumen/Colt traffic will be automatically rerouted over our other transits and peering connections.

Our transit provider Liberty Global has announced planned maintenance on their routers. Their reason is stated below:

"Liberty Global is planning to perform a firmware update on its router at Equinix AM7."

During this maintenance all Liberty Global traffic will be automatically rerouted over our other transits and peering connections.

Our transit provider Colt has announced planned maintenance on their routers. Their reason is stated below:

"Work is required to complete the code upgrades on network devices to the Colt domain to ensure resilience, scalability and continued integrity of the network."

During this maintenance all Lumen/Colt traffic will be automatically rerouted over our other transits and peering connections.

Final Notice: Departure from DE-CIX and LINX Internet Exchanges

Dear Valued Peers and Partners,

As part of our commitment to delivering a high-quality, globally interconnected network,
we are concluding our participation in the DE-CIX and LINX internet exchanges.
This decision aligns with our strategy to optimize connectivity, expand network diversity, and ensure the best possible performance for our customers.

Over the past 12 months, we initiated a phased process to remove peering on these exchanges
and have communicated with peers to migrate to alternative arrangements.
For peers who have not yet responded or transitioned, we will cease all peering over DE-CIX and LINX in the upcoming week.

Our Commitment to Quality and Growth

We remain highly diligent in expanding and improving our network. Our focus includes:

• Enhancing network diversity by continually evolving our strategy and partnerships with transit providers.


• Prioritizing geographic redundancy and connectivity to ensure our customers benefit from a stable, high-performance network.


• Actively engaging with peers and partners to foster a collaborative, well-distributed internet ecosystem.

Moving Forward

- We are available for peering via other exchanges such as NL-ix, AMS-ix, ERA-ix, and BBIX.

- We welcome peering requests according to our peering policy at PeeringDB and are happy to explore connections in any of our PoP locations.

If you wish to peer with us or discuss alternative arrangements, please contact us at
noc@nforce.com.
We are eager to collaborate with partners who share our vision of building a stronger, more interconnected internet.

Thank you for your understanding and cooperation as we continue to enhance the quality of our network and connectivity.

Best regards,

NOC Team
NFOrce Internet Services

Liberty Global is planning to perform a firmware upgrade which will cause downtime.

During this maintenance all Liberty Global traffic will be automatically rerouted over our other transits and peering connections.

As of September 24th, 19:25 our connection to LINX LON1 has degraded.

Our fiber provider's teams in both NL and the UK are currently investigating the issue.

We'll update this NOC post with updates as they come in. We're also monitoring traffic load to determine whether to re-route traffic to avoid congestion.

Update: Issue is resolved and our connection is back to full capacity.


Current updates so far (time in GMT):

Update 18 2024-09-25 16:26:06 : FINAL UPDATE. Please be advised that we have seen services restore. There was degradation on fibre and we also had to replace a faulted card in Zandvoort. All links are restored now and no further impact is expected. Will share the RFO in the coming days.
Update 17 2024-09-25 15:29:18 : Please be advised that we are continuing to work with our Engineering team and field team onsite. Investigation continues and we are working dilligently to resolve this issue. No ETR as of yet next update in 60 mins.
Update 16 2024-09-25 13:44:34 : Engineers are continuing their investigations on site and are working with our senior NOC team to resolve the ongoing fault. Exact ETR cannot be confirmed but we are working to resolve the issue as quickly as possible.
Update 15 2024-09-25 13:07:12 : Please be advised teams are onsite on both locations and we are now working to isolate this issue. Furhter updates to follow. Next update in 60 mins.
Update 14 2024-09-25 12:22:01 : Please be advised team are onsite on the Zandvoort side and we are pending the confirmation team are onsite in Leiston. Once onsite we will investigate and isolate the issue. Next update in 60 mins.
Update 13 2024-09-25 11:09:19 : Please be advised team remain on track to arrive at 12:00 UTC. At which time we will be able to test and isolate this issue. Next update in 60 mins.
Update 12 2024-09-25 09:38:03 : Please be advised that team have dispatched to site in Lesiton in order to test and locate the fault. ETA to site is 12:00 UTC. Next update in 60 mins.
Update 11 2024-09-25 08:17:32 : Please be advised that we have replaced the faulty cable in Zandvoort and have re-enabled the Amps. However we are still seeing an OLOS on the amp. The next step is to get team to site at the opposite end in Leiston to test end to end. No ETA as of yet. Next update in 60 mins.
Update 10 2024-09-25 06:09:30 : The ETR is estimated approximately 1.5h from now, at 07:30 UTC. New updates will be provided as soon as available.
Update 9 2024-09-25 05:21:04 : The fiber technician found a burnt connector onsite in Zandvoort. We are now checking availability to replace this and we will confirm as soon as an ETR is established.
Update 8 2024-09-25 03:51:49 : Please be advised that the Fiber Team is 15-30 minutes from site. We will provide updates as soon as information becomes available.
Update 7 2024-09-25 00:53:48 : Please be advised that no visible fault was found on the fiber patches for this link. We have also requested attendance from our Fiber Team in Zandvoort, they are en route to site at the moment. They will be able to perform OTDR tests on the fibres and confirm the location of the damage. Next update in 90 minutes.
Update 6 2024-09-24 23:06:01 : Our FLM technician have arrived on both sites, the investigation will be commenced shortly, more updates to follow. Next update in 60 minutes.
Update 5 2024-09-24 22:02:35 : Our FLM team advised the technician will arrive in Zandvoort shortly and the tech in Leiston experienced some delay but will arrive within an hour, further update will be shared once technician arrived site. Next update in 60 minutes.
Update 4 2024-09-24 20:49:38 : Our FLM advised the ETA to site in Zandvoort is at 22:00 UTC and Leiston at 22:15, further investigation will be commenced once tech arrived site. More updates to follow. Next update in 60 minutes.
Update 3 2024-09-24 19:43:49 : Kindly note that we're dispatching the field teams on the Zandvoot site to further investigate the issue. No ETA provided and will keep you posted with the latest updates Next update in 60 minutes.
Update 2 2024-09-24 18:26:56 : Kindly be informed that we're still investigating the degradation and enagaged our senior team No ETR orovided but will keep you posted with the latest updates Next update in 60 minutes.
Initial Update 1 2024-09-24 17:25:12 : Please be advised we are experiencing a heavy degradation on our span between Zandvoort, Netherlands and Leiston, UK since 16:29 UTC, we are currently investigating, update will be shared once available. Next update in 60 minutes.

We would like to inform you about a recent network outage and sincerely apologize for any inconvenience this may have caused. The issue arose when we needed to remove NTT as upstream from our network due to underlying issues with transit from them that we were experiencing. Following this removal and the subsequent changes to our route maps, the network began to spiral out of control, requiring an urgent reconfiguration.

Unfortunately, this reconfiguration process led to a temporary outage. We have since restored the network and are closely monitoring the situation to ensure stability.

Moving forward, we will conduct a thorough review of all changes and apply any necessary adjustments in a controlled and methodical manner to prevent similar issues from occurring.

Thank you for your understanding and continued support.

Because of some maintenance related tasks we will be performing an emergency reload on our GSA router on the 24th of August at 1:00 AM CEST.

Our peering traffic will automatically flow over our EQX-AM7 router.

Internet and Transit customers only having a connection to this router will experience some downtime during the reload.

FINAL UPDATE: Dear Customers, we have completed the scheduled maintenance, we thank you for your trust and patience.
--------------------------
UPDATE 2: Dear Customers, we are about to start our scheduled maintenance in the next couple of minutes. Our next update will be at 2 P.M. CEST
--------------------------
UPDATE 1: Dear Customers, we are in the preparation of our scheduled maintenance for ndu2.b-2-16.nzs which will start in approximately 15 Minutes. We expect a downtime of roughly 20-25 minutes. Our next update will be at 1:30 P.M. CEST when we will start our maintenance.
--------------------------
Dear Valued Customer,

We would like to inform you of a scheduled maintenance for ndu2 located in Rack B-2-16. We have observed unusual behavior, including intermittent ping drops, from this switch. To address these issues and ensure stable performance, we will be conducting maintenance next Thursday.

Impact:
There will be a brief downtime of approximately 20-30 minutes during the maintenance window. Services dependent on the affected unit may experience temporary interruptions during this period.

No Action Required:
Our technical team will manage the maintenance procedures and monitor the situation to ensure minimal disruption. We will do our best to restore services promptly.

Support:
If you have any questions or require further assistance, please contact our support team at noc@nforce.com.
We appreciate your understanding and cooperation as we work to improve our service reliability.

Thank you,
NOC Department
NFOrce Entertainment B.V

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Equinix-AM7 is having a planned maintenance.

"This work is necessary to continue guaranteeing the high quality and reliability of our network.
We will make sure to minimize the duration of any impact on your connection(s)."

The connectivity between NZS and EQX-AM7 will automatically reroute over our ring.

Customers having wave service will be down during this time.

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Equinix-AM7 is having a planned maintenance.

"During this maintenance, the connections over this route will be interrupted for some time.
It goes without saying that we do everything we can to keep the downtime as short as possible."

The connectivity between NZS and EQX-AM7 will automatically reroute over our ring.

Customers having wave service will be down during this time.

Wave service provider which connects Equinix AM7 (EQX-AM7) towards LINX is having a planned maintenance.

Their maintenance reason is "Test for power issues on chassis."

EXA has given this maintenance reference number: 00396869

All LINX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

Wave service provider which connects Equinix AM7 (EQX-AM7) towards DE-CIX is having a planned maintenance.

Their maintenance reason is "Fiber works on our network."

EXA has given this maintenance reference number: 00387835

All DE-CIX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

Wave service provider which connects Equinix AM7 (EQX-AM7) towards DE-CIX is having a planned maintenance.

Their maintenance reason is "Necessary cable relocation works due to Bridge demolition and reconstruction. Please note that date is set by the local authorities and any changes are out of our control."

EXA has given this maintenance reference number: 00403457

All DE-CIX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

UPDATE: Our darkfiber provider has let us known that the planned maintenance is cancelled.

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Equinix-AM7 is having a planned maintenance.

The connectivity between NZS and EQX-AM7 will automatically reroute over our ring.

Customers having wave service will be down during this time.

As of January 31st, 11:30 our connection to LINX LON1 has gone down.

Our fiber provider's teams in both NL and the UK are currently investigating the issue.

We'll update this NOC post with updates as they come in. The connection at our Globalswitch Amsterdam (GSA) location is restored as of February 2nd, 12:00 and peering has resumed.
The connection at Equinix AM7 (EQX-AM7) is restored as of February 16th, 4:00 and peering has resumed.

Current updates so far:
Update 35 2024-02-16 16:23:55 : FINAL UPDATE: EXA can confirm that the cable repair has been completed successfully and no further interruptions are expected. All customer services should now be restored, if you are experiencing any issues with yur service please contact EXA NOC.
Update 34 2024-02-16 11:44:34 : Please be advised that EXA can confirm that all repair work has been completed successfully. At this time, EXA are working with some customers to investigate their individual services which are still seeing an issue. A final notification will be sent once all investigations have been fully completed.
Update 33 2024-02-16 08:37:01 : Please be advised that we have seen our spans recover and light levels have restored, however, we see a lot of our waves are still recovering so you may still be experiencing outage, Please be advised our Engineering team are working through all these waves restoring services at this time. Next update in 60 mins.
Update 32 2024-02-16 05:03:54 : Dear Customers, EXA teams are seeing alarms clearing, and we are confirming the status with the remaining customers. Please contact EXA NOC if you are still seeing an impact on your service. The final update will be sent once all services are confirmed and restored. Next update is February 16th 09:00 GMT.
Update 31 2024-02-16 02:52:55 : Dear customers, Please be advised that EXA's field teams continue to connect the remaining customer fibres. We are seeing alarms beginning to clear but works continue, as some services are yet to recover. Next update February 16th 05:00 GMT.
Update 30 2024-02-16 00:58:03 : Dear customers, Be advised that final testing has been completed. At this time, EXA field teams are scoping and connecting the fibres on both terminal sites. Exa NOC will contact customers with dark fibre services to verify their status once fibres are reconnected. Next update February 16th 03:00 GMT.
Update 29 2024-02-15 22:54:54 : Dear customers, Be advised that final testing of the cable is continuing at this time. Once this has been completed successfully, EXA field teams will begin reconnecting all fibres on both terminal sites. At that time, EXA NOC will ask customers with dark fibre services to restore their lasers and verify their status. Please ensure that lasers are not turned back on until EXA confirms that fibres have been reconnected on-site. Next Update February 16th 01:00 GMT
Update 28 2024-02-15 17:19:18 : Please be advised that the final splice phase has been completed successfully. The cable repair vessel is now preparing to deploy the cable to the seabed. EXA field teams are currently on site at both UK and Netherlands terminal sites awaiting confirmation of deployment and will commence final testing of all fibers in the coming hours. Next Update February 15th 23:00 GMT
Update 27 2024-02-15 09:46:39 : Please be advised that the cable repair vessel replaced the faulty joint and the final splice phase has progressed well overnight. Post splice testing is ongoing at this time. EXA field teams are on site in both UK and Netherlands terminal stations preparing for final testing which expected to begin later today. Next Update February 15th 18:00 GMT
Update 26 2024-02-14 17:59:09 : Please be advised that during the final splice testing phase a fault was discovered with the joint of the final splice which needs to be removed and a new joint created. The cable repair vessel is working diligently to resolve this and expects to replace the joint and resplice the fibres over the next 24-30 hours. Next Update February 15th, 10:00 GMT
Update 25 2024-02-14 13:24:55 : Please be advised that the post splice testing is continuing at this time. EXA field teams are on site in both UK and Netherlands terminal stations preparing for final testing which expected to begin later today. Next Update February 14th, 18:00 GMT
Update 24 2024-02-14 08:20:58 : Please be advised that the final splice phase is progressing well and post splice testing is ongoing. Once final splice phase is completed, all fibres will undergo final testing from the terminal stations in UK and Netherlands. Next update February 14th 12:00 GMT
Update 23 2024-02-13 17:59:03 : Dear customers, Please be advised that the final splice phase is progressing well. The cable repair vessel expects to begin testing the fibres towards the UK terminal station later tonight. Next Update February 14th 08:00 GMT
Update 22 2024-02-13 10:15:39 : Please be advised that the intial splice phase was completed successfully overnight. The cable repair vessel has begun preparing the cable for the final splice phase. At present we expect the cable to be deployed and system normalized early morning on February 14th. Next Update February 13th 18:00 GMT
Update 21 2024-02-12 18:08:45 : Initial Splice phase operations are ongoing and expected to be completed later this evening. Upon completion of this phase, the cable repair vessel will retrieve the UK cable end for final splice operations. At present we expect the cable to be deployed and system normalized late evening on February 13th. Next Update February 13th 10:00 GMT
Update 20 2024-02-12 10:01:36 : Repair operations are progressing well over the last 24 hours. Expected to complete initial splice phase later today, 12th February. Upon completion of this phase, the cable repair vessel will retrieve the UK cable end for final splice operations. At present we expect the cable to be deployed and system normalized late evening on February 13th. Next Update February 12th 18:00 GMT
Update 19 2024-02-11 20:36:08 : Please be advised that the cable was successfully recovered and tested to the Netherlands terminal station. The initial splice phase is continuing at this time. Next update February 12th 10:00 GMT
Update 18 2024-02-11 10:09:33 : Please be advised that repair operations are progressing well with recovery of cable ongoing and initial splice phase to commence later today. Next update February 11th 20:00 GMT
Update 17 2024-02-10 21:07:52 : Please be advised that the cable repair vessel is continuing its operation to recover and test the cable to the Netherlands terminal station. Next update February 11th 10:00 GMT
Update 16 2024-02-10 10:07:13 : The cable repair vessel has confirmed that operations recommenced this morning following weather downtime. The vessel will continue operations to recover and test the Netherlands end of the cable today. Next update February 10th 20:00 GMT
Update 15 2024-02-09 10:01:14 : Please be advised that the cable repair vessel completed testing into the UK terminal station and has buoyed off the cable. At this time, the vessel is currently on weather standby due to large swells and strong winds in the area. At the next opportunity the vessel plans to recover and test the Netherlands end of the cable, and prepare for starting the initial splice. Next Update February 10th at 10:00 GMT.
Update 14 2024-02-08 09:55:21 : Please be advised that repair operations are progressing well. The cable repair vessel recovered the UK cable end overnight and is currently testing towards the UK terminal station. Next update: 9th Feb 10:00GMT.
Update 13 2024-02-07 18:14:25 : EXA's marine contractor has advised that the ROV has located the cable fault and the repair vessel is now working to retrieve the cable from the seabed. Once the cable is on board the vessel, the cable will be tested before being buoyed off. The repair vessel will then begin transit to recover the NL side of the cable. Next Update February 8th 10:00 GMT
Update 12 2024-02-07 09:41:38 : EXA's marine contractor has advised that they have successfully deployed the ROV overnight and work is ongoing to recover and test the UK end of the cable. Next update February 7th 18:00 GMT
Update 11 2024-02-06 18:04:25 : As per the latest update from our marine contractor, the repair vessel remains on weather standby. They have been closely monitoring the weather forecast and expect to be able to launch the ROV later this evening, to retrieve the cable from the seabed. Next update February 7th 10:00 GMT
Update 10 2024-02-06 09:51:56 : Please be advised that the cable repair vessel arrived on cable grounds yesterday evening however is currently on weather standby due to strong winds and large swells in the North sea. The cable repair vessel expects the earliest opportunity to launch the ROV to locate and retrieve the cable system will be later today, Feb 6th. We will continue to closely monitor the weather forecast and recommence operations at the earliest opportunity. Next Update February 6th at 18:00 GMT
Update 9 2024-02-05 09:42:06 : Dear customers, EXAs marine repair contractor has advised that the repair vessel is in transit and is due to arrive on the cable grounds today. Once on location at the cable grounds, the vessel will commence operations to recover cable from the seabed and test towards the NL side of the cable. Next update Feb 6th 10:00 GMT
Update 8 2024-02-04 10:28:59 : Exas marine repair contractor has advised that the repair vessel will complete loading of spares today and then depart from port once completed. The vessel is expected to arrive at the cable grounds on the morning of the 5th of February to commence repair operations. Next update Feb 5th at 10:00 GMT
Update 7 2024-02-03 09:42:34 : EXAs marine repair contractor has advised that the repair vessel will arrive in port today to begin loading of spares and equipment. Once this is complete it will then begin transit towards the cable grounds to commence repair operations. Next update Feb 4th 10am
Update 6 2024-02-02 10:29:28 : Please be advised that the repair vessel will arrive in port later today and begin loading necessary spares in preparation for the repair operations. The vessel is expected to depart for the cable grounds on Feb 3rd. EXA NOC will continue to share daily updates throughout to progress of repair operations. Next Update: Feb 3rd at 10:00GMT.
Update 5 2024-02-01 10:03:48 : EXA can confirm that our marine repair contractor has been engaged and that a repair vessel is being mobilised to support the upcoming repair works. As with all submarine cable repairs, it is difficult to provide an ETR and the repair timeline will be dependent on several factors including sea and weather conditions. Our next update will be provided on the 2nd of February at 10:00 GMT.
Update 4 2024-01-31 15:59:48 : EXA can confirm that OTDR testing has identified a break on our submarine cable between UK and Netherlands. EXA has engaged with our marine repair contractor to support repair operations. EXA requests that all customers with Dark Fibre services to please ensure your lasers are shut off and send confirmation to the EXA NOC that this has been done. As with all submarine cable repairs, the timeline for completion of the repair will be dependent on several factors including weather and sea conditions. Next Update will be provided Thursday the 1st February at 10:00 GMT
Update 3 2024-01-31 14:32:21 : EXA can confirm that OTDR testing verified that there is a fibre break on our submarine cable between UK and Netherlands. Technicians are continuing testing all fibres to confirm all fibres have been impacted. EXA requests that all customers with Dark Fibre services please ensure they have shut their Raman lasers off. Next Update 60 mins
Update 2 2024-01-31 12:46:56 : EXA can advise that our field engineer is on site in Zanvoort and will commence OTDR testing shortly. Further udpates will be available once testing has been performed. Next update in 60 mins
Initial Update 1 2024-01-31 11:48:59 : Please be advised we are experiencing a network outage between Leiston, UK and Zandvoort, NL. We have engaged our UK and NL Field Teams and technicians have been dispatched to investigate. The ETA for the Field Team in Zandvoort is approximately 60 minutes, and we are currently pending the ETA for the UK dispatch. We will be issuing our next update in 60 minutes.

We've identified traffic issue with the AMS-IX port located at GlobalSwitch. To ensure the stability of our network and to perform necessary diagnostic checks, we've taken the decision to temporarily disable this port.

We have reason to believe that the root cause of the issue is faulty 100GB optic, we will be scheduling a replacement for the optic later today to restore redundancy for AMS-IX

2023-12-30 02:00 - following extensive check in GSA and replacement of the optics, we have understood that our optics are not at fault and seems like this is an issue form the side of AMS-IX, we have opened a case with them and we are awaiting more information about resolution time from their side.

2023-12-31 13:30 - following a debug session with AMS-IX we have identified that the issue seems to be related to their line card acting up. when moving to the backup PE (we would use a different chassis, line card, port and optic, so it would help us check our internal domain. Please let us know if we can proceed to swap the service. The swap is performed by our photonic switches and takes less than 20 ms)
We can currently confirm that CRC errors have stopped and we will be re-enabling the traffic.
once an RFO(Reason For Outage) will arrive from AMS-IX we will provide an update.

2024-01-17 16:00 - after several tests and maintenance on AMS-IX' part, we've restored the connection fully and all peering is operational again.

Urgent Maintenance:

Maintenance Details:
Due to issues discovered in R1 DBA on 2023-12-15 which seem to have no obvious source, we have decided to take action in the form of diverting traffic away from this router until we have isolated the root cause and to ensure the stability and reliability of our network.

We have replaced multiple components in the router and aim to gradually restore it to circulation and monitor the stability as we move traffic over to check the stability of the replaced components.

Expected Impact:
As we aim to minimize risk, we will be spreading the maintenance windows over a few days, during the maintenance window DBA customers may experience high latency and potential packet loss. We aim to slowly increase traffic flow by groups of vlans and ranges. We will be closely monitoring for issues at each step of switching traffic back. We apologize for any inconvenience this may cause and appreciate your understanding as we work to enhance the reliability and performance of our network infrastructure. If you have any questions or require further assistance, please do not hesitate to contact our NOC Team

Due to a recent issue we are performing an emergency reload on router R1 in Databarn Amsterdam (DBA).

Since our datacenters are setup with 2 routers to handle situations like these, only customers having only one BGP session would be affected. Customers having a redundant BGP session with us should not experience any issues.

Maintenance Details:
Following the latest issues with AMS-IX, we have noticed some BGP issues originating in EQX-AM7 Router, to ensure redundancy, stability and reliability of our network, we will be conducting essential maintenance on our router located in EQX-AM7.

Expected Impact:
During the maintenance window, directly connected customers may experience a temporary loss of internet connectivity. This impact will primarily affect customers with a single BGP session. However, customers with redundant BGP sessions should not encounter any service disruptions.

Why This Time?
We have carefully selected this time slot to minimize the impact on our customers. It falls within our lowest network traffic period, ensuring that the maintenance is performed with the least possible disruption.

We apologize for any inconvenience this may cause and appreciate your understanding as we work to enhance the reliability and performance of our network infrastructure.

If you have any questions or require further assistance, please do not hesitate to contact our NOC Team

Due to prolonged issues at AMS-IX as of 19:05 and onward, we've turned off the peering sessions with direct peers as well as the AMS-IX route servers to prevent further packetloss.

We are monitoring the situation closely at AMS-IX Statistics and will re-enable BGP sessions over AMS-IX once the issue appears resolved.

AMS-IX Notification:
AMS-IX Outage on Amsterdam Peering Platform

We've found an issue with both optics leading to our rackswitch in Rack B-2-13, in Nedzone Steenbergen.

After replacing optics and fibers, as well as cleaning them, there are still errors occurring.

We're currently preparing a new switch to seamlessly move servers onto.

Please expect an update here once it is done.

Update 23:26 - Switch has been replaced and network traffic is restored. If you experience any issues, please contact us through Ticketing.

We've identified traffic issue with the AMS-IX port located at GlobalSwitch. To ensure the stability of our network and to perform necessary diagnostic checks, we've taken the decision to temporarily disable this port.

Our NOC team is currently investigating the root cause of the problem. We are aiming to conduct all required checks and, if deemed safe, plan to return the AMS-IX port to its operational state by tomorrow night.

We do not anticipate any capacity issues as a result of this action. Our network has been designed with redundancies in place to handle such situations, and we expect no interruption to the normal flow of data.

Update: We've troubleshooted the issue and determined a reload of the linecard could remedy this issue. The connection to AMS-IX has been restored and peering session operate nominally.

As of October 16th, 21:00 our connection to LINX LON1 has gone down.

Our fiber provider's teams in both NL and the UK are currently investigating the issue.

We'll update this NOC post with updates as they come in. We see the link is operational as of October 27th 6:35 and we'll monitor if it stays stable. Peering will resume automatically.

Current updates so far:

Update 34 2023-10-28 16:28:15 : Final update: EXA can confirm all final checks have been completed on remaining customer services. No further interruptions are expected.
Update 33 2023-10-27 12:14:36 : EXA team are seeing alarms clear and we are confirming status with the remaining customers. Please contact EXA NOC if you are still seeing an impact on your service. Final update will be sent once all services are confirmed restored.
Update 32 2023-10-27 10:08:36 : EXA teams are on our locations in UK and Netherlands and are currently working to complete the remaining patches to reconnect all fibres. Customers should start to see services recover in the next 30-60 minutes. Further update will be provided in the next 60 minute
Update 31 2023-10-27 07:52:48 : EXA has confirmed that testing has been completed and we will begin connecting fibres within the next 90 minutes. EXA will request all customers to verify connectivity has restored and will send a further update once all services have been confirmed.
Update 30 2023-10-26 19:14:03 : EXA's marine repair contractor has confirmed that the final splicing is progressing as planned. Testing of the final splice is expected to be completed on the morning of the 27th of October. EXA will then begin to contact customers to confirm status once the repairs are completed. Next Update: 27th October 09:00 GMT
Update 29 2023-10-26 09:55:11 : EXA's marine repair contractor has confirmed that the repair vessel is continuing repair operations on the final splice. Once this is completed, the cable will need to be fully tested to verify that the repair is successful and that all fibres have been repaired. Repair operations are expected to be completed in the early hours of Friday the 27th of October. Next Update: 26th October 18:00 GMT
Update 28 2023-10-25 15:00:14 : EXA's marine repair contractor has confirmed that the initial splice has been completed successfully and they are now proceeding with retrieving the UK end of the cable. The repair vessel will then commence the final splice phase of the repair operations. Repair operations are expected to be completed in the early hours of Friday the 27th of October. Further updates will be provided as the repair progresses. Next Update: 26th October 10:00 GMT
Update 27 2023-10-24 19:19:25 : EXAs marine repair contractor has advised that the repair vessel is continuing work on the initial splicing on the Netherlands side of the cable. The repair vessel expects to complete testing of the initial splice tomorrow morning. Once completed successfully, the vessel will transit to the UK side of the cable to perform splicing on that side. Next update: 25th October 18:00 GMT
Update 26 2023-10-24 10:10:47 : EXA's marine repair contractor has confirmed that initial splice operations by the repair vessel are continuing on the Netherlands side of the cable and are expected to be completed within the next 24 hours. Once this splice is completed and successfully tested, the repair vessel will then transit to commence splicing operations on the UK side of the cable. As with all submarine repairs, the progress of repair operations is dependent on suitable weather conditions and the repair vessel is closely monitoring the forecast for the coming days. Next Update: 24th October 18:00 GMT
Update 25 2023-10-23 17:32:05 : EXA's marine repair contractor has confirmed that they have successfully recovered the cable at the UK side of the fault. The cable has been buoyed off and the repair vessel has commenced working to retrieve the Netherlands side of the cable. Once the cable has been retrieved from the seabed and have the cable on board, the repair vessel team will commence preparations for the initial splice. As with all submarine repairs, the progress of repair operations is dependent on suitable weather conditions and the repair vessel is closely monitoring the forecast for tonight and the coming days. Next Update: 24th October 10:00 GMT
Update 24 2023-10-22 18:25:52 : EXAs marine repair contractor has advised that they were able to successfully launch the ROV to retrieve the Netherlands side of the submarine cable. Repair operations are progressing with good weather conditions today and good weather is forecasted for the next 24 hours. Next update: 23rd October 18:00 GMT
Update 23 2023-10-22 09:36:01 : Update 23 2023-10-22 09:35GMT: EXAs marine repair contractor has advised that weather conditions overnight were unsafe to proceed with repair operations. They have confirmed that weather conditions have improved this morning and the repair vessel is preparing to commence repair operations. Next update: 22nd October 18:00 GMT
Update 22 2023-10-21 18:35:27 : EXAs marine repair contractor has advised that the repair vessel remains on location at the cable grounds and monitoring weather forecast for suitable conditions to commence repair operations. Next Update: 22nd October 10:00 GMT
Update 21 2023-10-21 12:07:33 : EXA's marine repair contractor has advised that the vessel remains on location at the cable grounds and continuing to monitor the weather for suitable conditions to commence the repair operations. Based on the current weather forecast, the repair vessel is expecting to commence repair operations on the 22nd of October. Next Update: 21st October 18:00 GMT
Update 20 2023-10-20 17:35:04 : EXA's marine repair contractor has advised that the vessel remains on location at the cable grounds and continuing to monitor the weather for suitable conditions to commence the repair operations. Based on the current weather forecast, the repair vessel is expecting to commence repair operations on the 22nd of October. Next Update: 21st October 10:00 GMT
Update 19 2023-10-20 10:09:46 : EXA's marine repair contractor has confirmed that the repair vessel is due to arrive at the cable grounds shortly. The repair vessel will monitor weather conditions closely for a suitable window to commence the repair operations. The current weather forecast is expecting stormy conditions for the next few days and the repair vessel expects that they will be able to begin operations in the early hours of Sunday 22nd October. As with all submarine cable repairs, the estimated time to be completed is dependent on several factors, including suitable weather conditions. The estimated completion date is October the 25th . Next Update: 18:00 GMT 20th October
Update 18 2023-10-19 17:51:07 : EXA's marine repair contractor has confirmed that the repair vessel has departed from port earlier today and is now in transit to the cable grounds. The repair vessel is expected to reach the cable grounds late morning on the 20th of October. The estimated time for completion of the repair work is the 25th of October. Next Update: 20th October 10:00 GMT
Update 17 2023-10-19 10:28:44 : EXA's marine repair contractor has advised that the repair vessel's departure from port has been delayed due to storm conditions in the UK in the last 24 hours. The conditions were deemed unsafe for departure and the repair vessel remains on standby awaiting suitable conditions to leave port. As with all submarine cable repairs, the estimated time to be completed is dependent on several factors, including suitable weather conditions. The estimated completion date is early morning on October the 25th . Next Update: 18:00 GMT 19th October
Update 16 2023-10-18 20:03:28 : EXA marine repair contractor has advised that loading has been completed and the vessel will depart for the cable grounds in the coming hours. The vessel will then transit to the cable grounds to begin the repair process in the coming days. The current ETR is the 24th of October. Next update: 10:00 AM GMT 19th October
Update 15 2023-10-18 10:18:24 : EXA's marine repair contractor has advised that loading of the repair vessel was continuing this morning. The repair vessel is expected to depart from the harbour within the next 24 hours and begin transit to the cable grounds. The completion date for the repair is currently 24th of October 2023. We will continue to provide regular updates as the repair process progresses. Next Update: 18:00 GMT 18th October
Update 14 2023-10-17 18:49:37 : EXA's marine repair contractor has confirmed that the repair vessel has been mobilized. The repair vessel will load the necessary equipment on board before departing from dock and transiting towards the cable grounds. A tentative completion date of the 24th of October has been provided. As with all submarine cable repairs, the timeline for completion of the repair will be dependent on several factors including weather and sea conditions and may be subject to change. More updates to follow. Next Update: 10:00 AM GMT 18th October
Update 13 2023-10-17 10:16:53 : OTDR testing has been completed and we can confirm that the fault has been identified on our submarine cable approximately 121km from Netherlands towards the UK. This is a Force Majeure event and EXA have engaged with our marine repair contractor to mobilise a repair vessel. As with all submarine cable repairs, the timeline for completion will be dependent on several factors including weather and sea conditions. At this time an ETR cannot be provided. Next Update: 18:00 GMT 17th October
Update 12 2023-10-17 08:56:10 : Please be advised that our team is continuing with further testing and our senior engineers will review the results once obtained. The incident is escalated to the highest level within the senior leadership and are pending further feedback. More updates to follow. Next update approximately 60 minutes (Or earlier pending feedback)
Update 11 2023-10-17 07:15:15 : Engineers after OTDR the fiber have identified that no fibers are free and the spares are also affected by the outage. Unfortunately rolling of fibers is unavailable. Our team is working closely with field engineers at this time and will advise on further information from the field at the earliest. In the interim our senior technical team is exploring all possibilities to expedite the restoration and will advise on further updates as soon as possible. Please note this is escalated to the highest level in EXA and our team will continue to follow up with the teams for further information. There is no ETR at this time. More updates are to follow. Next update approximately 60 - 120 minutes (Or earlier)
Update 10 2023-10-17 05:42:17 : Dear customers, EXA has received the OTDR test results and is currently analysing these results to form a POA. An ETR is unknown at present. More updates will follow.
Update 9 2023-10-17 04:19:03 : Dear customers, Our field team are on site in Zandvoort and will begin to test the fibres shortly. We should receive these results shortly for analysis. More updates to follow
Update 8 2023-10-17 03:04:10 : Dear customers, Our technician in Leiston has performed the necessary OTDR. We are waiting for our tech in Zandvoort to perform his OTDR upon arrival at the site. EXA require both tests to isolate the fault across the backbone. More updates to follow.
Update 7 2023-10-17 02:02:04 : Dear customers, Our field tech in Leiston is currently performing the OTDR test on the affected fibres which are showing hard down. We have been given an ETA of 3 hours from our field team in the Netherlands to perform an additional OTDR test from the site in Zandvoort. We will keep you posted with further updates.
Update 6 2023-10-17 00:59:09 : Dear customers, Please be advised that our field tech has arrived on site in Leiston and will begin to OTDR the fibres. More updates to follow.
Update 5 2023-10-17 00:05:45 : Dear customers, Our field team are due on site in the UK shortly based on the prior ETA given to us. We are following up with the field team to confirm the ETA before OTDR testing commences. More updates to follow.
Update 4 2023-10-16 23:01:53 : Dear customers, As advised prior the ETA provided to us by our field team remains at an hour away. We will keep you posted with further updates regarding this fault. More updates to follow.
Update 3 2023-10-16 22:02:50 : Dear customers, As advised in the prior update, we have engaged the relevant field teams to investigate this outage on either side in the UK and Netherlands. We have been given an ETA of 2 hours from our UK counterparts to perform an OTDR test. More updates will follow.
Update 2 2023-10-16 21:08:57 : Dear customers, We're dispatching our field teams on the site to further investigate the issue and waiting for the ETA More updates to follow
Initial Update 1 2023-10-16 19:43:50 : Dear customers, Please be advised we are experiencing an outage between Leiston, UK to Zandvoort, Netherlands. We have contacted the relevant teams to investigate the ongoing fault. More updates to follow.

Because of soft errors in the memory of one of our linecards we will be replacing a linecard in our NZS R2 router on the 28th of October at 11:00 PM CET.

Update: 2023-10-28 22:30: Final update

The Linecard has been replaced successfully and working without any issues.

We have experienced no downtime, this is due to our already redundant network.

As of 18:56 Cloud VPS services have gone down.

Our engineers are currently investigating the issue.

Our sincere apologies for this inconvenience.

When we have more information on the exact issue we'll update the NOC post here.

Update 20:09:
We've found a DDoS that was targeting an IP and blocked it. Traffic is recovered now and situation looks stable. We'll keep monitoring for now.

Update 20:52:
We confirmed the problem to be resolved as of initial fix at 20:00.

Dear Valued Customer,

We regret to inform you that our Anti-DDoS services are currently suspended due to an unexpected hardware failure in our Distributed Denial of Service (DDoS) mitigation equipment. Unfortunately, the issue could not be resolved immediately, but we are actively collaborating with the vendor to identify a prompt solution. In the meantime, we have already initiated a migration plan to adopt a new solution, and we will provide further updates later this month.

Rest assured, our Network Operations Center (NOC) team is fully dedicated to supporting you during this period. They will be available to assist with special cases, such as implementing custom Access Control List (ACL) rules or finding quick resolutions for any DDoS incidents you might encounter.

We apologize for any inconvenience this may cause and appreciate your understanding and patience during this situation.

**This message pertains to customers who have an active DDoS protection plan**

Darkfiber provider ExaInfra which connects Databarn Amsterdam (DBA) towards Equinix AM7 (EQX-AM7) is having a planned maintenance. The connectivity between DBA and EQX-AM7 will automatically reroute over our GSA darkfiber during this maintenance.

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Equinix-AM7 is having a planned maintenance.

The connectivity between NZS and EQX-AM7 will automatically reroute over our ring.

Customers having wave service will be down during this time.

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Equinix-AM7 is having a planned maintenance.

The connectivity between NZS and EQX-AM7 will automatically reroute over our ring.

Customers having wave service will be down during this time.

Dear Valued Customers,

We wanted to update you on the removal of external access to our ticketing system.
As we have noticed that usage with our ticketing system is mostly operated via emails and with the following considerations in mind:

- Outdated Code-base: The system relies on an outdated code base that is no longer considered trustworthy.

- Unsupported Operating System: The ticketing system requires an outdated operating system that is no longer supported, leaving it vulnerable to security risks.

- Integration with Customer Portal: Our goal is to replace the ticketing system with an integrated view system for tickets within the customer portal. This streamlined approach will enhance user experience and improve ticket management efficiency.

For the reasons mentioned above we have decided to take immediate action to remove all access to the system.

History Access - While we are working to integrate a view system for tickets in our customer portal https://ssc.nforce.com if you have a specific request you may direct your request to our administration department and we may permit temporary access to view history.

Darkfiber provider EuNetworks which connects Databarn Capelle (DBC) towards Globalswitch Amsterdam (GSA) is having a planned maintenance.

The connectivity between DBC and GSA will automatically reroute over our ring.

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Equinix-AM7 is having a planned maintenance.

The connectivity between NZS and EQX-AM7 will automatically reroute over our ring.

Customers having wave service will be down during this time.

In finishing up our migrations we still need to migrate the IP addresses currently originating in DBC which means there will be an interruption in your traffic, we will do our best to keep this as short as possible which should not last longer then 30 minutes per IP block, in most case this will be much shorter.

If you currently have an IP in any of the following IP ranges, this will have an effect on your service.

Do keep in mind that even though the maintenance lists it for a week this will not mean you will have a weeks worth of downtime, it means that in this week at a moment you will have some downtime.

46.166.184.0/24
46.166.188.0/24
46.166.189.0/24
46.166.191.0/24
46.166.136.0/24
46.166.137.0/24
46.166.138.0/24
46.166.140.0/24
46.166.143.0/24
46.166.146.0/24
46.166.147.0/24
46.166.150.0/24

We will update this post once ranges have been moved.

In finishing up our migrations we still need to migrate the IP addresses currently originating in DBC which means there will be an interruption in your traffic, we will do our best to keep this as short as possible.

If you currently have an IP in any of the following IP ranges, this will have an effect on your service.

46.166.139.0/24
46.166.142.0/24
46.166.148.0/24
46.166.151.0/24
46.166.187.0/24

We will update this post once ranges have been moved.

Update 12:44 CET:
All ranges have been moved to the correct routers now.

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Equinix-AM7 is having a planned maintenance.

The connectivity between NZS and EQX-AM7 will automatically reroute over our ring.

Customers having wave service will be down during this time.

Darkfiber provider Relined which connects GlobalSwitch Amsterdam (GSA) towards Nikhef is having a planned maintenance.

Their reason for this maintenance:

"Relined performs maintenance work to maintain the availability and quality of its fiber optic network."

Their references is: PA-948002.

Darkfiber provider Relined which connects GlobalSwitch Amsterdam (GSA) towards Equinix AM5 (EQX-AM5) is having a planned maintenance.

Their reason for this maintenance:

"Relined performs maintenance work to maintain the availability and quality of its fiber optic network."

Their references is: PA-948002.

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Equinix-AM7 is having a planned maintenance.

The connectivity between NZS and EQX-AM7 will automatically reroute over our ring.

Customers having wave service will be down during this time.

Update 2023-01-31:
Due to unforeseen circumstances this will be postponed, once the new date has been known we will update this post.

Update 2023-02-03:
This maintenance has been rescheduled to 7 February

Update 2023-02-07:
During the migration of the links, one of the channels kept flapping, after a debugging session it was rebuild back to the old situation.

On the 15th of February we'll debug this further including separate cards and other channels.

Currently we have a switch outage in BNR (Bytesnet Rotterdam), we are sending out an engineer to investigate the situation.

Update 09:12:
SWe were notified by the datacenter that this was a power issue on one of the feeds in the datacenter which has now been solved.

Currently our darkfiber between Nedzone Steenbergen (NZS) and Equinix Amsterdam 7 (EQX-AM7) is down.

We've sent in a ticket with our darkfiber provider for more information, they have it currently under investigation.

All traffic for EQX-AM7 is re-routed over our darkfiber with GSA.

Direct wave customers will experience an outage for the duration of this issue.

Update 1:14:
Currently the connection is back online. We haven't had an RFO from our dark fiber provider yet and will monitor the situation.

We noticed one of the power distribution units (PDU) namely PDU2 not functioning as it should anymore. Therefore under emergency maintenance this PDU will be replaced, customers located on this PDU will have their servers rebooted.

As the switch is also located on this PDU customers in this rack will notice a short downtime in network.

Customer located in this rack will be informed by e-mail.

We are aware that currently customer having a connection over Levelfour have no connection to our network.

We are currently investigating into this matter.

Update 13.00:
Levelfour has debugged the issue and provided a fix for us, all connections are currently working again.

Due to BGP issues we are going to do maintenance of our router in EQX-AM7.

During the maintenance, Internet Customers will experience a loss of internet as well as customers having only one BGP session. Customers having a redundant BGP session with us should not experience any issues.

We've chosen this time to minimally impact our customers and be during our time of lowest network traffic.

We noticed one of the power distribution units (PDU) namely PDU1 not functioning as it should anymore. Therefore under emergency maintenance this PDU will be replaced, customers located on this PDU will have their servers rebooted.

As the switch is also located on this PDU customers in this rack will notice a short downtime in network.