Due to prolonged issues at AMS-IX as of 19:05 and onward, we've turned off the peering sessions with direct peers as well as the AMS-IX route servers to prevent further packetloss.

We are monitoring the situation closely at AMS-IX Statistics and will re-enable BGP sessions over AMS-IX once the issue appears resolved.

AMS-IX Notification:
AMS-IX Outage on Amsterdam Peering Platform

We've found an issue with both optics leading to our rackswitch in Rack B-2-13, in Nedzone Steenbergen.

After replacing optics and fibers, as well as cleaning them, there are still errors occurring.

We're currently preparing a new switch to seamlessly move servers onto.

Please expect an update here once it is done.

Update 23:26 - Switch has been replaced and network traffic is restored. If you experience any issues, please contact us through Ticketing.

We've identified traffic issue with the AMS-IX port located at GlobalSwitch. To ensure the stability of our network and to perform necessary diagnostic checks, we've taken the decision to temporarily disable this port.

Our NOC team is currently investigating the root cause of the problem. We are aiming to conduct all required checks and, if deemed safe, plan to return the AMS-IX port to its operational state by tomorrow night.

We do not anticipate any capacity issues as a result of this action. Our network has been designed with redundancies in place to handle such situations, and we expect no interruption to the normal flow of data.

Update: We've troubleshooted the issue and determined a reload of the linecard could remedy this issue. The connection to AMS-IX has been restored and peering session operate nominally.

As of October 16th, 21:00 our connection to LINX LON1 has gone down.

Our fiber provider's teams in both NL and the UK are currently investigating the issue.

We'll update this NOC post with updates as they come in. We see the link is operational as of October 27th 6:35 and we'll monitor if it stays stable. Peering will resume automatically.

Current updates so far:

Update 34 2023-10-28 16:28:15 : Final update: EXA can confirm all final checks have been completed on remaining customer services. No further interruptions are expected.
Update 33 2023-10-27 12:14:36 : EXA team are seeing alarms clear and we are confirming status with the remaining customers. Please contact EXA NOC if you are still seeing an impact on your service. Final update will be sent once all services are confirmed restored.
Update 32 2023-10-27 10:08:36 : EXA teams are on our locations in UK and Netherlands and are currently working to complete the remaining patches to reconnect all fibres. Customers should start to see services recover in the next 30-60 minutes. Further update will be provided in the next 60 minute
Update 31 2023-10-27 07:52:48 : EXA has confirmed that testing has been completed and we will begin connecting fibres within the next 90 minutes. EXA will request all customers to verify connectivity has restored and will send a further update once all services have been confirmed.
Update 30 2023-10-26 19:14:03 : EXA's marine repair contractor has confirmed that the final splicing is progressing as planned. Testing of the final splice is expected to be completed on the morning of the 27th of October. EXA will then begin to contact customers to confirm status once the repairs are completed. Next Update: 27th October 09:00 GMT
Update 29 2023-10-26 09:55:11 : EXA's marine repair contractor has confirmed that the repair vessel is continuing repair operations on the final splice. Once this is completed, the cable will need to be fully tested to verify that the repair is successful and that all fibres have been repaired. Repair operations are expected to be completed in the early hours of Friday the 27th of October. Next Update: 26th October 18:00 GMT
Update 28 2023-10-25 15:00:14 : EXA's marine repair contractor has confirmed that the initial splice has been completed successfully and they are now proceeding with retrieving the UK end of the cable. The repair vessel will then commence the final splice phase of the repair operations. Repair operations are expected to be completed in the early hours of Friday the 27th of October. Further updates will be provided as the repair progresses. Next Update: 26th October 10:00 GMT
Update 27 2023-10-24 19:19:25 : EXAs marine repair contractor has advised that the repair vessel is continuing work on the initial splicing on the Netherlands side of the cable. The repair vessel expects to complete testing of the initial splice tomorrow morning. Once completed successfully, the vessel will transit to the UK side of the cable to perform splicing on that side. Next update: 25th October 18:00 GMT
Update 26 2023-10-24 10:10:47 : EXA's marine repair contractor has confirmed that initial splice operations by the repair vessel are continuing on the Netherlands side of the cable and are expected to be completed within the next 24 hours. Once this splice is completed and successfully tested, the repair vessel will then transit to commence splicing operations on the UK side of the cable. As with all submarine repairs, the progress of repair operations is dependent on suitable weather conditions and the repair vessel is closely monitoring the forecast for the coming days. Next Update: 24th October 18:00 GMT
Update 25 2023-10-23 17:32:05 : EXA's marine repair contractor has confirmed that they have successfully recovered the cable at the UK side of the fault. The cable has been buoyed off and the repair vessel has commenced working to retrieve the Netherlands side of the cable. Once the cable has been retrieved from the seabed and have the cable on board, the repair vessel team will commence preparations for the initial splice. As with all submarine repairs, the progress of repair operations is dependent on suitable weather conditions and the repair vessel is closely monitoring the forecast for tonight and the coming days. Next Update: 24th October 10:00 GMT
Update 24 2023-10-22 18:25:52 : EXAs marine repair contractor has advised that they were able to successfully launch the ROV to retrieve the Netherlands side of the submarine cable. Repair operations are progressing with good weather conditions today and good weather is forecasted for the next 24 hours. Next update: 23rd October 18:00 GMT
Update 23 2023-10-22 09:36:01 : Update 23 2023-10-22 09:35GMT: EXAs marine repair contractor has advised that weather conditions overnight were unsafe to proceed with repair operations. They have confirmed that weather conditions have improved this morning and the repair vessel is preparing to commence repair operations. Next update: 22nd October 18:00 GMT
Update 22 2023-10-21 18:35:27 : EXAs marine repair contractor has advised that the repair vessel remains on location at the cable grounds and monitoring weather forecast for suitable conditions to commence repair operations. Next Update: 22nd October 10:00 GMT
Update 21 2023-10-21 12:07:33 : EXA's marine repair contractor has advised that the vessel remains on location at the cable grounds and continuing to monitor the weather for suitable conditions to commence the repair operations. Based on the current weather forecast, the repair vessel is expecting to commence repair operations on the 22nd of October. Next Update: 21st October 18:00 GMT
Update 20 2023-10-20 17:35:04 : EXA's marine repair contractor has advised that the vessel remains on location at the cable grounds and continuing to monitor the weather for suitable conditions to commence the repair operations. Based on the current weather forecast, the repair vessel is expecting to commence repair operations on the 22nd of October. Next Update: 21st October 10:00 GMT
Update 19 2023-10-20 10:09:46 : EXA's marine repair contractor has confirmed that the repair vessel is due to arrive at the cable grounds shortly. The repair vessel will monitor weather conditions closely for a suitable window to commence the repair operations. The current weather forecast is expecting stormy conditions for the next few days and the repair vessel expects that they will be able to begin operations in the early hours of Sunday 22nd October. As with all submarine cable repairs, the estimated time to be completed is dependent on several factors, including suitable weather conditions. The estimated completion date is October the 25th . Next Update: 18:00 GMT 20th October
Update 18 2023-10-19 17:51:07 : EXA's marine repair contractor has confirmed that the repair vessel has departed from port earlier today and is now in transit to the cable grounds. The repair vessel is expected to reach the cable grounds late morning on the 20th of October. The estimated time for completion of the repair work is the 25th of October. Next Update: 20th October 10:00 GMT
Update 17 2023-10-19 10:28:44 : EXA's marine repair contractor has advised that the repair vessel's departure from port has been delayed due to storm conditions in the UK in the last 24 hours. The conditions were deemed unsafe for departure and the repair vessel remains on standby awaiting suitable conditions to leave port. As with all submarine cable repairs, the estimated time to be completed is dependent on several factors, including suitable weather conditions. The estimated completion date is early morning on October the 25th . Next Update: 18:00 GMT 19th October
Update 16 2023-10-18 20:03:28 : EXA marine repair contractor has advised that loading has been completed and the vessel will depart for the cable grounds in the coming hours. The vessel will then transit to the cable grounds to begin the repair process in the coming days. The current ETR is the 24th of October. Next update: 10:00 AM GMT 19th October
Update 15 2023-10-18 10:18:24 : EXA's marine repair contractor has advised that loading of the repair vessel was continuing this morning. The repair vessel is expected to depart from the harbour within the next 24 hours and begin transit to the cable grounds. The completion date for the repair is currently 24th of October 2023. We will continue to provide regular updates as the repair process progresses. Next Update: 18:00 GMT 18th October
Update 14 2023-10-17 18:49:37 : EXA's marine repair contractor has confirmed that the repair vessel has been mobilized. The repair vessel will load the necessary equipment on board before departing from dock and transiting towards the cable grounds. A tentative completion date of the 24th of October has been provided. As with all submarine cable repairs, the timeline for completion of the repair will be dependent on several factors including weather and sea conditions and may be subject to change. More updates to follow. Next Update: 10:00 AM GMT 18th October
Update 13 2023-10-17 10:16:53 : OTDR testing has been completed and we can confirm that the fault has been identified on our submarine cable approximately 121km from Netherlands towards the UK. This is a Force Majeure event and EXA have engaged with our marine repair contractor to mobilise a repair vessel. As with all submarine cable repairs, the timeline for completion will be dependent on several factors including weather and sea conditions. At this time an ETR cannot be provided. Next Update: 18:00 GMT 17th October
Update 12 2023-10-17 08:56:10 : Please be advised that our team is continuing with further testing and our senior engineers will review the results once obtained. The incident is escalated to the highest level within the senior leadership and are pending further feedback. More updates to follow. Next update approximately 60 minutes (Or earlier pending feedback)
Update 11 2023-10-17 07:15:15 : Engineers after OTDR the fiber have identified that no fibers are free and the spares are also affected by the outage. Unfortunately rolling of fibers is unavailable. Our team is working closely with field engineers at this time and will advise on further information from the field at the earliest. In the interim our senior technical team is exploring all possibilities to expedite the restoration and will advise on further updates as soon as possible. Please note this is escalated to the highest level in EXA and our team will continue to follow up with the teams for further information. There is no ETR at this time. More updates are to follow. Next update approximately 60 - 120 minutes (Or earlier)
Update 10 2023-10-17 05:42:17 : Dear customers, EXA has received the OTDR test results and is currently analysing these results to form a POA. An ETR is unknown at present. More updates will follow.
Update 9 2023-10-17 04:19:03 : Dear customers, Our field team are on site in Zandvoort and will begin to test the fibres shortly. We should receive these results shortly for analysis. More updates to follow
Update 8 2023-10-17 03:04:10 : Dear customers, Our technician in Leiston has performed the necessary OTDR. We are waiting for our tech in Zandvoort to perform his OTDR upon arrival at the site. EXA require both tests to isolate the fault across the backbone. More updates to follow.
Update 7 2023-10-17 02:02:04 : Dear customers, Our field tech in Leiston is currently performing the OTDR test on the affected fibres which are showing hard down. We have been given an ETA of 3 hours from our field team in the Netherlands to perform an additional OTDR test from the site in Zandvoort. We will keep you posted with further updates.
Update 6 2023-10-17 00:59:09 : Dear customers, Please be advised that our field tech has arrived on site in Leiston and will begin to OTDR the fibres. More updates to follow.
Update 5 2023-10-17 00:05:45 : Dear customers, Our field team are due on site in the UK shortly based on the prior ETA given to us. We are following up with the field team to confirm the ETA before OTDR testing commences. More updates to follow.
Update 4 2023-10-16 23:01:53 : Dear customers, As advised prior the ETA provided to us by our field team remains at an hour away. We will keep you posted with further updates regarding this fault. More updates to follow.
Update 3 2023-10-16 22:02:50 : Dear customers, As advised in the prior update, we have engaged the relevant field teams to investigate this outage on either side in the UK and Netherlands. We have been given an ETA of 2 hours from our UK counterparts to perform an OTDR test. More updates will follow.
Update 2 2023-10-16 21:08:57 : Dear customers, We're dispatching our field teams on the site to further investigate the issue and waiting for the ETA More updates to follow
Initial Update 1 2023-10-16 19:43:50 : Dear customers, Please be advised we are experiencing an outage between Leiston, UK to Zandvoort, Netherlands. We have contacted the relevant teams to investigate the ongoing fault. More updates to follow.

As of 18:56 Cloud VPS services have gone down.

Our engineers are currently investigating the issue.

Our sincere apologies for this inconvenience.

When we have more information on the exact issue we'll update the NOC post here.

Update 20:09:
We've found a DDoS that was targeting an IP and blocked it. Traffic is recovered now and situation looks stable. We'll keep monitoring for now.

Update 20:52:
We confirmed the problem to be resolved as of initial fix at 20:00.