Status

[#1021] LINX LON1 connection down

Posted: 2024-01-31 15:51

Start: 2024-01-31 11:30:00
End : 2024-02-16 04:00:00

Affects: LINX peering traffic

As of January 31st, 11:30 our connection to LINX LON1 has gone down.

Our fiber provider's teams in both NL and the UK are currently investigating the issue.

We'll update this NOC post with updates as they come in. The connection at our Globalswitch Amsterdam (GSA) location is restored as of February 2nd, 12:00 and peering has resumed.
The connection at Equinix AM7 (EQX-AM7) is restored as of February 16th, 4:00 and peering has resumed.

Current updates so far:
Update 35 2024-02-16 16:23:55 : FINAL UPDATE: EXA can confirm that the cable repair has been completed successfully and no further interruptions are expected. All customer services should now be restored, if you are experiencing any issues with yur service please contact EXA NOC.
Update 34 2024-02-16 11:44:34 : Please be advised that EXA can confirm that all repair work has been completed successfully. At this time, EXA are working with some customers to investigate their individual services which are still seeing an issue. A final notification will be sent once all investigations have been fully completed.
Update 33 2024-02-16 08:37:01 : Please be advised that we have seen our spans recover and light levels have restored, however, we see a lot of our waves are still recovering so you may still be experiencing outage, Please be advised our Engineering team are working through all these waves restoring services at this time. Next update in 60 mins.
Update 32 2024-02-16 05:03:54 : Dear Customers, EXA teams are seeing alarms clearing, and we are confirming the status with the remaining customers. Please contact EXA NOC if you are still seeing an impact on your service. The final update will be sent once all services are confirmed and restored. Next update is February 16th 09:00 GMT.
Update 31 2024-02-16 02:52:55 : Dear customers, Please be advised that EXA's field teams continue to connect the remaining customer fibres. We are seeing alarms beginning to clear but works continue, as some services are yet to recover. Next update February 16th 05:00 GMT.
Update 30 2024-02-16 00:58:03 : Dear customers, Be advised that final testing has been completed. At this time, EXA field teams are scoping and connecting the fibres on both terminal sites. Exa NOC will contact customers with dark fibre services to verify their status once fibres are reconnected. Next update February 16th 03:00 GMT.
Update 29 2024-02-15 22:54:54 : Dear customers, Be advised that final testing of the cable is continuing at this time. Once this has been completed successfully, EXA field teams will begin reconnecting all fibres on both terminal sites. At that time, EXA NOC will ask customers with dark fibre services to restore their lasers and verify their status. Please ensure that lasers are not turned back on until EXA confirms that fibres have been reconnected on-site. Next Update February 16th 01:00 GMT
Update 28 2024-02-15 17:19:18 : Please be advised that the final splice phase has been completed successfully. The cable repair vessel is now preparing to deploy the cable to the seabed. EXA field teams are currently on site at both UK and Netherlands terminal sites awaiting confirmation of deployment and will commence final testing of all fibers in the coming hours. Next Update February 15th 23:00 GMT
Update 27 2024-02-15 09:46:39 : Please be advised that the cable repair vessel replaced the faulty joint and the final splice phase has progressed well overnight. Post splice testing is ongoing at this time. EXA field teams are on site in both UK and Netherlands terminal stations preparing for final testing which expected to begin later today. Next Update February 15th 18:00 GMT
Update 26 2024-02-14 17:59:09 : Please be advised that during the final splice testing phase a fault was discovered with the joint of the final splice which needs to be removed and a new joint created. The cable repair vessel is working diligently to resolve this and expects to replace the joint and resplice the fibres over the next 24-30 hours. Next Update February 15th, 10:00 GMT
Update 25 2024-02-14 13:24:55 : Please be advised that the post splice testing is continuing at this time. EXA field teams are on site in both UK and Netherlands terminal stations preparing for final testing which expected to begin later today. Next Update February 14th, 18:00 GMT
Update 24 2024-02-14 08:20:58 : Please be advised that the final splice phase is progressing well and post splice testing is ongoing. Once final splice phase is completed, all fibres will undergo final testing from the terminal stations in UK and Netherlands. Next update February 14th 12:00 GMT
Update 23 2024-02-13 17:59:03 : Dear customers, Please be advised that the final splice phase is progressing well. The cable repair vessel expects to begin testing the fibres towards the UK terminal station later tonight. Next Update February 14th 08:00 GMT
Update 22 2024-02-13 10:15:39 : Please be advised that the intial splice phase was completed successfully overnight. The cable repair vessel has begun preparing the cable for the final splice phase. At present we expect the cable to be deployed and system normalized early morning on February 14th. Next Update February 13th 18:00 GMT
Update 21 2024-02-12 18:08:45 : Initial Splice phase operations are ongoing and expected to be completed later this evening. Upon completion of this phase, the cable repair vessel will retrieve the UK cable end for final splice operations. At present we expect the cable to be deployed and system normalized late evening on February 13th. Next Update February 13th 10:00 GMT
Update 20 2024-02-12 10:01:36 : Repair operations are progressing well over the last 24 hours. Expected to complete initial splice phase later today, 12th February. Upon completion of this phase, the cable repair vessel will retrieve the UK cable end for final splice operations. At present we expect the cable to be deployed and system normalized late evening on February 13th. Next Update February 12th 18:00 GMT
Update 19 2024-02-11 20:36:08 : Please be advised that the cable was successfully recovered and tested to the Netherlands terminal station. The initial splice phase is continuing at this time. Next update February 12th 10:00 GMT
Update 18 2024-02-11 10:09:33 : Please be advised that repair operations are progressing well with recovery of cable ongoing and initial splice phase to commence later today. Next update February 11th 20:00 GMT
Update 17 2024-02-10 21:07:52 : Please be advised that the cable repair vessel is continuing its operation to recover and test the cable to the Netherlands terminal station. Next update February 11th 10:00 GMT
Update 16 2024-02-10 10:07:13 : The cable repair vessel has confirmed that operations recommenced this morning following weather downtime. The vessel will continue operations to recover and test the Netherlands end of the cable today. Next update February 10th 20:00 GMT
Update 15 2024-02-09 10:01:14 : Please be advised that the cable repair vessel completed testing into the UK terminal station and has buoyed off the cable. At this time, the vessel is currently on weather standby due to large swells and strong winds in the area. At the next opportunity the vessel plans to recover and test the Netherlands end of the cable, and prepare for starting the initial splice. Next Update February 10th at 10:00 GMT.
Update 14 2024-02-08 09:55:21 : Please be advised that repair operations are progressing well. The cable repair vessel recovered the UK cable end overnight and is currently testing towards the UK terminal station. Next update: 9th Feb 10:00GMT.
Update 13 2024-02-07 18:14:25 : EXA's marine contractor has advised that the ROV has located the cable fault and the repair vessel is now working to retrieve the cable from the seabed. Once the cable is on board the vessel, the cable will be tested before being buoyed off. The repair vessel will then begin transit to recover the NL side of the cable. Next Update February 8th 10:00 GMT
Update 12 2024-02-07 09:41:38 : EXA's marine contractor has advised that they have successfully deployed the ROV overnight and work is ongoing to recover and test the UK end of the cable. Next update February 7th 18:00 GMT
Update 11 2024-02-06 18:04:25 : As per the latest update from our marine contractor, the repair vessel remains on weather standby. They have been closely monitoring the weather forecast and expect to be able to launch the ROV later this evening, to retrieve the cable from the seabed. Next update February 7th 10:00 GMT
Update 10 2024-02-06 09:51:56 : Please be advised that the cable repair vessel arrived on cable grounds yesterday evening however is currently on weather standby due to strong winds and large swells in the North sea. The cable repair vessel expects the earliest opportunity to launch the ROV to locate and retrieve the cable system will be later today, Feb 6th. We will continue to closely monitor the weather forecast and recommence operations at the earliest opportunity. Next Update February 6th at 18:00 GMT
Update 9 2024-02-05 09:42:06 : Dear customers, EXAs marine repair contractor has advised that the repair vessel is in transit and is due to arrive on the cable grounds today. Once on location at the cable grounds, the vessel will commence operations to recover cable from the seabed and test towards the NL side of the cable. Next update Feb 6th 10:00 GMT
Update 8 2024-02-04 10:28:59 : Exas marine repair contractor has advised that the repair vessel will complete loading of spares today and then depart from port once completed. The vessel is expected to arrive at the cable grounds on the morning of the 5th of February to commence repair operations. Next update Feb 5th at 10:00 GMT
Update 7 2024-02-03 09:42:34 : EXAs marine repair contractor has advised that the repair vessel will arrive in port today to begin loading of spares and equipment. Once this is complete it will then begin transit towards the cable grounds to commence repair operations. Next update Feb 4th 10am
Update 6 2024-02-02 10:29:28 : Please be advised that the repair vessel will arrive in port later today and begin loading necessary spares in preparation for the repair operations. The vessel is expected to depart for the cable grounds on Feb 3rd. EXA NOC will continue to share daily updates throughout to progress of repair operations. Next Update: Feb 3rd at 10:00GMT.
Update 5 2024-02-01 10:03:48 : EXA can confirm that our marine repair contractor has been engaged and that a repair vessel is being mobilised to support the upcoming repair works. As with all submarine cable repairs, it is difficult to provide an ETR and the repair timeline will be dependent on several factors including sea and weather conditions. Our next update will be provided on the 2nd of February at 10:00 GMT.
Update 4 2024-01-31 15:59:48 : EXA can confirm that OTDR testing has identified a break on our submarine cable between UK and Netherlands. EXA has engaged with our marine repair contractor to support repair operations. EXA requests that all customers with Dark Fibre services to please ensure your lasers are shut off and send confirmation to the EXA NOC that this has been done. As with all submarine cable repairs, the timeline for completion of the repair will be dependent on several factors including weather and sea conditions. Next Update will be provided Thursday the 1st February at 10:00 GMT
Update 3 2024-01-31 14:32:21 : EXA can confirm that OTDR testing verified that there is a fibre break on our submarine cable between UK and Netherlands. Technicians are continuing testing all fibres to confirm all fibres have been impacted. EXA requests that all customers with Dark Fibre services please ensure they have shut their Raman lasers off. Next Update 60 mins
Update 2 2024-01-31 12:46:56 : EXA can advise that our field engineer is on site in Zanvoort and will commence OTDR testing shortly. Further udpates will be available once testing has been performed. Next update in 60 mins
Initial Update 1 2024-01-31 11:48:59 : Please be advised we are experiencing a network outage between Leiston, UK and Zandvoort, NL. We have engaged our UK and NL Field Teams and technicians have been dispatched to investigate. The ETA for the Field Team in Zandvoort is approximately 60 minutes, and we are currently pending the ETA for the UK dispatch. We will be issuing our next update in 60 minutes.

[#1020] AMS-IX Port Malfunction at GlobalSwitch

Posted: 2023-12-29 14:22

Start: 2023-12-29 13:40:00
End : 2024-01-17 16:00:00

Affects: AMS-IX Reduncancy

We've identified traffic issue with the AMS-IX port located at GlobalSwitch. To ensure the stability of our network and to perform necessary diagnostic checks, we've taken the decision to temporarily disable this port.

We have reason to believe that the root cause of the issue is faulty 100GB optic, we will be scheduling a replacement for the optic later today to restore redundancy for AMS-IX

2023-12-30 02:00 - following extensive check in GSA and replacement of the optics, we have understood that our optics are not at fault and seems like this is an issue form the side of AMS-IX, we have opened a case with them and we are awaiting more information about resolution time from their side.

2023-12-31 13:30 - following a debug session with AMS-IX we have identified that the issue seems to be related to their line card acting up. when moving to the backup PE (we would use a different chassis, line card, port and optic, so it would help us check our internal domain. Please let us know if we can proceed to swap the service. The swap is performed by our photonic switches and takes less than 20 ms)
We can currently confirm that CRC errors have stopped and we will be re-enabling the traffic.
once an RFO(Reason For Outage) will arrive from AMS-IX we will provide an update.

2024-01-17 16:00 - after several tests and maintenance on AMS-IX' part, we've restored the connection fully and all peering is operational again.

[#1018] Emergency reload R1 DBA

Posted: 2023-12-15 11:11

Start: 2023-12-15 10:00:00
End : 2023-12-15 12:00:00

Affects: Customer BGP DBA, Traffic DBA

Due to a recent issue we are performing an emergency reload on router R1 in Databarn Amsterdam (DBA).

Since our datacenters are setup with 2 routers to handle situations like these, only customers having only one BGP session would be affected. Customers having a redundant BGP session with us should not experience any issues.

[#1016] AMS-IX Issues

Posted: 2023-11-22 22:02

Start: 2023-11-22 21:50:00
End :

Affects: AMS-IX Traffic

Due to prolonged issues at AMS-IX as of 19:05 and onward, we've turned off the peering sessions with direct peers as well as the AMS-IX route servers to prevent further packetloss.

We are monitoring the situation closely at AMS-IX Statistics and will re-enable BGP sessions over AMS-IX once the issue appears resolved.

AMS-IX Notification:
AMS-IX Outage on Amsterdam Peering Platform

[#1015] Emergency Switch Replacement

Posted: 2023-10-29 22:17

Start: 2023-10-29 22:00:00
End : 2023-10-29 23:20:00

Affects: Servers Rack B-2-13 NZS

We've found an issue with both optics leading to our rackswitch in Rack B-2-13, in Nedzone Steenbergen.

After replacing optics and fibers, as well as cleaning them, there are still errors occurring.

We're currently preparing a new switch to seamlessly move servers onto.

Please expect an update here once it is done.

Update 23:26 - Switch has been replaced and network traffic is restored. If you experience any issues, please contact us through Ticketing.