[#911] Outage cooling systems DBC

Posted: 2020-05-12 03:27

Start: 2020-05-12 02:45:00
End : 2020-05-12 05:50:00

Affects: Services DBC

We are experiencing since 02:45 an outage of the cooling systems in Databarn Capelle (DBC).

Currently, engineers of the datacenter are resolving the issue.

Systems may affect shutdown due to overheating, few of our systems are currently unavailable, such as ticketing.

Update 05:55:
Currently all issues should be solved, we will keep monitoring the situation for any changes.

[#874] Outage connection towards Londen Internet Exchange

Posted: 2020-01-06 07:22

Start: 2020-01-06 05:06:00
End : 2020-01-13 02:17:00

Affects: Traffic LINX via Global Switch Amsterdam

Our wave service provider GTT reported that there is a possible fiber cut in the sea, disrupting our services between Global Switch Amsterdam and LINX.

All traffic towards LINX is now automatically rerouted over EQX-AM7.

Their reference: 4817514

Their current updates:
09-01-2020 at 3:30:
The damaged cable was found.
Repair works will start during good weather conditions.
Tentative ETR is 14th January.
Next update once every 24 hours or if any new changes appear.

13-01-2020 at 4:30:
All traffic has been restored and no further interruptions are expected.
If you continue to experience any issues with your dark fiber service, please check your RAMAN amplifiers first and in case no issue observed, feel free to contact our Support team on the mentioned phone numbers at the bottom of the email.

We can confirm that all traffic carried over our DWDM network is restored since 1:17 UTC.
We are pending confirmation from the on-site engineers that the customers' equipment on site is connected and powered one.
Once that is confirmed, dark fiber customers will be contacted for confirmation on the status of their service.

07-01-2020 at 1:19:
NMC managed to protect several more lines. Further migration options are reviewed. Repair ship has departed last night and if the weather allows, restoration work will commence this evening.

All possible migrations were applied.
GTT will review if other migration options are possible.
Next update will be available tomorrow.

Cut is confirmed to be 121 km away from Zandvoort side.
Colocation services will be shut down in order to operate normally with the cable.
Tier 3 engineers are reviewing options for traffic migration, depending on the available capacity in the network.
The whole cable is cut, so dark fiber service migration is not possible.
Repair ship dispatch is confirmed by the contractor. Departure expected later in the evening/tomorrow, depending on the completion of the spares loading.

06-01-2020 at 6:00:
Fault Description:Fiber Break

GTT is experiencing an outage on the subsea section of our transmission network between UK and Netherlands.

Initial Update:
Fibre break detected between Leiston and Zandvoort.
Our systems detect the cut at 91.4km from Leiston.
Engineers are sent to both locations for precise measurement and dispatch of the repair ship.
No ETA yet.

For transparency purposes we decided to publish this notification.

[#870] Outage rack b2.z5 DBA

Posted: 2019-12-19 17:35

Start: 2019-12-19 17:20:00
End : 2019-12-19 18:02:00

Affects: Customers located in b2.z5 DBA

Currently we are having an outage on the rack b2.z5 in Databarn Amsterdam (DBA).

We've gotten a datacenter engineer on route to check on the issue at hand, once we have an update we'll update this post as well.

Update 18:02:
There was a server in this same rack which PSU caused a short circuit on the breaker of this PDU, after the engineer enabled the breaker the PDU came back online.

[#861] Outage VPLS ring EQX-AM7

Posted: 2019-11-25 13:17

Start: 2019-11-25 12:33:00
End : 2019-11-25 14:30:00

Affects: IP access Customers / IP transit Customers EQX-AM7 / Traffic EQX-AM7

At 12:33 CET we noticed an issue on our VPLS ring through EQX-AM7.

During debugging it got very unstable to the point we have rebooted the entire stack.
We are still debugging as to what triggered the outage in the first place together with our vendor.

At the moment everything should be stable again but we continue to monitor the situation for the next couple of hours.

Update 13:30 CET:
Internet Customers may still experience issues with packetloss at this moment, our team is currently working around the clock on getting this solved as well.

Update 14:30 CET:
Also for Internet customers the issue should now be resolved.

Update 15:00 CET:
From investigation, the instability was present since the power outage of equinix AM7, during the black building test.

[#859] Outage Equinix AM7

Posted: 2019-11-21 22:57

Start: 2019-11-21 22:09:00
End : 2019-11-21 22:35:00

Affects: Internet Access / Transits located on EQX-AM7

At 22:09 CET we noticed a complete outage on our Equinix Amsterdam 7 datacenter which appeared to be because of an outage during their annual blackbuilding test.

Equinix AM7 is a big connecting hub for Europe and the Netherlands and while our hosting datacenters did not have any issues but because this was such a big outage for a lot of party's involved at Equinix there will be many re-routes throughout the internet and a lot of packetloss happening or happened.