We are currently investigating an issue with the power distribution at DBC. The power started to flap at 12:50 and went completely down at 13:00.

We have contacted the datacenter for more information on the issue and are waiting for a response.

Update 13:29:
There is an outage reported by Stedin, we have received notification of an issue in a 100kv substation. More information can be found at https://www.stedin.net/storing-en-onderhoud/overzicht .

Update 17:17:

We received a message from the datacenter:

"We just had a major impact in our power supply due to a problem with Stedin.

The power was cut of at 12:34 and returns after several seconds. The next problem that occurred was a low voltages on 1 feed

https://www.stedin.net/storing-en-onderhoud/storingshistorie

The ups stabilised the drops as it kept on coming and going. For safety measurements we tried to bypass the UPS and put the full load on our generators. But since the power did not fully dropped it declined to bypass. Due to an error the power was cut of for several minutes. "

As of 08:43 a rack in Databarn Amsterdam (DBA) has gone down.

Our engineers are currently investigating the issue.

Our sincere apologies for this inconvenience.

When we have more information on the exact issue we'll update the noc post here.

UPDATE 11:55: Switch has been replaced and connectivity should be restored. If your servers' are still offline, please make a support ticket.

We are experiencing since 23:30 an outage of the cooling systems in Databarn Amsterdam (DBA).

Currently, engineers of the datacenter are resolving the issue.

Systems may affect shutdown due to overheating.

We are chasing them for an ETR and solution.

Update 00:10:
Currently a backup system has been started to sent as much cold air into the rooms as possible while the vendor is debugging the primary cooling system.

Update 00:40:
Currently we see a drop in temperatures throughout all dc rooms, since all hot air needs to be replaced by cold air it might take time before it returns to previous temperatures.

Update 00:55:
At the moment the primary and backup cooling is running to cool off the datacenter as quicky as possible and we've seen temperatures dropping throughout all rooms back to the normal values.

In case you have one or more servers that went into a thermal shutdown please let us know by emailing support@nforce.com so we can follow up for you.

Our sincere apologies for this inconvenience.

When we have more information on the exact issue with the primary cooling we'll update the noc post here.

We've gotten reports of packet loss in DBC.

It appears an optic has failed between our core-switch and one of our routers, causing packet loss.

We have properly disabled the troubling link and will schedule a replacement in the oncoming week.

We have noticed a big outage on one of the dutch Internet Exchanges named NL-ix.

On their traffic graphs it is clearly visible an outage is visible https://www.nl-ix.net/network/traffic-stats/

The issue started at 13:15 and is resolved around 13:40.

We will be monitor the situation and if the issue returns, we will temporary disable our sessions with NL-ix.

Update sent by the NLix 15:07:
Investigation about cause is still ongoing, but is seems this is related to the general issue going on in Nikhef.