We've noticed some packet loss in NZS.

One of the optics in a bundle of multiple Hundred Gbit optics started giving issues without going down causing packet loss, because of it not going down the issue was hard to pin-point.

We have now disabled the troubling link and will schedule a debug session of this link without affecting production traffic.

Today we had packet loss in our datacenter Nedzone Steenbergen.

After investigation we have noticed a hardware defect in one of the linecards.
The card was only using limited memory space for the routes installed (1M vs 2M).

The issue was not clear at first as everything was looking correctly including optical values. After further investigation we see that the card is not functioning correctly because of the above reason and have currently disabled the ports affected until we have a fix at hand.

Currently we are seeing a DDOS against our DNS resolvers, as many servers are using these you may be having issues as well.

While we are currently focusing on solving these issues a solution to get it solved instantly would be to set any of the resolvers from one of the following:

Google 8.8.8.8,8.8.4.4
OpenDNS 208.67.222.222,208.67.220.220
Cloudflare 1.1.1.1

Update 15:35:
Currently DDOS is mitigated, we will keep monitoring the situation.

We are currently investigating an issue with the power distribution at DBC. The power started to flap at 12:50 and went completely down at 13:00.

We have contacted the datacenter for more information on the issue and are waiting for a response.

Update 13:29:
There is an outage reported by Stedin, we have received notification of an issue in a 100kv substation. More information can be found at https://www.stedin.net/storing-en-onderhoud/overzicht .

Update 17:17:

We received a message from the datacenter:

"We just had a major impact in our power supply due to a problem with Stedin.

The power was cut of at 12:34 and returns after several seconds. The next problem that occurred was a low voltages on 1 feed

https://www.stedin.net/storing-en-onderhoud/storingshistorie

The ups stabilised the drops as it kept on coming and going. For safety measurements we tried to bypass the UPS and put the full load on our generators. But since the power did not fully dropped it declined to bypass. Due to an error the power was cut of for several minutes. "

As of 08:43 a rack in Databarn Amsterdam (DBA) has gone down.

Our engineers are currently investigating the issue.

Our sincere apologies for this inconvenience.

When we have more information on the exact issue we'll update the noc post here.

UPDATE 11:55: Switch has been replaced and connectivity should be restored. If your servers' are still offline, please make a support ticket.