We've noticed some packet loss in NZS.

One of the optics in a bundle of multiple Hundred Gbit optics started giving issues without going down causing packet loss, because of it not going down the issue was hard to pin-point.

We have now disabled the troubling link and will schedule a debug session of this link without affecting production traffic.

We have planned an upgrade of network capacity towards TI Sparkle.

During this maintenance all TI Sparkle traffic will be automatically rerouted over our other transits and peering connections.

Today we had packet loss in our datacenter Nedzone Steenbergen.

After investigation we have noticed a hardware defect in one of the linecards.
The card was only using limited memory space for the routes installed (1M vs 2M).

The issue was not clear at first as everything was looking correctly including optical values. After further investigation we see that the card is not functioning correctly because of the above reason and have currently disabled the ports affected until we have a fix at hand.

Currently we are seeing a DDOS against our DNS resolvers, as many servers are using these you may be having issues as well.

While we are currently focusing on solving these issues a solution to get it solved instantly would be to set any of the resolvers from one of the following:

Google 8.8.8.8,8.8.4.4
OpenDNS 208.67.222.222,208.67.220.220
Cloudflare 1.1.1.1

Update 15:35:
Currently DDOS is mitigated, we will keep monitoring the situation.

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Globalswitch Amsterdam (GSA) is having a planned maintenance. The connectivity between NZS and GSA will automatically reroute over our EQX-AM7 darkfiber during this maintenance.

Customers having their own waves from NZS to GSA will experience an outage during this time.

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Equinix AM7 (EQX-AM7) is having a planned maintenance. The connectivity between NZS and EQX-AM7 will automatically reroute over our GSA darkfiber during this maintenance.

Customers having their own waves from NZS to EQX-AM7 will experience an outage during this time.

Darkfiber provider GTT which connects Databarn Amsterdam (DBA) towards Equinix AM7 (EQX-AM7) is having a planned maintenance. The connectivity between DBA and EQX-AM7 will automatically reroute over our GSA darkfiber during this maintenance.

We are currently investigating an issue with the power distribution at DBC. The power started to flap at 12:50 and went completely down at 13:00.

We have contacted the datacenter for more information on the issue and are waiting for a response.

Update 13:29:
There is an outage reported by Stedin, we have received notification of an issue in a 100kv substation. More information can be found at https://www.stedin.net/storing-en-onderhoud/overzicht .

Update 17:17:

We received a message from the datacenter:

"We just had a major impact in our power supply due to a problem with Stedin.

The power was cut of at 12:34 and returns after several seconds. The next problem that occurred was a low voltages on 1 feed

https://www.stedin.net/storing-en-onderhoud/storingshistorie

The ups stabilised the drops as it kept on coming and going. For safety measurements we tried to bypass the UPS and put the full load on our generators. But since the power did not fully dropped it declined to bypass. Due to an error the power was cut of for several minutes. "

As of 08:43 a rack in Databarn Amsterdam (DBA) has gone down.

Our engineers are currently investigating the issue.

Our sincere apologies for this inconvenience.

When we have more information on the exact issue we'll update the noc post here.

UPDATE 11:55: Switch has been replaced and connectivity should be restored. If your servers' are still offline, please make a support ticket.

We are experiencing since 23:30 an outage of the cooling systems in Databarn Amsterdam (DBA).

Currently, engineers of the datacenter are resolving the issue.

Systems may affect shutdown due to overheating.

We are chasing them for an ETR and solution.

Update 00:10:
Currently a backup system has been started to sent as much cold air into the rooms as possible while the vendor is debugging the primary cooling system.

Update 00:40:
Currently we see a drop in temperatures throughout all dc rooms, since all hot air needs to be replaced by cold air it might take time before it returns to previous temperatures.

Update 00:55:
At the moment the primary and backup cooling is running to cool off the datacenter as quicky as possible and we've seen temperatures dropping throughout all rooms back to the normal values.

In case you have one or more servers that went into a thermal shutdown please let us know by emailing support@nforce.com so we can follow up for you.

Our sincere apologies for this inconvenience.

When we have more information on the exact issue with the primary cooling we'll update the noc post here.

We have received a notification of a maintenance by one of our transit providers, NTT.

During their maintenance all NTT traffic will be automatically rerouted over our other transits and peering connections.

Their maintenance reason: Services migration

For transparency purposes we decided to publish this notification.

We have received a notification of a maintenance by one of our transit providers, NTT.

During their maintenance all NTT traffic will be automatically rerouted over our other transits and peering connections.

Their maintenance reason: Services migration

For transparency purposes we decided to publish this notification.

We've gotten reports of packet loss in DBC.

It appears an optic has failed between our core-switch and one of our routers, causing packet loss.

We have properly disabled the troubling link and will schedule a replacement in the oncoming week.

We have received reports of packetloss towards Facebook (AS32934)

We have removed the path over LINX in the routing tables.

The best selected path is now over Cogent

Because of some maintenance related tasks we will be reloading our GSA router on the 28th of February at 0:30.

Our peering traffic will automatically flow over our EQX-AM7 router.

Internet and Transit customers only having a connection to this router will experience some downtime during the reload.

We have noticed a big outage on one of the dutch Internet Exchanges named NL-ix.

On their traffic graphs it is clearly visible an outage is visible https://www.nl-ix.net/network/traffic-stats/

The issue started at 13:15 and is resolved around 13:40.

We will be monitor the situation and if the issue returns, we will temporary disable our sessions with NL-ix.

Update sent by the NLix 15:07:
Investigation about cause is still ongoing, but is seems this is related to the general issue going on in Nikhef.

We have received reports of high latency towards Global Connect (AS2116)

We have preferred the path over NTT in the routing tables.

The previous best selected path was over Lumen

Wave service provider which connects Equinix AM7 (EQX-AM7) towards LINX is having a planned maintenance.

Their maintenance reason is "Maintenance on our Fibre infrastructure platform related to Cable Splicing."

GTT has given this maintenance reference number: 5639371

All LINX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Equinix AM7 (EQX-AM7) is having a planned maintenance. The connectivity between NZS and EQX-AM7 will automatically reroute over our GSA darkfiber during this maintenance.

Customers having their own waves from NZS to EQX-AM7 will experience an outage during this time.

Wave service provider which connects Global Switch Amsterdam (GSA) towards FranceIX is having a planned maintenance.

Their maintenance reason is "Maintenance on our Transmission platform related to Hardware Replacement."

GTT has given this maintenance reference number: 5669155

All FranceIX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

Around 9:00 we've noticed our connection from EQX-AM7 towards DECIX went down.

Wave service provider which connects Equinix AM7 (EQX-AM7) towards DECIX is investigating the issue.

Update 14:00: The 3rd Party Engineer´s performed OTDR measurements and showing an interruption between Cologne and Dusseldorf.
Fibre team on the way to break location , no ETA/ETR yet.

Final update from our wave service provider: After concluding our investigation, the GTT optical NOC can report back that we observed outage on our DWDM backhaul between Dusseldorf and Linz through which your circuit traverses. Further case was opened with the underlying fibre provider.Our partner immediately started their investigation and dispatched techs to Dusseldorf to troubleshoot the fiber. They performed OTDR test and identified possible fiber cut. Our partner dispatched the engineer and found a fiber cut due to construction work at Bergisch Gladbacher Str., 51063 Cologne. Our partner splices the fiber which restored the services. Restoration work delay due to our partner need to installed approximately 400ft of new cable with 6 Tubes needs to spliced to restore connectivity. Our partner confirmed this is permanent fixed.

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Globalswitch Amsterdam (GSA) is having a planned maintenance. The connectivity between NZS and GSA will automatically reroute over our EQX-AM7 darkfiber during this maintenance.

Customers having their own waves from NZS to GSA will experience an outage during this time.

Update 2020-11-13:
This maintenance has been postponed by Eurofiber for now, once a new date is available we'll create a new post.

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Globalswitch Amsterdam (GSA) is having a planned maintenance. The connectivity between NZS and GSA will automatically reroute over our EQX-AM7 darkfiber during this maintenance.

Customers having their own waves from NZS to GSA will experience an outage during this time.

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Equinix AM7 (EQX-AM7) is having a planned maintenance. The connectivity between NZS and EQX-AM7 will automatically reroute over our GSA darkfiber during this maintenance.

Customers having their own waves from NZS to EQX-AM7 will experience an outage during this time.

Wave service provider which connects Global Switch Amsterdam (GSA) towards DECIX is having a planned maintenance.

Their maintenance reason is "Fiber optic works needs to take place in our network. "

GTT has given this maintenance reference number: 5494823

All DECIX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification

We have received reports of slow speeds towards Vodafone Germany (AS3209)

We have preferred the path over Liberty Global in the routing tables.

The previous best selected path was over Level3.

Darkfiber provider EUNetworks which connects Databarn Capelle (DBC) towards Equinix AM7 (EQX-AM7) is having a planned maintenance. The connectivity between DBC and EQX-AM7 will automatically reroute over our GSA darkfiber during this maintenance.

At 10.20 we had one of the ring switches in DBC reload itself, where we have 2 switches running in a stack. The moment the second switch came back online we started seeing packetloss throughout DBC.

For now we disabled all ports of the second switch while we investigate into this matter, currently seeing a steady ping again.

Update 12:20:
Half of the ports have been enabled again, after resetting the bridges on the VPLS ring.

Update 16:15:
The other half has been enabled now as well.

We've made a case at the vendor to debug this in more detail.

Due to the on-going issues following in post #936 we are going to do an emergency reload of our router in EQX-AM7.

Internet Customers will experience a loss of internet as well as customers having only one BGP session. Customers having a redundant BGP session with us should not experience any issues.

We are aware it is the middle of the day for most but due to the on-going issues there is little choice left for us to do it at a very short timespan, we will try our best to keep the downtime to an absolute minimum.

Update 15:30:
The router has been reload and is back online.

We've became aware of multiple issues affecting our FTTB customers and are currently investigating.

Once we have an update it will be posted here.

Update 15:42:
After the reload of our router in the previous ticket we see most of the issues resolved, a few remain and we'll continu on those.

Update 17:22:
The last few we had now also report it to be working again.

In case you still notice any issues please let us know.

Currently we are experiencing an outage on the darkfiber between GSA and DBC, carrying 400gbps of connectivity. All traffic is diverted over our darkfiber between DBC and Equinix AM7.

At this moment we are inspecting the issue and keep this post updated with our findings.

Update 15:20 CEST:
We've been notified by our darkfiber party that there is currently a darkfiber break on the path between Rotterdam and Amsterdam.

Update 2020-09-17 21:30 CEST:
Currently our darkfiber is back online but we are currently awaiting a crew hands-off from our darkfiber party before closing up this case.

We have received a notification of a maintenance by one of our transit providers, Liberty Global.

During their maintenance all Liberty Global traffic will be automatically rerouted over our other transits and peering connections.

Their maintenance reason: Hotcut fibers in AM5 towards Liberty Global.

For transparency purposes we decided to publish this notification.

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Equinix AM7 (EQX-AM7) is having a planned maintenance. The connectivity between NZS and EQX-AM7 will automatically reroute over our GSA darkfiber during this maintenance.

Customers having their own waves from NZS to EQX-AM7 will experience an outage during this time.

We have been in contact with ForeFreedom about a possible network outage within their platform.

As we have investigated together, we have noticed that everything at our side is running optimal but may be resident in their platform.

The engineers of ForeFreedom is currently looking into this issue at this moment.

Once we received new updates from ForeFreedom, we will update this post accordingly.

Please note that this outage is only for Fiber to the Home/Business networks connected through ForeFreedom and does not affect datacenter, transit connectivity and hosting or other Fiber to the Home/Business connectivity.

Update 2020-09-01 10:35:
We have received an update from ForeFreedom, ForeFreedom rerouted the traffic as there is a major outage on the darkfiber.

At 19:25 we have received notifications from our monitoring system about a minor outage on connectivity at DBC.

After investigation we have found one of the optics reported fault causing several VLANS to flap between VRRP primary to VRRP secondary which caused some connectivity issues.

The offending port is shutdown at 19:30 and VRRP completely restored at 19:40.

Currently we will be monitoring the situation and we will be planning in to replace the defective optic.

We have received yet another DDoS attack towards a IP transit customer.

The DDoS was 100+ GBPS in size and we have taken steps to the IP address that is being attacked.
We will also be taking additional steps.

All systems should now run normal again.

We are sorry for the inconvenience.

We have received a DDoS attack towards a IP transit customer.

The DDoS was 100+ GBPS in size and we have taken steps to the IP address that is being attacked.

All systems should now run normal again.

We are sorry for the inconvenience.

We experienced an issue with the feed row above one of our core racks hitting both of our core switches causing a complete outage of our core switches.

Currently with the help of engineers we've brought them back online and will now look at individual issues still happening.

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Equinix AM7 (EQX-AM7) is having a planned maintenance. The connectivity between NZS and EQX-AM7 will automatically reroute over our GSA darkfiber during this maintenance.

Customers having their own waves from NZS to EQX-AM7 will experience an outage during this time.

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Equinix AM7 (EQX-AM7) is having a planned maintenance. The connectivity between NZS and EQX-AM7 will automatically reroute over our GSA darkfiber during this maintenance.

Customers having their own waves from NZS to EQX-AM7 will experience an outage during this time.

Darkfiber provider GTT which connects Databarn Amsterdam (DBA) towards Equinix AM7 (EQX-AM7) is having a planned maintenance. The connectivity between DBA and EQX-AM7 will automatically reroute over our GSA darkfiber during this maintenance.

We have received a notification of a maintenance by one of our transit providers, Centurylink.

During their maintenance all Centurylink traffic will be automatically rerouted over our other transits and peering connections.

Their maintenance reason: The nature of this work is to perform a Network Element software upgrade.

Their reference number: 19350931

For transparency purposes we decided to publish this notification.

One of our hypervisors crashed at 20:30 which has been solved and the hypervisor is back online for some time, currently we are running a disk repair over the cluster, which is needed for vpsses to being able to boot, if yours currently is not able to boot this is because it's disk hasn't been fully synched yet.

Once that has been done it should boot up again, currently it's almost at 50%.

Update 22:46:
Disk repair is currently at 60%

Update 23:04:
Disk repair is currently at 70%

Update 23:18:
Disk repair is currently at 80%

Update 23:29:
Disk repair is currently at 90%

Update 00:53:
Currently all disks have been repaired and synched and all vpsses should work again.

Update 2020/08/07 08:00:
Currently we're having issues on the platform and are working very hard with the Onapp support in solving these issues.

Update 15:33:

Currently all VPSses should be working again, as our vendor is still investigating into these crashes we will not close this case just yet. We will update this case when we are ready to give the all clear.

We have received reports of slow speeds towards Reliance Jio Infocomm Limited(AS55836)

We have denied the path over NL-IX in the routing tables.

The best selected path is now going over AMS-IX.

We have received reports of slow speeds towards Reliance Jio (AS55836)

We have denied the path over NL-IX in the routing tables.

The best selected path is now going over AMS-IX.

We became aware through our monitoring that one of the PDU's in rack a8.z4 (pdu2) went down and sent an engineer to the location to inspect.

After checking it was noticed the breaker was off, the engineer switched the breaker back on and got the pdu powered back on.

It was found out the breaker tripped because of a bad PSU in one of the servers in the same rack.

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Equinix AM7 (EQX-AM7) is having a planned maintenance. The connectivity between NZS and EQX-AM7 will automatically reroute over our GSA darkfiber during this maintenance.

Customers having their own waves from NZS to EQX-AM7 will experience an outage during this time.

We have received reports of high latency towards Vodafone Telekomunikasyon A.S. (AS15897)

We have denied the path over NTT in the routing tables.

The best selected path is now going over Cogent (AS174).

Tomorrow on the 17th of June between 10.30 AM and 2 PM CEST we will upgrade the capacity of the rack A-2-3 and at the same exchange the many amount of 10Gbit optics for 40Gbit optics to keep it in check for future server additions.

This may cause some minor packetloss or ping timeouts while the switch transfers it's mac address over from the 10Gbit ports to the 40Gbit ones.

This only affects servers in rack A-2-3 on Nedzone Steenbergen (NZS).

We have received a notification of a maintenance by one of our transit providers, Cogent.

During their maintenance all Cogent traffic will be automatically rerouted over our other transits and peering connections.

Their maintenance reason: Service migration to a new linecard

Their reference number: EU6162

For transparency purposes we decided to publish this notification.