We have received reports of packetloss towards Facebook (AS32934)

We have removed the path over LINX in the routing tables.

The best selected path is now over Cogent

Because of some maintenance related tasks we will be reloading our GSA router on the 28th of February at 0:30.

Our peering traffic will automatically flow over our EQX-AM7 router.

Internet and Transit customers only having a connection to this router will experience some downtime during the reload.

We have noticed a big outage on one of the dutch Internet Exchanges named NL-ix.

On their traffic graphs it is clearly visible an outage is visible https://www.nl-ix.net/network/traffic-stats/

The issue started at 13:15 and is resolved around 13:40.

We will be monitor the situation and if the issue returns, we will temporary disable our sessions with NL-ix.

Update sent by the NLix 15:07:
Investigation about cause is still ongoing, but is seems this is related to the general issue going on in Nikhef.

We have received reports of high latency towards Global Connect (AS2116)

We have preferred the path over NTT in the routing tables.

The previous best selected path was over Lumen

Wave service provider which connects Equinix AM7 (EQX-AM7) towards LINX is having a planned maintenance.

Their maintenance reason is "Maintenance on our Fibre infrastructure platform related to Cable Splicing."

GTT has given this maintenance reference number: 5639371

All LINX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Equinix AM7 (EQX-AM7) is having a planned maintenance. The connectivity between NZS and EQX-AM7 will automatically reroute over our GSA darkfiber during this maintenance.

Customers having their own waves from NZS to EQX-AM7 will experience an outage during this time.

Wave service provider which connects Global Switch Amsterdam (GSA) towards FranceIX is having a planned maintenance.

Their maintenance reason is "Maintenance on our Transmission platform related to Hardware Replacement."

GTT has given this maintenance reference number: 5669155

All FranceIX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

Around 9:00 we've noticed our connection from EQX-AM7 towards DECIX went down.

Wave service provider which connects Equinix AM7 (EQX-AM7) towards DECIX is investigating the issue.

Update 14:00: The 3rd Party Engineer´s performed OTDR measurements and showing an interruption between Cologne and Dusseldorf.
Fibre team on the way to break location , no ETA/ETR yet.

Final update from our wave service provider: After concluding our investigation, the GTT optical NOC can report back that we observed outage on our DWDM backhaul between Dusseldorf and Linz through which your circuit traverses. Further case was opened with the underlying fibre provider.Our partner immediately started their investigation and dispatched techs to Dusseldorf to troubleshoot the fiber. They performed OTDR test and identified possible fiber cut. Our partner dispatched the engineer and found a fiber cut due to construction work at Bergisch Gladbacher Str., 51063 Cologne. Our partner splices the fiber which restored the services. Restoration work delay due to our partner need to installed approximately 400ft of new cable with 6 Tubes needs to spliced to restore connectivity. Our partner confirmed this is permanent fixed.

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Globalswitch Amsterdam (GSA) is having a planned maintenance. The connectivity between NZS and GSA will automatically reroute over our EQX-AM7 darkfiber during this maintenance.

Customers having their own waves from NZS to GSA will experience an outage during this time.

Update 2020-11-13:
This maintenance has been postponed by Eurofiber for now, once a new date is available we'll create a new post.

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Globalswitch Amsterdam (GSA) is having a planned maintenance. The connectivity between NZS and GSA will automatically reroute over our EQX-AM7 darkfiber during this maintenance.

Customers having their own waves from NZS to GSA will experience an outage during this time.

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Equinix AM7 (EQX-AM7) is having a planned maintenance. The connectivity between NZS and EQX-AM7 will automatically reroute over our GSA darkfiber during this maintenance.

Customers having their own waves from NZS to EQX-AM7 will experience an outage during this time.

Wave service provider which connects Global Switch Amsterdam (GSA) towards DECIX is having a planned maintenance.

Their maintenance reason is "Fiber optic works needs to take place in our network. "

GTT has given this maintenance reference number: 5494823

All DECIX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification

We have received reports of slow speeds towards Vodafone Germany (AS3209)

We have preferred the path over Liberty Global in the routing tables.

The previous best selected path was over Level3.

Darkfiber provider EUNetworks which connects Databarn Capelle (DBC) towards Equinix AM7 (EQX-AM7) is having a planned maintenance. The connectivity between DBC and EQX-AM7 will automatically reroute over our GSA darkfiber during this maintenance.

At 10.20 we had one of the ring switches in DBC reload itself, where we have 2 switches running in a stack. The moment the second switch came back online we started seeing packetloss throughout DBC.

For now we disabled all ports of the second switch while we investigate into this matter, currently seeing a steady ping again.

Update 12:20:
Half of the ports have been enabled again, after resetting the bridges on the VPLS ring.

Update 16:15:
The other half has been enabled now as well.

We've made a case at the vendor to debug this in more detail.

Due to the on-going issues following in post #936 we are going to do an emergency reload of our router in EQX-AM7.

Internet Customers will experience a loss of internet as well as customers having only one BGP session. Customers having a redundant BGP session with us should not experience any issues.

We are aware it is the middle of the day for most but due to the on-going issues there is little choice left for us to do it at a very short timespan, we will try our best to keep the downtime to an absolute minimum.

Update 15:30:
The router has been reload and is back online.

We've became aware of multiple issues affecting our FTTB customers and are currently investigating.

Once we have an update it will be posted here.

Update 15:42:
After the reload of our router in the previous ticket we see most of the issues resolved, a few remain and we'll continu on those.

Update 17:22:
The last few we had now also report it to be working again.

In case you still notice any issues please let us know.

Currently we are experiencing an outage on the darkfiber between GSA and DBC, carrying 400gbps of connectivity. All traffic is diverted over our darkfiber between DBC and Equinix AM7.

At this moment we are inspecting the issue and keep this post updated with our findings.

Update 15:20 CEST:
We've been notified by our darkfiber party that there is currently a darkfiber break on the path between Rotterdam and Amsterdam.

Update 2020-09-17 21:30 CEST:
Currently our darkfiber is back online but we are currently awaiting a crew hands-off from our darkfiber party before closing up this case.

We have received a notification of a maintenance by one of our transit providers, Liberty Global.

During their maintenance all Liberty Global traffic will be automatically rerouted over our other transits and peering connections.

Their maintenance reason: Hotcut fibers in AM5 towards Liberty Global.

For transparency purposes we decided to publish this notification.

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Equinix AM7 (EQX-AM7) is having a planned maintenance. The connectivity between NZS and EQX-AM7 will automatically reroute over our GSA darkfiber during this maintenance.

Customers having their own waves from NZS to EQX-AM7 will experience an outage during this time.

We have been in contact with ForeFreedom about a possible network outage within their platform.

As we have investigated together, we have noticed that everything at our side is running optimal but may be resident in their platform.

The engineers of ForeFreedom is currently looking into this issue at this moment.

Once we received new updates from ForeFreedom, we will update this post accordingly.

Please note that this outage is only for Fiber to the Home/Business networks connected through ForeFreedom and does not affect datacenter, transit connectivity and hosting or other Fiber to the Home/Business connectivity.

Update 2020-09-01 10:35:
We have received an update from ForeFreedom, ForeFreedom rerouted the traffic as there is a major outage on the darkfiber.

At 19:25 we have received notifications from our monitoring system about a minor outage on connectivity at DBC.

After investigation we have found one of the optics reported fault causing several VLANS to flap between VRRP primary to VRRP secondary which caused some connectivity issues.

The offending port is shutdown at 19:30 and VRRP completely restored at 19:40.

Currently we will be monitoring the situation and we will be planning in to replace the defective optic.

We have received yet another DDoS attack towards a IP transit customer.

The DDoS was 100+ GBPS in size and we have taken steps to the IP address that is being attacked.
We will also be taking additional steps.

All systems should now run normal again.

We are sorry for the inconvenience.

We have received a DDoS attack towards a IP transit customer.

The DDoS was 100+ GBPS in size and we have taken steps to the IP address that is being attacked.

All systems should now run normal again.

We are sorry for the inconvenience.

We experienced an issue with the feed row above one of our core racks hitting both of our core switches causing a complete outage of our core switches.

Currently with the help of engineers we've brought them back online and will now look at individual issues still happening.

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Equinix AM7 (EQX-AM7) is having a planned maintenance. The connectivity between NZS and EQX-AM7 will automatically reroute over our GSA darkfiber during this maintenance.

Customers having their own waves from NZS to EQX-AM7 will experience an outage during this time.

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Equinix AM7 (EQX-AM7) is having a planned maintenance. The connectivity between NZS and EQX-AM7 will automatically reroute over our GSA darkfiber during this maintenance.

Customers having their own waves from NZS to EQX-AM7 will experience an outage during this time.

Darkfiber provider GTT which connects Databarn Amsterdam (DBA) towards Equinix AM7 (EQX-AM7) is having a planned maintenance. The connectivity between DBA and EQX-AM7 will automatically reroute over our GSA darkfiber during this maintenance.

We have received a notification of a maintenance by one of our transit providers, Centurylink.

During their maintenance all Centurylink traffic will be automatically rerouted over our other transits and peering connections.

Their maintenance reason: The nature of this work is to perform a Network Element software upgrade.

Their reference number: 19350931

For transparency purposes we decided to publish this notification.

One of our hypervisors crashed at 20:30 which has been solved and the hypervisor is back online for some time, currently we are running a disk repair over the cluster, which is needed for vpsses to being able to boot, if yours currently is not able to boot this is because it's disk hasn't been fully synched yet.

Once that has been done it should boot up again, currently it's almost at 50%.

Update 22:46:
Disk repair is currently at 60%

Update 23:04:
Disk repair is currently at 70%

Update 23:18:
Disk repair is currently at 80%

Update 23:29:
Disk repair is currently at 90%

Update 00:53:
Currently all disks have been repaired and synched and all vpsses should work again.

Update 2020/08/07 08:00:
Currently we're having issues on the platform and are working very hard with the Onapp support in solving these issues.

Update 15:33:

Currently all VPSses should be working again, as our vendor is still investigating into these crashes we will not close this case just yet. We will update this case when we are ready to give the all clear.

We have received reports of slow speeds towards Reliance Jio Infocomm Limited(AS55836)

We have denied the path over NL-IX in the routing tables.

The best selected path is now going over AMS-IX.

We have received reports of slow speeds towards Reliance Jio (AS55836)

We have denied the path over NL-IX in the routing tables.

The best selected path is now going over AMS-IX.

We became aware through our monitoring that one of the PDU's in rack a8.z4 (pdu2) went down and sent an engineer to the location to inspect.

After checking it was noticed the breaker was off, the engineer switched the breaker back on and got the pdu powered back on.

It was found out the breaker tripped because of a bad PSU in one of the servers in the same rack.

Darkfiber provider Eurofiber which connects Nedzone Steenbergen (NZS) towards Equinix AM7 (EQX-AM7) is having a planned maintenance. The connectivity between NZS and EQX-AM7 will automatically reroute over our GSA darkfiber during this maintenance.

Customers having their own waves from NZS to EQX-AM7 will experience an outage during this time.

We have received reports of high latency towards Vodafone Telekomunikasyon A.S. (AS15897)

We have denied the path over NTT in the routing tables.

The best selected path is now going over Cogent (AS174).

Tomorrow on the 17th of June between 10.30 AM and 2 PM CEST we will upgrade the capacity of the rack A-2-3 and at the same exchange the many amount of 10Gbit optics for 40Gbit optics to keep it in check for future server additions.

This may cause some minor packetloss or ping timeouts while the switch transfers it's mac address over from the 10Gbit ports to the 40Gbit ones.

This only affects servers in rack A-2-3 on Nedzone Steenbergen (NZS).

We have received a notification of a maintenance by one of our transit providers, Cogent.

During their maintenance all Cogent traffic will be automatically rerouted over our other transits and peering connections.

Their maintenance reason: Service migration to a new linecard

Their reference number: EU6162

For transparency purposes we decided to publish this notification.

We have received reports of high latency towards Vodafone Limited (AS5378)

We have denied the path over Linx in the routing tables.

The best selected path is now going over NTT (AS2914).

Wave service provider which connects Equinix Amsterdam 7 (EQX-AM7) towards LINX is having a planned maintenance.

Their maintenance reason is "Fiber optic works needs to take place in our network. "

KPN has given this maintenance reference number: 111117

All LINX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

We have received reports of packetloss towards China Mobile communications corporation (AS56046)

We have denied the path over Cogent in the routing tables.

The best selected path is now going over NTT (AS2914).

We have received reports of slow speeds towards Altibox (AS29695)

We have denied the path over NTT in the routing tables.

The best selected path is now going over Cogent (AS174).

We are experiencing since 02:45 an outage of the cooling systems in Databarn Capelle (DBC).

Currently, engineers of the datacenter are resolving the issue.

Systems may affect shutdown due to overheating, few of our systems are currently unavailable, such as ticketing.

Update 05:55:
Currently all issues should be solved, we will keep monitoring the situation for any changes.

We have received reports of slow speeds towards Netia SA (AS12741)

We have denied the path over DEC-IX in the routing tables.

The best selected path is now going over Cogent (AS174).

Wave service provider which connects Globalswitch Amsterdam (GSA) towards LINX is having a planned maintenance.

Their maintenance reason is "Maintenance on our Transmission platform related to Hardware Removal"

GTT has given this maintenance reference number: 5103739

All LINX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

Wave service provider which connects Equinix (AM7) towards DECIX is having a planned maintenance.

Their maintenance reason is "Maintenance Provider"

KPN International has given these maintenance reference numbers: 110208, 110226

All DECIX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

Wave service provider which connects Globalswitch Amsterdam (GSA) towards LINX is having a planned maintenance.

Their maintenance reason is "Maintenance on our Transmission platform related to Hardware Replacement"

GTT has given this maintenance reference number: 5035949

All LINX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

Wave service provider which connects Globalswitch Amsterdam (GSA) towards LINX is having a planned maintenance.

Their maintenance reason is "card needs replacement"

GTT has given this maintenance reference number: 5003442

All LINX routed traffic is automatically routed over our other exchanges and IP transits.

For transparency purposes we decided to publish this notification.

We have received reports of packet loss towards Charter Communications (AS11351)

We have denied the path over Century Link in the routing tables.

The best selected path is now going over Liberty Global.

We have received reports of packet loss towards Charter Communications (AS33363)

We have denied the path over Century Link in the routing tables.

The best selected path is now going over Cogent.

We have received reports of high latency towards Vodafone Deutschland (AS31334)

We have denied the path over Century Link in the routing tables.

The best selected path is now going over NTT.