Currently we are experiencing an outage at our router in AM7 with the AMS-IX Peering platform.

Yesterday night AMS-IX performed a maintenance and we expect that this issue we are experiencing is caused by the maintenance.

AMS-IX is informed with additional details so they can investigate what happened.

As we have redundancy in place, our AMS-IX peering traffic is automatically rerouted over GSA. We do not expect any issues.

For transparency purposes we decided to publish this notification.

Update 2019-02-21, 11:28:
AMS-IX found out that the linecard they connected us to is defective and put us back on the primary path.

All traffic and BGP sessions of AMS-IX at AM7 are now restored.

At 6:35 CET we had a crash of the linecards in both routers R1 and R2 DBA causing also the BGP to our transit providers to flap.

After the cards came back online all routes also need to be relearned from all of our transit providers where packetloss or unreachabillity might have occured.

We've gathered logs for our vendor who is assisting us in debugging this issue.

Our sincere apologies for any issues caused by this outage.

At this moment the debugging process is still ongoing.

Update: 2019-02-21, 12:11:
While having close contact with our vendor, they have reported that the crash is happening within the IPv6 stack. They however were not able to resolve the issue at this moment.

We have looked for alternative options and did some tests. As this phase is completed, we would like to inform you that we are going to separate the IPv6 from our current routers away from IPv4. IPv6 will run independently.

Due to the recent crashes and our vendor yet to provide a patch against this issue, we will be performing the move of customer IPv6 to the new platform on DBA tomorrow (Friday, 22 Februari 2019) starting from 7:00. We expect to finish up around 12:00.

During this time, each IPv6 block can expect a downtime up to 5 minutes.

The IPv4 will be untouched and unaffected by this urgent maintenance.
There are no changes required at the customer side, please do note that when you do have IPv6 BGP sessions with us, 1 session will stay down until a later maintenance.


Update: 2019-02-22, 11:43:
We have finished the move of the customers IPv6 to the new platform.

At Monday, 25 Februari 2019, starting from 7:00, we will be moving the transit IPv6 sessions. We do not expect any downtimes and will be carefully moved. The process may take up until at latest 17:00.

As this is IPv6 only, IPv4 is not affected by the move.

At 22:40 CET, 1 Februari 2019 we have noticed a linecard crash on Router 1 of the datacenter Databarn Amsterdam.

As the linecard restored quickly after the crash there might have been interruptions of services due to the failover process.

We have made dumps of the crash logs and provided the information to the vendor for further investigation while we continue to monitor stability of the linecard.

Update 2019-02-02 8:35 CET:
We experienced another linecard crash at 2019-02-01 22:45 CET, which may briefly (~ 2 minutes) caused packetloss on your services in DBA.

Update 2019-02-07 12:46 CET:
We experienced another linecard crash last night at 3:13 AM CET, which may briefly (~ 2 minutes) caused packetloss on your services in DBA, this crashdump was added to the case with our vendor.

Update 2019-02-13 8:03 CET:
Around 18:28 CET, 12 Februari 2019, we have found our other linecard of our DBA router has crashed. We have submitted crashdump as well as other dumps to provide as much information as we can.

Update 2019-02-16 06:35 CET:
We experienced a linecard crash of both linecards in R1 and R2 of DBA. where also the BGP with our transits was dropped which needed to be re-converged after coming back online, this may have caused packetloss or unreachabillity.

Update 2019-02-18 04:52 CET:
We experienced linecard crashes both on R1 and R2 of DBC. As the stacktraces are exactly the same as the crashes of DBA, we expect this to be the same issue. Currently the vendor is still investigating the debugs.

Wave service provider which connects Globalswitch Amsterdam (GSA) towards Paris is having an outage.

Their outage reason is "GTT is experiencing a fibre break between Antwerp and Gent. Engineers are dispatched. Our systems show fault on 39km from Antwerp."

All FRANCE-IX routed traffic is automatically routed over our other exchanges and IP transits.

Update 21:26:
The fiber is back online again and the repairs are completed on the fiber.

At 22:35 CET a darkfiber between DBC (Datbarn Capelle) and EQX-AM7 (Equinix AM7) started flapping, carrying 400 Gbps traffic.

Currently all traffic is sent over the alternative path on our ring namely DBC - GSA while we are investigating the cause into this.

While testing what the issue is on the path some packetloss may appear.

Update 01:45:
At 23:30 CET the connections went physically down and haven't come back, we are still investigating to find the root cause of this issue to being able to solve it in getting the link back online.

Update 2018-12-17 15:49:

After a lot of debugging last night and today we noticed one of the amplifiers at DBC giving out a much higher value then what was measured using a power meter, where we would have strong light levels from one side, but not from the other side.

We've replaced the amplifier in question and currently the connection is up and running again, we'll keep monitoring this line for some time.