At 6:35 CET we had a crash of the linecards in both routers R1 and R2 DBA causing also the BGP to our transit providers to flap.

After the cards came back online all routes also need to be relearned from all of our transit providers where packetloss or unreachabillity might have occured.

We've gathered logs for our vendor who is assisting us in debugging this issue.

Our sincere apologies for any issues caused by this outage.

At this moment the debugging process is still ongoing.

Update: 2019-02-21, 12:11:
While having close contact with our vendor, they have reported that the crash is happening within the IPv6 stack. They however were not able to resolve the issue at this moment.

We have looked for alternative options and did some tests. As this phase is completed, we would like to inform you that we are going to separate the IPv6 from our current routers away from IPv4. IPv6 will run independently.

Due to the recent crashes and our vendor yet to provide a patch against this issue, we will be performing the move of customer IPv6 to the new platform on DBA tomorrow (Friday, 22 Februari 2019) starting from 7:00. We expect to finish up around 12:00.

During this time, each IPv6 block can expect a downtime up to 5 minutes.

The IPv4 will be untouched and unaffected by this urgent maintenance.
There are no changes required at the customer side, please do note that when you do have IPv6 BGP sessions with us, 1 session will stay down until a later maintenance.


Update: 2019-02-22, 11:43:
We have finished the move of the customers IPv6 to the new platform.

At Monday, 25 Februari 2019, starting from 7:00, we will be moving the transit IPv6 sessions. We do not expect any downtimes and will be carefully moved. The process may take up until at latest 17:00.

As this is IPv6 only, IPv4 is not affected by the move.

Update: 2019-02-25, 14:42:
All transits and peering connectivity is moved and officially disconnected IPv6 from the old platform. IPv6 is now running independently.

For the status on the bug, we recommend you to keep an eye on the post: https://noc.nforce.com/notifications/item/749