Status

#560 Packetloss several vlans DBC

Posted: 2017-11-08 11:29

Start: 2017-11-08 08:45:00
End : 2017-11-08 11:15:00

Affects: DBC vlans 1018, 1069

At 08.45 we received notifications from our monitoring that for vlan 1018 and 1069 at the DBC facility there is packetloss. At 09.00 this automatically recovered.

We are currently investigating the cause.


Update 11.30:
During the timeframes 09.45 - 09.55 and 11.00 - 11.15 we have received the same reports.

Update 11.42:
Investigation is still ongoing, VRRP-E flaps due to duplicated mac-address. The issue seems to originate between the R1 <-> core-sw1.z1 and R2 <-> core-sw1.z1. We do not see any loops and our ports facing the routers are configured as route-only as well. Our debugging shows that it is most likely caused by a defective port which is not shutting itself down and still forwarding partial traffic. We are currently going through all 96x 10 Gbps ports connecting the core-sw1.z1 and the R1/R2 routers to locate the problematic port.

An engineer was already on-site since 09.00 for regular maintenance, that maintenance has been prosponed to later today and the engineer is working together with remote engineers to debug and resolve this issue.

The issue is not continuously ongoing, making the issue more difficult to identify. Hence we have enabled real-time monitoring to find the culprit.

We expect an update/resolve of the issue within approximately 1-2 hours.

Update 13.05:
We have found the problematic bridge and are currently moving away the vlans to debug this further. No incidents have occurred since 11.15.

Update 14.45:
The incident has been resolved, everything has been restored to primary paths. There we no more incidents since last incident which was at 11.15.

#558 Darkfiber outage DBC - GSA

Posted: 2017-11-08 00:26

Start: 2017-11-08 00:26:41
End : 2017-11-08 01:40:00

Affects: Redundancy DBC

At 00.12 CET a darkfiber between DBC and Globalswitch Amsterdam (GSA) went down, carrying 400 Gbps traffic.

Currently all traffic is send over alternative paths on our ring, DBC - EQX-AM7 and DBC - NZS.

From DBC you might experience increased latency and lower throughput towards routes terminated at our GSA pop.

We have contacted the darkfiber provider euNetworks and requested a repair.

Update 1:40:
At this moment the darkfiber is back online, we will keep monitoring the connection.

According to euNetworks this was due to an unscheduled emergency maintenance.

#555 Darkfiber outage DBC - EQX-AM7

Posted: 2017-11-01 13:49

Start: 2017-11-01 14:08:00
End : 2017-11-01 14:17:00

Affects: Redundancy DBC

At 14.08 CEST a darkfiber between DBC and Equinix-AM7 (EQX-AM7) went down, carrying 200 Gbps traffic.

Currently all traffic is send over alternative paths on our ring, DBC - GSA and DBC - NZS.

From DBC you might experience increased latency and lower throughput towards routes terminated at our EQX-AM7 pop.

We have contacted the darkfiber provider euNetworks and requested a repair.

At 14.17 the darkfiber recovered, therefore this has not been sent though email.

#554 VPLS outage EQX-AM7

Posted: 2017-10-31 05:30

Start: 2017-10-31 02:30:00
End : 2017-10-31 05:55:00

Affects: VPLS & IPT EQX-AM7

At 02.30 we received error notifications from ring-sw2 at EQX-AM7. Traffic passing through this location (VPLS & IPT) are partially affected.

We have gather debugging reports for our vendor and currently in the process of debugging.

Update 05.45:
ISSU with failover upgrade/reload did not resolve the issue. We will now be reloading the chassis as whole.

Update 05.55:
Ring-sw2 is reloaded and mpls is being established. Running tests and restoring traffic.

Update 06.20:
Traffic has been restroing since 05.55. We will reach out to the vendor with tech logs for debugging further. Incident itself is closed for now.

#546 LINX Outage GSA

Posted: 2017-10-18 09:03

Start: 2017-10-17 23:21:00
End : 2017-10-18 13:02:00

Affects: Routing LINX through GSA

At 23:21 we lost signal with LINX at our router in Global Switch Amsterdam (GSA), we have directly investigated the issue and suspect the issue to be with the wave service provider.

We have informed the wave service provider to inspect the issue and are awaiting their feedback.

As our setup is redundant, the traffic is automatically being rerouted over EQX-AM7

Update 2017-10-18 9:17:
The wave server provider informed us that the issue is at the side of London, confirming that the issue is not with the NFOrce Entertainment B.V. Network Infrastructure.

Update 2017-10-18 13:02:
The connection is now back up, and all BGP sessions are restored on the LINX.