Incident Date: Wednesday, February 25
Estimated Start: 05:20
Traffic Mitigated: 06:18 (R2 became master)
Redundancy Restored: 06:39

Summary
Intermittent packet loss impacted several NZS Datacenter prefixes. The issue was flow-dependent (specific SrcIP-DstIP pairings) due to internal load-balancing. Average packet loss during the window was ~3% for affected flows.

Timeline (GMT+1 - NL Time)
05:20 -- Issue onset (estimated)
05:20-06:18 -- Investigation: no CRC errors, alarms, or logs; slightly reduced bandwidth observed
06:18 -- Traffic migrated to R2 (master); packet loss immediately ceased
06:18-06:39 -- Deep diagnostics on R1; issue isolated to Switch Fabric Module (SFM) links
06:39 -- Faulty SFM links disabled; redundancy restored; monitoring continued

Root Cause
Failing/degraded internal Switch Fabric Module (SFM) links in router R1. The platform normally operates with 20 fabric links; several links were internally shut down, leaving 18 active. Because traffic is load-balanced across fabric paths, only flows hashed across the degraded links experienced loss. The fault was intermittent and did not trigger system logs or automated alerts.

Resolution
Traffic remains stable via R2. Faulty SFM links on R1 were identified using controlled testing with traffic offloaded and were administratively disabled to prevent reactivation via failover.

Preventative Actions
Hardware replacement is being expedited.
Migration to a new routing platform is in progress, targeted for introduction within the next ~2 months.
Continued monitoring to confirm stability.

1. Incident Summary
Earlier today, around 17:30, our network monitoring detected a significant decline in traffic and elevated latency affecting services originating from our NZS location.

2. Root Cause Analysis
Following immediate investigation, our team has clearly identified the configuration errors responsible for the congestion:

Split-Horizon Misconfiguration: The newly deployed router was missing the required Split-Horizon configuration. This allowed routes learned via the new path to be re-advertised back onto the same ring, creating a suboptimal routing loop.

OSPF Metric Issue: The Open Shortest Path First (OSPF) cost was also incorrectly configured. This low metric amplified the Split-Horizon issue, causing the new, suboptimal path to be immediately advertised as the preferred path by the routing protocol.

These combined errors resulted in traffic converging onto the new connection, causing rapid saturation and the subsequent high latency observed by users.

3. Resolution Steps
Our Network Operations Team took immediate action to mitigate the issue and restore full stability:

The faulty routing source was identified and isolated.

The new router was promptly removed from our ring switches network.

Traffic immediately rerouted back to the stable, existing paths.

Full service stability and normal performance have been restored to the NZS location.

4. Next Steps & Assurance
The device remains isolated while its configuration is thoroughly reviewed and corrected to include the necessary Split-Horizon and OSPF settings.

We are currently closely monitoring the network to guarantee sustained optimal performance and prevent any recurrence.

We sincerely apologize for the inconvenience and service interruption this incident may have caused.

Currently we have a fiber outage in BNR (Bytesnet Rotterdam), we have contacted the darkfiber provider to investigate the situation.

Update 23:55 - We observe the connection coming back up now.

We've observed at 05:06, NL time, that a linecard responsible for AMS-IX connectivity on our EQX-AM7 router went down. After checking that the linecard indeed was unresponsive, we powercycled it. We saw the ports come back up at 05:25 and BGP sessions over AMS-IX were restored. This proved not to remain stable and we are looking at replacing the linecard.

The AMS-IX sessions with our other router are not affected by this.

Update 2025-10-23, 7:26 We have replaced the linecard and are monitoring the situation. AMS-IX port is not coming up despite showing good lightlevels. We will follow-up with AMS-IX to resolve this.

Update 2026-02-19, 13:10 Optic has been replaced and we saw the link coming back up. BGP is re-established.

Currently we have a fiber outage in BNR (Bytesnet Rotterdam), we have contacted the darkfiber provider to investigate the situation.

Update 21:20 - We see the connection restored. We will continue to monitor and seek the cause of this.

Update 23-10-2025 10:00 - We once again had this fault occur and are urging our provider to find the cause of this.