1. Incident Summary
Earlier today, around 17:30, our network monitoring detected a significant decline in traffic and elevated latency affecting services originating from our NZS location.

2. Root Cause Analysis
Following immediate investigation, our team has clearly identified the configuration errors responsible for the congestion:

Split-Horizon Misconfiguration: The newly deployed router was missing the required Split-Horizon configuration. This allowed routes learned via the new path to be re-advertised back onto the same ring, creating a suboptimal routing loop.

OSPF Metric Issue: The Open Shortest Path First (OSPF) cost was also incorrectly configured. This low metric amplified the Split-Horizon issue, causing the new, suboptimal path to be immediately advertised as the preferred path by the routing protocol.

These combined errors resulted in traffic converging onto the new connection, causing rapid saturation and the subsequent high latency observed by users.

3. Resolution Steps
Our Network Operations Team took immediate action to mitigate the issue and restore full stability:

The faulty routing source was identified and isolated.

The new router was promptly removed from our ring switches network.

Traffic immediately rerouted back to the stable, existing paths.

Full service stability and normal performance have been restored to the NZS location.

4. Next Steps & Assurance
The device remains isolated while its configuration is thoroughly reviewed and corrected to include the necessary Split-Horizon and OSPF settings.

We are currently closely monitoring the network to guarantee sustained optimal performance and prevent any recurrence.

We sincerely apologize for the inconvenience and service interruption this incident may have caused.

Currently we have a fiber outage in BNR (Bytesnet Rotterdam), we have contacted the darkfiber provider to investigate the situation.

Update 23:55 - We observe the connection coming back up now.

We've observed at 05:06, NL time, that a linecard responsible for AMS-IX connectivity on our EQX-AM7 router went down. After checking that the linecard indeed was unresponsive, we powercycled it. We saw the ports come back up at 05:25 and BGP sessions over AMS-IX were restored. This proved not to remain stable and we are looking at replacing the linecard.

The AMS-IX sessions with our other router are not affected by this.

Update 2025-10-23, 7:26 We have replaced the linecard and are monitoring the situation. AMS-IX port is not coming up despite showing good lightlevels. We will follow-up with AMS-IX to resolve this.

Currently we have a fiber outage in BNR (Bytesnet Rotterdam), we have contacted the darkfiber provider to investigate the situation.

Update 21:20 - We see the connection restored. We will continue to monitor and seek the cause of this.

Update 23-10-2025 10:00 - We once again had this fault occur and are urging our provider to find the cause of this.

Due to power issues, several of our websites went down. Some emails sent to our ticketing system may have been delayed.

We are sorry for this inconvenience and aim to fully restore power, as we are currently running on a single feed.

No customer equipment has been affected by this.