Incident began

Between approximately 12:00AM PST and 12:30AM PST, an outage occurred which affected nearly all of our online users. Long time service members will know that incidents of this scale are incredibly rare, and have only happened a few times in the history of our service. The effects of this outage were wide-ranging, and include any of the following conditions: - Dashboard control unusable - Streams to active destinations failing (causing VOD splits) - Ingest feeds going black and not coming back This incident happened due to a bug in the firmware of our two core routers which was exposed when we attempted a seemingly harmless configuration change. This led to a loss of communication between the two routers. Due to maintenance upstream of us, one of our two main network uplinks were already shut down. When the loss of communication happened between our routers, it resulted in traffic which was previously being rerouted due to the maintenance to instead be blackholed (sent to nowhere). This is what resulted in the affects described above. We immediately worked to begin troubleshooting the issues, and with the help of our upstream provider, were able to get the second upstream link online temporarily in order to minimize the loss of communication between the two core routers. This allowed us enough connectivity to migrate all online users off of the LAX PoP and onto a different one. By 12:30AM, all users were migrated to other PoPs and were able to continue their broadcasts. We have taken immediate action to prevent an issue of this kind from happening again. We completed a pending infrastructure migration which would have allowed us to resolve the issue within only a few minutes, and we have begun seeking alternative hardware vendors to upgrade the two core routers. For any further questions or concerns, please reach out to me at tt2468@irltoolkit.com - Kyle (tt2468) | CEO/Co-founder