[Resolved] Major Outage - Multiple Providers

We are receiving calls from many of our clients experiencing issues across multiple internet providers. The issues appear to be related to some kind of routing issue between providers. We are investigating the issue and will be posting all updates here as well as hourly emails. 

Update 1:10PM:

Comcast has declared a mass outage that may be affecting all of their clients. Estimated time of recovery is 5:00PM.

We are still working with the other ISPs to gain information. If you are experiencing issues feel free to call the help desk at 650-454-8850 with any questions. 

Update 2:10PM

Comcast is still experiencing a mass outage. Estimated time of recovery has remained at 5:00pm. We will be moving all clients that have secondary circuits over to their backups.

Level 3 has confirmed that they experienced a nationwide outage at 9:00AM as well. However, this was cleared at about 12:00PM. 

We are currently still investigating the remaining ISPs to verify that the signals and outages are clear. If there are any questions or further issues please call the help desk at 650-454-8850 for assistance.

Update 4:22PM

After investigation it appears that the source of this outage was related to a nationwide DNS server issue that affected most, if not all providers. Service has been returned to many clients, and several ISPs like Level 3 and AT&T have cleared their outages.

While many clients of Comcast have service once more, they have extended the outage to 7:00PM. If service has still not returned, please call the help desk at 650-454-8850 for further support. We will continue to monitor and update this page once the Comcast outage has been cleared.

Update 8:43PM

After speaking with Comcast the outage has been officially cleared. While we were informed by a Comcast technician that the outage was related to a DNS server issue. We continued investigation and found an article related to today's outage. This article described the issue as a "configuration issue from Level 3". While the issue originated at the provider Level 3, it affected other vendors like Comcast due to Level 3's infrastructure which distributes for other services.

Service should now be returned to normal. If any issues remain please reach out to the help desk at 650-454-8550 for assistance. We thank you for your patience.

Best,

Kinetix Support