[Resolved] Major Outage - Multiple Providers

We are receiving calls from many of our clients experiencing issues across multiple internet providers. The issues appear to be related to some kind of routing issue between providers. We are investigating the issue and will be posting all updates here as well as hourly emails. 

Update 1:10PM:

Comcast has declared a mass outage that may be affecting all of their clients. Estimated time of recovery is 5:00PM.

We are still working with the other ISPs to gain information. If you are experiencing issues feel free to call the help desk at 650-454-8850 with any questions. 

Update 2:10PM

Comcast is still experiencing a mass outage. Estimated time of recovery has remained at 5:00pm. We will be moving all clients that have secondary circuits over to their backups.

Level 3 has confirmed that they experienced a nationwide outage at 9:00AM as well. However, this was cleared at about 12:00PM. 

We are currently still investigating the remaining ISPs to verify that the signals and outages are clear. If there are any questions or further issues please call the help desk at 650-454-8850 for assistance.

Update 4:22PM

After investigation it appears that the source of this outage was related to a nationwide DNS server issue that affected most, if not all providers. Service has been returned to many clients, and several ISPs like Level 3 and AT&T have cleared their outages.

While many clients of Comcast have service once more, they have extended the outage to 7:00PM. If service has still not returned, please call the help desk at 650-454-8850 for further support. We will continue to monitor and update this page once the Comcast outage has been cleared.

Update 8:43PM

After speaking with Comcast the outage has been officially cleared. While we were informed by a Comcast technician that the outage was related to a DNS server issue. We continued investigation and found an article related to today's outage. This article described the issue as a "configuration issue from Level 3". While the issue originated at the provider Level 3, it affected other vendors like Comcast due to Level 3's infrastructure which distributes for other services.

Service should now be returned to normal. If any issues remain please reach out to the help desk at 650-454-8550 for assistance. We thank you for your patience.

Best,

Kinetix Support

[RESOLVED] PG&E Power Outage - Menlo Park

At this time, there is a current power outage in the Menlo Park Area. It appears the power will be out for at least two hours. We are investigating this further, and update here as new information is gained. Let us know if you have any questions by calling at 650-454-8850. 

Update 9/28 @ 3:30pm PST

PG&E estimates the power will return around 5:30pm PST.

Update 9/28 @ 4:01pm PST

Power has returned.

Best,

Kinetix Support

[RESOLVED] 8/3 - Shoretel Major Outage

Hello Shoretel Supported Clients,

Shoretel is currently experiencing a major outage that affects most (if not all) customers. There is no ETA at this time, but we will keep checking each hour until the issue is resolved. Thank you for your patience during this downtime.

Update - 8/3 @ 9:55AM PST

The issue is still ongoing, but we will check on the status again around 11:00AM PST.

Update - 8/3 @ 11:42AM PST

The Shoretel outage has been resolved.

If you have any questions, please reach out to us at the helpdesk at 650-454-8850.

Best,

Kinetix Support

[RESOLVED] 8/1 Major Outage - Slack

Dear Slack Supported Clients, we have become aware that there is a major outage affecting Slack.  This affects our ability to respond to incoming Slack messages from our clients in real time.  We are working with Slack support to resolve this issue, and will provide an estimated time of repair as soon as it is known.  In the meantime, direct all urgent help requests to 650.454.8850, and non-urgent requests to help@kinetix.com.

Update - 8/2 @ 5:16PM PST

We have resolved the issues related to Slack messages. 

Thanks,

Kinetix Support

7/21 Helpdesk Phone System Outage - RESOLVED

 

Hello Kinetix Client,

It has come to our attention that our helpdesk phone system is experiencing an intermittent outage. When calling our 650-454-8850 number, you may be unable to get through. We are working on resolving this issue with our phone system provider at a high priority.

In the interim, please direct any urgent, business hours requests to our live chat service. You may find a link to our chat service service here - http://help.kinetix.com/

Thanks,

Kinetix Support

 

UPDATE - 1:35 PST

The issue with the Kinetix phone system has been resolved by the our phone provider. You may have your team resume opening cases, 24/7, at 650-454-8850

 

 

 

 

[RESOLVED] Egnyte Authentication issues - 7/3

Hello Kinetix Client,

It has been brought to our attention that Egnyte, a cloud hosted filesharing service, is currently undergoing issues prohibiting users from authenticating to their systems. Egnyte support is currently working on a process for remediation.

We will continue to stay in touch with Egnyte during the remediation process. For future updates on this issue, please visit Kinetix.com/status. If you have any questions, please give us a ring at 650-454-8850.

Below are all related updates from Egnyte:

At 7:13am PST- We are investigating an issue causing authentication issues with the Egnyte Drive app and Desktop Connect app. Please contact our support if you have experienced any issues.

At 7:55am PST - Customers from West coast region may be facing this issue. We are working on the mitigation process to bring our services back to operation status. We know some customer are facing login issues with their domain in the Web coast region. If you are experiencing any issues, please contact the Egnyte support team.

At 10:13am PST - We were able to identify the issues affecting Egnyte Drive services for some of our customer in the West coast. We have mitigated the issues and our EGD service is now operational. In the meantime we are monitoring the situation and our engineers are working on the root cause. We will continue to update EGD status and provide you with a postmortem report today.


Best,

Kinetix Support

Kinetix datacenter issue 6/27

We are aware of a momentary network issue associated with services hosted in the Kinetix datacenter and are investigating the root cause with our infrastructure provider. At this time all services are back online.

We will provide the root cause as soon as our investigation is complete.

If you have any questions or are noticing any further issues, please reach out to our helpdesk at 650-454-8850. We appreciate your understanding.

Major Outage - Slack

According to Slack support at 1:20pm PDT, "We are currently seeing a connectivity problem where some teams are unable to load Slack. We are very sorry for this disruption. We're working hard to solve it as quickly as we can. We will update this page with more information as it becomes available."

As of 1:38pm PDT: This issue is specifically affecting IBM, so this outage is considered resolved for all other users.

Please let us know if you have any questions by calling us at 650-454-8850.

Thanks,

Kinetix Support

Large Scale Spear Phishing Attack

UPDATE: Google has officially released a statement that they have stopped the attack.

-----------------------------------------------------------

Currently there is a large scale phishing attack that is propagating throughout the internet.  The attack appears as a legitimate email from an employee within your company or address list, and instructs the recipient to click on a link that looks like a legitimate link to a Google Document.  Please be aware and inform your employees to not click on this link.  If clicked, you are asked to grant permissions to an illegitimate Google Drive application that then parses your emails and contacts to send additional messages to. 

If you have clicked this link, please mitigate the attack by doing the following:

“[I]f you have clicked on the link, go to your Google account's page (https://myaccount.google.com/permissions) where you can manage the permissions you've granted to apps. Then locate the "Google Doc" app. This looks totally legitimate, but it's actually not. If that's the malicious app that's gotten access to your account after you clicked on the link it should have a recent "Authorization Time." Now, click on that Google Docs app and click Remove.”

More details can be found here: https://motherboard.vice.com/en_us/article/massive-gmail-google-doc-phishing-email

Please do not hesitate to reach out to our helpdesk directly at 650.454.8850 for assistance with this.

Thanks,

Kinetix Support

Power Outage in San Francisco

Hi Kinetix Clients,

We believe the PG&E Power outage is now resolved.

Issue: PG&E had a power outage affecting portions of San Francisco.

Root Cause: PG&E power outage was caused by an equipment failure at an electrical substation in the Tenderloin. It took them several hours to repair and this affected over 90,000 customers.

Please let us know if you have any questions by calling us at 650-454-8850.

Thanks,

Kinetix Support

3/23 - Office 365 Login Issues

 Hello Kinetix Client,

Currently, Microsoft is having an outage related to their Office365 portal logins.  Users are often unable to log into their online portal and are given an error message stating that there are some transient issues with the server. We will continue to monitor this outage and will send notice when Microsoft resolves this issue. 

Please reach out to the helpdesk with any questions at 650-454-8850

Thanks,
Kinetix Support