Incident Report Number: 2015-024

Google Apps

Ticket Number: INC0034779
Problem Number: PRB0010123
Major Incident Number: MIR0001079
What happened?

Google Apps such as Gmail, Google Drive, Google Docs, Google Hangouts, YouTube, and Google Maps which provide users with email, personal storage, chat and media content experienced an issue. uLearn also experienced issues while on campus.

Who was affected?

Google App users located around campus were affected by this outage. Some users of uLearn on campus also were affected.

What was the impact?

The affected users experienced difficulty using some Google Apps and/or latency issues. uLean users located on campus reported latency.

What was the timeline of the incident?

Start: 2015/09/15 12:55 - Users began reporting Google Apps such as Gmail, Contacts and Drive were experiencing an issue. IST began investigating.
2015/09/15 13:30 - IST analysts determined the issue was related to overutilization of the internet on campus.
2015/09/15 16:20 - Service was confirmed restored.
2015/09/16 12:15 - Issue reoccurred, users began reporting Google Apps such as Gmail and Drive were experiencing an issue. IST escalated to third party vendors for investigation and resolution.
2015/09/16 13:25 - Users of uLearn on campus reported latency issues. 
2015/09/16 14:45 - Impacted Google Apps include, but were not limited to: Google Drive, Google Docs, Google Hangouts, YouTube and Google Maps.
End: 2015/09/16 15:15 - Service was restored.

What was the root cause of the incident?

Root cause determined to be insufficient network capacity on our internet connection triggering changes within Google systems.

What was the work around and resolution for the incident?
Work Around

Not Applicable



Resolution

No actions were taken by IST. Once the bandwidth utilization returned to normal, Google reverted changes within Google systems.

What are any recommendations to prevent this incident from occurring again?

Obtain increased bandwidth capacity.

Updates

Not Applicable