Incident Report Number: 2015-005

Network Outage in North East Quadrant of North Campus

Ticket Number: ​INC0021661
Problem Number: ​PRB0010089
Major Incident Number: ​MIR0001050
What happened?

Users were experiencing network connectivity issues.

Who was affected?

Users located in the following locations were affected:

  • Brain and Aging Research Building
  • Earth Sciences
  • Education ­ School of Library and Information Studies
  • Fine Arts Building (FAB)
  • Head House
  • Humanities Center
  • HUB Mall
  • International House
  • Killam
  • Law Center
  • Law Library
  • Parking Lot U in HUB
  • Power Plant
  • Rutherford Library
  • School of Business
  • St. Stephen's College
  • Telus Center
  • Timms Center
  • Tory Building
  • Central Administration Building (CAB) ­ 2nd Floor (Women in Scholarship, Engineering, Science & Technology [WISEST]; Math and Applied Sciences Center (MASC) and Specialized Support and Disability Services (SSDS)

Learnnet classrooms and labs:

  • Business
  • Earth Sciences
  • Fine Arts Building (FAB)
  • Humanities Center
  • Law Center
  • Rutherford Library
  • Tory Building
What was the impact?

Users were experiencing network connectivity issues with wired networks, University Wireless Service (UWS) and Voice over Internet Protocol (VOIP)

What was the timeline of the incident?

Start: 2015/02/02 06:55 ­ Monitoring systems detected an alert with the UPS (uninterruptible power supply) located in the Faculty of Business.
2015/02/02 08:30 ­ IST began to check the UPS status to investigate the cause of the alert.
2015/02/02 08:55 ­ Monitoring systems detected the Business distribution switch and all connected devices were unreachable. Analysts were dispatched to investigate the issue on site.
2015/02/02 09:15 ­ A UPS was discovered to be powered off. UPS was powered back on.
2015/02/02 09:20 ­ The distribution switch services was restored.
End: 2015/02/02 09:25 ­ Service was confirmed to be restored.

What was the root cause of the incident?

The UPS inexplicably turned itself off. The distribution switch became unavailable as both power connections were connected to the failed UPS.

What was the work around and resolution for the incident?
Work Around

Not Applicable



Resolution

The UPS was turned back on to restore power to the distribution switch which restored the affected IT services. The second power supply on the distribution switch was disconnected from the UPS and plugged directly into the building power to provide redundant power.

What are any recommendations to prevent this incident from occurring again?
  • Update Cause and Effect to change IST’s response window to 24 hours x7 days a week.
  • Replace the potentially faulty UPS.
  • Confirm all distribution switches are connected to redundant power sources.
Updates

Not Applicable