Incident Report Number: 2016-018

UWS & LAN

Ticket Number: INC0061262

Major Incident Number: MIR0001102
What happened?

University Wireless Service (UWS) and Local Area Network (LAN)  connections experienced an issue. As a result, University IT services either could not be reached or experienced slow access.

Who was affected?

All University IT service users around campus were potentially affected by this degradation. Users located off campus also may have experienced difficulty connecting to IT services hosted on campus.

What was the impact?

The affected users experienced difficulty connecting to IT services such as UWS, wired LAN, Voice over Internet Protocol telephones (VoIP) and internet based applications. In addition computers in Labs and Classrooms were unavailable for use. ONEcard distribution, Bear Tracks and PeopleSoft Campus Solutions Applications were also affected.

What was the timeline of the incident?

Start: 2016/09/01 11:00 – Multiple reports of University Wireless Service (UWS) and Local Area Network (LAN) connections experiencing issues were received. Affected users experienced difficulty connecting to various IT services. IST began investigating.
2016/09/01 11:45 – IST engaged vendor to assist with the investigation.
2016/09/01 13:00 – Services were restored by resetting the configuration on a network device. IT support teams continued to investigate the cause of the problem and monitored services for further issues.
2016/09/01 22:00 – IST created plans to reconfigure a network device affecting University Wireless Service (UWS) and Firewall Services to resolve the issue.
2016/09/02 00:00 – IST upgraded and reconfigured the network device to address the issue. University Wireless Service (UWS) and Local Area Network (LAN) connections were stable. IST continued monitoring for further issues.
2016/09/02 11:10 – Reports were received of the same issue occurring again across campus. IST support teams engaged vendor for further investigation.
2016/09/02 12:45 – Services were restored by restarting a network device. IST support teams continued to investigate with the vendor and monitored services for further issues.
2016/09/02 15:05 – Reports were received of the same issue occurring again across campus. IST support teams continue to work with vendor for investigation and resolution.
2016/09/02 15:45 – Service was restored and the issue was resolved by implementing configuration changes. IST support teams continued to monitor services for any further issues. 
End: 2016/09/06 12:00 – No further issues have been reported after services were restored on Friday afternoon.

What was the root cause of the incident?

IST completed upgrades in May 2016 to provide new functionality. Testing of the changes throughout the summer did not identify any issues with the upgrade. With the start of the Fall term and the increased load of over 30,000 users the network experienced performance issues and became unresponsive. Working with the vendor IST identified the cause of the issue to be a problem with the software changes to support new functionality.

What was the work around and resolution for the incident?
Work Around

Not Applicable



Resolution

Removing the change to provide the new functionality resolved the issue.

What are any recommendations to prevent this incident from occurring again?

IST will work with the vendor to understand what caused the incident and develop an updated implementation approach and testing to mitigate the issue from occurring.

 

Updates

None