Incident Report Number: 2015-013

VoIP Telephones

Ticket Number: ​INC0026424
Problem Number: ​PRB0010108
Major Incident Number: ​MIR0001063
What happened?

Voice Over Internet Protocol (VoIP) Cisco telephones were experiencing an issue.

Who was affected?

Some users located on North Campus and Enterprise Square (ESQ) were affected by this outage.

What was the impact?

The affected users were not able to make inbound and outbound calls. Some users had continuous restarts of their VoIP telephones.

What was the timeline of the incident?

Start: 2015/04/13 09:30 ­ Users reported their VoIP phones were not working properly.
2015/04/13 09:30 ­ IT support analysts began working on the issue and engaged vendor for assistance.
2015/04/13 10:30 ­ The root cause was identified and steps were taken towards resolution.
2015/04/13 12:15 ­ Service was restored.
End: 2015/04/13 13:15 ­ Users confirmed service was restored.

What was the root cause of the incident?

Call Manager system failure due to database synchronization issues. Compounding the issue a phone software upgrade resulted in some client phones restarting several times. That action triggered a precautionary measure whereby network access was disabled to VoIP phones connections.

What was the work around and resolution for the incident?
Work Around

Not Applicable



Resolution

Call Manager database was resynchronized and VoIP connections were re­enabled.

What are any recommendations to prevent this incident from occurring again?
  • Enhance monitoring of the Call Manager systems to determine service state automatically rather than requiring manual verification of each system.
  • Investigate if event logs from the VoIP environment can be utilized by monitoring systems to provide more proactive notifications of system state.
Updates

Not Applicable