Incident Report Number: 2015-006

uLearn

Ticket Number: ​INC0021939
Problem Number: ​PRB0010090
Major Incident Number: ​MIR0001051
What happened?

The uLearn (Blackboard) application, a learning management system, was experiencing a service degradation.

Who was affected?

Users of uLearn (School of Business) were affected by this outage.

What was the impact?

The affected users were unable to see or access their courses from the current term once logged into uLearn. Users who only have enrollments in the current term could no longer log in to the system. During the restoration some students regained access to courses that they had dropped.

What was the timeline of the incident?

Start: 2015/02/05 12:15 ­ Users of uLearn reported that course information was missing.
2015/02/05 12:45 ­ IT support began investigating this issue.
2015/02/05 12:58 ­ The issue was determined to be with the LDAP (Lightweight Directory Access Protocol) System.
2015/02/05 13:08 ­ A data rollback in uLearn was started in order to restore missing data for the affected users. 2015/02/05 13:30 ­ The initial data rollback was completed. This resolved issues for all but 122 users.
2015/02/05 13:35 ­ An automated tool to resolve the remaining issues was built and tested successfully.
2015/02/05 13:45 ­ The missing data for the remaining 122 users was successfully restored and a full verification was started to ensure the issue was resolved.
End: 2015/02/05 14:00 ­ The issue was confirmed resolved.

What was the root cause of the incident?

The root cause of the issue was an import script failed when it was unable to connect to the LDAP system. The import operation caused all enrollment records for instructors and students to be dropped.

What was the work around and resolution for the incident?
Work Around

A backup of the records from the previous day was restored. This recovered the missing records. Any changes to student records on the day of the incident still existed in LDAP.



Resolution

LDAP began a restore process. Once LDAP completed the data restore and updated, the records in uLearn were correct.

What are any recommendations to prevent this incident from occurring again?
  • Replace LDAP with PeopleSoft as the data source for uLearn.
  • Investigate adding a locking mechanism to prevent scripts from failing when they do not successfully connect to LDAP.
Updates

Not Applicable